- Flutter: language='auto' omits the language field → backend receives none
- Backend: no language field → passes undefined to STT service
- STT service: language=undefined → omits language param from Whisper request
- Whisper auto-detects language per utterance when no hint is provided
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reads settingsProvider.language (BCP-47 code) and passes it to the
Whisper transcribe call instead of hardcoding 'zh'.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When backgrounded, the periodic TCP ping times out causing isOnline=false.
On resume, immediately re-check so the banner clears as soon as the app
is foregrounded rather than waiting up to 30s for the next scheduled check.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Node 18 native fetch (undici) ignores https.Agent, causing fetch failed
on the self-signed proxy at 67.223.119.33:8443. Switch to https.request
with rejectUnauthorized: false which works reliably.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GestureDetector was fighting with IconButton's inner Tooltip gesture
recognizer — onLongPressStart was never called (only vibration from
tooltip). Replaced with Listener (raw pointer events) + manual 500ms
Timer, which bypasses the gesture arena entirely.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
OPENAI_BASE_URL=https://67.223.119.33:8443/v1 already includes /v1,
so the URL was being built as .../v1/v1/audio/transcriptions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
multer was only transitively available; pnpm strict mode blocks it.
Also adds @types/multer for TypeScript compilation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add POST /api/v1/agent/transcribe endpoint (STT only, no agent trigger)
- Add transcribeAudio() to chat datasource and provider
- VoiceMicButton now fills the text input field with transcript;
user reviews and sends manually
- Add OPENAI_API_KEY/OPENAI_BASE_URL to agent-service in docker-compose
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Default to PTT (push-to-talk) on call connect: mic muted until user holds button
- Toggle switch between PTT and free voice mode in active call controls
- PTT button: press-and-hold unmutes mic, release mutes again
- Voice message bubble (waveform + duration) appears after each PTT send
- Mute button hidden in PTT mode (mic controlled by PTT button)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three coordinated fixes to make in-app APK download work end-to-end:
1. version-service/main.ts: serve uploaded files as static assets via
NestExpressApplication.useStaticAssets('/data/versions', prefix:
'/downloads/versions'), so GET /downloads/versions/{platform}/{file}
returns the actual APK stored in the Docker volume.
2. kong.yml: add /downloads/versions route to Kong so requests from
the Flutter app can reach version-service through the API gateway.
Previously only /api/v1/versions and /api/app/version were routed;
the download URL returned by the check endpoint was unreachable (404).
3. download_manager.dart: skip SHA-256 verification when sha256Expected
is empty string. The check endpoint always returns sha256:"" because
version-service doesn't store file hashes. The previous code compared
actual_hash == "" which always failed, causing the downloaded file to
be deleted after a successful download.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add verbose debugPrint logs throughout VersionChecker to diagnose why
app update check isn't triggering:
- Log apiBaseUrl and full request URL + query params before the request
- Log response status code and raw response body
- Log explicit needUpdate=true/false with version details
- Log version code comparison (server versionCode vs local buildNumber)
- Add stack trace to all catch blocks for better error diagnosis
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add /api/app/version route to Kong declarative config so that the
Flutter app's GET /api/app/version/check?platform=¤t_version_code=
request can reach version-service through the API gateway.
Previously only /api/v1/versions was routed; the public check endpoint
served by AppVersionCheckController was unreachable (Kong returned
"no Route matched with those values").
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Flutter VersionChecker was calling GET /api/app/version/check but this
endpoint didn't exist — only the admin CRUD /api/v1/versions was there.
New: AppVersionCheckController (@Controller('api/app/version'))
GET /api/app/version/check?platform=android¤t_version_code=N
- Finds latest enabled version for the platform (highest buildNumber)
- Returns { needUpdate: false } when already up to date
- Returns full VersionInfo payload when update is available
Response fields match Flutter VersionInfo.fromJson exactly:
needUpdate, version, versionCode, downloadUrl, fileSize,
fileSizeFriendly (computed), sha256 (empty — not stored),
forceUpdate, updateLog, releaseDate
Also: AppVersionRepository.findLatestEnabled(platform) — queries all
enabled versions for platform, picks the one with the highest buildNumber
(parsed as int, robust against varchar storage).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ports the APK/IPA upgrade management UI from rwadurian/mobile-upgrade
into it0-web-admin, adapted exclusively for IT0 App's version-service.
New files:
- src/domain/entities/app-version.ts
Domain entity matching version-service response schema:
platform returned as ANDROID/IOS (normalized to lowercase),
fileSize as number (bigint), no versionCode/fileSha256 fields.
- src/infrastructure/repositories/api-app-version.repository.ts
CRUD via existing apiClient (→ /api/proxy/api/v1/versions).
Upload/parse use dedicated Next.js routes (/api/app-versions/*)
because the existing proxy uses request.text() which corrupts binary.
- src/app/api/app-versions/upload/route.ts
Multipart FormData upload proxy → API_BASE_URL/api/v1/versions/upload
maxDuration=300s for large APK files (up to 500 MB).
- src/app/api/app-versions/parse/route.ts
Multipart proxy → API_BASE_URL/api/v1/versions/parse
Forwards APK/IPA file to version-service for auto-parsing.
- src/app/(admin)/app-versions/page.tsx
Admin page: react-query list, platform filter (all/android/ios),
upload button, loading skeleton, delete/toggle with confirm.
Single-app (IT0 only) — no multi-app switcher from mobile-upgrade.
- src/presentation/components/app-versions/version-card.tsx
Version card with enable/disable/edit/delete/download actions.
Uses dark-theme CSS variables (bg-card, text-muted-foreground, etc.)
- src/presentation/components/app-versions/upload-modal.tsx
Upload modal: auto-detects platform from .apk/.ipa extension,
auto-parses version info via /parse endpoint, sonner toasts.
- src/presentation/components/app-versions/edit-modal.tsx
Edit modal: update changelog, force-update flag, enabled state,
min OS version. Loads version data on open via getVersionById.
Modified:
- sidebar.tsx: added Smartphone icon + appVersions nav item → /app-versions
- locales/zh/sidebar.json: "appVersions": "App 版本管理"
- locales/en/sidebar.json: "appVersions": "App Versions"
Backend: IT0 version-service at /api/v1/versions (no auth guard required)
Flutter: it0_app/lib/core/updater/version_checker.dart calls
GET /api/app/version/check (public) for client-side update check.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New VoiceMicButton widget (press-and-hold to record, release to send):
- Records audio to a temp .m4a file via the `record` package
- Slide-up gesture cancels recording without sending
- Pulsing red mic icon + "松开发送/松开取消" feedback during recording
New flow for voice messages:
1. Temp "🎤 识别中..." bubble shown immediately
2. Audio uploaded to POST /api/v1/agent/sessions/:id/voice-message
(multipart/form-data; backend runs Whisper STT)
3. Placeholder replaced with real transcript
4. WS stream subscribed via new subscribeExistingTask() to receive
agent's streaming response — same pipeline as text chat
Voice messages act as async interrupts: if the agent is mid-task the
backend hard-cancels it before processing the new voice command,
so whoever presses the mic button always takes priority.
Files changed:
chat_remote_datasource.dart — sendVoiceMessage() multipart upload
chat_repository.dart — subscribeExistingTask() interface method
chat_repository_impl.dart — implement subscribeExistingTask(); fix
sendVoiceMessage() stub
chat_providers.dart — ChatNotifier.sendVoiceMessage()
voice_mic_button.dart — NEW press-and-hold recording widget
chat_page.dart — mic button added to input area
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New endpoint: POST /api/v1/agent/sessions/:sessionId/voice-message
- Accepts multipart/form-data audio file (any format Whisper supports)
- Transcribes via OpenAI Whisper API (routed through existing proxy)
- If a task is currently running in the session → hard-interrupts it first
(same cancel+inject pattern as text inject, triggered by voice command)
- Otherwise → starts a fresh task with the transcript
- Returns { sessionId, taskId, transcript } so client can subscribe to WS stream
This enables WhatsApp-style push-to-talk and doubles as an async voice
interrupt into any active agent workflow, bypassing the need for speaker
diarization (whoever presses record owns the message).
New files:
infrastructure/stt/openai-stt.service.ts — OpenAI Whisper client,
manually builds multipart/form-data, supports self-signed proxy cert
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements a two-level abort controller design to support real-time
interruption when the user speaks while the agent is still responding:
sessionAbortController (session-scoped)
- Created once when startSession() is called
- Fired only by terminateSession() (user hangs up)
- Propagated into each turn via addEventListener
turnAbort (per-turn, stored as handle.currentTurnAbort)
- Created fresh at the start of each executeTurn() call
- Stored on the VoiceSessionHandle so injectMessage() can abort it
- When a new inject arrives while a turn is running, injectMessage()
calls turnAbort.abort() BEFORE enqueuing the new message
Interruption flow:
1. User speaks mid-response → LiveKit stops TTS playback (client-side)
2. STT utterance → POST voice/inject → injectMessage() fires
3. handle.currentTurnAbort.abort() called → sets aborted flag
4. for-await loop checks turnAbort.signal.aborted on next SDK event → break
5. catch block NOT reached (break ≠ exception) → no error event emitted
6. finally block saves partial text with "[中断]" suffix to history
7. New message dequeued → fresh executeTurn() starts immediately
Why no "Agent error" message plays to the user:
- break exits the for-await loop silently, not via exception
- The catch block's error-event emission is guarded by err?.name !== 'AbortError'
AND requires an actual exception; a plain break never enters catch
- Empty or partial responses are filtered by `if response:` in agent.py
Also update module-level JSDoc with full architecture explanation covering
the long-lived run loop design, two-level abort hierarchy, tenant context
injection pattern, and SDK session resume across turns.
Update agent.py module docstring to document voice session lifecycle and
interruption flow for future maintainers.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the per-turn POST /tasks approach for voice calls with a
long-lived agent run loop tied to the call lifecycle:
agent-service:
- Add AsyncQueue<T> utility for blocking message relay
- Add VoiceSessionManager: spawns one background run loop per voice call,
accepts injected messages, terminates cleanly on hangup
- Add VoiceSessionController with 3 endpoints:
POST /api/v1/agent/sessions/voice/start (call start)
POST /api/v1/agent/sessions/:id/voice/inject (each speech turn)
DELETE /api/v1/agent/sessions/:id/voice (user hung up)
- Register VoiceSessionManager + VoiceSessionController in agent.module.ts
voice-agent:
- AgentServiceLLM: add start_voice_session(), terminate_voice_session(),
inject_text_message() (voice/inject-aware), _do_inject_voice()
- AgentServiceLLMStream._run(): use voice/inject path when voice session
is active; fall back to per-task POST for text-chat / non-SDK engines
- entrypoint(): call start_voice_session() after session.start();
register _on_room_disconnect that calls terminate_voice_session()
so the agent is always killed when the user hangs up
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two issues fixed:
1. agent.controller.ts — on the FIRST task of each session, write title+voiceMode
into session.metadata so the client can display a meaningful conversation title:
- Text sessions: metadata.title = first 40 chars of user prompt
- Voice sessions: metadata.title = '' + metadata.voiceMode = true
(Flutter renders these as '语音对话 M/D HH:mm')
titleSet flag prevents overwriting the title on subsequent turns of the same session.
2. session.controller.ts — listSessions() now returns a DTO instead of the raw entity.
systemPrompt is an internal engine instruction and is explicitly excluded from the
response. The client receives { id, status, engineType, metadata, createdAt, updatedAt }.
Voice sessions set systemPrompt to the voice-mode instruction string,
causing every voice conversation to display '你正在通过语音与用户实时对话。请…'
as its title in the chat history list.
Title derivation priority (highest to lowest):
1. metadata.title — explicit title saved by backend on first task
2. metadata.voiceMode == true → '语音对话 M/D HH:mm'
3. Fallback → '对话 M/D HH:mm' based on session createdAt
The billing-service tsconfig.json was missing the TypeScript path aliases
required for the workspace build (turbo builds shared packages first, then
resolves @it0/* via paths). Without these, nest build fails with
'Cannot find module @it0/database'.
Also disables overly strict checks (strictNullChecks, strictPropertyInitialization,
useUnknownInCatchVariables) to match the lenient settings used by other services.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Comprehensive fix of 124 TS errors across the billing-service:
Entity fixes:
- invoice.entity.ts: add InvoiceStatus/InvoiceCurrency const objects,
rename fields to match DB schema (subtotalCents, taxCents, totalCents,
amountDueCents), add OneToMany items relation
- invoice-item.entity.ts: add InvoiceItemType const object, add column
name mappings and currency field
- payment.entity.ts: add PaymentStatus const, rename amount→amountCents
with column name mapping, add paidAt field
- subscription.entity.ts: add SubscriptionStatus const object
- usage-aggregate.entity.ts: rename periodYear/Month→year/month to match
DB columns, add periodStart/periodEnd fields
- payment-method.entity.ts: add displayName, expiresAt, updatedAt fields
Port/Provider fixes:
- payment-provider.port.ts: make PaymentProviderType a const object (not
just a type), add PaymentSessionRequest alias, rename WebhookEvent with
correct field shape (type vs eventType), make providerPaymentId optional
- All 4 providers: replace PaymentSessionRequest→CreatePaymentParams,
fix amountCents→amount, remove sessionId from PaymentSession return,
add confirmPayment() stub, fix Stripe API version to '2023-10-16'
Use case fixes:
- aggregate-usage.use-case.ts: replace 'redis' with 'ioredis' (workspace
standard); rewrite using ioredis xreadgroup API
- change/check/generate use cases: fix Plan field names
(monthlyPriceCentsUsd, includedTokens, overageRateCentsPerMTokenUsd)
- generate-monthly-invoice: fix SubscriptionStatus/InvoiceCurrency as
values (now const objects)
- handle-payment-webhook: fix WebhookResult import, result.type usage,
payment.paidAt
Controller/Repository fixes:
- plan.controller.ts, plan.repository.ts: fix Plan field names
- webhook.controller.ts: remove express import, use any for req type
- invoice-generator.service.ts: fix overageAmountCents→overageCentsUsd,
monthlyPriceCny→monthlyPriceFenCny, includedTokensPerMonth→includedTokens
Dependencies:
- billing-service/package.json: replace redis with ioredis dependency
- pnpm-lock.yaml: regenerated after ioredis addition
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Dockerfile.service: add COPY lines for billing-service/package.json in
both build and production stages so pnpm install includes its deps
(omission caused 'node_modules missing' turbo build error)
- pnpm-lock.yaml: regenerated after running pnpm install to include all
billing-service dependencies (stripe, alipay-sdk, wechat-pay-v3, etc.)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- 005-create-billing-tables.sql: replace all `it0_shared.tenants` with
`public.tenants` and all `tenant_id VARCHAR(20)` with `tenant_id UUID`
to match the actual server DB schema (public schema, UUID primary key)
- packages/shared/testing src/test-utils.ts: add new quota fields
(maxServers, maxUsers, maxStandingOrders, maxAgentTokensPerMonth) to
TEST_TENANT mock to satisfy the extended TenantInfo interface, fixing
the @it0/testing TypeScript build error
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ensure /data/versions/android and /data/versions/ios directories are
created with correct appuser ownership during image build, fixing
EACCES permission error when version-service starts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The entrypoint.sh expects dist/services/${SERVICE_NAME}/src/main, but
nest build with inline TypeORM config produces dist/main directly.
Using DatabaseModule from @it0/database forces tsc to emit the nested
path structure (since it references shared packages), matching the
entrypoint path convention used by all other services.
Also gains SnakeNamingStrategy and autoLoadEntities from the shared module.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
voice-agent agent.py:
- Module docstring explains lk.agent.state lifecycle
(initializing → listening → thinking → speaking)
- Explains how RoomIO publishes state as participant attribute
- Documents BackgroundAudioPlayer with all available built-in clips
Flutter agent_call_page.dart:
- Documents _agentState field and all possible values
- Documents ParticipantAttributesChanged listener with UI mapping
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Import from livekit.agents.voice.background_audio submodule directly,
as it's not re-exported from livekit.agents.voice.__init__.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- voice-agent: enable BackgroundAudioPlayer with keyboard typing sound
during LLM thinking state (auto-plays when agent enters "thinking",
stops when "speaking" starts)
- Flutter: monitor lk.agent.state participant attribute from LiveKit
agent, show pulsing dots animation + "思考中..." text when thinking,
avatar border changes to warning color with pulsing glow ring
- Both call mode and chat mode headers show thinking state
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detailed record of why livekit-plugins-speechmatics was removed:
- EXTERNAL: no FINAL_TRANSCRIPT (framework never sends FlushSentinel)
- ADAPTIVE: zero output (dual Silero VAD conflict)
- SMART_TURN: fragments Chinese speech into tiny pieces
- FIXED: finalize() async race condition with session teardown
All tested on 2026-03-03, none viable with LiveKit agents v1.4.4.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SMART_TURN fragments continuous speech into tiny pieces, each triggering
an LLM request that aborts the previous one. FIXED mode waits for a
configurable silence duration (1.0s) before emitting FINAL_TRANSCRIPT
via the built-in END_OF_UTTERANCE handler.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Document all findings from the integration process directly in the
source code for future reference:
1. Language code mapping: Speechmatics uses ISO 639-3 "cmn" for
Mandarin, but LiveKit LanguageCode auto-normalizes it to "zh".
Must override stt._stt_options.language after construction.
2. Turn detection modes (critical):
- EXTERNAL: unusable — LiveKit never sends FlushSentinel, only
pushes silence frames, so FINAL_TRANSCRIPT never arrives
- ADAPTIVE: unusable — client-side Silero VAD conflicts with
LiveKit's own VAD, produces zero transcription output
- SMART_TURN: correct choice — server-side intelligent turn
detection, auto-emits FINAL_TRANSCRIPT, fully compatible
3. Speaker diarization: is_active flag distinguishes primary speaker
from TTS echo, solving the "speaker confusion" problem
4. Docker deployment: SPEECHMATICS_API_KEY in .env, watch for
COPY layer cache when rebuilding
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace EXTERNAL mode + monkey-patch hack with SMART_TURN mode.
SMART_TURN uses Speechmatics server-side turn detection that properly
emits AddSegment (FINAL_TRANSCRIPT) when the user finishes speaking.
No client-side finalize or debounce timer needed.
Ref: https://docs.speechmatics.com/integrations-and-sdks/livekit
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Speechmatics re-sends identical partial segments during silence, causing
the debounce timer to fire multiple times with the same text. Each
duplicate FINAL aborts the in-flight LLM request and restarts it.
Replace time-based cooldown with text comparison: skip finalization if
the segment text matches the last finalized text. Also skip starting
new timers when partial text hasn't changed from last finalized.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Reduce debounce delay from 700ms to 400ms for faster response
- Add 1.5s cooldown after emitting FINAL to prevent duplicate triggers
that cause LLM abort/retry cycles
- Enable speaker diarization (enable_diarization=True)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The LiveKit framework never sends FlushSentinel to the STT stream.
Instead it pushes silence frames and waits for FINAL_TRANSCRIPT events.
In EXTERNAL turn-detection mode, Speechmatics only emits partials.
New approach: each partial transcript restarts a 700ms debounce timer.
When partials stop (user stops speaking), the timer fires and promotes
the last partial to FINAL_TRANSCRIPT, unblocking the pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Trace _patched_process_audio lifecycle and FlushSentinel handling
to diagnose why final transcripts are not being promoted.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
VoiceAgentClient.finalize() schedules an async task chain that often
loses the race against session teardown. Instead, intercept partial
segments as they arrive, stash them, and synchronously emit them as
FINAL_TRANSCRIPT when FlushSentinel fires.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the SpeechStream._process_audio patch from container runtime
into our own source code so it survives Docker rebuilds. The patch
adds client.finalize() on FlushSentinel so EXTERNAL mode produces
final transcripts when LiveKit's VAD detects end of speech.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
EXTERNAL mode produces partial transcripts but livekit-plugins-speechmatics
does not call finalize() when receiving a flush sentinel from the framework.
A runtime monkey-patch on the plugin's SpeechStream._process_audio adds the
missing finalize() call so final transcripts are generated.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Speechmatics handles end-of-utterance natively via its Voice Agent
API (ADAPTIVE mode). Use turn_detection="stt" on AgentSession so
LiveKit delegates turn boundaries to the STT engine instead of
conflicting with its own VAD-based turn detection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
hailin