The livekit package is the client SDK and doesn't include the server-side
API module. Switch to livekit-api which provides AccessToken, VideoGrants,
RoomAgentDispatch, and RoomConfiguration needed for token generation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Upgrade websockets from ==12.0 to >=13.0 (openai[realtime] requires >=13)
- Install torch CPU-only build separately in Dockerfile to avoid ~2GB CUDA download
- Remove torch from requirements.txt (installed via --index-url cpu wheel)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
FileType.custom with allowedExtensions causes Android system picker
to hide subdirectories on some devices. Changed to FileType.any with
post-selection extension validation instead.
- Unsupported file types are skipped with a SnackBar hint
- Allowed: jpg, jpeg, png, gif, webp, pdf
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Claude API supports up to 32MB PDFs; base64 encoding adds ~33% overhead.
50mb body limit covers the maximum single-document upload case.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PDF files were incorrectly wrapped as type:'image' content blocks,
causing Claude API to reject them as "Invalid image data".
- conversation-context.service: check mediaType for application/pdf,
use type:'document' block (Anthropic native PDF support) instead
- claude-agent-sdk-engine: detect both 'image' and 'document' blocks
when deciding to build multimodal SDK prompt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- AppBar background transparent, merges with scaffold for seamless look
- toolbarHeight reduced from 64dp to 44dp (~20dp screen space saved)
- scrolledUnderElevation: 0 prevents Material 3 shadow on scroll
- Icons 24→20px with VisualDensity.compact for tighter action buttons
- Title fontSize 16 w600, less visual weight
- Both dark and light themes updated consistently
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Input area redesign (ChatGPT/Claude App style):
- Replace fixed bottom bar with floating pill overlay using Stack+Positioned
- Semi-transparent background (surface 92% opacity) with rounded corners (28px)
- Drop shadow for depth separation from content
- Remove inner TextField border (InputBorder.none) for cleaner look
- ListView bottom padding increased to 80px to leave room under the pill
- Input pill floats 12px from edges, 8px from bottom
History scroll fix:
- Add jump parameter to _scrollToBottom() for instant positioning
- When loading conversation history (empty→many messages), use jumpTo
instead of animateTo to avoid incomplete scroll on large message lists
- Double-frame jumpTo ensures layout settles before final scroll position
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove redundant "从剪贴板粘贴" option from attachment menu (long-press to paste natively)
- Remove super_clipboard dependency (no longer needed)
- Fix timeline vertical line overlapping icon nodes by using dynamic dotRadius
- Dim input field placeholder color to AppColors.textMuted
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
getFile requires two positional args: format and callback.
Wrapped in Completer for async/await usage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add super_clipboard and file_picker dependencies
- Clipboard paste: reads PNG/JPEG image data from system clipboard
- Multi-image: pickMultiImage with remaining count limit
- File picker: supports images (jpg/png/gif/webp) and PDF files
- Updated attachment preview to show file icon for non-image types
- Bottom sheet now shows 4 options: gallery, camera, clipboard, file
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update migration files to include the attachments column for
multimodal image storage. Also add ALTER TABLE migration for
existing deployments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The direct `import * as express from 'express'` caused a
MODULE_NOT_FOUND error in the Docker production image since express
is only available as a transitive dependency via @nestjs/platform-express.
Use NestExpressApplication.useBodyParser() which is the official NestJS API.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SDK engine now constructs AsyncIterable<SDKUserMessage> with image
content blocks when attachments are present in conversationHistory,
using the SDK's native multimodal prompt format
- CLI engine logs a warning when images are detected, since the `-p`
flag only accepts text (upstream Claude CLI limitation)
- Both SDK and API engines now fully support multimodal image input
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two major features in this commit:
1. Streaming Markdown Rendering Optimization
- Replace deprecated flutter_markdown with gpt_markdown (active, AI-optimized)
- Real-time markdown rendering during streaming (was showing raw syntax)
- Solid block cursor (█) instead of AnimationController blink
- 80ms token throttle buffer reducing rebuilds from per-token to ~12.5/sec
- RepaintBoundary isolation for markdown widget repaints
- StreamTextWidget simplified from StatefulWidget to StatelessWidget
2. Multimodal Image Input (camera + gallery + display)
- Flutter: image_picker for gallery/camera, base64 encoding, attachment
preview strip with delete, thumbnails in sent messages
- Data layer: List<String>? → List<Map<String, dynamic>>? for structured
attachment payloads through datasource/repository/usecase
- ChatAttachment model with base64Data, mediaType, fileName
- ChatMessage entity + ChatMessageModel both support attachments field
- Backend DTO, Entity (JSONB), Controller, ConversationContextService
all extended to receive, store, and reconstruct Anthropic image
content blocks in loadContext()
- Claude API engine skips duplicate user message when history already
ends with multimodal content blocks
- NestJS body parser limit raised to 10MB for base64 image payloads
- Android CAMERA permission added to manifest
- Image.memory uses cacheWidth/cacheHeight for memory efficiency
- Max 5 images per message enforced in UI
Data flow:
ImagePicker → base64Encode → ChatAttachment → POST body →
DB (JSONB) → loadContext → Anthropic image content blocks → Claude API
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three root causes fixed:
1. TimelineEventNode: Replaced IntrinsicHeight (which forces intrinsic
height calculation on unbounded content) with CustomPaint-based
_TimelineLinePainter that draws vertical lines based on actual
rendered widget size. Also added maxLines/ellipsis to label text
and mainAxisSize.min on inner Column.
2. ApprovalActionCard: Changed countdown + action buttons layout from
Row with Spacer (which requires infinite width) to Wrap with
spacing, preventing horizontal overflow on narrow screens.
3. AnimatedCrossFade in _CollapsibleCodeBlock and _CollapsibleThinking:
Wrapped with ClipRect and added sizeCurve: Curves.easeInOut to
prevent the outgoing child from extending beyond parent bounds
during the cross-fade transition.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
6 rounds of systematic audit identified and fixed 14 bugs across
backend controller and Flutter client:
## Backend (agent.controller.ts)
Security & Tenant Isolation:
- Add @TenantId + ForbiddenException check to cancelTask, injectMessage,
approveCommand — all 4 write endpoints now enforce tenant isolation
- Add tenantId check on session reuse in executeTask to prevent
cross-tenant session hijacking
Architecture & Correctness:
- Extract shared runTaskStream() from inline fire-and-forget block,
used by both executeTask and injectMessage to reduce duplication
- Use session.engineType (not getActiveEngine()) in cancelTask,
injectMessage, approveCommand — fixes wrong-engine-cancel when
global engine config is switched after task creation
- Add concurrent task prevention: executeTask checks for existing
RUNNING task on same session and cancels it before starting new one
- Add runningTasks Map to track task promises, awaitTaskCleanup()
helper with 3s timeout for inject to wait for partial text save
- captureSdkSessionId() captures SDK session ID into metadata
without DB save (callers persist), preventing fire-and-forget race
Cancel/Reject Improvements:
- cancelTask: idempotent (returns early if already CANCELLED/COMPLETED),
session stays 'active' (was 'cancelled'), emits cancelled WS event
- approveCommand reject: session stays 'active' (was 'cancelled'),
now emits cancelled WS event so Flutter stream listeners clean up
- approveCommand approved: collect text events and save assistant
response to conversation history on completion (was missing)
Minor:
- task.result! non-null assertion → task.result ?? 'Unknown error'
- Add findRunningBySessionId() to TaskRepository
## Flutter
API Contract Fix:
- approveCommand: route changed from /api/v1/ops/approvals/:id/approve
to /api/v1/agent/tasks/:id/approve with {approved: true} body
- rejectCommand: route changed from /api/v1/ops/approvals/:id/reject
to /api/v1/agent/tasks/:id/approve with {approved: false} body
Resource Management:
- ChatNotifier.dispose() now disconnects WebSocket to prevent
connection leak when navigating away from chat
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backend (agent-engine.port.ts):
- Add `cancelled` event type: emitted when a task is cancelled (user-initiated
or injection), so Flutter can close the old stream cleanly
- Add `task_info` event type: emitted after inject to pass the new taskId to
the client, enabling cancel/re-inject on the replacement task
Flutter (features/chat/):
- ChatState: track current `taskId` alongside `sessionId`; clear on completion
or error
- Handle `TaskInfoEvent`: update taskId in state when server issues a new task
- Handle `CancelledEvent`: treat as stream termination (agentStatus → idle)
- MessageType.interrupted: new UI node (warning style) for mid-stream cancels
- _inject(): send text as an inject request while streaming; backend cancels
the current task and starts a new one with the injected message
- Input area: during streaming, hint changes to "追加指令...", Enter key calls
_inject() instead of _send(), and both inject-send + stop buttons are shown
- isAwaitingApproval kept separate from isStreaming so approval flow is not
blocked by inject mode
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously AgentSkillService wrote skills to public.agent_skills (TypeORM
entity with tenantId column filter), while ClaudeAgentSdkEngine read from
it0_t_{tenantId}.skills (per-tenant schema). The two tables were never
connected, so any skill added via the CRUD API was invisible to the agent.
This fix:
- Rewrites AgentSkillService to use DataSource + raw SQL against the
per-tenant schema it0_t_{tenantId}.skills
- Maps API fields: script→content, enabled→is_active
- Removes AgentSkillRepository and AgentSkill entity from module (no longer needed)
- CRUD API response shape is unchanged (fields mapped back to script/enabled)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Load active skills from the tenant's schema `skills` table and append
them to the system prompt before passing to the Claude Agent SDK. This
closes the gap where skills existed in the DB but were never surfaced
to the agent during task execution.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove iproute2/NET_ADMIN (no longer needed)
- Remove ip route hack from entrypoint.sh
- rwa-colocation-2 server record updated to use Docker gateway IP
since 14.215.128.96 is a host-local NIC on the IT0 server
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
14.215.128.96 is bound to a host NIC (enp5s0) and unreachable from
Docker bridge via default NAT. Add NET_ADMIN + ip route via gateway.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The IT0 server has its own id_ed25519 which differs from the local
key that's authorized on RWADurian servers. Use a dedicated key file.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The bind-mounted SSH key is owned by host uid (1000/node) but the
service runs as appuser (uid 1001). Use su-exec in entrypoint.sh
to copy the key as root, fix ownership, then drop privileges.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Mount host key to /tmp/host-ssh-key (read-only), then copy to
appuser's .ssh directory with correct ownership at container start.
Fixes "Permission denied" due to uid mismatch on bind mount.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add openssh-client to Dockerfile.service (alpine)
- Create .ssh directory with correct permissions for appuser
- Mount host SSH key into agent-service container (read-only)
This allows the Agent SDK to SSH into managed servers using the Bash tool.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The createCredential method was missing the tenantId assignment,
causing a NOT NULL constraint violation on the credentials table.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Server side (session_router.py):
- /reconnect now accepts sessions in "active" state (not just "disconnected")
- When client reconnects to an active session, the old WebSocket/pipeline is
automatically replaced when the new WebSocket connects
- Only truly terminal states (e.g. "ended") return 409
Flutter side (agent_call_page.dart):
- Distinguish terminal errors (404 session gone, 409 ended) from transient
errors (network timeout, server unreachable) in reconnect loop
- Terminal errors break immediately instead of wasting retry attempts
- Extract _connectWebSocket() helper for cleaner reconnect flow
- Add DioException handling for proper HTTP status code inspection
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
FlutterSoundPlayer.feedUint8FromStream() requires interleaved mode.
With interleaved=false, every feed() call threw:
"Cannot feed with UInt8 with non interleaved mode"
- feedUint8FromStream (Uint8List) → requires interleaved: true
- feedFromStream (Float32List) → requires interleaved: false
Since we feed raw PCM bytes (Uint8List), interleaved must be true.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: PcmPlayer called openPlayer() without audio session config,
so Android defaulted to earpiece-only mode. When the mic was actively
recording, playback was silently suppressed — the agent's TTS audio was
sent successfully over WebSocket but never reached the speaker.
Changes:
1. PcmPlayer (pcm_player.dart):
- Added audio_session package for proper audio session management
- Configure AudioSession with playAndRecord category so mic + speaker
work simultaneously
- Set voiceCommunication usage to enable Android hardware AEC (echo
cancellation) — prevents feedback loops when speaker is active
- defaultToSpeaker routes output to loudspeaker instead of earpiece
- Restored setSpeakerOn() method stub (used by UI toggle)
2. AgentCallPage (agent_call_page.dart):
- Fixed fire-and-forget bug: _pcmPlayer.feed() returns Future but was
called without await, causing interleaved feedUint8FromStream calls
- Added _feedChain serializer to guarantee sequential audio feeding
3. Dependencies:
- Added audio_session package to pubspec.yaml
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
OpenAI TTS returns 24kHz audio which Android MediaPlayer can't play
via FlutterSound's pcm16WAV codec. Request raw PCM and resample to
16kHz before wrapping in WAV header, matching the local TTS format.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Local /synthesize and /transcribe endpoints now auto-load Kokoro/Whisper
models on first call instead of returning 503 when not pre-loaded at
startup. This allows switching between Local and OpenAI providers in the
Flutter test page without requiring server restart.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: IOWebSocketChannel.sink.close() can hang indefinitely
(dart-lang/web_socket_channel#185). Previous fix used unawaited close
but didn't cancel the stream subscription, so the old listener could
still push events to _messageController.
Fix: Extract _closeCurrentConnection() that:
1. Cancels StreamSubscription first (stops duplicate events immediately)
2. Fire-and-forget sink.close(goingAway) (frees underlying socket)
This follows the workaround recommended in the official issue tracker.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The await on sink.close() blocks indefinitely when the server doesn't
respond to the close handshake. Use fire-and-forget with unawaited()
so the new connection can proceed immediately.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When sending a second message in the same session, the old WebSocket
connection was not closed, causing both connections to subscribe to the
same session room. This resulted in each text event being received twice,
producing garbled/duplicated output text.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Without this, the SDK engine fails to create tenant HOME directories
because the Docker volume mount point doesn't exist and appuser lacks
write permissions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace prompt-prefix workaround with SDK's native resume mechanism.
Each tenant gets isolated HOME directory (/data/claude-tenants/{tenantId})
to prevent cross-tenant session file mixing. SDK session IDs are persisted
in session.metadata for cross-request resume support.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement DB-based conversation message storage (engine-agnostic) that
works across both Claude API and Agent SDK engines. Add ChatGPT/Claude-style
conversation history drawer in Flutter with date-grouped session list,
session switching, and new chat functionality.
Backend: entity, repository, context service, migration 004, session/message
API endpoints. Flutter: ConversationDrawer, sessionId flow from backend
response via SessionInfoEvent, session list/switch/delete support.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove shared interceptors from the binary Dio instance to prevent
request dedup/retry interceptors from interfering with audio downloads.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add STT_PROVIDER/TTS_PROVIDER config (local or openai) in settings
- Pipeline uses OpenAI API for STT/TTS when provider is "openai"
- Skip loading local models (Kokoro/faster-whisper) when using OpenAI
- VAD (Silero) always loads for speech detection
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace hardcoded Colors.grey with Theme.of(context).colorScheme for
result containers and status text so they're readable in both light
and dark themes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add openai package to voice-service requirements
- Add /api/v1/test/tts/synthesize-openai (tts-1/tts-1-hd/gpt-4o-mini-tts)
- Add /api/v1/test/stt/transcribe-openai (gpt-4o-transcribe/whisper-1)
- Add OPENAI_API_KEY and OPENAI_BASE_URL env vars to voice-service
- Flutter test page: SegmentedButton to toggle Local/OpenAI provider
- All endpoints maintain same response format for easy comparison
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove record package dependency, use FlutterSoundRecorder instead
- Use permission_handler for microphone permission (already in pubspec)
- Proper temp file path via path_provider
- Cleanup temp files after upload
- Single package (flutter_sound) handles both recording and playback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The record package requires a valid file path. Empty string caused
ENOENT (No such file or directory) on Android.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
hailin