hailin
/
it0
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
193 Commits 1 Branch 0 Tags 8 MiB
Commit Graph

9 Commits

Author SHA1 Message Date
hailin e302891f16 fix: disable SSL verify for self-signed OpenAI proxy + handle no-user-msg 
- Pass httpx.AsyncClient(verify=False) to OpenAI STT/TTS to support
  self-signed certificate on OPENAI_BASE_URL proxy
- Handle generate_reply calls with no user message by falling back to
  system/developer instructions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 21:39:49 -08:00
hailin 4d47c6a955 fix: remove wait_for_participant — room not connected in rtc_session mode 
In livekit-agents v1.x @server.rtc_session() pattern, ctx.room is not
yet connected when entrypoint is called. session.start() handles room
connection internally.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 21:15:37 -08:00
hailin 2112445191 fix: voice-agent crash — add room I/O options and filter AgentConfigUpdate 
- Add room_input_options/room_output_options to session.start() so agent
  binds audio I/O and stays in the room
- Add wait_for_participant() before starting session
- Filter AgentConfigUpdate items in agent_llm.py (no 'role' attribute)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 21:08:07 -08:00
hailin 00be878a95 fix: refactor voice-agent to official LiveKit v1.x AgentServer pattern 
Replace deprecated WorkerOptions(entrypoint_fnc=...) with AgentServer() +
@server.rtc_session() decorator. Use server.setup_fnc for prewarm. Remove
manual ctx.connect() and ctx.wait_for_participant() calls that prevented
the pipeline from properly wiring up VAD→STT→LLM→TTS.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 12:31:31 -08:00
hailin 75b14d5200 fix: use RoomOptions instead of deprecated RoomInputOptions 
RoomInputOptions is deprecated in livekit-agents 1.4.x. Switch to
RoomOptions with explicit audio_input/audio_output enabled.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 11:32:36 -08:00
hailin 23b5bce983 fix: extract auth header from job.metadata instead of agent_dispatch 
LiveKit passes RoomAgentDispatch metadata through as job.metadata
(protobuf field), not via a separate agent_dispatch object. Also
use room_io.RoomInputOptions for participant targeting (livekit-agents 1.x).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 11:04:02 -08:00
hailin f1d50e43f1 fix: update AgentSession.start() for livekit-agents 1.x API 
livekit-agents 1.x removed the 'participant' parameter from
AgentSession.start(). Use room_input_options with participant_identity
instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 10:31:04 -08:00
hailin 112c445143 fix: resolve websockets version conflict and use CPU-only torch 
- Upgrade websockets from ==12.0 to >=13.0 (openai[realtime] requires >=13)
- Install torch CPU-only build separately in Dockerfile to avoid ~2GB CUDA download
- Remove torch from requirements.txt (installed via --index-url cpu wheel)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 09:02:31 -08:00
hailin 94a14b3104 feat: migrate voice call from WebSocket/PCM to LiveKit WebRTC 
实时语音对话架构迁移：WebSocket → LiveKit WebRTC

## 背景
原语音通话架构基于 FastAPI WebSocket 传输原始 PCM，管道串行执行
（VAD → 批量STT → Agent → 攒句 → 批量TTS），首音频延迟约 6 秒。
迁移到 LiveKit Agents 框架后，利用 WebRTC 传输 + 流水线并行，
预期延迟降至 1.5-2 秒。

## 架构
Flutter App ←── WebRTC (Opus/UDP) ──→ LiveKit Server ←──→ Voice Agent
  livekit_client                      (自部署, Go)       (Python, LiveKit Agents SDK)
                                                          ├─ VAD (Silero)
                                                          ├─ STT (faster-whisper / OpenAI)
                                                          ├─ LLM (自定义插件 → agent-service)
                                                          └─ TTS (Kokoro / OpenAI)

关键设计：LLM 不直接调用 Claude API，而是通过自定义插件代理到现有
agent-service，保留 Tool Use、会话历史、租户隔离等能力。

## 新增服务

### voice-agent (packages/services/voice-agent/)
LiveKit Agent Worker，包含：
- agent.py: 入口，prewarm() 预加载模型，entrypoint() 编排会话
- plugins/agent_llm.py: 自定义 LLM 插件，代理 agent-service API
  - POST /api/v1/agent/tasks 创建任务
  - WS /ws/agent 订阅流式事件 (stream_event)
  - 跨轮复用 session_id 保持对话上下文
- plugins/whisper_stt.py: 本地 faster-whisper STT (批量识别)
- plugins/kokoro_tts.py: 本地 Kokoro-82M TTS (24kHz PCM)
- config.py: pydantic-settings 配置

### LiveKit Server (deploy/docker/)
- livekit.yaml: 信令端口 7880, RTC TCP 7881, UDP 50000-50200
- docker-compose.yml: 新增 livekit-server + voice-agent 容器

### LiveKit Token 端点
- voice-service/src/api/livekit_token.py:
  POST /api/v1/voice/livekit/token
  生成 Room JWT，嵌入 auth_header 到 AgentDispatch metadata

## Flutter 客户端改造
- agent_call_page.dart: 从 ~814 行简化到 ~380 行
  - 替换: WebSocketChannel, AudioRecorder, PcmPlayer, 手动心跳/重连
  - 使用: Room.connect(), setMicrophoneEnabled(true), LiveKit 事件监听
  - 波形动画改用 participant.audioLevel
- pubspec.yaml: 添加 livekit_client: ^2.3.0
- app_config.dart: 增加 livekitUrl 字段
- api_endpoints.dart: 增加 livekitToken 端点

## 配置说明 (环境变量)
- STT_PROVIDER: local (默认, faster-whisper) / openai
- TTS_PROVIDER: local (默认, Kokoro) / openai
- WHISPER_MODEL: base (默认) / small / medium / large
- WHISPER_LANGUAGE: zh (默认)
- KOKORO_VOICE: zf_xiaoxiao (默认)
- DEVICE: cpu (默认) / cuda

## 不变的部分
- agent-service: 完全不改，voice-agent 通过现有 API 调用
- voice-service 核心: pipeline/STT/TTS/VAD 保留 (Twilio 备用)
- Kong 网关: 现有路由不变
- 数据库: 无 schema 变更

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 08:55:33 -08:00