Commit Graph

47 Commits

Author SHA1 Message Date
wizardchen 492e92580b feat(observability): integrate Langfuse for LLM token tracking and tracing
Closes #620 #497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).

Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
  LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
  the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
  so logger.CloneContext can preserve it across handler / goroutine
  boundaries (otherwise each LLM call auto-created an orphan trace,
  fragmenting one request into many)

Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes

Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.

Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
  reuses WeKnora's existing PostgreSQL (separate database via an idempotent
  init container that fixes ICU collation drift) and Redis (separate DB
  number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
  to avoid Prisma P1013 when passwords contain @ / # / etc.

Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
2026-04-24 10:29:19 +08:00
wizardchen 53e8e17df4 feat(logger): implement LLM debug logging functionality
- Added a new logger for LLM calls, enabling detailed logging of model interactions, including request and response data.
- Introduced configuration options for enabling and specifying the log directory for LLM debug logs.
- Implemented cleanup for old log files to manage disk space effectively.
- Wrapped existing chat implementations to log calls when debug logging is enabled, enhancing traceability for model interactions.
2026-04-16 23:15:02 +08:00
AndyYang d5ecc150e0 feat(agent): support customizable LLM call timeout and add docker-compose mapping 2026-04-07 11:26:52 +08:00
nullkey 29de7dfbbd fix: allow MINIO_ENDPOINT to be configured via environment variable
Previously MINIO_ENDPOINT was hardcoded to minio:9000 in docker-compose.yml,
preventing users from connecting to an external MinIO service. Now it supports
override via .env while keeping the same default for backward compatibility.
2026-04-03 18:41:26 +08:00
Windfarer 54da98fc24 feat: add docx max pages env config 2026-04-02 10:31:52 +08:00
nullkey 0b64ef9bd1 fix(security): support IPv6 in SSRF validation via whitelist mechanism
- Keep strict mode blocking all direct IPs (IPv4 and IPv6 uniformly)
- Unify all SSRF call sites to use ValidateURLForSSRF (whitelist-aware)
- Add Teredo (2001:0000::/32) and 6to4 (2002::/16) tunnel detection
- Make redirect handler and DNS pinning respect SSRF_WHITELIST
- Unexport isSSRFSafeURL to prevent future callers bypassing whitelist
- Add scheme validation for whitelisted redirect targets
- Document IPv6 whitelist syntax in .env.example
- Add comprehensive IPv6 test coverage
2026-03-31 20:45:15 +08:00
Tavily PR Agent b84ca0b72a feat: add Tavily as web search provider option 2026-03-31 12:24:39 +08:00
Windfarer c1816fe6d6 add oidc 2026-03-30 11:13:44 +08:00
wizardchen 3a8bd36d8a feat(env): add SSRF whitelist configuration to .env.example and docker-compose.yml 2026-03-25 22:08:29 +08:00
Dounx 8df12aeee2 fix: make Milvus vector metric type configurable via MILVUS_METRIC_TYPE 2026-03-25 21:20:49 +08:00
Dounx fc5c405639 feat(dev): add milvus service and db host config 2026-03-20 15:50:16 +08:00
wizardchen 139a9c40ff feat: enhance localization and configuration support
- Added timezone and language settings to the environment configuration.
- Introduced built-in agent configurations with multilingual support for various agents.
- Updated Docker Compose to utilize new environment variables for timezone and language.
- Created new prompt templates for question generation, summary generation, and keywords extraction.
2026-03-17 22:23:24 +08:00
DaWesen e309e0bed8 feat(storage): 集成S3存储适配器
添加对AWS S3及兼容存储服务的支持:
- 实现完整的S3FileService接口
- 支持文件上传、下载、删除功能
- 添加配置支持和环境变量检查
- 实现连接测试功能
- 遵循与其他存储适配器相同的代码风格
2026-03-09 10:39:46 +08:00
MaoMengww c4a5a4d99b feat: support weaviate vectordb for knowledge retrieve 2026-03-09 10:36:16 +08:00
AndyYang 6c69de2df1 feat(security): add AES-256-GCM encryption for API keys at rest
- Add crypto utility (internal/utils/crypto.go) with AES-256-GCM encrypt/decrypt
  using SYSTEM_AES_KEY env var, with "enc:v1:" prefix for versioned ciphertext
- Encrypt tenant API key via GORM BeforeSave/AfterFind hooks and manual
  encryption in CreateTenant/UpdateAPIKey (db.Updates bypasses hooks)
- Encrypt model API key in ModelParameters Value/Scan (driver.Valuer)
- Widen api_key column from varchar(64) to varchar(256) across all DB dialects
  (MySQL, ParadeDB, SQLite) and add versioned migration 000018
- Propagate SYSTEM_AES_KEY through docker-compose, Helm secrets and values
- Fix migration 000017 PL/pgSQL dollar-quoting syntax ($ -> $$)
2026-03-09 10:35:07 +08:00
wizardchen 397689d2f3 feat: introduce WeKnora Lite edition with lightweight configuration and deployment
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.

This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
2026-03-02 21:21:49 +08:00
joeyczheng 1b56e99cac feat: support milvus vectordb for knowledge retrieve
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-02-27 09:51:17 +08:00
Dounx 6adecdb30e feat: add volcengine tos support 2026-02-25 14:24:50 +08:00
wizardchen bfab05972f feat: support remote backend and HTTPS proxy 2026-02-09 17:58:35 +08:00
Dounx 66756de19f feat(frontend): allow configurable backend host and port 2026-02-06 20:05:42 +08:00
wizardchen 2d6efec84f feat: enhance agent skills sandbox configuration and availability
- Updated .env.example to set default sandbox mode to 'docker' and added timeout and docker image variables.
- Modified docker-compose files to include a sandbox service for building and pulling the sandbox image.
- Adjusted frontend API to reflect sandbox availability for skills, ensuring UI elements are conditionally displayed based on sandbox status.
- Implemented backend logic to disable skills when the sandbox is not enabled, improving error handling and user experience.
2026-02-04 20:58:24 +08:00
nullkey 03446f35d3 feat: adjust sandbox setting of agentConfig 2026-02-04 20:08:49 +08:00
Dounx 6e03f1ea79 feat(redis): add REDIS_USERNAME support for Redis ACL 2026-02-04 19:38:40 +08:00
begoniezhao 10c2be0e6e feat: Add configurable global log level via env 2026-02-04 19:18:35 +08:00
begoniezhao 1abdaa5d5c feat: Make OCR and task concurrency configurable 2026-01-15 10:56:09 +08:00
wizardchen 5152f37195 feat: 添加 WEKNORA_VERSION 环境变量支持
允许通过环境变量统一控制所有服务的镜像版本标签,默认使用 latest
2025-12-31 18:25:58 +08:00
begoniezhao a13c4a7af7 feat: add APK mirror configuration and enhance chunking separator options 2025-12-30 17:19:21 +08:00
wizardchen 6cf7cbcb9c feat: 添加可配置的文件上传大小限制
新增 MAX_FILE_SIZE_MB 环境变量统一控制文件上传大小,默认 50MB
2025-12-30 14:35:10 +08:00
begoniezhao 907e9a5522 feat: Add DataSchema tool for retrieving schema information from CSV and Excel files 2025-12-29 20:03:51 +08:00
wizardchen a36bf67166 feat: 新增环境变量DISABLE_REGISTRATION控制用户注册开关 2025-12-22 15:01:09 +08:00
wizardchen d6c3636b20 refactor: 调整默认生产环境配置,禁用非生产环境下的Swagger文档访问 2025-12-22 15:01:09 +08:00
wizardchen c67a5d3949 feat: 支持数据库迁移脏状态自动恢复和Neo4j连接重试机制 2025-12-08 19:38:16 +08:00
comqx d23863db63 chore: Update .env.example by removing unused DB_PORT variable and add highlight.js dependency in package.json and package-lock.json 2025-12-08 19:37:38 +08:00
comqx a15e9cd1e6 change env postgress port 2025-12-08 19:37:38 +08:00
wizardchen b1c79ccd4d Merge remote-tracking branch 'github_public/main' 2025-12-05 14:21:56 +08:00
Anush008 c7659c3d19 feat: Qdrant Vector Search support
Signed-off-by: Anush008 <anushshetty90@gmail.com>
2025-11-27 11:17:23 +08:00
wizardchen d41f8afe55 feat: Add JWT_SECRET configuration to .env.example and enhance pagination validation in chunk handler 2025-11-26 17:58:31 +08:00
begoniezhao af620806e0 docs: 新增 Docker Compose 启动配置说明,调整 docker-compose.yml 配置 2025-11-18 18:31:12 +08:00
begoniezhao 02b78a5908 feat: 新增异步任务提取服务 2025-10-16 17:48:21 +08:00
wizardchen fd6c50059e feat: support modify listening port 2025-09-10 10:14:43 +08:00
wizardchen 785261313f feat: make CONCURRENCY_POOL_SIZE configurable 2025-08-16 13:27:01 +08:00
begoniezhao f77720155c feat: Added WEB_PROXY environment variable to optimize web content processing 2025-08-14 17:09:11 +08:00
wizardchen 57c6e04e54 fix: 调整构建镜像采用的镜像源 2025-08-14 12:16:08 +08:00
wizardchen 8b43931886 feat: support minio storage 2025-08-14 12:16:08 +08:00
wizardchen bdabed6bfa feat: Added web page for configuring model information 2025-08-10 17:11:07 +08:00
begoniezhao 24c190c492 feat: 新增多模态模型配置及 VLM 模型认证 2025-08-08 17:05:24 +08:00
wizardchen 56eb2bce33 init commit 2025-08-05 15:08:07 +08:00