feat(observability): extend Langfuse tracing across asynq pipeline

Previously the Langfuse integration only traced in-process HTTP requests
(chat / search / eval), so file uploads and every downstream asynq task
(document parse, chunk embedding, OCR/VLM, summary / question gen, wiki
ingest, datasource sync, etc.) produced either disconnected shallow
traces or no observation at all.

This change threads one trace end-to-end:

- tracer: add SPAN observation type and StartSpan; add ResumeTrace so a
  worker can attach to an upstream trace without emitting a duplicate
  trace-create; StartGeneration now auto-picks parentObservationId from
  ctx so nested trace -> span -> generation trees render correctly.
- types.TracingContext + LangfuseTracingCarrier: embed on all 17 asynq
  payloads so trace_id / parent_obs_id / user_id / session_id serialise
  into every job.
- langfuse.InjectTracing: injected at 28 enqueue sites before json.Marshal
  so the HTTP-layer trace survives the Redis hop.
- langfuse.AsynqMiddleware: mux.Use hook that peeks the payload, either
  resumes the upstream trace or opens a standalone asynq.<type> trace
  for scheduled jobs, and wraps the handler in a SPAN with task metadata
  (id / queue / retry / payload_bytes) plus ERROR level on failure.
- GinMiddleware.shouldTrace: whitelist ingestion / knowledge-mutation /
  FAQ / wiki / datasource endpoints so the root trace actually starts.
- Tests: tracer_test.go covers span nesting, error status, and
  ResumeTrace no-trace-create guarantee; asynq_test.go covers
  InjectTracing round-trip, middleware resume path, and standalone
  trace fallback.
- Docs: docs/Langfuse\u96c6\u6210.md now lists the covered task types
  and documents the cross-process propagation model.

No behavioural change when Langfuse is disabled (all new code paths are
no-ops and carriers serialise to empty strings with omitempty).
This commit is contained in:
wizardchen
2026-04-24 12:23:30 +08:00
committed by lyingbug
parent 59fbf77524
commit beca2b89a3
23 changed files with 1077 additions and 40 deletions
+32 -7
View File
@@ -5,8 +5,9 @@ WeKnora 内置了对 [Langfuse](https://langfuse.com) 的轻量级集成,用
## 1. 特性
- 自动上报 **chat / embedding / rerank / VLM(视觉语言模型)/ ASR(语音识别)** 全部 5 类模型调用的 prompt、响应和 token 使用量。
- 为每个对话或检索请求创建一条 **trace**,对应 HTTP 请求级别,方便按 session / user 聚合。
- 为每个对话、检索、**文件上传及后续异步处理**创建一条端到端 **trace**。HTTP 请求是根,asynq 任务以 SPAN 的形式挂在同一条 trace 下,文档解析 → chunk embedding → 多模态 OCR/Caption → 摘要 / 问题生成全部在同一棵树里可见。
- 支持 **流式响应**:记录首 token 延迟(Time-To-First-Token),完整响应在流结束后一次性写入。
- **跨进程 trace 透传**:HTTP 层把 `trace_id` / `parent_observation_id` 注入 asynq payload,worker 在 asynq middleware 层自动 resume;定时任务(例如数据源同步)则退化为独立 trace,依然按任务类型(`asynq.<type>`)聚合。
- **完全可选**:不配置 `LANGFUSE_*` 环境变量时,Langfuse 相关代码路径是 no-op,不产生任何性能开销。
- **异步批量上报**:不阻塞业务请求;队列满时静默丢弃,观测数据不会影响用户对话。
- **开箱即用的部署方式**:Docker Compose(`docker-compose.yml` 已内置环境变量)、Helm Chart(通过 `extraEnv`)、Lite 版本(本地单机)均支持。
@@ -226,15 +227,34 @@ Dev 相关容器都带 `-dev` 后缀、用独立网络 `WeKnora-network-dev`,
| Langfuse 概念 | WeKnora 对应 | 备注 |
| --- | --- | --- |
| Trace | 一次 HTTP 请求 | 自动在 `/api/v1/knowledge-chat`、`/api/v1/agent-chat`、`/api/v1/knowledge-search`、`generate_title`、模型连通性测试等端点创建。 |
| Generation(type=GENERATION) | 每次 chat / embedding / rerank / VLM / ASR 调用 | 包含 model、prompt、响应、token usage、modelParameters(temperature / top_p / max_tokens 等)。 |
| Trace | 一次 HTTP 请求(含其触发的所有 asynq 任务) | 对于 `knowledge-chat`、`agent-chat`、`knowledge-search`、`generate_title`、`evaluation`、模型连通性测试等在线请求;以及文件上传/URL 入库/manual/reparse/move/copy、FAQ 导入、知识修改、wiki auto-fix、数据源手工触发等入库请求,HTTP 层都会开启 trace,并把 `trace_id` / `parent_observation_id` 注入 asynq payload。 |
| Span(type=SPAN) | 每个 asynq 任务的执行窗口 | 由 `internal/tracing/langfuse/AsynqMiddleware` 在 `mux.Use` 注册;对每个 handler 自动创建 `asynq.<task_type>` 的 SPAN,并记录 `task_id` / `queue` / `retry` / `payload_bytes`。定时任务(无上游 trace)会退化为 `asynq.<task_type>` 独立 trace。 |
| Generation(type=GENERATION) | 每次 chat / embedding / rerank / VLM / ASR 调用 | 若位于 span 下会自动设置 `parentObservationId`,所以 Langfuse UI 呈现 trace → asynq-span → generation 的树状结构。 |
| Input Tokens | `TokenUsage.PromptTokens` | 来自模型返回的 usage 字段。 |
| Output Tokens | `TokenUsage.CompletionTokens` | 来自模型返回的 usage 字段。 |
| Total Tokens | `TokenUsage.TotalTokens` | 大多数厂商返回;未返回时自动求和。 |
| `userId` | `X-User-ID` / 租户 ID | 未登录时退化为 `tenant:<id>`,方便按租户汇总消耗。 |
| `sessionId` | URL 中的 `:session_id` | 可以在 Langfuse 的 Sessions 视图聚合一整场对话。 |
| `userId` | `X-User-ID` / 租户 ID | 未登录时退化为 `tenant:<id>`,方便按租户汇总消耗;enqueue 时会写入 payload,worker 在无上游 trace 的场景也能保留归属。 |
| `sessionId` | URL 中的 `:session_id`(或 `RequestID` 兜底) | 可以在 Langfuse 的 Sessions 视图聚合一整场对话,或按单次异步批次聚合。 |
| Time-To-First-Token | 流式调用首条有效 chunk 的时间 | 通过 `generation-update.completionStartTime` 上报。 |
### 覆盖到的 asynq 任务类型
下表列出当前会在 Langfuse 里自动出现对应 SPAN 的 asynq 任务;每种任务的 payload 均已嵌入 `types.TracingContext`,enqueue 时由 `langfuse.InjectTracing(ctx, &payload)` 从当前 HTTP trace 拷出 `trace_id` / `parent_observation_id`。
| 任务类型常量 | Handler | 典型触发来源 |
| --- | --- | --- |
| `document:process` | `knowledgeService.ProcessDocument` | 文件 / URL / 文本 / file_url 四种入库;reparse;知识库克隆内部重派发 |
| `manual:process` | `knowledgeService.ProcessManualKnowledge` | 手工知识新建 / 更新 |
| `image:multimodal` | `ImageMultimodalService.Handle` | 文档解析时发现图片 |
| `knowledge:post_process` | `KnowledgePostProcessService.Handle` | 文档解析完成后统一调度 summary/question |
| `summary:generation` / `question:generation` | `KnowledgePostProcessService` 子任务 | 由 `knowledge:post_process` 派发 |
| `chunk:extract` | `ChunkExtractor.Handle` | 图谱提取(NEO4J 启用时) |
| `datatable:summary` | `DataTableSummaryService.Handle` | 表格文件解析 |
| `faq:import` | FAQ 批量导入 handler | FAQ 导入 / 批量创建 |
| `knowledge:move` / `knowledge:list_delete` / `index:delete` / `kb:clone` / `kb:delete` | 知识移动 / 批量删除 / 索引清理 / 知识库复制 / 知识库删除 | 对应 HTTP 路由 |
| `wiki:ingest` | `wikiIngestService.ProcessWikiIngest` | Wiki auto-fix / 重建链接 |
| `datasource:sync` | `dataSourceSyncService.Handle` | 数据源手动触发 + 定时调度(定时场景下 trace 为 standalone) |
### 各模型的 usage 处理策略
| 模型类型 | 上报名称 | Token 计量方式 | 备注 |
@@ -270,12 +290,17 @@ Dev 相关容器都带 `-dev` 后缀、用独立网络 `WeKnora-network-dev`,
## 8. 代码位置
- `internal/tracing/langfuse/` — Langfuse 客户端、异步批量上报、Gin 中间件。
- `internal/tracing/langfuse/` — Langfuse 客户端、异步批量上报、Gin 中间件、**asynq middleware**、Span / Trace resume 实现。
- `tracer.go` — 暴露 `Trace` / `Span` / `Generation` + `StartTrace` / `StartSpan` / `StartGeneration` / `ResumeTrace`。
- `asynq.go` — `AsynqMiddleware()` 统一在 mux 上包 handler;`InjectTracing(ctx, payload)` 在 enqueue 侧把 trace/span ID 注入 payload。
- `middleware.go` — Gin 中间件 + `shouldTrace` 白名单(覆盖 chat / 入库 / FAQ / wiki / 数据源等路径)。
- `internal/types/tracing.go` — `TracingContext` POCO,所有 asynq payload 通过嵌入此结构携带 `lf_trace_id` / `lf_parent_obs_id` / `lf_user_id` / `lf_session_id`。
- `internal/models/chat/langfuse_wrapper.go` — Chat 调用装饰器(含流式)。
- `internal/models/embedding/langfuse_wrapper.go` — Embedding 调用装饰器。
- `internal/models/rerank/langfuse_wrapper.go` — Rerank 调用装饰器。
- `internal/models/vlm/langfuse_wrapper.go` — VLM(视觉语言模型)调用装饰器。
- `internal/models/asr/langfuse_wrapper.go` — ASR(语音识别)调用装饰器。
- `internal/router/router.go` — 注册 `langfuse.GinMiddleware()`。
- `internal/router/task.go` — 在 asynq mux 上 `mux.Use(langfuse.AsynqMiddleware())`,使所有 handler 自动被 trace。
- `internal/container/container.go` — 初始化 + 资源清理。
- `internal/router/router.go` — 自动为 LLM 相关路由创建 trace。
- `docker-compose.yml` / `.env.example` / `.env.lite.example` — 预置 `LANGFUSE_*` 环境变量直通。
@@ -12,6 +12,7 @@ import (
"github.com/Tencent/WeKnora/internal/datasource"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/hibiken/asynq"
@@ -298,6 +299,7 @@ func (s *DataSourceService) ManualSync(ctx context.Context, dsID string) (*types
SyncLogID: syncLog.ID,
ForceFull: false,
}
langfuse.InjectTracing(ctx, payload)
payloadJSON, _ := json.Marshal(payload)
task := asynq.NewTask(types.TypeDataSourceSync, payloadJSON)
+10 -4
View File
@@ -16,6 +16,7 @@ import (
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/models/embedding"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/google/uuid"
@@ -91,11 +92,13 @@ func NewChunkExtractTask(
logger.Warn(ctx, "NEO4J is not enabled, skip chunk extract task")
return nil
}
payload, err := json.Marshal(types.ExtractChunkPayload{
taskPayload := types.ExtractChunkPayload{
TenantID: tenantID,
ChunkID: chunkID,
ModelID: modelID,
})
}
langfuse.InjectTracing(ctx, &taskPayload)
payload, err := json.Marshal(taskPayload)
if err != nil {
return err
}
@@ -118,12 +121,14 @@ func NewDataTableSummaryTask(
summaryModel string,
embeddingModel string,
) error {
payload, err := json.Marshal(DataTableSummaryPayload{
taskPayload := DataTableSummaryPayload{
TenantID: tenantID,
KnowledgeID: knowledgeID,
SummaryModel: summaryModel,
EmbeddingModel: embeddingModel,
})
}
langfuse.InjectTracing(ctx, &taskPayload)
payload, err := json.Marshal(taskPayload)
if err != nil {
return err
}
@@ -238,6 +243,7 @@ func (s *ChunkExtractService) Handle(ctx context.Context, t *asynq.Task) error {
// DataTableExtractPayload represents the table extract task payload
type DataTableSummaryPayload struct {
types.TracingContext
TenantID uint64 `json:"tenant_id"`
KnowledgeID string `json:"knowledge_id"`
SummaryModel string `json:"summary_model"`
@@ -14,6 +14,7 @@ import (
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/models/utils/ollama"
"github.com/Tencent/WeKnora/internal/models/vlm"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
secutils "github.com/Tencent/WeKnora/internal/utils"
@@ -419,6 +420,7 @@ func (s *ImageMultimodalService) enqueueKnowledgePostProcessTask(ctx context.Con
KnowledgeBaseID: payload.KnowledgeBaseID,
Language: payload.Language,
}
langfuse.InjectTracing(ctx, &taskPayload)
payloadBytes, err := json.Marshal(taskPayload)
if err != nil {
logger.Warnf(ctx, "[ImageMultimodal] Failed to marshal post process payload: %v", err)
+16 -2
View File
@@ -31,6 +31,7 @@ import (
"github.com/Tencent/WeKnora/internal/models/embedding"
"github.com/Tencent/WeKnora/internal/searchutil"
"github.com/Tencent/WeKnora/internal/tracing"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
secutils "github.com/Tencent/WeKnora/internal/utils"
@@ -429,6 +430,7 @@ func (s *knowledgeService) CreateKnowledgeFromFile(ctx context.Context,
Language: lang,
}
langfuse.InjectTracing(ctx, &taskPayload)
payloadBytes, err := json.Marshal(taskPayload)
if err != nil {
logger.Errorf(ctx, "Failed to marshal document process task payload: %v", err)
@@ -603,6 +605,7 @@ func (s *knowledgeService) CreateKnowledgeFromURL(ctx context.Context,
Language: lang,
}
langfuse.InjectTracing(ctx, &taskPayload)
payloadBytes, err := json.Marshal(taskPayload)
if err != nil {
logger.Errorf(ctx, "Failed to marshal URL process task payload: %v", err)
@@ -825,6 +828,7 @@ func (s *knowledgeService) createKnowledgeFromFileURL(
Language: lang,
}
langfuse.InjectTracing(ctx, &taskPayload)
payloadBytes, err := json.Marshal(taskPayload)
if err != nil {
logger.Errorf(ctx, "Failed to marshal file URL process task payload: %v", err)
@@ -1043,6 +1047,7 @@ func (s *knowledgeService) createKnowledgeFromPassageInternal(ctx context.Contex
Language: lang,
}
langfuse.InjectTracing(ctx, &taskPayload)
payloadBytes, err := json.Marshal(taskPayload)
if err != nil {
logger.Errorf(ctx, "Failed to marshal passage process task payload: %v", err)
@@ -2206,12 +2211,14 @@ func (s *knowledgeService) processChunks(ctx context.Context,
} else {
// If there are no multimodal tasks, enqueue the post process task immediately
lang, _ := types.LanguageFromContext(ctx)
payloadBytes, err := json.Marshal(types.KnowledgePostProcessPayload{
postProcessPayload := types.KnowledgePostProcessPayload{
TenantID: knowledge.TenantID,
KnowledgeID: knowledge.ID,
KnowledgeBaseID: knowledge.KnowledgeBaseID,
Language: lang,
})
}
langfuse.InjectTracing(ctx, &postProcessPayload)
payloadBytes, err := json.Marshal(postProcessPayload)
if err == nil {
task := asynq.NewTask(types.TypeKnowledgePostProcess, payloadBytes, asynq.Queue("default"), asynq.MaxRetry(3))
if _, err := s.task.Enqueue(task); err != nil {
@@ -3076,6 +3083,7 @@ func (s *knowledgeService) enqueueManualProcessing(ctx context.Context,
Content: content,
NeedCleanup: needCleanup,
}
langfuse.InjectTracing(ctx, &payload)
payloadBytes, err := json.Marshal(payload)
if err != nil {
return fmt.Errorf("failed to marshal manual process payload: %w", err)
@@ -3202,6 +3210,7 @@ func (s *knowledgeService) ReparseKnowledge(ctx context.Context, knowledgeID str
Language: lang,
}
langfuse.InjectTracing(ctx, &taskPayload)
payloadBytes, err := json.Marshal(taskPayload)
if err != nil {
logger.Errorf(ctx, "Failed to marshal reparse task payload: %v", err)
@@ -3254,6 +3263,7 @@ func (s *knowledgeService) ReparseKnowledge(ctx context.Context, knowledgeID str
Language: lang,
}
langfuse.InjectTracing(ctx, &taskPayload)
payloadBytes, err := json.Marshal(taskPayload)
if err != nil {
logger.Errorf(ctx, "Failed to marshal file URL reparse task payload: %v", err)
@@ -3299,6 +3309,7 @@ func (s *knowledgeService) ReparseKnowledge(ctx context.Context, knowledgeID str
Language: lang,
}
langfuse.InjectTracing(ctx, &taskPayload)
payloadBytes, err := json.Marshal(taskPayload)
if err != nil {
logger.Errorf(ctx, "Failed to marshal URL reparse task payload: %v", err)
@@ -4077,6 +4088,7 @@ func (s *knowledgeService) UpsertFAQEntries(ctx context.Context,
taskPayload.Entries = payload.Entries
}
langfuse.InjectTracing(ctx, &taskPayload)
payloadBytes, err := json.Marshal(taskPayload)
if err != nil {
logger.Errorf(ctx, "Failed to marshal FAQ import task payload: %v", err)
@@ -8461,6 +8473,7 @@ func (s *knowledgeService) enqueueImageMultimodalTasks(
ImageSourceType: metadata["image_source_type"],
}
langfuse.InjectTracing(ctx, &payload)
payloadBytes, err := json.Marshal(payload)
if err != nil {
logger.Warnf(ctx, "Failed to marshal image multimodal payload: %v", err)
@@ -9676,6 +9689,7 @@ func (s *knowledgeService) moveKnowledgeReparse(
Language: lang,
}
langfuse.InjectTracing(ctx, &taskPayload)
payloadBytes, err := json.Marshal(taskPayload)
if err != nil {
return fmt.Errorf("failed to marshal document process payload: %w", err)
@@ -7,6 +7,7 @@ import (
"time"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/hibiken/asynq"
@@ -142,6 +143,7 @@ func (s *KnowledgePostProcessService) enqueueSummaryGenerationTask(ctx context.C
KnowledgeID: payload.KnowledgeID,
Language: payload.Language,
}
langfuse.InjectTracing(ctx, &taskPayload)
payloadBytes, err := json.Marshal(taskPayload)
if err != nil {
logger.Warnf(ctx, "[KnowledgePostProcess] Failed to marshal summary generation payload: %v", err)
@@ -180,6 +182,7 @@ func (s *KnowledgePostProcessService) enqueueQuestionGenerationIfEnabled(ctx con
QuestionCount: questionCount,
Language: payload.Language,
}
langfuse.InjectTracing(ctx, &taskPayload)
payloadBytes, err := json.Marshal(taskPayload)
if err != nil {
logger.Warnf(ctx, "[KnowledgePostProcess] Failed to marshal question generation payload: %v", err)
@@ -8,6 +8,7 @@ import (
"github.com/Tencent/WeKnora/internal/application/service/retriever"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/google/uuid"
@@ -384,6 +385,7 @@ func (s *knowledgeBaseService) DeleteKnowledgeBase(ctx context.Context, id strin
KnowledgeBaseID: id,
EffectiveEngines: tenantInfo.GetEffectiveEngines(),
}
langfuse.InjectTracing(ctx, &payload)
payloadBytes, err := json.Marshal(payload)
if err != nil {
+3
View File
@@ -10,6 +10,7 @@ import (
"github.com/Tencent/WeKnora/internal/application/service/retriever"
werrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/google/uuid"
@@ -282,6 +283,7 @@ func (s *knowledgeTagService) DeleteTag(ctx context.Context, id string, force bo
TenantID: tenantID,
KnowledgeIDs: knowledgeIDs,
}
langfuse.InjectTracing(ctx, &payload)
payloadBytes, err := json.Marshal(payload)
if err != nil {
logger.Errorf(ctx, "Failed to marshal knowledge list delete payload: %v", err)
@@ -351,6 +353,7 @@ func (s *knowledgeTagService) enqueueIndexDeleteTask(ctx context.Context,
ChunkIDs: chunkIDs,
EffectiveEngines: effectiveEngines,
}
langfuse.InjectTracing(ctx, &payload)
payloadBytes, err := json.Marshal(payload)
if err != nil {
logger.Errorf(ctx, "Failed to marshal index delete payload: %v", err)
@@ -14,6 +14,7 @@ import (
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/searchutil"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/hibiken/asynq"
@@ -92,6 +93,7 @@ func WikiDeletedTombstoneKey(kbID, knowledgeID string) string {
// The actual document IDs are stored in a Redis list (wiki:pending:{kbID}).
// KnowledgeID is only used as fallback in Lite mode (no Redis).
type WikiIngestPayload struct {
types.TracingContext
TenantID uint64 `json:"tenant_id"`
KnowledgeBaseID string `json:"knowledge_base_id"`
Language string `json:"language,omitempty"`
@@ -101,6 +103,7 @@ type WikiIngestPayload struct {
// WikiRetractPayload is the asynq task payload for wiki content retraction
type WikiRetractPayload struct {
types.TracingContext
TenantID uint64 `json:"tenant_id"`
KnowledgeBaseID string `json:"knowledge_base_id"`
KnowledgeID string `json:"knowledge_id"`
@@ -208,6 +211,7 @@ func EnqueueWikiIngest(ctx context.Context, task interfaces.TaskEnqueuer, redisC
}}
}
langfuse.InjectTracing(ctx, &payload)
payloadBytes, _ := json.Marshal(payload)
t := asynq.NewTask(types.TypeWikiIngest, payloadBytes,
@@ -248,6 +252,7 @@ func EnqueueWikiRetract(ctx context.Context, task interfaces.TaskEnqueuer, redis
ingestPayload.LiteOps = []WikiPendingOp{op}
}
langfuse.InjectTracing(ctx, &ingestPayload)
payloadBytes, _ := json.Marshal(ingestPayload)
t := asynq.NewTask(types.TypeWikiIngest, payloadBytes,
asynq.Queue("low"),
@@ -11,6 +11,7 @@ import (
"github.com/Tencent/WeKnora/internal/agent"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/google/uuid"
"github.com/hibiken/asynq"
@@ -29,6 +30,7 @@ func (s *wikiIngestService) scheduleFollowUp(ctx context.Context, payload WikiIn
logger.Infof(ctx, "wiki ingest: %d more documents pending for KB %s, scheduling follow-up", count, payload.KnowledgeBaseID)
langfuse.InjectTracing(ctx, &payload)
payloadBytes, _ := json.Marshal(payload)
t := asynq.NewTask(types.TypeWikiIngest, payloadBytes,
asynq.Queue("low"),
+2
View File
@@ -8,6 +8,7 @@ import (
"time"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/hibiken/asynq"
@@ -168,6 +169,7 @@ func (s *Scheduler) triggerSync(dataSourceID string, tenantID uint64) {
SyncLogID: syncLog.ID,
ForceFull: false,
}
langfuse.InjectTracing(ctx, payload)
payloadJSON, _ := json.Marshal(payload)
task := asynq.NewTask(types.TypeDataSourceSync, payloadJSON)
+3
View File
@@ -18,6 +18,7 @@ import (
"github.com/Tencent/WeKnora/internal/application/service"
"github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/Tencent/WeKnora/internal/utils"
@@ -686,6 +687,7 @@ func (h *KnowledgeHandler) ClearKnowledgeBaseContents(c *gin.Context) {
TenantID: effectiveTenantID,
KnowledgeIDs: knowledgeIDs,
}
langfuse.InjectTracing(ctx, &payload)
payloadBytes, err := json.Marshal(payload)
if err != nil {
logger.Errorf(ctx, "Failed to marshal knowledge list delete payload: %v", err)
@@ -1559,6 +1561,7 @@ func (h *KnowledgeHandler) MoveKnowledge(c *gin.Context) {
TargetKBID: req.TargetKBID,
Mode: req.Mode,
}
langfuse.InjectTracing(ctx, &payload)
payloadBytes, err := json.Marshal(payload)
if err != nil {
+2
View File
@@ -13,6 +13,7 @@ import (
"github.com/Tencent/WeKnora/internal/errors"
apperrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/Tencent/WeKnora/internal/utils"
@@ -655,6 +656,7 @@ func (h *KnowledgeBaseHandler) CopyKnowledgeBase(c *gin.Context) {
SourceID: req.SourceID,
TargetID: req.TargetID,
}
langfuse.InjectTracing(ctx, &payload)
payloadBytes, err := json.Marshal(payload)
if err != nil {
+9
View File
@@ -8,6 +8,7 @@ import (
"time"
"github.com/Tencent/WeKnora/internal/application/service"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/hibiken/asynq"
@@ -100,6 +101,14 @@ func RunAsynqServer(params AsynqTaskParams) *asynq.ServeMux {
// Create a new mux and register all handlers
mux := asynq.NewServeMux()
// Install Langfuse middleware BEFORE handler registration so every task
// type is automatically wrapped. When Langfuse is disabled the middleware
// is a pass-through; when enabled it resumes the upstream HTTP trace (if
// the payload carries one) or opens a standalone trace, then wraps the
// handler execution in a SPAN so all child generations (embedding / VLM /
// chat / rerank / ASR) nest correctly in the Langfuse UI.
mux.Use(langfuse.AsynqMiddleware())
// Register extract handlers - router will dispatch to appropriate handler
mux.HandleFunc(types.TypeChunkExtract, params.ChunkExtractor.Handle)
mux.HandleFunc(types.TypeDataTableSummary, params.DataTableSummary.Handle)
+209
View File
@@ -0,0 +1,209 @@
package langfuse
import (
"context"
"encoding/json"
"strconv"
"github.com/Tencent/WeKnora/internal/types"
"github.com/hibiken/asynq"
)
// InjectTracing stamps the current trace/span ids (and a best-effort
// user/session label) from ctx onto the given payload, provided the payload
// embeds types.TracingContext (and thus implements LangfuseTracingCarrier).
//
// Safe to call unconditionally: when Langfuse is disabled or no trace is
// present on ctx, it writes a zero-valued TracingContext — which round-trips
// through JSON as absent fields and therefore costs nothing.
//
// Call sites live at every asynq.NewTask creation point. We deliberately
// keep the API synchronous and mutating (rather than returning a new payload
// copy) so that existing enqueue code needs only a single added line just
// before json.Marshal, minimizing review surface and the risk of regressions.
func InjectTracing(ctx context.Context, carrier types.LangfuseTracingCarrier) {
if carrier == nil {
return
}
mgr := GetManager()
if !mgr.Enabled() {
return
}
tc := types.TracingContext{}
if trace, ok := TraceFromContext(ctx); ok && trace != nil {
tc.LangfuseTraceID = trace.ID
}
if obs, ok := parentObservationFromCtx(ctx); ok {
tc.LangfuseParentObservationID = obs
}
tc.LangfuseUserID = userIDFromCtx(ctx)
tc.LangfuseSessionID = sessionIDFromCtx(ctx)
carrier.SetLangfuseTracing(tc)
}
// peekTracingContext pulls just the Langfuse tracing fields out of a raw
// asynq payload. It's deliberately lax: every payload type in the project
// is a JSON object at the top level, and absent/mismatched fields decode to
// the zero value. If unmarshalling fails entirely (e.g. the payload isn't
// JSON at all) we return a zero TracingContext and let the main handler
// deal with its own error — we never want an observability bug to mask a
// real task failure.
func peekTracingContext(payload []byte) types.TracingContext {
if len(payload) == 0 {
return types.TracingContext{}
}
var tc types.TracingContext
_ = json.Unmarshal(payload, &tc)
return tc
}
// AsynqMiddleware is the worker-side counterpart of GinMiddleware. It:
//
// 1. Parses any tracing ids embedded in the task payload and either
// resumes the originating trace (so the Langfuse UI stitches the HTTP
// request and the async processing into one tree), or — for tasks that
// came from a scheduled job with no originating HTTP request — creates
// a standalone trace tagged with the task type.
//
// 2. Opens a SPAN around the handler execution so every child generation
// (embedding / VLM / chat / rerank / ASR) auto-attaches to it via
// parentObservationId.
//
// 3. Enriches the span with asynq's own metadata: task id, queue, retry
// count, payload size.
//
// When the manager is disabled it degrades to a pass-through; failure of
// the Langfuse path never blocks task execution.
//
// Register it once in router/task.go via mux.Use.
func AsynqMiddleware() asynq.MiddlewareFunc {
return func(next asynq.Handler) asynq.Handler {
return asynq.HandlerFunc(func(ctx context.Context, task *asynq.Task) error {
mgr := GetManager()
if !mgr.Enabled() {
return next.ProcessTask(ctx, task)
}
tc := peekTracingContext(task.Payload())
taskID, _ := asynq.GetTaskID(ctx)
retryCount, _ := asynq.GetRetryCount(ctx)
maxRetry, _ := asynq.GetMaxRetry(ctx)
queueName, _ := asynq.GetQueueName(ctx)
meta := map[string]interface{}{
"task_type": task.Type(),
"task_id": taskID,
"queue": queueName,
"retry": retryCount,
"max_retry": maxRetry,
"payload_bytes": len(task.Payload()),
}
// If the upstream enqueuer stamped a trace id onto the payload,
// we graft under that trace. Otherwise (scheduled jobs, legacy
// in-flight tasks that predate this code, tests, etc.) we start
// a standalone trace named after the task type so at least the
// worker-side work is observable.
var trace *Trace
shouldFinishTrace := false
if tc.LangfuseTraceID != "" {
ctx, trace = mgr.ResumeTrace(ctx, tc.LangfuseTraceID, tc.LangfuseParentObservationID)
} else {
ctx, trace = mgr.StartTrace(ctx, TraceOptions{
Name: "asynq." + task.Type(),
UserID: firstNonEmptyString(tc.LangfuseUserID, userIDFromCtx(ctx)),
SessionID: firstNonEmptyString(tc.LangfuseSessionID, sessionIDFromCtx(ctx)),
Metadata: meta,
Tags: []string{"asynq", task.Type()},
})
shouldFinishTrace = true
}
ctx, span := mgr.StartSpan(ctx, SpanOptions{
Name: "asynq." + task.Type(),
Input: spanInputFromPayload(task.Payload()),
Metadata: meta,
})
err := next.ProcessTask(ctx, task)
outcome := "success"
if err != nil {
outcome = "error"
}
span.Finish(map[string]interface{}{
"outcome": outcome,
}, map[string]interface{}{
"outcome": outcome,
}, err)
if shouldFinishTrace {
trace.Finish(map[string]interface{}{
"outcome": outcome,
}, map[string]interface{}{
"task_type": task.Type(),
"outcome": outcome,
})
}
return err
})
}
}
// spanInputFromPayload surfaces a compact, human-readable summary of the
// task payload for the Langfuse "Input" pane. We deliberately do NOT send
// the full JSON blob because:
//
// - Manual/text-ingest payloads can be many kilobytes of prose;
// - FAQ import payloads embed the full entry list;
// - Document-process payloads contain file URLs that, while small, may
// include presigned query strings that rotate (adding diff noise).
//
// Instead we preview the first ~1KB verbatim, which matches what Langfuse
// itself would display and keeps ingestion bandwidth predictable.
func spanInputFromPayload(payload []byte) interface{} {
const preview = 1024
if len(payload) == 0 {
return nil
}
if len(payload) <= preview {
return string(payload)
}
return map[string]interface{}{
"preview": string(payload[:preview]) + "...",
"bytes": len(payload),
}
}
// userIDFromCtx mirrors middleware.extractUserID but accepts a raw context
// (no gin.Context) so both HTTP and asynq paths can share the same fallback
// logic: explicit UserID → tenant:<id> → empty.
func userIDFromCtx(ctx context.Context) string {
if v, ok := ctx.Value(types.UserIDContextKey).(string); ok && v != "" {
return v
}
if v, ok := ctx.Value(types.TenantIDContextKey).(uint64); ok && v != 0 {
return "tenant:" + strconv.FormatUint(v, 10)
}
return ""
}
// sessionIDFromCtx pulls a best-effort "session" label. For HTTP chat this
// is already set by GinMiddleware; for async work we fall back to the
// request id so retries of the same logical task group together.
func sessionIDFromCtx(ctx context.Context) string {
if v, ok := types.RequestIDFromContext(ctx); ok && v != "" {
return v
}
return ""
}
func firstNonEmptyString(vals ...string) string {
for _, v := range vals {
if v != "" {
return v
}
}
return ""
}
+272
View File
@@ -0,0 +1,272 @@
package langfuse
import (
"context"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"sync"
"testing"
"time"
"github.com/Tencent/WeKnora/internal/types"
"github.com/hibiken/asynq"
)
// dummyPayload is a minimal payload that embeds TracingContext, mirroring
// how real asynq payloads opt into trace propagation.
type dummyPayload struct {
types.TracingContext
KnowledgeID string `json:"knowledge_id"`
}
// TestInjectTracing_DisabledIsZero verifies InjectTracing is a no-op when
// Langfuse is disabled: no panics, no trace fields written.
func TestInjectTracing_DisabledIsZero(t *testing.T) {
// Explicitly install a disabled manager to shadow any globally-enabled
// one from a previous test in the same package.
_, _ = Init(Config{Enabled: false})
p := &dummyPayload{KnowledgeID: "k1"}
InjectTracing(context.Background(), p)
if p.LangfuseTraceID != "" || p.LangfuseParentObservationID != "" {
t.Fatalf("expected no tracing fields on disabled manager, got %+v", p.TracingContext)
}
}
// TestInjectTracing_PopulatesFromContext checks that when a trace is active
// on the context, its id is copied onto the payload and a subsequent
// peekTracingContext round-trips it correctly.
func TestInjectTracing_PopulatesFromContext(t *testing.T) {
// Stand up a fake ingestion endpoint — Init starts a real flush worker
// and we don't want it to panic on connection refused during the test.
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = io.Copy(io.Discard, r.Body)
w.WriteHeader(200)
_, _ = w.Write([]byte(`{}`))
}))
defer srv.Close()
m, err := Init(Config{
Enabled: true,
Host: srv.URL,
PublicKey: "pk",
SecretKey: "sk",
FlushAt: 16,
FlushInterval: 50 * time.Millisecond,
QueueSize: 16,
RequestTimeout: 2 * time.Second,
SampleRate: 1.0,
})
if err != nil {
t.Fatalf("init: %v", err)
}
defer m.Shutdown(context.Background())
ctx, trace := m.StartTrace(context.Background(), TraceOptions{Name: "parent"})
ctx, span := m.StartSpan(ctx, SpanOptions{Name: "wrap"})
if span == nil || span.ID == "" {
t.Fatalf("expected a span with id, got %+v", span)
}
p := &dummyPayload{KnowledgeID: "k1"}
InjectTracing(ctx, p)
if p.LangfuseTraceID != trace.ID {
t.Errorf("trace id mismatch: got %q want %q", p.LangfuseTraceID, trace.ID)
}
if p.LangfuseParentObservationID != span.ID {
t.Errorf("parent observation id mismatch: got %q want %q", p.LangfuseParentObservationID, span.ID)
}
// Round-trip the payload through JSON, which is what asynq does on the
// wire, and make sure peekTracingContext recovers the ids.
b, err := json.Marshal(p)
if err != nil {
t.Fatalf("marshal: %v", err)
}
got := peekTracingContext(b)
if got.LangfuseTraceID != trace.ID || got.LangfuseParentObservationID != span.ID {
t.Errorf("peek round-trip lost ids: %+v", got)
}
}
// TestAsynqMiddleware_ResumeTrace asserts the middleware grafts a resumed
// trace onto the child handler's context, and that the ingestion endpoint
// receives span-create / span-update events for the wrapper — without
// emitting a duplicate trace-create (which would split the Langfuse UI
// tree into two disconnected roots).
func TestAsynqMiddleware_ResumeTrace(t *testing.T) {
var mu sync.Mutex
var batches []ingestionRequest
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
var req ingestionRequest
_ = json.Unmarshal(body, &req)
mu.Lock()
batches = append(batches, req)
mu.Unlock()
w.WriteHeader(200)
_, _ = w.Write([]byte(`{}`))
}))
defer srv.Close()
m, err := Init(Config{
Enabled: true,
Host: srv.URL,
PublicKey: "pk",
SecretKey: "sk",
FlushAt: 1,
FlushInterval: 5 * time.Millisecond,
QueueSize: 32,
RequestTimeout: 2 * time.Second,
SampleRate: 1.0,
})
if err != nil {
t.Fatalf("init: %v", err)
}
// Build a payload that already carries a trace id (as if the HTTP layer
// injected one at enqueue time).
payload := &dummyPayload{KnowledgeID: "k1"}
payload.LangfuseTraceID = "trace-xyz"
payload.LangfuseParentObservationID = "parent-span-abc"
raw, _ := json.Marshal(payload)
var receivedTraceID, receivedParentID string
var sawTrace bool
handler := asynq.HandlerFunc(func(ctx context.Context, task *asynq.Task) error {
if tr, ok := TraceFromContext(ctx); ok && tr != nil {
sawTrace = true
receivedTraceID = tr.ID
}
if pid, ok := parentObservationFromCtx(ctx); ok {
receivedParentID = pid
}
return nil
})
mw := AsynqMiddleware()
wrapped := mw(handler)
task := asynq.NewTask("test:type", raw)
if err := wrapped.ProcessTask(context.Background(), task); err != nil {
t.Fatalf("handler err: %v", err)
}
if !sawTrace {
t.Fatal("expected handler ctx to carry a resumed trace")
}
if receivedTraceID != "trace-xyz" {
t.Errorf("trace id mismatch: got %q want trace-xyz", receivedTraceID)
}
if receivedParentID == "" {
t.Error("expected parent observation to be set to the wrapper span id")
}
if err := m.Shutdown(context.Background()); err != nil {
t.Fatalf("shutdown: %v", err)
}
mu.Lock()
defer mu.Unlock()
var sawSpanCreate, sawSpanUpdate, sawTraceCreate bool
for _, b := range batches {
for _, ev := range b.Batch {
switch ev.Type {
case "span-create":
sawSpanCreate = true
// Resumed span should attach to the supplied trace id, not
// a fresh one we made up in the worker.
body, _ := json.Marshal(ev.Body)
var ob observationBody
_ = json.Unmarshal(body, &ob)
if ob.TraceID != "trace-xyz" {
t.Errorf("span-create trace id = %q, want trace-xyz", ob.TraceID)
}
if ob.ParentObservationID != "parent-span-abc" {
t.Errorf("span-create parentObservationId = %q, want parent-span-abc", ob.ParentObservationID)
}
case "span-update":
sawSpanUpdate = true
case "trace-create":
sawTraceCreate = true
}
}
}
if !sawSpanCreate {
t.Error("missing span-create event")
}
if !sawSpanUpdate {
t.Error("missing span-update event")
}
if sawTraceCreate {
t.Error("resumed trace should not emit trace-create events (parent owns them)")
}
}
// TestAsynqMiddleware_StandaloneTrace asserts that when the payload carries
// NO upstream trace id (e.g. a scheduled job), the middleware opens a
// standalone trace tagged with the task type, so the worker-side work
// still shows up in Langfuse.
func TestAsynqMiddleware_StandaloneTrace(t *testing.T) {
var mu sync.Mutex
var batches []ingestionRequest
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
var req ingestionRequest
_ = json.Unmarshal(body, &req)
mu.Lock()
batches = append(batches, req)
mu.Unlock()
w.WriteHeader(200)
_, _ = w.Write([]byte(`{}`))
}))
defer srv.Close()
m, err := Init(Config{
Enabled: true, Host: srv.URL, PublicKey: "pk", SecretKey: "sk",
FlushAt: 1, FlushInterval: 5 * time.Millisecond, QueueSize: 32,
RequestTimeout: 2 * time.Second, SampleRate: 1.0,
})
if err != nil {
t.Fatalf("init: %v", err)
}
payload := &dummyPayload{KnowledgeID: "kX"}
raw, _ := json.Marshal(payload)
handler := asynq.HandlerFunc(func(ctx context.Context, task *asynq.Task) error { return nil })
wrapped := AsynqMiddleware()(handler)
if err := wrapped.ProcessTask(context.Background(), asynq.NewTask("scheduled:ping", raw)); err != nil {
t.Fatalf("handler err: %v", err)
}
if err := m.Shutdown(context.Background()); err != nil {
t.Fatalf("shutdown: %v", err)
}
mu.Lock()
defer mu.Unlock()
// Langfuse uses trace-create events both on open and on update (the
// server merges by id), so we filter to the initial event that carries
// the Name — the finalizing update has no name.
var sawNamedTraceCreate bool
for _, b := range batches {
for _, ev := range b.Batch {
if ev.Type != "trace-create" {
continue
}
body, _ := json.Marshal(ev.Body)
var tb traceBody
_ = json.Unmarshal(body, &tb)
if tb.Name == "asynq.scheduled:ping" {
sawNamedTraceCreate = true
}
}
}
if !sawNamedTraceCreate {
t.Error("standalone run should emit a trace-create with name=asynq.scheduled:ping")
}
}
+27
View File
@@ -15,6 +15,15 @@ import (
// into many unrelated traces in the Langfuse UI.
var traceCtxKey = types.LangfuseTraceContextKey
// parentObsCtxKey tracks the current "parent observation" id — i.e. the span
// that encloses any generation/span started under it. Unlike traceCtxKey, this
// key is private: spans are a pure tracer concern and logger.CloneContext
// should reset the parent across request boundaries (we don't want a leftover
// span id from one request leaking into another's LLM calls).
type parentObsCtxKeyType struct{}
var parentObsCtxKey = parentObsCtxKeyType{}
// withTrace stores a *Trace on the context so downstream LLM wrappers can
// attach their generations to it.
func withTrace(ctx context.Context, t *Trace) context.Context {
@@ -38,3 +47,21 @@ func traceFromCtx(ctx context.Context) (*Trace, bool) {
func TraceFromContext(ctx context.Context) (*Trace, bool) {
return traceFromCtx(ctx)
}
// withParentObservation stores the id of the enclosing span so children
// (sub-spans or generations) can attach via parentObservationId.
func withParentObservation(ctx context.Context, id string) context.Context {
if ctx == nil || id == "" {
return ctx
}
return context.WithValue(ctx, parentObsCtxKey, id)
}
// parentObservationFromCtx returns the enclosing span id if any.
func parentObservationFromCtx(ctx context.Context) (string, bool) {
if ctx == nil {
return "", false
}
v, ok := ctx.Value(parentObsCtxKey).(string)
return v, ok && v != ""
}
+57 -3
View File
@@ -57,14 +57,32 @@ func GinMiddleware() gin.HandlerFunc {
}
}
// shouldTrace restricts tracing to chat / search / agent / evaluation
// endpoints where LLMs run. Everything else (auth, list, config, static…)
// is skipped to keep the Langfuse dashboard signal-to-noise high.
// shouldTrace restricts tracing to endpoints where LLM work (or the asynq
// jobs that will run LLM work) originates. Everything else — auth, list,
// config fetches, static assets, health checks — is skipped to keep the
// Langfuse dashboard's signal-to-noise ratio high.
//
// The list below is grouped by purpose:
// - online inference: knowledge-chat / agent-chat / knowledge-search /
// generate_title are the existing chat/retrieval surface.
// - ingestion: POST on knowledge-bases/:id/knowledge (file / url / manual)
// and reparse / move / copy all kick off asynq jobs that later run
// embedding/VLM/chat calls. Tracing the HTTP side means the Langfuse UI
// shows a parent trace whose children are the worker spans.
// - batch ops: FAQ import + knowledge batch delete also enqueue jobs.
// - model/setup diagnostics: initialization endpoints exercise live
// models; evaluation runs arbitrary chat pipelines.
// - wiki: auto-fix kicks off wiki ingest, which calls embedding.
//
// Read-only listing / GET endpoints are deliberately excluded; they never
// trigger LLM work and would only add noise.
func shouldTrace(c *gin.Context) bool {
path := c.FullPath()
if path == "" {
return false
}
method := c.Request.Method
// Online inference
switch {
case strings.HasPrefix(path, "/api/v1/knowledge-chat"),
strings.HasPrefix(path, "/api/v1/agent-chat"),
@@ -72,9 +90,45 @@ func shouldTrace(c *gin.Context) bool {
strings.HasPrefix(path, "/api/v1/sessions") && strings.Contains(path, "generate_title"),
strings.HasPrefix(path, "/api/v1/initialization/remote/check"),
strings.HasPrefix(path, "/api/v1/initialization/embedding/test"),
strings.HasPrefix(path, "/api/v1/initialization/rerank/check"),
strings.HasPrefix(path, "/api/v1/initialization/asr/check"),
strings.HasPrefix(path, "/api/v1/initialization/multimodal/test"),
strings.HasPrefix(path, "/api/v1/initialization/extract/"),
strings.HasPrefix(path, "/api/v1/evaluation"):
return true
}
// Ingestion (all POST/PUT that enqueue LLM-backed async work)
if method == "POST" || method == "PUT" {
switch {
// Per-knowledge-base ingestion surface.
case strings.Contains(path, "/knowledge-bases/") && strings.Contains(path, "/knowledge/"):
return true
// Knowledge-level mutations that trigger re-processing.
case strings.HasPrefix(path, "/api/v1/knowledge/") &&
(strings.HasSuffix(path, "/reparse") ||
strings.HasSuffix(path, "/move") ||
strings.Contains(path, "/manual/")):
return true
// Knowledge base copy (clones an entire KB, fanning out documents).
case path == "/api/v1/knowledge-bases/copy":
return true
// FAQ bulk import.
case strings.Contains(path, "/faq/entries") ||
strings.Contains(path, "/faq/entry") ||
strings.Contains(path, "/faq/import"):
return true
// Wiki auto-fix enqueues wiki ingest.
case strings.Contains(path, "/wiki/auto-fix") ||
strings.Contains(path, "/wiki/rebuild-links"):
return true
// Chunk-level mutations that rerun embeddings on update.
case strings.HasPrefix(path, "/api/v1/chunks/") && method == "PUT":
return true
// Manual data source sync triggers asynq sync.
case strings.Contains(path, "/datasource/") && strings.HasSuffix(path, "/sync"):
return true
}
}
return false
}
+149 -24
View File
@@ -16,13 +16,27 @@ type Trace struct {
// Generation represents a single model invocation (LLM / embedding / VLM).
type Generation struct {
ID string
TraceID string
manager *Manager
sampled bool
startTime time.Time
model string
name string
ID string
TraceID string
ParentObservationID string
manager *Manager
sampled bool
startTime time.Time
model string
name string
}
// Span represents a logical unit of work that isn't itself an LLM call — for
// example an asynq task execution, a pipeline stage, or a document-processing
// step. Generations and nested spans attach to it via parentObservationId.
type Span struct {
ID string
TraceID string
ParentObservationID string
manager *Manager
sampled bool
startTime time.Time
name string
}
// TraceOptions configures a new trace.
@@ -46,6 +60,13 @@ type GenerationOptions struct {
ModelParameters map[string]interface{}
}
// SpanOptions configures a new SPAN observation.
type SpanOptions struct {
Name string
Input interface{}
Metadata map[string]interface{}
}
// StartTrace opens a new trace, stores its ID in the returned ctx, and returns
// a handle callers can finish with FinishTrace. When the manager is disabled
// or sampling excludes the trace, the returned *Trace is non-nil but all
@@ -110,8 +131,109 @@ func (t *Trace) Finish(output interface{}, metadata map[string]interface{}) {
})
}
// ResumeTrace reconstructs a *Trace handle from an upstream-provided trace id
// (and optional parent observation id), without emitting a new trace-create
// event. Used by the asynq middleware to graft async work onto the HTTP-level
// trace that originated it: the HTTP layer already issued trace-create, and
// the worker only needs to add child observations.
//
// When traceID is empty the returned *Trace is nil, signalling the caller
// should fall back to StartTrace if it wants a standalone root.
func (m *Manager) ResumeTrace(ctx context.Context, traceID, parentObservationID string) (context.Context, *Trace) {
if m == nil || !m.cfg.Enabled || traceID == "" {
return ctx, nil
}
t := &Trace{ID: traceID, manager: m, sampled: true}
ctx = withTrace(ctx, t)
if parentObservationID != "" {
ctx = withParentObservation(ctx, parentObservationID)
}
return ctx, t
}
// StartSpan opens a SPAN observation under the trace (and optional parent
// observation) carried by ctx. When no trace is present, a shallow trace is
// auto-created, mirroring StartGeneration's behaviour. Returns a ctx whose
// parentObservationId is set to this span's id, so any nested span or
// generation will properly attach as a child.
func (m *Manager) StartSpan(ctx context.Context, opts SpanOptions) (context.Context, *Span) {
if m == nil || !m.cfg.Enabled {
return ctx, &Span{}
}
trace, ok := traceFromCtx(ctx)
if !ok || trace == nil {
newCtx, t := m.StartTrace(ctx, TraceOptions{Name: opts.Name})
ctx = newCtx
trace = t
}
if !trace.sampled {
return ctx, &Span{}
}
parent, _ := parentObservationFromCtx(ctx)
now := time.Now()
s := &Span{
ID: newID(),
TraceID: trace.ID,
ParentObservationID: parent,
manager: m,
sampled: true,
startTime: now,
name: opts.Name,
}
body := observationBody{
ID: s.ID,
TraceID: s.TraceID,
ParentObservationID: parent,
Type: "SPAN",
Name: opts.Name,
StartTime: isoTime(now),
Input: opts.Input,
Metadata: opts.Metadata,
}
m.enqueue(ingestionEvent{
ID: newID(),
Timestamp: isoTime(now),
Type: "span-create",
Body: body,
})
ctx = withParentObservation(ctx, s.ID)
return ctx, s
}
// Finish updates a span with its final output, extra metadata and any error.
// A non-nil err marks the span as ERROR level in Langfuse.
func (s *Span) Finish(output interface{}, metadata map[string]interface{}, err error) {
if s == nil || s.manager == nil || !s.sampled {
return
}
level := "DEFAULT"
var statusMsg string
if err != nil {
level = "ERROR"
statusMsg = err.Error()
}
body := observationBody{
ID: s.ID,
TraceID: s.TraceID,
Type: "SPAN",
EndTime: isoTime(time.Now()),
Output: output,
Metadata: metadata,
Level: level,
StatusMessage: statusMsg,
}
s.manager.enqueue(ingestionEvent{
ID: newID(),
Timestamp: isoTime(time.Now()),
Type: "span-update",
Body: body,
})
}
// StartGeneration opens a generation observation under the trace carried by
// ctx (or a newly auto-created trace if none is present).
// ctx (or a newly auto-created trace if none is present). If a parent span
// is present on ctx, the generation attaches under it via parentObservationId
// so the Langfuse tree shows: trace → span → generation.
func (m *Manager) StartGeneration(ctx context.Context, opts GenerationOptions) (context.Context, *Generation) {
if m == nil || !m.cfg.Enabled {
return ctx, &Generation{}
@@ -128,26 +250,29 @@ func (m *Manager) StartGeneration(ctx context.Context, opts GenerationOptions) (
if !trace.sampled {
return ctx, &Generation{}
}
parent, _ := parentObservationFromCtx(ctx)
now := time.Now()
g := &Generation{
ID: newID(),
TraceID: trace.ID,
manager: m,
sampled: true,
startTime: now,
model: opts.Model,
name: opts.Name,
ID: newID(),
TraceID: trace.ID,
ParentObservationID: parent,
manager: m,
sampled: true,
startTime: now,
model: opts.Model,
name: opts.Name,
}
body := observationBody{
ID: g.ID,
TraceID: g.TraceID,
Type: "GENERATION",
Name: opts.Name,
StartTime: isoTime(now),
Input: opts.Input,
Metadata: opts.Metadata,
Model: opts.Model,
ModelParameters: opts.ModelParameters,
ID: g.ID,
TraceID: g.TraceID,
ParentObservationID: parent,
Type: "GENERATION",
Name: opts.Name,
StartTime: isoTime(now),
Input: opts.Input,
Metadata: opts.Metadata,
Model: opts.Model,
ModelParameters: opts.ModelParameters,
}
m.enqueue(ingestionEvent{
ID: newID(),
+195
View File
@@ -0,0 +1,195 @@
package langfuse
import (
"context"
"encoding/json"
"errors"
"io"
"net/http"
"net/http/httptest"
"sync"
"testing"
"time"
)
// newTestServer spins up a fake Langfuse ingestion endpoint and returns the
// collected events + a cleanup func.
func newTestServer(t *testing.T) (*httptest.Server, func() []ingestionEvent) {
t.Helper()
var mu sync.Mutex
var batches []ingestionRequest
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
var req ingestionRequest
_ = json.Unmarshal(body, &req)
mu.Lock()
batches = append(batches, req)
mu.Unlock()
w.WriteHeader(200)
_, _ = w.Write([]byte(`{}`))
}))
drain := func() []ingestionEvent {
mu.Lock()
defer mu.Unlock()
var out []ingestionEvent
for _, b := range batches {
out = append(out, b.Batch...)
}
return out
}
return srv, drain
}
func newTestManager(t *testing.T, host string) *Manager {
t.Helper()
m, err := Init(Config{
Enabled: true,
Host: host,
PublicKey: "pk",
SecretKey: "sk",
FlushAt: 1,
FlushInterval: 5 * time.Millisecond,
QueueSize: 32,
RequestTimeout: 2 * time.Second,
SampleRate: 1.0,
})
if err != nil {
t.Fatalf("init: %v", err)
}
return m
}
// TestSpan_NestedHierarchy verifies that nested StartSpan calls produce a
// trace → span₁ → span₂ → generation hierarchy, with parentObservationId
// pointing to the direct ancestor at each level.
func TestSpan_NestedHierarchy(t *testing.T) {
srv, drain := newTestServer(t)
defer srv.Close()
m := newTestManager(t, srv.URL)
ctx, trace := m.StartTrace(context.Background(), TraceOptions{Name: "root"})
ctx, outer := m.StartSpan(ctx, SpanOptions{Name: "outer"})
ctx, inner := m.StartSpan(ctx, SpanOptions{Name: "inner"})
_, gen := m.StartGeneration(ctx, GenerationOptions{Name: "llm", Model: "m"})
gen.Finish("out", &TokenUsage{Input: 1, Output: 2, Total: 3}, nil)
inner.Finish("inner-out", nil, nil)
outer.Finish("outer-out", nil, nil)
trace.Finish("root-out", nil)
if err := m.Shutdown(context.Background()); err != nil {
t.Fatalf("shutdown: %v", err)
}
events := drain()
parentOf := map[string]string{}
traceOf := map[string]string{}
kindOf := map[string]string{}
for _, ev := range events {
if ev.Type != "span-create" && ev.Type != "generation-create" {
continue
}
b, _ := json.Marshal(ev.Body)
var ob observationBody
_ = json.Unmarshal(b, &ob)
parentOf[ob.ID] = ob.ParentObservationID
traceOf[ob.ID] = ob.TraceID
kindOf[ob.ID] = ob.Type
}
if traceOf[outer.ID] != trace.ID || parentOf[outer.ID] != "" {
t.Errorf("outer span should sit directly under the trace, got parent=%q trace=%q", parentOf[outer.ID], traceOf[outer.ID])
}
if parentOf[inner.ID] != outer.ID {
t.Errorf("inner span parent mismatch: got %q want %q", parentOf[inner.ID], outer.ID)
}
if parentOf[gen.ID] != inner.ID {
t.Errorf("generation should nest under inner span, got parent=%q want %q", parentOf[gen.ID], inner.ID)
}
if kindOf[outer.ID] != "SPAN" || kindOf[inner.ID] != "SPAN" {
t.Errorf("expected both wrappers to be SPAN observations, got outer=%q inner=%q", kindOf[outer.ID], kindOf[inner.ID])
}
if kindOf[gen.ID] != "GENERATION" {
t.Errorf("expected generation kind, got %q", kindOf[gen.ID])
}
}
// TestSpan_FinishWithError records an error status on the span so failures in
// asynq handlers surface as red observations in Langfuse.
func TestSpan_FinishWithError(t *testing.T) {
srv, drain := newTestServer(t)
defer srv.Close()
m := newTestManager(t, srv.URL)
ctx, _ := m.StartTrace(context.Background(), TraceOptions{Name: "root"})
_, span := m.StartSpan(ctx, SpanOptions{Name: "boom"})
span.Finish(nil, nil, errors.New("kaboom"))
if err := m.Shutdown(context.Background()); err != nil {
t.Fatalf("shutdown: %v", err)
}
var sawError bool
for _, ev := range drain() {
if ev.Type != "span-update" {
continue
}
b, _ := json.Marshal(ev.Body)
var ob observationBody
_ = json.Unmarshal(b, &ob)
if ob.Level == "ERROR" && ob.StatusMessage == "kaboom" {
sawError = true
}
}
if !sawError {
t.Fatal("expected span-update with ERROR level and status message")
}
}
// TestResumeTrace_NoTraceCreateEvent verifies that ResumeTrace does NOT
// emit a trace-create event — the originating HTTP handler already did,
// and a duplicate would register as an orphan root in the Langfuse UI.
func TestResumeTrace_NoTraceCreateEvent(t *testing.T) {
srv, drain := newTestServer(t)
defer srv.Close()
m := newTestManager(t, srv.URL)
ctx, trace := m.ResumeTrace(context.Background(), "upstream-trace", "upstream-span")
if trace == nil || trace.ID != "upstream-trace" {
t.Fatalf("expected resumed trace with id upstream-trace, got %+v", trace)
}
if pid, ok := parentObservationFromCtx(ctx); !ok || pid != "upstream-span" {
t.Errorf("expected parent observation upstream-span on ctx, got %q (ok=%v)", pid, ok)
}
// Emit one child so there's *something* to flush, proving only child
// observations reach the wire, not a trace-create for the resume.
_, span := m.StartSpan(ctx, SpanOptions{Name: "child"})
span.Finish(nil, nil, nil)
if err := m.Shutdown(context.Background()); err != nil {
t.Fatalf("shutdown: %v", err)
}
for _, ev := range drain() {
if ev.Type == "trace-create" {
t.Fatalf("ResumeTrace must not emit trace-create events, got %+v", ev)
}
}
}
// TestResumeTrace_DisabledIsSafe guards against nil deref when Langfuse is
// off: ResumeTrace should return a nil *Trace and the original ctx unchanged.
func TestResumeTrace_DisabledIsSafe(t *testing.T) {
m, err := Init(Config{Enabled: false})
if err != nil {
t.Fatalf("init: %v", err)
}
ctx, trace := m.ResumeTrace(context.Background(), "x", "y")
if trace != nil {
t.Errorf("expected nil trace when disabled, got %+v", trace)
}
if _, ok := parentObservationFromCtx(ctx); ok {
t.Error("disabled ResumeTrace should not attach parent observation")
}
}
+2
View File
@@ -314,6 +314,8 @@ type SyncResult struct {
// DataSourceSyncPayload represents the asynq task payload for data source sync
type DataSourceSyncPayload struct {
TracingContext
// Data source ID to sync
DataSourceID string `json:"data_source_id"`
+13
View File
@@ -21,6 +21,7 @@ const (
// ExtractChunkPayload represents the extract chunk task payload
type ExtractChunkPayload struct {
TracingContext
TenantID uint64 `json:"tenant_id"`
ChunkID string `json:"chunk_id"`
ModelID string `json:"model_id"`
@@ -28,6 +29,7 @@ type ExtractChunkPayload struct {
// DocumentProcessPayload represents the document process task payload
type DocumentProcessPayload struct {
TracingContext
RequestId string `json:"request_id"`
TenantID uint64 `json:"tenant_id"`
KnowledgeID string `json:"knowledge_id"`
@@ -46,6 +48,7 @@ type DocumentProcessPayload struct {
// FAQImportPayload represents the FAQ import task payload (including dry run mode)
type FAQImportPayload struct {
TracingContext
TenantID uint64 `json:"tenant_id"`
TaskID string `json:"task_id"`
KBID string `json:"kb_id"`
@@ -60,6 +63,7 @@ type FAQImportPayload struct {
// QuestionGenerationPayload represents the question generation task payload
type QuestionGenerationPayload struct {
TracingContext
TenantID uint64 `json:"tenant_id"`
KnowledgeBaseID string `json:"knowledge_base_id"`
KnowledgeID string `json:"knowledge_id"`
@@ -70,6 +74,7 @@ type QuestionGenerationPayload struct {
// SummaryGenerationPayload represents the summary generation task payload
type SummaryGenerationPayload struct {
TracingContext
TenantID uint64 `json:"tenant_id"`
KnowledgeBaseID string `json:"knowledge_base_id"`
KnowledgeID string `json:"knowledge_id"`
@@ -78,6 +83,7 @@ type SummaryGenerationPayload struct {
// KBClonePayload represents the knowledge base clone task payload
type KBClonePayload struct {
TracingContext
TenantID uint64 `json:"tenant_id"`
TaskID string `json:"task_id"`
SourceID string `json:"source_id"`
@@ -86,6 +92,7 @@ type KBClonePayload struct {
// IndexDeletePayload represents the index delete task payload
type IndexDeletePayload struct {
TracingContext
TenantID uint64 `json:"tenant_id"`
KnowledgeBaseID string `json:"knowledge_base_id"`
EmbeddingModelID string `json:"embedding_model_id"`
@@ -96,6 +103,7 @@ type IndexDeletePayload struct {
// KBDeletePayload represents the knowledge base delete task payload
type KBDeletePayload struct {
TracingContext
TenantID uint64 `json:"tenant_id"`
KnowledgeBaseID string `json:"knowledge_base_id"`
EffectiveEngines []RetrieverEngineParams `json:"effective_engines"`
@@ -103,12 +111,14 @@ type KBDeletePayload struct {
// KnowledgeListDeletePayload represents the batch knowledge delete task payload
type KnowledgeListDeletePayload struct {
TracingContext
TenantID uint64 `json:"tenant_id"`
KnowledgeIDs []string `json:"knowledge_ids"`
}
// KnowledgeMovePayload represents the knowledge move task payload
type KnowledgeMovePayload struct {
TracingContext
TenantID uint64 `json:"tenant_id"`
TaskID string `json:"task_id"`
KnowledgeIDs []string `json:"knowledge_ids"`
@@ -136,6 +146,7 @@ type KnowledgeMoveProgress struct {
// ManualProcessPayload represents the manual knowledge processing task payload.
// Used for both create (publish) and update operations.
type ManualProcessPayload struct {
TracingContext
RequestId string `json:"request_id"`
TenantID uint64 `json:"tenant_id"`
KnowledgeID string `json:"knowledge_id"`
@@ -146,6 +157,7 @@ type ManualProcessPayload struct {
// ImageMultimodalPayload represents the image multimodal processing task payload.
type ImageMultimodalPayload struct {
TracingContext
TenantID uint64 `json:"tenant_id"`
KnowledgeID string `json:"knowledge_id"`
KnowledgeBaseID string `json:"knowledge_base_id"`
@@ -160,6 +172,7 @@ type ImageMultimodalPayload struct {
// KnowledgePostProcessPayload represents the knowledge post process task payload.
type KnowledgePostProcessPayload struct {
TracingContext
TenantID uint64 `json:"tenant_id"`
KnowledgeID string `json:"knowledge_id"`
KnowledgeBaseID string `json:"knowledge_base_id"`
+60
View File
@@ -0,0 +1,60 @@
package types
// TracingContext is an embeddable struct that carries observability context
// (currently Langfuse trace/span ids plus user/session hints) across process
// boundaries — specifically, from an HTTP request into an asynq task payload
// and back out inside the worker.
//
// It lives in the types package, not the langfuse package, so that:
//
// - asynq payload structs can embed it without pulling the langfuse
// package into every service/handler import graph;
// - the langfuse package can remain a leaf dependency that only types
// (and its own tests) reference directly.
//
// The JSON tags all use the "lf_" prefix and omitempty so that payloads
// constructed before the Langfuse feature landed remain byte-compatible
// (empty fields collapse to nothing in the serialized output) and so that
// Langfuse-specific columns don't collide with business fields that may
// happen to be named similarly.
type TracingContext struct {
// LangfuseTraceID is the id of the root trace that originated this task.
// When set, workers attach their observations to the same trace instead
// of creating a standalone one, which is what makes the Langfuse UI
// show the asynq work as a child of the originating HTTP request.
LangfuseTraceID string `json:"lf_trace_id,omitempty"`
// LangfuseParentObservationID, when set, points to the span that
// enclosed the enqueue call. The worker's own span/generation will list
// it as parentObservationId so Langfuse renders the tree as:
// http-trace → http-span → asynq-span → generation.
LangfuseParentObservationID string `json:"lf_parent_obs_id,omitempty"`
// LangfuseUserID preserves the userId / tenant label across the async
// boundary so that orphan async traces (when no upstream trace id is
// available) still show up in the Langfuse "Users" view under the
// right tenant.
LangfuseUserID string `json:"lf_user_id,omitempty"`
// LangfuseSessionID preserves the sessionId for the same reason.
LangfuseSessionID string `json:"lf_session_id,omitempty"`
}
// SetLangfuseTracing overwrites the embedded TracingContext. Method is
// exported so helpers in internal/tracing/langfuse can populate it via the
// LangfuseTracingCarrier interface without reflection.
func (tc *TracingContext) SetLangfuseTracing(other TracingContext) {
*tc = other
}
// GetLangfuseTracing returns a copy of the embedded TracingContext.
func (tc TracingContext) GetLangfuseTracing() TracingContext {
return tc
}
// LangfuseTracingCarrier is implemented automatically by any struct that
// embeds TracingContext (thanks to Go's method promotion rules). The asynq
// enqueue helper in internal/tracing/langfuse uses this interface to inject
// the current trace/span ids into the payload without caring about the
// concrete payload type.
type LangfuseTracingCarrier interface {
SetLangfuseTracing(TracingContext)
GetLangfuseTracing() TracingContext
}