feat(observability): extend Langfuse tracing across asynq pipeline

Previously the Langfuse integration only traced in-process HTTP requests
(chat / search / eval), so file uploads and every downstream asynq task
(document parse, chunk embedding, OCR/VLM, summary / question gen, wiki
ingest, datasource sync, etc.) produced either disconnected shallow
traces or no observation at all.

This change threads one trace end-to-end:

- tracer: add SPAN observation type and StartSpan; add ResumeTrace so a
  worker can attach to an upstream trace without emitting a duplicate
  trace-create; StartGeneration now auto-picks parentObservationId from
  ctx so nested trace -> span -> generation trees render correctly.
- types.TracingContext + LangfuseTracingCarrier: embed on all 17 asynq
  payloads so trace_id / parent_obs_id / user_id / session_id serialise
  into every job.
- langfuse.InjectTracing: injected at 28 enqueue sites before json.Marshal
  so the HTTP-layer trace survives the Redis hop.
- langfuse.AsynqMiddleware: mux.Use hook that peeks the payload, either
  resumes the upstream trace or opens a standalone asynq.<type> trace
  for scheduled jobs, and wraps the handler in a SPAN with task metadata
  (id / queue / retry / payload_bytes) plus ERROR level on failure.
- GinMiddleware.shouldTrace: whitelist ingestion / knowledge-mutation /
  FAQ / wiki / datasource endpoints so the root trace actually starts.
- Tests: tracer_test.go covers span nesting, error status, and
  ResumeTrace no-trace-create guarantee; asynq_test.go covers
  InjectTracing round-trip, middleware resume path, and standalone
  trace fallback.
- Docs: docs/Langfuse\u96c6\u6210.md now lists the covered task types
  and documents the cross-process propagation model.

No behavioural change when Langfuse is disabled (all new code paths are
no-ops and carriers serialise to empty strings with omitempty).
This commit is contained in:
wizardchen
2026-04-24 12:23:30 +08:00
committed by lyingbug
parent 59fbf77524
commit beca2b89a3
23 changed files with 1077 additions and 40 deletions
+2
View File
@@ -13,6 +13,7 @@ import (
"github.com/Tencent/WeKnora/internal/errors"
apperrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/Tencent/WeKnora/internal/utils"
@@ -655,6 +656,7 @@ func (h *KnowledgeBaseHandler) CopyKnowledgeBase(c *gin.Context) {
SourceID: req.SourceID,
TargetID: req.TargetID,
}
langfuse.InjectTracing(ctx, &payload)
payloadBytes, err := json.Marshal(payload)
if err != nil {