Migrate WeKnora's Langfuse tracing from the legacy /api/public/ingestion
batch API to the OpenTelemetry Go SDK emitting OTLP/HTTP to
/api/public/otel/v1/traces — the Langfuse v3+ standard ingestion protocol.
Why: Langfuse v3 deprecated the legacy ingestion API in favor of OTLP, and
OTel-only backends (e.g. the LiteFuse Doris fork) dropped it entirely, so the
legacy client was incompatible. Migrating to the OTel SDK also adds W3C
traceparent cross-service propagation: an upstream caller's trace id is
inherited so its run and the WeKnora call land under one trace in the UI.
What:
- Manager owns an OTel TracerProvider backed by an otlptracehttp exporter
(Basic auth + the x-langfuse-ingestion-version gate header). Spans are
buffered by BatchSpanProcessor and exported complete on End, so a long
trace no longer fragments across flush batches.
- *Trace/*Span/*Generation wrap OTel spans; the public API (StartTrace/
Span/Generation, handles, InjectTracing, asynq/gin middleware, Config,
TokenUsage) is unchanged, so callers (agent/chat/handler/scheduler/asynqdl)
need no edits.
- W3C traceparent: GinMiddleware extracts traceparent from incoming requests;
AsynqMiddleware + InjectTracing propagate it across the asynq payload.
types.TracingContext gains a LangfuseTraceparent field.
- logger.CloneContext preserves the active OTel span across context rebuilds
so child spans (agent engine, summary generation) stitch to the HTTP trace
instead of orphaning into separate traces.
- StartSpan/StartGeneration re-establish the parent from the *Trace handle
when the ctx lost the active OTel span, covering background/derived contexts.
Compatibility: works with Langfuse v3+ (cloud / self-hosted / the bundled
docker-compose profile) and LiteFuse. Drops Langfuse v2 (legacy-only) support.
Uses the OTel Go SDK already (indirectly) in go.mod, promoting
otlptracehttp to direct; no third-party langfuse-go library.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- docreader: serialize all pdfium access behind a global lock. pypdfium2
is process-global and not thread-safe, so concurrent PDF uploads could
corrupt its shared state and deadlock the whole process (requests stuck
in "Parsing document" forever). Non-PDF parsers still run concurrently.
- logger: strip ANSI color escapes from file logs so on-disk logs stay
plain text while stdout keeps terminal colors.
- storage: log baseDir/resolvedPath on file-open failure and when the
multimodal service resolves a file service, so a LOCAL_STORAGE_BASE_DIR
mismatch between writer and reader is immediately visible.
- docker: set TZ (default Asia/Shanghai) for the docreader container.
Replace scopes JSON with a viewer/contributor/admin role column, move
unsafe-method authorization from path-prefix middleware to declarative
APIKey* guards on routes, and close remaining KB-scope gaps in handlers.
Enforce per-key knowledge base restrictions on batch/search read paths,
centralize scope checks in shared KB access validators, and propagate
TenantAPIKeyScope through logger.CloneContext so async QA cannot bypass it.
The runnable_by_viewer field was never populated for built-in agents,
so TenantRoleViewer users always hit 403 on agent-chat. Drop the field
and RBAC check so agent-chat matches knowledge-chat access for viewers.
- Add CustomFormatter.Template driven by LOG_FORMAT env var with
placeholders %d/%level/%thread/%logger/%traceId/%msg; default
format unchanged for backward compatibility.
- Replace chained ReplaceAll with strings.NewReplacer for single-pass
substitution, avoiding accidental re-substitution when a field value
contains a literal placeholder string.
- Inject ANSI color at the %level substitution step; removes the old
whole-line ReplaceAll(line, "INFO", ...) which would mis-color
literal level tokens appearing inside messages.
- Cache threadNeeded on the formatter so runtime.Stack only runs when
the template references %thread.
logger.CloneContext rebuilds a fresh context.Background() and only
copies a hand-curated allow-list of context keys (LoggerContextKey,
TenantIDContextKey, UserContextKey, etc.) before passing it to
downstream handlers — the chat / agent QA pipelines all do
`ctx := logger.CloneContext(c.Request.Context())` at the top of
parseQARequest.
TenantRoleContextKey was missing from that allow-list. Any call site
that read the role via TenantRoleFromContext(reqCtx.ctx) saw the
type-zero TenantRole and the helper's least-privilege fallback
returned Viewer. The auth middleware had set role=Owner correctly
on the parent context, but the cloned context lost it.
Symptom: an Owner of their home tenant got 403 on POST /agent-chat
when the agent had runnable_by_viewer=false, because the gate at
session/qa.go:520 saw role=viewer despite middleware logging
role=owner two ms earlier.
Diagnostic log lines added in the previous commits (auth.go:145
"[auth] resolved role=...", auth.go:283 "[auth] resolveTenantRole
step1 hit row_role=...") made this immediately visible — a request
showed role=owner at the middleware layer and role=viewer at the
handler layer, which can only mean ctx propagation lost it. Keeping
the diagnostic logs in place so similar regressions surface fast.
Fix is one line: add TenantRoleContextKey to the CloneContext
allow-list, with a comment pinning it to the bug it fixes so a
future drive-by removal has to weigh that loss.
Wires KnowledgeBase.VectorStoreID and the ownership-aware retrieve factory
into the user-facing knowledge-base lifecycle:
- POST /knowledge-bases validates the requested vector_store_id against
the caller's tenant scope and the engine registry. New error codes
ErrVectorStoreBindingInvalid (2200) and ErrVectorStoreUnavailable (2201)
distinguish the typed branches without echoing UUIDs to the client.
- GET / POST / PUT / PUT-pin responses embed the bound store's display
metadata (name, source, engine_type, status) without exposing any
connection credentials. Cross-tenant shared KBs receive a suppressed
payload (vector_store_id stripped, source="shared") so operator-chosen
store names cannot be enumerated across tenants.
- POST /knowledge-bases/copy synchronously rejects clones whose target
has a different embedding model or vector store, before the async
clone task is enqueued. The async clone worker re-applies the same
checks for defense in depth.
- DELETE /vector-stores/:id refuses to remove a store with bound KBs,
inside a transaction that row-locks the store on PostgreSQL and
serializes via WAL on SQLite. unregister-from-registry is wrapped in
defer/recover so a panic surfaces as a structured warning instead of
silently leaking a stale engine.
- vector_store_id is immutable after creation. The GORM <-:create tag
blocks every ORM update path; the service-layer DTO omits the field
entirely; a reflection-based regression test catches any future
maintainer who adds it back to either layer.
- Empty-string vector_store_id is normalized to nil at both the create
path and inside SharesStoreWith, so rows persisted by callers that
did not run Normalize first cannot trip false same-store comparisons.
Part of #993. Depends on #994 and #1310.
Closes#620#497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).
Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
so logger.CloneContext can preserve it across handler / goroutine
boundaries (otherwise each LLM call auto-created an orphan trace,
fragmenting one request into many)
Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes
Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.
Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
reuses WeKnora's existing PostgreSQL (separate database via an idempotent
init container that fixes ICU collation drift) and Redis (separate DB
number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
to avoid Prisma P1013 when passwords contain @ / # / etc.
Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
Syncs personal and team Yuque knowledge bases into WeKnora as Markdown,
with full and incremental sync + deletion detection. Mirrors the Feishu
connector pattern; fetches V2Doc.body directly (no async export step).
Config: api_token (required) + base_url (optional, defaults to public cloud).
401/403 wraps ErrInvalidCredentials; 429 honors Retry-After; filters to
status=1 Doc-type at list stage to conserve API quota. Body is trusted as
Markdown for format=markdown|lake — empirically verified against the v2
API: Lake XML lives separately in body_lake, body itself is the Markdown
view. Other formats (e.g. html) are skipped with a placeholder item so
they surface in sync logs instead of silently feeding HTML to the parser.
v2Doc.status accepts both int and string shapes — Yuque's OpenAPI spec
declares string but the live API returns int. sanitizeFileName truncates
at a UTF-8 rune boundary so multi-byte titles don't emit invalid UTF-8
that downstream filename validation rejects.
Also adds logger.SetOutput test hook, used by the token-redaction test.
- Added a new logger for LLM calls, enabling detailed logging of model interactions, including request and response data.
- Introduced configuration options for enabling and specifying the log directory for LLM debug logs.
- Implemented cleanup for old log files to manage disk space effectively.
- Wrapped existing chat implementations to log calls when debug logging is enabled, enhancing traceability for model interactions.
- Added critical language rules to fallback response templates to ensure consistent language usage in responses.
- Updated logger context to include language and session-related keys for improved context management.
- Updated configuration to improve handling of multimodal queries, including clearer guidelines for when to use [NO_SEARCH] in responses.
- Enhanced the user interface to include dedicated sections for multimodal configuration, allowing users to manage image uploads and related settings more effectively.
- Added localization support for new multimodal features in both English and Chinese, improving accessibility for a broader audience.
- Implemented backend changes to support image analysis and extraction of OCR text, enriching the interaction experience for users.
These enhancements significantly improve the application's capability to process and respond to multimodal queries, fostering a more interactive and user-friendly environment.
- Added state management for pending document requests to enhance user experience during chunk loading.
- Implemented error handling for chunk loading failures, providing user feedback through warnings.
- Adjusted scroll handling logic to ensure smoother pagination and prevent multiple concurrent requests.
This update enhances the reliability of document loading and improves user interaction during scrolling.
- Introduced a new API for organization management, including creation, updating, and deletion of organizations.
- Implemented organization sharing functionality for knowledge bases, allowing users to share knowledge bases with specific organizations and set permissions.
- Added UI components for organization selection and sharing dialogs.
- Enhanced existing components to support organization-related features, including shared knowledge base handling and organization role management.
This update enhances collaboration capabilities within the application by allowing users to manage and share knowledge bases across different organizations.
- Introduced a new endpoint to fetch chunks by ID, improving access to specific knowledge base entries.
- Updated tool result types to include web search results, enhancing the display capabilities in the chat component.
- Enhanced the AgentStreamDisplay component to support rendering web search results with improved HTML formatting.
- Implemented logic for managing web search result summaries and citations, ensuring a cohesive user experience.
- Refactored session management to accommodate new web search functionalities, improving overall system integration.