- Refactored checkStorageEngineConfigured to be a method of knowledgeService, enhancing encapsulation and readability.
- Updated logic to allow fallback to global file service when no storage provider is configured at the KB or tenant level, improving error handling.
- Added detailed comments to clarify the method's behavior and internal logic, ensuring better understanding for future maintenance.
Before this change, the `organization_members` table keyed on user_id —
joining an org as a Viewer in tenant T meant only that one user got
visibility on shared KBs/agents, while the rest of T (including its
Owner) saw nothing. The mismatch surfaced as confusing UX bugs:
"my colleague shared this with us, why can't I see it?"
This change replaces `organization_members` with
`organization_tenant_members`. Membership and Org-RBAC role now key on
(org_id, tenant_id); `representative_user_id` stays as a display-only
label of the user who first brought the tenant in.
Permission resolution gains a 3-D cap so tenant-RBAC stays honest under
cross-tenant sharing:
effective = min(share.Permission, tenant_org_role, tenant_role_cap)
where tenant_role_cap pins tenant Viewers to OrgRoleViewer regardless
of the org-level grant. A Viewer in their own tenant cannot edit a
shared KB even if the org granted Editor — preserves the
"Viewer = read-only everywhere" promise.
Migration 000045 backfills `organization_tenant_members` from the old
table by collapsing each (org, tenant) group to its max role, then
renames the old table to `organization_members_pre_plan3` for safe
rollback. Sqlite mirror (Lite mode) is fresh-bootstrap: the init
schema replaces the old table directly.
Routes: member-management URLs change from `:user_id` to `:tenant_id`.
Org create / join / leave now require Admin+ in the caller's tenant —
joining an Org affects everyone in T, so the decision can't sit with
a Viewer.
Frontend: OrganizationSettingsModal + store update to address members
by tenant_id.
PR 1–4 of the multi-tenant RBAC series gated KB-level and Agent-level
mutations behind a "creator OR Admin+" check via OwnedKBOrAdmin /
OwnedAgentOrAdmin. The layer underneath — knowledge entries, chunks,
wiki pages — was still gated only by Contributor, which is wider than
intended: any tenant Contributor could edit/delete documents in a
private KB they didn't own. PR 2 / wiki author left explicit follow-up
breadcrumbs at the three call sites in router.go acknowledging this gap.
This PR closes those breadcrumbs by reusing the existing scaffolding:
- KnowledgeService.GetOwningKBCreatorID walks knowledge_id -> kb_id ->
KB.CreatorID, tenant-scoped through GetKnowledgeByID. The chain lives
in the service layer so the handler-side lookups stay pure adapters.
- Three new CreatorLookup closures (KBCreatorLookupFromKnowledgeID,
KBCreatorLookupFromKnowledgeIDParam, KBCreatorLookupFromKBPath) are
added in handler/rbac_lookups.go alongside the existing
KBCreatorLookup / AgentCreatorLookup. Each translates the repository
sentinels into middleware.ErrResourceNotFound and re-checks tenant
scope as defence-in-depth, mirroring KBCreatorLookup exactly.
- router/rbac.go gains three guard factories — OwnedKnowledgeKBOrAdmin,
OwnedChunkKBOrAdmin, OwnedWikiKBOrAdmin — wired through the same
rbacGuards bundle.
- Mutating routes in RegisterKnowledgeRoutes / RegisterChunkRoutes /
RegisterWikiPageRoutes switch from g.Contributor() to the new
ownership guards. Cross-KB batch operations (POST /knowledge/move,
POST /knowledge/batch-delete, PUT /knowledge/tags) stay
Contributor-gated since there's no single owning KB to check
against. KB-scoped uploads (POST /knowledge-bases/:id/knowledge/...)
reuse the existing OwnedKBOrAdmin because the URL :id is the KB id
directly. The chunks.DELETE("/by-id/:id/questions") route addresses
chunks by their own id with no knowledge id in the URL, so it stays
Contributor-gated as a documented carve-out.
No schema changes: documents inside a shared KB are KB-owned content,
not personal uploads — that's how the product is positioned and matches
the original "per-KB-ownership granularity" wording from the PR 2
breadcrumbs. Legacy/built-in KBs with empty CreatorID surface as
tenant-owned (Admin+ only), same rule as built-in agents.
Tests:
- 9 new test cases in handler/rbac_lookups_test.go covering the three
new lookups: NotFoundMapsToSentinel (each repository sentinel),
CrossTenantIsHiddenAsNotFound, OwnerMatchReturnsCreatorID,
MissingTenantContext (fail-closed), and ChunkRouteShape (param-name
regression guard).
- 5 new test cases in service/knowledge_owner_chain_test.go covering
the chain helper: knowledge-not-found, kb-not-found, nil-kb-without-
err, happy path, and legacy KB with empty CreatorID.
Build, vet, and the targeted ./internal/middleware/...
./internal/handler/... ./internal/router/...
./internal/application/service test runs are clean. Pre-existing
TestCreateStore_* / TestOssEnsureBucket_CreateFails failures on main
are unrelated and reproduce on a clean checkout.
Extracts the 25 repeated NewCompositeRetrieveEngine call sites across
seven services into two factory functions with tenant-ownership
verification: CreateRetrieveEngineForKB (synchronous) and
CreateRetrieveEngineFromPayload (async task handlers). Promotes
GetByStoreID from concrete-only to interfaces.RetrieveEngineRegistry.
Extends KBDeletePayload / IndexDeletePayload with an omitempty
VectorStoreID snapshot.
Design highlights
- Factory verifies tenant ownership of the resolved store
(defense-in-depth) so a gap in PR3 validation or a tampered Asynq
payload cannot cross tenants. Cross-tenant attempts return
ErrVectorStoreForbidden with a structured log entry.
- Sentinel errors (ErrTenantInfoMissing, ErrVectorStoreNotFound,
ErrVectorStoreForbidden) let async handlers classify non-retryable
failures as asynq.SkipRetry.
- nil and empty-string pointers for vectorStoreID normalize to
"unbound" so callers never send an empty UUID to GetByStoreID.
- resolveBoundEngine constructs engineInfos directly (with
slices.Clone of Support()) instead of going through
NewCompositeRetrieveEngine. The latter only reads from the
byEngineType map and cannot reach DB stores in byStoreID -- multiple
stores can share the same engine type. Unbound fallbacks still reuse
NewCompositeRetrieveEngine so the tenant effective-engines path is
byte-for-byte unchanged.
- DeleteKnowledgeBase now reads the KB before soft-delete so the
enqueued KBDeletePayload carries a VectorStoreID snapshot; once the
row is soft-deleted GORM's default scope hides it. Cost: +1 SELECT
per KB-delete request (a rare admin operation).
- enqueueIndexDeleteTask gains a vectorStoreID *string parameter that
the single caller (tag.go) populates from the owning KB. The worker
(ProcessIndexDelete) validates ownership via the factory.
- cleanupKnowledgeResources loads the KB so bound-store cleanup routes
correctly. If the load fails it warns and falls back to tenant
effective engines; orphan vectors become observable in logs instead
of disappearing silently.
- DeleteKnowledges (batch) stays on the tenant effective-engines path
-- batches can span stores and multi-store fan-out is PR4 scope.
Noted inline.
Merge safety
All knowledge bases have VectorStoreID = NULL at merge time (introduced
nullable in PR1, Tencent/WeKnora#994). Every factory call falls back to
NewCompositeRetrieveEngine(registry, tenantInfo.GetEffectiveEngines()),
which is the existing code path for these KBs. Async payloads from
before this change decode to VectorStoreID = nil (omitempty) and take
the same fallback.
Testing
- go build ./...
- go vet ./...
- go test -race -count=1 ./internal/application/service/retriever/...
covers unbound (nil/empty), bound, cross-tenant, unregistered store,
ownership infra error, legacy payload shapes (missing field/null/
empty string/nil), tampered payload, and parallel invocation.
- grep -r NewCompositeRetrieveEngine internal/application/service
returns zero results outside factory.go/composite.go.
Refs: Part of Tencent/WeKnora#993. Depends on Tencent/WeKnora#994.
Add three new optional filters to the document list under a knowledge base
detail page — parse status, source/channel, and updated time range — and
rework multi-select to no longer cause the card title to jitter on hover.
Backend
- Introduce types.KnowledgeListFilter to aggregate optional filter dimensions
(tag, keyword, file_type, parse_status, source, updated_from/to) and switch
ListPagedKnowledgeByKnowledgeBaseID (repository/service/interface) to accept
it instead of a growing positional parameter list.
- The ListKnowledge HTTP handler accepts new parse_status, source, start_time
and end_time query params; time params accept RFC3339, "YYYY-MM-DD HH:MM:SS"
and "YYYY-MM-DD". The repository routes source="manual"/"url" onto the type
column to stay consistent with file_type semantics; other source values match
the channel column.
- Update the four other callers (agent_service, initialization) to pass an
empty filter struct, preserving prior behavior.
Frontend
- Add three controls in the doc-filter-bar (status select, source select,
date-range picker with future-date disabled) wired through getKnowled /
listKnowledgeFiles into the new backend params.
- Replace the hover-triggered card checkbox with an explicit "批量管理" mode
(mirrors the session list UX): in card view the checkbox only renders while
batch mode is on, entered via the per-card "..." menu; the list view keeps
its leading checkbox column. Switching from list to grid auto-enables batch
mode when something is already selected, so the selection stays visible.
- DocumentBatchBar now stays open whenever batch mode or selection > 0, and
its "取消选择" button both clears the selection and exits batch mode.
API surface sync
- Regenerate Swagger artifacts (docs/docs.go / swagger.json / swagger.yaml).
- Update docs/api/knowledge.md with the new query parameters.
- Add backward-compatible ListKnowledgeWithFilter + KnowledgeListFilter to the
Go SDK; the existing ListKnowledge keeps its signature.
i18n
- New filter labels in zh-CN / en-US / ko-KR / ru-RU; reuse existing
menu.batchManage / batchManage.cancel for the multi-select strings.
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.
This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.
- migration 000041: task_pending_ops + task_dead_letters tables, plus
three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
source_refs text fallback, lower(title) trgm).
- TaskPendingOpsRepository + TaskDeadLetterRepository in
internal/application/repository/task_queue.go: cursor-paginated
list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
delete that refuses empty keys so a buggy caller can't wipe a KB's
queue.
- internal/middleware/asynqdl: writes a task_dead_letters row when an
asynq task exhausts its retry budget. Payload-agnostic — a small
probe struct extracts TenantID + scope hints across every existing
payload type, so summary:generation / image:multimodal /
faq_import / kb:clone / etc. all dead-letter without per-handler
code. Best-effort: an insert failure never masks the underlying
task error. Installed in router/task.go before the langfuse mw so
it sees raw errors.
- Pending queue moved Redis → PG (no TTL, restart-durable). Redis
keeps just the active-batch lock and the delete tombstone.
- In-batch retry budget (wikiMaxFailRetries=5) tracked via
pendingRepo.IncrFailCount; over the cap the row is moved to
task_dead_letters with task_type=wiki:ingest, related_id=knowledge
id. Asynq retries (10) are handled by the new middleware.
- Removed the per-batch ListAllPages. WikiBatchContext now carries
lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
mutex-protected caches; reduce reaches for titles / summaries on
demand instead of pre-loading the whole KB.
- getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
which the new GIN index on source_refs serves as a Bitmap Index
Scan instead of a sequential text LIKE.
- Dedup pre-filter uses pg_trgm via FindSimilarPages
(idx_wiki_pages_title_trgm). For each new entity/concept (and each
of its aliases) we ask the DB for the top-K trigram-similar
existing pages and union the results — bounded prompt size, no Go-
side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
- cleanDeadLinks / injectCrossLinks are scoped to the batch's
affected slugs (a few dozen) instead of every page in the KB. Both
use the new lite ListBySlugs / ExistsSlugs repo methods so they
pull only slug + title + outlinks, not full content.
- Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
[[bad-slug|display]] the cleanup path now tries display-text
reverse lookup → hyphen/case normalized equality → char-bigram
Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
break drift case ("shang-hai-tower" vs "shanghai-tower") in place
instead of replacing the link with plain text.
- rebuildIndexPage uses ListByTypeRecent(200) for the first-time
intro and drops the full DocumentSummaries blob from the
incremental update prompt, so its context stays bounded regardless
of KB size.
- Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
IngestMapParallel / IngestReduceParallel, with sensible defaults
via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
follow-ups don't waste asynq retry slots bouncing on the active
lock.
- RunLint walks pages via the new ListPagesCursor in 200-page
windows and computes the live-slug set with a one-column
ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
every node + edge. Memory is now bounded; 40k pages walks in
constant ~4MB.
- 14 GORM tests for both repos against an in-memory SQLite mirror of
the production DDL (task_queue_test.go).
- 6 tests for the dead-letter middleware covering retry budget
detection, payload-agnostic scope inference, error truncation, and
repo-failure isolation.
- 7 tests for the slug fuzzy resolve helper covering the three
resolution stages, display-text priority over normalized equality,
bigram fallback acceptance, and rejection of unrelated slugs.
- Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
new fetcher / cursor APIs.
internal/application/service/knowledge.go had grown to 9883 lines /
149 functions spanning CRUD, document processing, summary/question
generation, clone/move, FAQ (CRUD+import+index+export), wiki cleanup,
multimodal image, and file utilities. Navigating it, or diffing
changes in it, was increasingly painful.
Purely mechanical split into 8 files in the same package. No
signatures, behaviors, or types were modified. knowledgeService
struct, NewKnowledgeService, and package-level errors remain in
knowledge.go unchanged. Per-file imports were trimmed by goimports.
knowledge.go 497 struct + constructor + errs + CRUD reads + Search
knowledge_create.go 1144 CreateFrom{File,URL,Passage,Manual} + helpers
knowledge_delete.go 686 Delete + wiki cleanup + ProcessKnowledgeListDelete
knowledge_process.go 2407 ProcessDocument/Summary/Question + Reparse + UpdateImageInfo
knowledge_clone_move.go 1049 Clone* + Move* + progress persistence
knowledge_faq.go 1932 FAQ CRUD + Search + Export
knowledge_faq_import.go 1939 FAQ import/validate/index + runningFAQImportInfo
knowledge_util.go 352 file-type/url/hash + VLM+Storage config + resolveFileService
Verification: go build, go vet, gofmt on the 8 files, and
golangci-lint all clean (22 issues post-split matches 22 pre-split
in knowledge.go — no new issues introduced). The pre-existing
unused warnings on getVLMConfig / buildStorageConfig were left as-is
rather than removing dead code beyond the scope of this task.
Resolves the issues surfaced by a self-review of the adaptive chunking
work. Tests covering each regression were added first to prove the
fix.
Critical
- Heading splitter no longer prepends breadcrumb to Content. The
breadcrumb now lives on Chunk.ContextHeader; EmbeddingContent()
combines header + content for the embedder. This restores the
End-Start == len(Content) invariant relied on by knowledge.go's
document-reconstruction path (summary generation, UI highlight) and
also eliminates the duplicate-heading regression where the section
heading appeared twice in a chunk's body.
- splitBySeparators is now genuinely recursive: when one separator
leaves a piece > ChunkSize, the next-priority separator is applied
inside that piece. Mirrors the Python reference. Without this, a
document with one paragraph break followed by a long run of
newline-separated lines could emit a single ~1950-rune chunk for
ChunkSize=300.
Medium
- SplitParentChild forces children onto the recursive tier, skipping
the per-parent profile pass (was N extra O(N) scans).
- Heuristic splitter snaps overlap start to the nearest semantic
boundary or newline instead of slicing mid-line / mid-word.
- Strategy.Split returns the legacy tier's output directly when the
whole chain rejects rather than running SplitText a second time.
Low
- ensureDefaults caps ChunkOverlap at ChunkSize/2 so pathological
configs (TokenLimit clamping ChunkSize tiny while Overlap stays
high) cannot produce ~97% overlap chunks.
- DefaultChunkOverlap docstring spells out the migration story for
KBs that stored ChunkOverlap=0.
ContextHeader plumbing
- chunker.Chunk and types.ParsedChunk gain ContextHeader plus an
EmbeddingContent() helper.
- types.Chunk gets a transient (gorm:"-") ContextHeader field so the
knowledge service can build embedding inputs as
titlePrefix + chunk.EmbeddingContent() without a DB schema change.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
The Go splitter previously combined all separators into one regex, losing
the priority-based recursive behavior of the Python reference. This pass
walks the separator list in order: the first separator that yields >1
piece wins, mirroring docreader/splitter/splitter.py. Larger semantic
boundaries (\n\n) now win over smaller ones (\n).
Additional cross-cutting changes:
- ChunkingConfig gains Strategy / TokenLimit / Languages fields so the
per-KB config can pick a tier and pin embedding token budgets.
- buildSplitterConfig in the knowledge service uses the new
chunker.DefaultChunkSize / DefaultChunkOverlap constants (overlap
consolidated to 80 ≈ 15%) instead of the previous hardcoded 50.
- ensureDefaults clamps ChunkSize to a per-language character budget
derived from TokenLimit so chunks stay safe under embedding model
token caps. Chinese (~1.7 chars/tok) gets a tighter clamp than
English (~4 chars/tok).
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Adds the building blocks for tiered chunking strategies — multilingual
patterns, token approximation, document profiling, validation, and a
strategy resolver that today still falls through to the legacy
SplitText. Default Strategy is empty/legacy so existing knowledge bases
continue to chunk identically.
- tokens.go: char/token approximation per language (en/de/zh/mixed)
and a coarse DetectLanguage heuristic
- patterns.go: source-of-truth regex bibliothek for headings, numbered
sections, multilingual chapter markers, visual separators, etc.
- profiler.go: single-pass document scan + SelectStrategy chain
- validator.go: rejects obviously broken chunk-sets to drive fallbacks
- strategy.go: public Split / SplitParentChild entry points; resolves
StrategyAuto via the profiler, anything else maps to a fixed tier
- splitter.go: SplitterConfig gains Strategy / TokenLimit / Languages
fields and DefaultChunkSize / DefaultChunkOverlap constants (overlap
raised to 80 ≈ 15%)
- knowledge.go: switches both call sites to chunker.Split /
SplitParentChild — behavior unchanged for empty Strategy
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
knowledgeService.buildStorageConfig had hardcoded switches in two places that
only handled local/minio/cos. Knowledge bases configured for tos, s3, oss, or
ks3 reached the docreader with a DocParserStorageConfig containing only the
Provider tag — bucket, endpoint, and credentials were silently dropped, and
extraction stalled or fetched from the wrong location.
- Tenant-merge switch: add tos/s3/oss/ks3 cases mapping
StorageEngineConfig.{TOS,S3,OSS,KS3} fields onto DocParserStorageConfig.
Same shape as the existing minio/cos branches; AccessKey -> AccessKeyID,
SecretKey -> SecretAccessKey, no AppID.
- Legacy compat switch: list tos/s3/oss/ks3 alongside local with hasKBFull=false
so fall-through to the tenant-merge path is intentional rather than an
unrecognised provider sliding past the switch. The legacy `cos_config` column
predates these providers and they should never resolve via that path.
- Doc drift: types/custom_agent.go ImageStorageProvider comment and
handler/system.go StorageEngineStatusItem.Name comment now list all 7
providers (matched against types.StorageEngineConfig and ParseProviderScheme).
Tests (knowledge_storage_config_test.go, +268 LOC, 17 cases):
- TestBuildStorageConfig_TenantMergeAllProviders pins per-provider field
population for all 7 providers; the tos/s3/oss/ks3 sub-tests are the
regression guard for #1117.
- TestBuildStorageConfig_LegacyPathOnlyForCOSAndMinIO pins that legacy
StorageConfig data is only consumed for cos/minio and skipped for the
others (intentional fall-through).
- TestBuildStorageConfig_NoTenantFallsThroughToEmpty pins the no-tenant case.
Out of scope (called out in the linked issue):
- The systemic refactor (single source-of-truth provider registry) — left as
a follow-up; this PR keeps changes local to the buggy code path.
- initialization.go validateStorageEngineConfig multimodal validation gap —
the InitializationRequest struct does not carry tos/s3/oss/ks3 fields, so
fixing it requires API surface additions; can ship separately.
Closes#1117 (the buildStorageConfig switches; remaining items deferred).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
internal/application/service/knowledge.go had grown to 9883 lines /
149 functions spanning CRUD, document processing, summary/question
generation, clone/move, FAQ (CRUD+import+index+export), wiki cleanup,
multimodal image, and file utilities. Navigating it, or diffing
changes in it, was increasingly painful.
Purely mechanical split into 8 files in the same package. No
signatures, behaviors, or types were modified. knowledgeService
struct, NewKnowledgeService, and package-level errors remain in
knowledge.go unchanged. Per-file imports were trimmed by goimports.
knowledge.go 497 struct + constructor + errs + CRUD reads + Search
knowledge_create.go 1144 CreateFrom{File,URL,Passage,Manual} + helpers
knowledge_delete.go 686 Delete + wiki cleanup + ProcessKnowledgeListDelete
knowledge_process.go 2407 ProcessDocument/Summary/Question + Reparse + UpdateImageInfo
knowledge_clone_move.go 1049 Clone* + Move* + progress persistence
knowledge_faq.go 1932 FAQ CRUD + Search + Export
knowledge_faq_import.go 1939 FAQ import/validate/index + runningFAQImportInfo
knowledge_util.go 352 file-type/url/hash + VLM+Storage config + resolveFileService
Verification: go build, go vet, gofmt on the 8 files, and
golangci-lint all clean (22 issues post-split matches 22 pre-split
in knowledge.go — no new issues introduced). The pre-existing
unused warnings on getVLMConfig / buildStorageConfig were left as-is
rather than removing dead code beyond the scope of this task.
Addresses two pre-merge review notes:
1. Extract the sentinel-returning threshold check from getSummary into
a small package-private helper (checkSufficientSummaryContent) and
add a unit test for it. The new test pins the behavior the caller
in ProcessSummaryGeneration relies on: the sentinel is returned for
empty / image-only / sub-threshold / scanned-PDF-without-OCR inputs,
and nil is returned for legitimate short notes and scanned PDFs
where VLM OCR succeeded.
The downstream "errors.Is(err, errInsufficientSummaryContent) →
SummaryStatusFailed + skip summary chunk" branch is mechanical (no
branching beyond the errors.Is check); covering it end-to-end would
require mocking KnowledgeRepository / ChunkRepository / chat.Chat,
for which there is no precedent in this package.
2. Switch minTextContentRunes from const to var so tests can override
the threshold and so future config plumbing can adjust it at
runtime without a rebuild. The new test exercises the override.
Self-review of the prior commit surfaced one critical regression and
several smaller issues. This change addresses them.
Critical: the old stripImageMarkup stripped the WHOLE <image>...</image>
block, but searchutil.EnrichContentWithImageInfo wraps successful VLM OCR
output in exactly that shape — so a scanned PDF where OCR worked was
being rejected by hasSufficientTextContent and routed to the same
"failed" path as the OCR-empty case. The fix splits stripping into:
* <image_original>...</image_original> blocks removed entirely
(they only carry the redundant Markdown ref already stripped above)
* <image>, <images>, <image_caption>, <image_ocr> tags removed at
the tag level, INNER TEXT PRESERVED, so OCR / caption text reaches
the threshold check and the LLM call.
A regression test pins this down with realistic enriched-image XML that
the previous code failed silently on.
Other fixes:
* Threshold lowered from 50 to 10 runes — short legitimate documents
(brief memos, single-line notes) must still pass.
* Summary chunk content no longer prefixes "# Document\n<filename>";
only the LLM-generated summary is indexed, so filenames cannot
leak back into RAG retrieval context.
* Wiki summary slug is now derived from the knowledge ID rather than
the upload filename, eliminating filename-based slugs like
"summary/mx5280-pdf" that downstream wiki-edit prompts can read.
* Citation sourceRef likewise drops the "|<docTitle>" suffix.
* Dead docTitle parameter removed from extractCandidateSlugs,
extractEntitiesAndConceptsNoUpsert, classifyChunkCitations.
* The redundant stripImageMarkup pass in the warning log is gone;
realTextRuneCount caches the count once and uses
utf8.RuneCountInString to avoid the rune-slice allocation.
Wiki-page display titles still flow through DocTitle (filename) into
later wiki-edit / index-rebuild prompts. That is a larger refactor of
the wiki pipeline and is intentionally left for a separate PR.
Scanned PDFs without a successful OCR pass (e.g. when no VLM is configured
on the KB, or when the VLM returned a "no text content" reply that the
sanitizer discarded) reach the summary and wiki-ingest pipelines with
nothing but markdown image references for content. The downstream LLM
prompts used to receive the filename, file type, and title alongside that
empty content, and the generate_summary template explicitly forbade the
model from refusing — so the model would invent a topic from the filename
alone, producing e.g. a "Canon scanner manual" summary for an MX5280.pdf
that is actually a scanned legal letter.
Two changes shut this down:
1. Drop filename / title / file-type from every LLM prompt input. The
wiki summary, knowledge extract, candidate slug, and chunk citation
prompts now see only the document content. The knowledge summary
path no longer prepends the "Document Type: X / File Name: Y" intro.
2. Add a content-sufficiency guard in front of the LLM call. A new
helper strips markdown image references and HTML image tags and
refuses to invoke the model when fewer than 50 real-text runes
remain. getSummary returns a sentinel error in that case and the
caller marks the knowledge as SummaryStatusFailed instead of
surfacing the first chunk's bare image reference. Wiki ingest skips
LLM extraction with a clear log message.
The generate_summary.yaml prompt is also updated: the "NEVER output
refusal phrases" clause is removed and replaced with an explicit
empty-content rule that returns a deterministic placeholder line.
Real OCR (Tesseract / PaddleOCR) and filename-pattern sanitisation are
intentionally left for separate PRs.
Previously the Langfuse integration only traced in-process HTTP requests
(chat / search / eval), so file uploads and every downstream asynq task
(document parse, chunk embedding, OCR/VLM, summary / question gen, wiki
ingest, datasource sync, etc.) produced either disconnected shallow
traces or no observation at all.
This change threads one trace end-to-end:
- tracer: add SPAN observation type and StartSpan; add ResumeTrace so a
worker can attach to an upstream trace without emitting a duplicate
trace-create; StartGeneration now auto-picks parentObservationId from
ctx so nested trace -> span -> generation trees render correctly.
- types.TracingContext + LangfuseTracingCarrier: embed on all 17 asynq
payloads so trace_id / parent_obs_id / user_id / session_id serialise
into every job.
- langfuse.InjectTracing: injected at 28 enqueue sites before json.Marshal
so the HTTP-layer trace survives the Redis hop.
- langfuse.AsynqMiddleware: mux.Use hook that peeks the payload, either
resumes the upstream trace or opens a standalone asynq.<type> trace
for scheduled jobs, and wraps the handler in a SPAN with task metadata
(id / queue / retry / payload_bytes) plus ERROR level on failure.
- GinMiddleware.shouldTrace: whitelist ingestion / knowledge-mutation /
FAQ / wiki / datasource endpoints so the root trace actually starts.
- Tests: tracer_test.go covers span nesting, error status, and
ResumeTrace no-trace-create guarantee; asynq_test.go covers
InjectTracing round-trip, middleware resume path, and standalone
trace fallback.
- Docs: docs/Langfuse\u96c6\u6210.md now lists the covered task types
and documents the cross-process propagation model.
No behavioural change when Langfuse is disabled (all new code paths are
no-ops and carriers serialise to empty strings with omitempty).
- Updated the scrubWikiPendingIngest method to accept a reason parameter, improving logging clarity during wiki cleanup and reparse operations.
- Introduced prepareWikiForReparse to maintain the integrity of the pending ingest queue during reparse events, preventing stale operations from executing.
- Enhanced the mapping logic in the wiki ingest process to differentiate between stale and reparse updates, ensuring accurate handling of knowledge contributions.
- Improved logging to provide better insights into the number of reparse and stale slugs processed during wiki ingestion.
- Deleted the RebuildKnowledgeBaseIndex function from the knowledge service, which was responsible for reprocessing all documents in a knowledge base.
- Removed the RebuildIndex handler from the KnowledgeBaseHandler, eliminating the endpoint that triggered the rebuild process.
- Updated the router to remove the route for rebuilding the index, streamlining the knowledge base management functionality.
- Updated the DeleteKnowledge and DeleteKnowledgeList functions to skip vector store cleanup for knowledge entries without an embedding model, preventing unnecessary errors.
- Enhanced logging to provide clear information when skipping cleanup for knowledge without embeddings, improving overall robustness and clarity in the deletion process.
These changes ensure that the deletion process is more efficient and user-friendly, particularly for knowledge bases that do not utilize embeddings.
- Enhanced the `cleanupWikiOnKnowledgeDelete` function to pass the full knowledge object, allowing for more accurate title and summary retrieval during wiki cleanup.
- Introduced a tombstone mechanism in Redis to mark recently deleted knowledge, enabling in-flight wiki ingest tasks to skip processing for deleted documents.
- Updated the `isKnowledgeGone` function to check for deleted knowledge using the tombstone, improving efficiency in handling concurrent delete and ingest operations.
- Added logic in the `wikiIngestService` to filter out updates for deleted knowledge, ensuring that no ghost references are created during the ingest process.
- Implemented lint checks for stale source references in the `WikiLintService`, allowing for automatic cleanup of pages referencing deleted knowledge.
These changes significantly enhance the robustness of the wiki system, ensuring consistency and accuracy in handling knowledge deletions and related wiki pages.
- Introduced multiple new documentation files, including a complete documentation index, quick reference guides, and comprehensive analyses of the knowledge base and wiki settings.
- The new files provide detailed insights into the architecture, data models, API endpoints, and implementation strategies for the knowledge base and wiki functionalities.
- Enhanced navigation and usability for developers and team members by summarizing key components, configurations, and data flows related to the knowledge base system.
These additions significantly improve the clarity and accessibility of documentation, supporting better onboarding and reference for current and future team members.
- Renamed `WikiPageUpdatePrompt` to `WikiPageModifyPrompt` to better reflect its functionality of adding new information and removing outdated content in a single operation.
- Enhanced the instructions for both `WikiPageModifyPrompt` and `WikiPageRetractPrompt` to clarify the handling of additions and retractions, ensuring accurate content updates.
- Consolidated the logic for enqueuing retraction tasks in the `cleanupWikiOnKnowledgeDelete` method to streamline the process of managing affected pages.
- Removed the `rebuildWikiIndexSimple` function to simplify the codebase, as its functionality is no longer needed with the new ingestion approach.
These changes improve the clarity and efficiency of the wiki content management process, enhancing overall system performance.
- Introduced `pending_tasks` and `is_active` fields in the `WikiStats` interface to track the number of tasks waiting for processing and the current activity status of the wiki ingestion.
- Updated localization files for English, Korean, Russian, and Chinese to include a new message for displaying the number of pending tasks in the wiki queue.
- Enhanced the `WikiBrowser` component to display the queue status, improving user awareness of ongoing tasks.
- Implemented polling logic to refresh wiki stats periodically when there are pending tasks or active ingestion, ensuring up-to-date information is presented.
These changes enhance the functionality and user experience of the wiki management system by providing real-time insights into task status.
- Introduced a new method to list all wiki pages without pagination, facilitating index rebuilding and cross-linking.
- Updated the wiki ingest and retract processes to include document summaries, improving clarity in log entries and index updates.
- Enhanced the cleanup process for deleted knowledge, ensuring accurate summary retention and index regeneration.
- Streamlined the handling of page updates and deletions, optimizing the overall wiki management workflow.
These changes improve the efficiency and reliability of wiki content management and indexing.
- Modified the `rebuildWikiIndexSimple` function to include `knowledgeID` in the parameters, ensuring accurate logging of deleted pages.
- Enhanced the log entry format in `appendLogEntry` to include the action type and improved timestamp formatting, providing clearer context for actions taken on wiki pages.
- Adjusted the `ProcessWikiRetract` and `ProcessWikiIngest` methods to align with the new log entry structure, ensuring consistency across logging actions.
These changes improve the clarity and accuracy of wiki activity logs, facilitating better tracking of content changes.
Implemented new features in the WikiBrowser component to allow users to toggle the visibility of arrows on graph edges. This includes:
1. **Localization Updates**: Added translations for "Show arrows" and "Hide arrows" in English, Korean, Russian, and Chinese.
2. **Graph Rendering Enhancements**: Introduced logic to dynamically update SVG line attributes based on the toggle state, improving visual clarity in graph representations.
3. **State Management**: Added reactive state management for arrow visibility, ensuring seamless user interaction.
These changes enhance the user experience by providing more control over graph visualizations.
Added functionality to enable updates for the `wiki_config` field in the `KnowledgeBaseConfig` struct. This includes:
1. **Type Definition Update**: Added `WikiConfig` field to `KnowledgeBaseConfig` to allow JSON unmarshaling during update requests.
2. **Service Layer Enhancement**: Updated the `UpdateKnowledgeBase` method to persist changes to `wiki_config` when provided in the request.
3. **Frontend TypeScript Adjustments**: Enhanced type definitions in the frontend to support `wiki_config`, improving IDE autocomplete and type checking.
This fix addresses the critical issue where `wiki_config` was not being updated due to its absence in the `KnowledgeBaseConfig` struct, ensuring proper handling of wiki configurations during knowledge base updates.
- Updated the prompt template to clarify the task of understanding and rewriting user questions, improving the instructions for coreference resolution and ellipsis completion.
- Renamed sections for better clarity, changing "Rewriting Goals" to "Query Understanding."
- Added critical notes for intent classification regarding image and document attachments, ensuring accurate processing based on the presence of attachments.
- Improved examples to reflect the updated intent classification logic, enhancing the overall user experience in query handling.
Introduce KnowledgePostProcess task to decouple image multimodal
processing from downstream operations like summary, question
generation, and knowledge graph extraction.
This allows unifying the processing logic across documents and
images and ensures all text content, including from images,
is properly extracted before post-processing.
Multiple fixes for WeKnora Lite mode (SQLite, no Redis):
1. Redis nil pointer panic on FAQ import (knowledge.go)
- Add in-memory fallback (sync.Map) for FAQ import progress tracking
- Add nil guards for running task locks and progress queries
2. seq_id not auto-incrementing in SQLite (chunk.go, tag.go)
- GORM autoIncrement on non-PK columns doesn't work in SQLite
- Add AssignChunkSeqIDs() to pre-assign seq_ids before batch insert
- Add BeforeCreate hook for KnowledgeTag
- Use Unscoped() to include soft-deleted records in MAX(seq_id) query
3. NOW() function not available in SQLite (chunk.go)
- Replace raw SQL NOW() with time.Now() or datetime('now')
- Use dialector check for raw SQL that must differ between PG and SQLite
4. FAQ KB should reject file uploads (knowledge.go)
- Add kb.Type == "faq" check in CreateKnowledgeFromFile
5. Linux/macOS build compatibility (Makefile)
- Conditionally apply macOS-only linker flag based on uname
6. Add SQLite integration tests (chunk_sqlite_test.go)
- TestCreateChunks_SQLite_SeqIDAutoAssigned
- TestCreateChunks_SQLite_SeqIDContinuesFromExisting
- TestCreateChunks_SQLite_SeqIDUniqueAcrossKBs
- TestCreateChunks_SQLite_SeqIDAfterSoftDelete
- TestKnowledgeTag_SQLite_SeqIDAutoAssigned
- TestUpdateChunk_SQLite_NoNOWError
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replaced references to Docreader credentials with WeKnoraCloud credentials across multiple services and handlers.
- Introduced a new CredentialsConfig structure to manage third-party provider credentials, specifically for WeKnoraCloud.
- Updated database schema to include a credentials column for storing WeKnoraCloud AppID and AppSecret.
- Enhanced methods for retrieving and utilizing WeKnoraCloud credentials, ensuring proper encryption and decryption during storage and retrieval.
This update improves the management of WeKnoraCloud credentials, streamlining the integration and enhancing security measures.
- Removed support for video file uploads across the application, including updates to the knowledge base and document processing services.
- Updated internationalization files to reflect the removal of video-related terminology, changing labels and descriptions to focus solely on audio processing.
- Adjusted file type handling to exclude video formats, ensuring users receive accurate feedback regarding supported file types.
- Enhanced error handling to provide clear messages when users attempt to upload unsupported video files.
This update streamlines the application by focusing on audio processing capabilities and improving user experience through clearer messaging.
- Changed 'Audio Processing' to 'Audio & Video' in English, Korean, Russian, and Chinese translations to reflect expanded functionality.
- Updated ASR configuration titles and descriptions to include video processing capabilities, enhancing user understanding of the feature.
- Renamed file type labels from 'Audio Files' to 'Audio & Video' across multiple languages for consistency and clarity.
- Enhanced document upload handling in the KnowledgeBase to filter unsupported file types, providing user feedback for filtered images, videos, and audio files.
This update improves the internationalization of the application and clarifies the capabilities of the ASR feature.
- Introduced a new WeKnoraCloudSettings component for managing APPID and APPSECRET.
- Added functionality to save WeKnoraCloud credentials without automatically creating models.
- Implemented credential status checks across various components to provide user feedback on configuration status.
- Updated API endpoints to handle credential saving and status retrieval.
- Enhanced UI elements to guide users in configuring WeKnoraCloud settings.
This update improves the integration of WeKnoraCloud by allowing users to manage their credentials more effectively and receive real-time feedback on their configuration status.
- Added ASR configuration options to the knowledge base and model settings.
- Implemented ASR model selection and transcription capabilities in the knowledge processing pipeline.
- Enhanced file upload support to include audio formats for ASR processing.
- Updated localization files to include ASR-related labels and descriptions.
- Refactored model handling to accommodate ASR as a new model type.
- Added max_input_chars configuration to limit input size for summary generation.
- Updated generate_summary.yaml to clarify summary generation steps and requirements.
- Introduced a new summary section in the document content view to display generated summaries or loading states.
- Refactored doc-content.vue to utilize computed properties for improved performance and readability.
- Enhanced knowledge base hooks to include description and summary status fields for better data handling.
- Updated internationalization files to include new summary-related labels in multiple languages.
Previously, only the original uploaded file was deleted via knowledge.FilePath.
Images extracted by the builtin parser (stored as individual S3/MinIO/local files
via ImageResolver) were never cleaned up, causing orphan files to accumulate.
- Add ListImageInfoByKnowledgeIDs repository method (no chunk_type filter)
- Add collectImageURLs utility with URL deduplication and JSON parse logging
- Add deleteExtractedImages helper (standalone, best-effort with error logging)
- Fix DeleteKnowledge: collect image URLs before errgroup, delete in file goroutine
- Fix DeleteKnowledgeList: bulk collect with KB-level fileSvc mapping
- Fix cleanupKnowledgeResources: collect before chunk deletion, delete after
- Fix ProcessKBDelete: collect before chunk deletion for KB-wide cleanup
- Add DefaultLanguage() that reads WEKNORA_LANGUAGE env with zh-CN fallback
- Replace hardcoded zh-CN in LanguageNameFromContext with DefaultLanguage()
- Unify middleware/language.go to use types.DefaultLanguage()
- Add Language field to DocumentProcessPayload (8 enqueue sites + worker restore)
- Add Language field to ImageMultimodalPayload (enqueue + worker restore)
- Propagate language from ImageMultimodalPayload to QuestionGenerationPayload
- Updated UpdateKnowledgeTagBatch method to include an authorizedKBID parameter for restricting updates to specific knowledge items.
- Implemented validation to ensure that all requested knowledge IDs belong to the authorized knowledge base.
- Enhanced GetKnowledgeBatch and UpdateKnowledgeTagBatch methods to incorporate agent-specific knowledge base access restrictions.
- Added utility functions to resolve allowed knowledge base IDs for agents based on their configuration.
- Updated the question generation template to clarify the role of surrounding context and main content.
- Enhanced quality rules for generated questions to better align with user search intent.
- Revised output format and added explicit instructions on what not to generate.
- Improved logging and output in the web parser for better visibility of parsed content and metadata.
- Updated DocReaderServicer to pass metadata in responses.
- Modified PipelineParser to accumulate and merge metadata from all parsers.
- Enhanced StdWebParser to extract and log the title from web page content.
- Implemented logic in knowledge service to update knowledge title based on extracted metadata.
- Keep strict mode blocking all direct IPs (IPv4 and IPv6 uniformly)
- Unify all SSRF call sites to use ValidateURLForSSRF (whitelist-aware)
- Add Teredo (2001:0000::/32) and 6to4 (2002::/16) tunnel detection
- Make redirect handler and DNS pinning respect SSRF_WHITELIST
- Unexport isSSRFSafeURL to prevent future callers bypassing whitelist
- Add scheme validation for whitelisted redirect targets
- Document IPv6 whitelist syntax in .env.example
- Add comprehensive IPv6 test coverage
The file upload pipeline (ProcessDocument) only called ResolveAndStore(),
which handles inline bytes and base64 data URIs but explicitly skips
http/https URLs. This meant markdown files with external image URLs
(e.g. GitHub issue screenshots) were never downloaded, stored, or
processed by VLM multimodal.
Add ResolveRemoteImages() call after ResolveAndStore() in the file
upload path, matching the existing manual processing path behavior.
When LLM summary generation fails, the fallback logic truncates
the first chunk content to 500 characters. Previously used byte-based
slicing (summary[:500]) which could cut multi-byte UTF-8 characters
in half, causing PostgreSQL to reject invalid byte sequences.
Changed to rune-based truncation to ensure valid UTF-8 after truncation.
Fixes: Documents stuck in "Generating summary" status when containing
Chinese/emoji characters and summary generation fails.
- Updated search queries to filter by both file name and title when a keyword is provided, improving search accuracy.
- Added support for filtering by URL and HTML file types in search queries, enhancing the flexibility of the search functionality.
- Refactored query construction for file types to streamline the inclusion of additional conditions, ensuring robust handling of various file types.
- Introduced a new `channel` field in the Knowledge struct and associated request types to track the source channel (e.g., "web", "api", "browser_extension").
- Updated various frontend components to display channel information and enhance user experience with channel labels.
- Enhanced localization files to support channel labels in English and Chinese.
- Modified backend services and database migrations to accommodate the new channel feature, ensuring consistent tracking across knowledge entries.
- Refactored related functions to integrate channel handling, improving overall knowledge management and context.