resolveFileServiceForKnowledge was reading os.Getenv("LOCAL_STORAGE_BASE_DIR")
on every request. This causes a mismatch when the variable is not in the
process environment at request time (e.g. it was set programmatically before
app startup but not exported, or the DI-constructed fileService uses a
different configured path).
Fix: store the value in a new localBaseDir field at NewDataAnalysisTool
construction time — the same moment the DI container assembles the local
FileService — and use t.localBaseDir in resolveFileServiceForKnowledge
instead. This guarantees the data-analysis tool always resolves files
under the same base directory that the local FileService was configured
with, eliminating the "path not found" errors when LOCAL storage is used
for a knowledge base while the global storage backend is minio/oss/cos.
Fixes#1040
Dev-mode repro from the issue reporter's logs:
[Tool][DataAnalysis] Failed to create table from Excel: IO Error:
GDAL Error (4): Failed to open file
'local://10000/.../1777030910246871000.xlsx': No such file or directory
The local file service returns a custom 'local://' URL from GetFileURL,
which the duckdb spatial/excel extensions can't resolve (they expect
plain paths or http(s)/s3:// style URIs). Presigned HTTPS URLs from
cloud backends worked by accident; the 'local://' dev path has always
been broken for the Data Analysis tool.
Instead of per-scheme adapters, stream the file through
FileService.GetFile into a temp file and hand DuckDB the resulting
filesystem path. This works uniformly across every backend (local,
OSS, S3, MinIO, COS) and survives future scheme changes without any
changes to the Data Analysis tool.
- Preserve the original file extension on the temp file so DuckDB's
format auto-detection (csv / xlsx / xls) still kicks in.
- Clean up the temp file when LoadFromKnowledge returns, including on
every error branch. Cleanup is idempotent (double-invoke safe).
- New tests stub FileService to lock in that (a) we never leak a
provider:// URL to DuckDB, (b) GetFile errors propagate, (c) the
extension survives case normalization.
Refs: https://github.com/Tencent/WeKnora/issues/1007
Catching up on a real-world check: when a multi-sheet .xlsx fixture is
fed through LoadFromExcel against an in-memory DuckDB with the spatial
and excel extensions, the first iteration's sheet enumeration quietly
fell back because st_read_meta doesn't expose a scalar 'layer_name'
column — it returns a LIST<STRUCT> column called 'layers'. We now
UNNEST(layers).name so enumeration actually works, and assert the full
data path with three DuckDB-backed tests:
- multi-sheet workbook yields the sum of all rows and per-sheet
breakdown via __sheet_name (including schema drift: columns only
present in one sheet are NULL for the other)
- single-sheet workbook still works and tags rows with its sheet name
- sheet name containing a single quote (Q1'24) survives the SQL
literal round-trip
excelize/v2 is added as a test-time helper to build the fixtures
deterministically. If DuckDB's spatial/excel extensions can't be
installed (offline CI), the tests skip rather than fail.
Refs: https://github.com/Tencent/WeKnora/issues/1007
DuckDB's st_read (spatial) only reads the first layer/sheet of a .xlsx
workbook, so every sheet beyond Sheet1 was silently dropped from the
DuckDB table the Data Analysis tool builds. Users trying to analyse
multi-sheet workbooks could only see the first sheet.
Switch to DuckDB's dedicated 'excel' extension (read_xlsx) for the
actual data load, enumerate sheets via the spatial extension's
st_read_meta, and UNION ALL BY NAME the rows of every sheet into one
table. A synthetic __sheet_name column records the source so the LLM
can still filter/aggregate per sheet; schema drift between sheets is
tolerated via UNION BY NAME. If enumeration fails (older DuckDB, local
filesystem errors, …) we fall back to reading the first sheet so the
tool stays usable.
- Install & LOAD the 'excel' extension alongside 'spatial' both at
startup (internal/container) and in the offline prefetch binary
(cmd/download/duckdb).
- Harden sheet/path handling against single quotes.
- Update the tool description so the agent knows about __sheet_name.
- Add unit tests for the CREATE TABLE SQL builder covering 0 / 1 / N
sheets and quote escaping.
Refs: https://github.com/Tencent/WeKnora/issues/1007
Convert all LLM-facing prompts, templates, and tool output labels to
language-neutral English so the model responds in the user's input
language. Update frontend relevance/match-type mappings to match.
Output language strategy:
- User-facing: reply in user's input language
- Vector-indexed: match source document language
- LLM-internal: fixed English
- Introduced a new API for organization management, including creation, updating, and deletion of organizations.
- Implemented organization sharing functionality for knowledge bases, allowing users to share knowledge bases with specific organizations and set permissions.
- Added UI components for organization selection and sharing dialogs.
- Enhanced existing components to support organization-related features, including shared knowledge base handling and organization role management.
This update enhances collaboration capabilities within the application by allowing users to manage and share knowledge bases across different organizations.
guangyang1206