Weknora/knowledge-adapter-service

Files

T

姚章浩 693cd952cd feat(asset): 清理被截断的结尾图片引用，修复整单 502

仓库首次入库（此前仅存在于本地工作区与测试服构建产物中，未上传）。

改动（src/knowledge_adapter/asset_content_processor.py）：
- 新增 _TRUNCATED_TRAILING_MARKDOWN_RESOURCE_IMAGE 规则，清除 chunk 末尾被截断的
  ![](resource://xxxx 残片（WeKnora 分块在图片引用中间截断时产生，旧逻辑无法匹配完整引用，
  残留 resource:// 触发 ContentProcessingError，导致整单 502）
- 回归测试 tests/test_asset_content_processor.py::
  test_truncated_trailing_markdown_resource_image_is_removed

验证证据：
- 测试服 120.76.217.51，镜像 knowledge-adapter-service:reviewed-test-aeab4b5fca2d
  （release aeab4b5fca2d8661）：KB d015d05c-25c8-4d63-b2ab-e8590d3c2dfb 5 个查询全部
  http=200、records=5、残留 resource:// = 0；KB b9dacd62 无退化；容器日志无 error/exception
- 本地：pytest 141 项全通过、ruff 全通过
- 该文件与 258c31e0fe07fd08 源码的差异经逐文件 md5 比对，为唯一变更文件

2026-09-16 10:01:27 +08:00

5.8 KiB

Raw Blame History

Security Model

Security objectives

Keep the OSS bucket Private while permitting stable customer-facing UUID image links.
Keep all raw API keys, database credentials, OSS AccessKeys, object keys, and Presigned URL query parameters out of Git and logs.
Restrict retrieval to configured knowledge bases and image signing to active Adapter mappings.
Fail closed when validation, a database, WeKnora, OSS credential files, or signing is unavailable.

Credential model

Because the target ECS instances are not permitted to receive RAM roles, the Adapter uses a dedicated RAM user's static AK. Test and production use different identities and different Private OSS buckets. The AccessKey ID and secret are mounted as separate read-only files; they are never accepted as Compose environment values, build arguments, API fields, or command-line arguments.

The production RAM user policy grants only oss:GetObject. Alibaba Cloud documents oss:GetObject as the permission required to download an object.[4] The supplied policy narrows this action to:

acs:oss:*:*:${OSS_BUCKET}/weknora/${WEKNORA_TENANT_ID}/*

It intentionally omits List, Put, Delete, ACL, bucket-management, RAM management, and KMS permissions. A higher-privilege test AK may be used only for the explicitly approved test window; it must never be copied to production and must be replaced by a dedicated read-only identity before production traffic. If the reviewed bucket uses SSE-KMS, do not add kms:Decrypt automatically; review a separately scoped KMS policy first.

Private OSS and temporary access

A V4 Presigned GET URL grants temporary access to one object without requiring credentials at the recipient.[2] V4 URLs expose x-oss-expires, use OSS4-HMAC-SHA256, and carry a signature in the query string.[2] Therefore:

the URL lifetime is fixed at 600 seconds;
the generated scheme and hostname are revalidated as the configured public regional OSS host;
the URL appears only in the 302 Location header;
the response is no-store and no-referrer;
the URL is never persisted or logged;
a stable /i/{UUID} remains reusable after the temporary URL expires.

The bucket must remain Private. Anonymous direct OSS access must return denial during Slice 9 acceptance.

Secret storage

Raw secrets are mounted as read-only files owned by container UID 10001 with mode 0400:

/run/secrets/weknora_api_key
/run/secrets/asset_database_url
/run/secrets/weknora_readonly_database_url
/run/secrets/oss_access_key_id
/run/secrets/oss_access_key_secret

They must not appear in .env, Compose environment, Docker build args, image layers, source, fixtures, documentation examples, or shell command history. Dify stores only current/next SHA-256 hashes in non-secret runtime configuration.

Input and SSRF controls

The Adapter does not perform arbitrary HTTP/HTTPS downloads. Callers cannot provide bucket, endpoint, object key, expiry, or destination URL. Only canonical resource:// handles are accepted for first resolution; fixed tenant, storage backend, provider, bucket, and object-key rules are checked before persistence or signing.

This removes the general outbound URL-fetch primitive that would create SSRF risk. Outbound application destinations are limited to configured WeKnora, PostgreSQL, and the SDK-derived regional OSS endpoint.

Authentication and authorization

POST /retrieval requires a Dify Bearer key matching current or next hash.
Requested knowledge bases must be allowlisted before search.
Every returned chunk must repeat the requested knowledge-base UUID.
GET /i/{UUID} is intentionally anonymous; possession of a valid UUID grants access.
Missing and disabled UUIDs have the same empty 404 behavior.
GET /healthz discloses only {"status":"ok"}.

Logging and error redaction

The application writes one JSON request summary with only:

request ID;
method;
matched route template;
status code;
duration in milliseconds;
fixed error code for failures.

The following are prohibited from application and Nginx access logs:

Dify or WeKnora API Key;
Authorization header;
request query/body and complete knowledge content;
resource:// locator;
object_key;
database DSN;
OSS AccessKey ID/secret;
Presigned URL or signature fields.

Nginx access logging is disabled for this virtual host. Unexpected exceptions return a fixed 500 without traceback or exception text in Adapter request logs.

Container and network controls

runtime UID/GID 10001; no root process;
read-only root filesystem and no-exec temporary filesystem;
all Linux capabilities dropped;
no-new-privileges enabled;
process, CPU, and memory limits set;
Adapter published only on loopback;
TLS 1.2/1.3 terminated by host Nginx;
route/method allowlists, request-size limit, and rate limits at Nginx;
Uvicorn raw access log disabled.

Review checklist

Private bucket confirmed; anonymous object GET denied.
Dedicated test/production RAM users and AccessKeys are separate.
Production RAM policy has only oss:GetObject for the reviewed bucket/tenant prefix.
No other policy grants overlapping OSS write/list access.
Five Secret files are owned by UID 10001 and mode 0400.
Adapter DB and WeKnora DB use different least-privilege accounts.
Nginx certificate paths and public hostname are reviewed.
Sanitized logs contain request IDs but none of the prohibited values.
Arbitrary URL, bucket, object-key, and endpoint override tests pass.
Slice 9 real test-environment E2E and audit report are accepted.

Sources

[2] https://alibabacloud.com/help/en/oss/developer-reference/python-download-using-a-presigned-url — Alibaba Cloud OSS Python presigned GET URL [4] https://www.alibabacloud.com/help/en/oss/developer-reference/get-object — Alibaba Cloud OSS GetObject permission