姚章浩
693cd952cd
仓库首次入库（此前仅存在于本地工作区与测试服构建产物中，未上传）。 改动（src/knowledge_adapter/asset_content_processor.py）： - 新增 _TRUNCATED_TRAILING_MARKDOWN_RESOURCE_IMAGE 规则，清除 chunk 末尾被截断的 ![](resource://xxxx 残片（WeKnora 分块在图片引用中间截断时产生，旧逻辑无法匹配完整引用， 残留 resource:// 触发 ContentProcessingError，导致整单 502） - 回归测试 tests/test_asset_content_processor.py:: test_truncated_trailing_markdown_resource_image_is_removed 验证证据： - 测试服 120.76.217.51，镜像 knowledge-adapter-service:reviewed-test-aeab4b5fca2d （release aeab4b5fca2d8661）：KB d015d05c-25c8-4d63-b2ab-e8590d3c2dfb 5 个查询全部 http=200、records=5、残留 resource:// = 0；KB b9dacd62 无退化；容器日志无 error/exception - 本地：pytest 141 项全通过、ruff 全通过 - 该文件与 258c31e0fe07fd08 源码的差异经逐文件 md5 比对，为唯一变更文件
115 lines
5.8 KiB
Markdown
115 lines
5.8 KiB
Markdown
# Security Model
|
|
|
|
## Security objectives
|
|
|
|
1. Keep the OSS bucket Private while permitting stable customer-facing UUID image links.
|
|
2. Keep all raw API keys, database credentials, OSS AccessKeys, object keys, and Presigned URL query parameters out of Git and logs.
|
|
3. Restrict retrieval to configured knowledge bases and image signing to active Adapter mappings.
|
|
4. Fail closed when validation, a database, WeKnora, OSS credential files, or signing is unavailable.
|
|
|
|
## Credential model
|
|
|
|
Because the target ECS instances are not permitted to receive RAM roles, the Adapter uses a dedicated RAM user's static AK. Test and production use different identities and different Private OSS buckets. The AccessKey ID and secret are mounted as separate read-only files; they are never accepted as Compose environment values, build arguments, API fields, or command-line arguments.
|
|
|
|
The production RAM user policy grants only `oss:GetObject`. Alibaba Cloud documents `oss:GetObject` as the permission required to download an object.[4] The supplied policy narrows this action to:
|
|
|
|
```text
|
|
acs:oss:*:*:${OSS_BUCKET}/weknora/${WEKNORA_TENANT_ID}/*
|
|
```
|
|
|
|
It intentionally omits List, Put, Delete, ACL, bucket-management, RAM management, and KMS permissions. A higher-privilege test AK may be used only for the explicitly approved test window; it must never be copied to production and must be replaced by a dedicated read-only identity before production traffic. If the reviewed bucket uses SSE-KMS, do not add `kms:Decrypt` automatically; review a separately scoped KMS policy first.
|
|
|
|
## Private OSS and temporary access
|
|
|
|
A V4 Presigned GET URL grants temporary access to one object without requiring credentials at the recipient.[2] V4 URLs expose `x-oss-expires`, use `OSS4-HMAC-SHA256`, and carry a signature in the query string.[2] Therefore:
|
|
|
|
- the URL lifetime is fixed at 600 seconds;
|
|
- the generated scheme and hostname are revalidated as the configured public regional OSS host;
|
|
- the URL appears only in the 302 `Location` header;
|
|
- the response is `no-store` and `no-referrer`;
|
|
- the URL is never persisted or logged;
|
|
- a stable `/i/{UUID}` remains reusable after the temporary URL expires.
|
|
|
|
The bucket must remain Private. Anonymous direct OSS access must return denial during Slice 9 acceptance.
|
|
|
|
## Secret storage
|
|
|
|
Raw secrets are mounted as read-only files owned by container UID 10001 with mode `0400`:
|
|
|
|
- `/run/secrets/weknora_api_key`
|
|
- `/run/secrets/asset_database_url`
|
|
- `/run/secrets/weknora_readonly_database_url`
|
|
- `/run/secrets/oss_access_key_id`
|
|
- `/run/secrets/oss_access_key_secret`
|
|
|
|
They must not appear in `.env`, Compose environment, Docker build args, image layers, source, fixtures, documentation examples, or shell command history. Dify stores only current/next SHA-256 hashes in non-secret runtime configuration.
|
|
|
|
## Input and SSRF controls
|
|
|
|
The Adapter does not perform arbitrary HTTP/HTTPS downloads. Callers cannot provide bucket, endpoint, object key, expiry, or destination URL. Only canonical `resource://` handles are accepted for first resolution; fixed tenant, storage backend, provider, bucket, and object-key rules are checked before persistence or signing.
|
|
|
|
This removes the general outbound URL-fetch primitive that would create SSRF risk. Outbound application destinations are limited to configured WeKnora, PostgreSQL, and the SDK-derived regional OSS endpoint.
|
|
|
|
## Authentication and authorization
|
|
|
|
- `POST /retrieval` requires a Dify Bearer key matching current or next hash.
|
|
- Requested knowledge bases must be allowlisted before search.
|
|
- Every returned chunk must repeat the requested knowledge-base UUID.
|
|
- `GET /i/{UUID}` is intentionally anonymous; possession of a valid UUID grants access.
|
|
- Missing and disabled UUIDs have the same empty 404 behavior.
|
|
- `GET /healthz` discloses only `{"status":"ok"}`.
|
|
|
|
## Logging and error redaction
|
|
|
|
The application writes one JSON request summary with only:
|
|
|
|
- request ID;
|
|
- method;
|
|
- matched route template;
|
|
- status code;
|
|
- duration in milliseconds;
|
|
- fixed error code for failures.
|
|
|
|
The following are prohibited from application and Nginx access logs:
|
|
|
|
- Dify or WeKnora API Key;
|
|
- `Authorization` header;
|
|
- request query/body and complete knowledge content;
|
|
- `resource://` locator;
|
|
- `object_key`;
|
|
- database DSN;
|
|
- OSS AccessKey ID/secret;
|
|
- Presigned URL or signature fields.
|
|
|
|
Nginx access logging is disabled for this virtual host. Unexpected exceptions return a fixed 500 without traceback or exception text in Adapter request logs.
|
|
|
|
## Container and network controls
|
|
|
|
- runtime UID/GID 10001; no root process;
|
|
- read-only root filesystem and no-exec temporary filesystem;
|
|
- all Linux capabilities dropped;
|
|
- `no-new-privileges` enabled;
|
|
- process, CPU, and memory limits set;
|
|
- Adapter published only on loopback;
|
|
- TLS 1.2/1.3 terminated by host Nginx;
|
|
- route/method allowlists, request-size limit, and rate limits at Nginx;
|
|
- Uvicorn raw access log disabled.
|
|
|
|
## Review checklist
|
|
|
|
- [ ] Private bucket confirmed; anonymous object GET denied.
|
|
- [ ] Dedicated test/production RAM users and AccessKeys are separate.
|
|
- [ ] Production RAM policy has only `oss:GetObject` for the reviewed bucket/tenant prefix.
|
|
- [ ] No other policy grants overlapping OSS write/list access.
|
|
- [ ] Five Secret files are owned by UID 10001 and mode `0400`.
|
|
- [ ] Adapter DB and WeKnora DB use different least-privilege accounts.
|
|
- [ ] Nginx certificate paths and public hostname are reviewed.
|
|
- [ ] Sanitized logs contain request IDs but none of the prohibited values.
|
|
- [ ] Arbitrary URL, bucket, object-key, and endpoint override tests pass.
|
|
- [ ] Slice 9 real test-environment E2E and audit report are accepted.
|
|
|
|
## Sources
|
|
|
|
[2] https://alibabacloud.com/help/en/oss/developer-reference/python-download-using-a-presigned-url — Alibaba Cloud OSS Python presigned GET URL
|
|
[4] https://www.alibabacloud.com/help/en/oss/developer-reference/get-object — Alibaba Cloud OSS GetObject permission
|