为什么:测试 0.9.0 栈(Compose 项目 weknora-v090)没有源库容器,治理库也不在 容器里,两者都是 RDS。旧实现用 -source-pg/-governance-pg 的容器地址拼 DSN、 用 docker exec psql 读路由绑定,在这类栈上采集根本起不来——不是权限问题,是 地址解析方式错了。 怎么改: - 新增 internal/librarydsn 收敛"这次 DSN 从哪来":容器模式保持原形状 (sslmode=disable,目标不变);-source-pg 为空时改用源应用容器 env 的 DB_* 直连托管实例,TLS 取 LIBRARY_SOURCE_DB_SSLMODE(默认 require,prefer/allow 这类会静默降级的取值直接拒);-governance-pg 为空时从 LIBRARY_GOVERNANCE_DSN_FILE 的 0600 文件读 DSN(复用 dobotaiops.ReadDSNFile, 不另写一份同义实现)。两条路径都不进日志/错误/describe,错误只点名缺哪个变量、 缺哪个文件。 - libraryctl 的路由绑定读取按模式分派,只共用行解析;外部路径用只读会话。 libraryagent 的 identity/health 增报 source_db/governance_db 模式;-describe 只打这两项,不再 dump 配置。 - 新增 scripts/library-capture-v090.sh:weknora-v090 的唯一写入者 app + 逐个 声明的 non-writers,两个 -pg 传空走外部模式,归档根可环境变量覆盖, LIBRARYCTL_BIN 缺失即失败;0.7.1 入口脚本一字未动。 怎么验:internal/librarydsn 三条路径(容器/RDS/文件)各有用例,缺变量、缺文件、 0644 文件、非 PostgreSQL 内容均失败关闭且断言错误不含口令;cmd/libraryctl 侧 钉住文件模式接线与"错误不带地址";v090 脚本进 CI 做 bash -n 加标志/服务声明 断言,并拒绝 0.7.1 容器名出现在新入口。make verify 17 项全绿。 未做:真实采集、生产 executor、Key/KB 绑定。
This commit is contained in:
@@ -1,5 +1,29 @@
|
||||
# Changelog
|
||||
|
||||
## 开发快照 0.1.379(采集接入 RDS 型 0.9.0 栈)
|
||||
|
||||
### Added
|
||||
|
||||
- `libraryctl` / `libraryagent` 的数据库来源可选:`-source-pg` 为空时用 `-source-app`
|
||||
容器环境里的 `DB_HOST/DB_PORT/DB_NAME/DB_USER/DB_PASSWORD` 直连托管实例(RDS),
|
||||
TLS 模式取 `LIBRARY_SOURCE_DB_SSLMODE`(默认 `require`,`disable` 仅在显式测试时允许);
|
||||
`-governance-pg` 为空时从 `LIBRARY_GOVERNANCE_DSN_FILE` 指向的 0600 文件读治理库 DSN。
|
||||
两个都配置时行为与原来完全一致(容器模式),0.7.1 栈不受影响。
|
||||
- 新增 `scripts/library-capture-v090.sh`:Compose 项目 `weknora-v090` 的整库采集入口,
|
||||
唯一写入者 `app`,其余服务逐个声明理由;两个数据库走外部模式(RDS),归档/日志根
|
||||
默认 `/var/lib/knowledge-governance/{library,freeze}` 并可用环境变量覆盖;
|
||||
`LIBRARYCTL_BIN` 未设置或不可执行时明确失败,不做"同目录二进制"回退。
|
||||
- `libraryctl status` 增印 `databases.source_db/governance_db` 模式(`container|external|file`),
|
||||
`libraryagent -describe` 输出同样两项;地址与凭据一律不出现。
|
||||
|
||||
### Tests
|
||||
|
||||
- DSN 来源三条路径(容器 / RDS / 治理库文件)各有用例;缺变量、缺文件、权限过宽、
|
||||
非 PostgreSQL 内容都要求失败关闭,且断言错误里不含口令;`sslmode=prefer` 这类会静默
|
||||
降级的取值直接拒绝。
|
||||
- v090 采集脚本进 CI:`bash -n` 语法检查 + 标志/服务声明断言,并拒绝 0.7.1 的
|
||||
容器名与项目名出现在新入口里。
|
||||
|
||||
## 开发快照 0.1.378(发布写入支持 WeKnora 0.9.0)
|
||||
|
||||
### Added
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/libraryagent"
|
||||
"knowledge-governance/internal/librarydsn"
|
||||
"knowledge-governance/internal/librarymaintenance"
|
||||
)
|
||||
|
||||
@@ -25,10 +26,10 @@ func main() {
|
||||
flag.StringVar(&config.ArchiveRoot, "archive", filepath.Join(home, ".local/share/knowledge-governance/library"), "sealed artifact root")
|
||||
flag.StringVar(&config.JournalRoot, "journal", filepath.Join(home, ".local/share/knowledge-governance/freeze"), "freeze journal root")
|
||||
flag.StringVar(&config.SourceApp, "source-app", "WKNFull-app", "source container describing the stack")
|
||||
flag.StringVar(&config.SourcePG, "source-pg", "WKNFull-postgres", "source PostgreSQL container")
|
||||
flag.StringVar(&config.SourcePG, "source-pg", "WKNFull-postgres", "source PostgreSQL container; empty = the source DB lives on a managed instance and is reached with the source app env (DB_HOST/...)")
|
||||
flag.StringVar(&config.SourceES, "source-es", "WKNFull-elasticsearch", "source Elasticsearch container")
|
||||
flag.StringVar(&config.SourceS3Endpoint, "source-s3-endpoint", "", "host-reachable source object-store endpoint (overrides the container-internal one)")
|
||||
flag.StringVar(&config.GovernancePG, "governance-pg", "knowledge-governance-p1-postgres-1", "governance PostgreSQL container")
|
||||
flag.StringVar(&config.GovernancePG, "governance-pg", "knowledge-governance-p1-postgres-1", "governance PostgreSQL container; empty = read the DSN from LIBRARY_GOVERNANCE_DSN_FILE")
|
||||
flag.StringVar(&config.Project, "project", "weknora071full", "Compose project owning the writers")
|
||||
flag.StringVar(&config.Writers, "writers", "app", "declared writer services")
|
||||
nonWriters := flag.String("non-writers", defaultNonWriters, "service=reason for every other project service")
|
||||
@@ -41,6 +42,13 @@ func main() {
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
if *describe {
|
||||
// describe 只回答"这两个库从哪来"。地址与凭据一律不出现:这段输出会被贴进
|
||||
// 工单与日志,而 RDS 的连接串里带口令。
|
||||
log.Printf("source_db=%s governance_db=%s",
|
||||
librarydsn.SourceMode(config.SourcePG), librarydsn.GovernanceMode(config.GovernancePG))
|
||||
return
|
||||
}
|
||||
// 实例级维护接线(入口闸门 + 后台队列)不在 agent 的 Config 里:它由同一份
|
||||
// 环境声明提供给子进程。启动时先说清楚有没有——缺了封存会在开窗口前被拒,
|
||||
// 不允许"以为接好了"跑到一半才发现没有排空证据。
|
||||
@@ -49,10 +57,6 @@ func main() {
|
||||
} else {
|
||||
log.Printf("library agent: 实例级维护目标 %s 入口 %s 闸门 %s", target.Instance, target.IngressURL, target.IngressService)
|
||||
}
|
||||
if *describe {
|
||||
log.Printf("library agent config: %+v", agent.Describe())
|
||||
return
|
||||
}
|
||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
log.Printf("library agent listening on %s (project %s, writers %s)", config.Socket, config.Project, config.Writers)
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// 文件模式没配置:必须失败关闭并点名缺的那个变量——"静默退回容器模式"会让
|
||||
// 操作员以为在读 RDS,实际在读本机容器(或者干脆读到不存在的容器)。
|
||||
func TestGovernanceFileModeFailsClosedWithoutDSNFile(t *testing.T) {
|
||||
t.Setenv("LIBRARY_GOVERNANCE_DSN_FILE", "")
|
||||
_, err := readRoutingBindings(context.Background(), options{governancePG: ""})
|
||||
if err == nil {
|
||||
t.Fatal("没有 DSN 文件却读成功了")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "LIBRARY_GOVERNANCE_DSN_FILE") {
|
||||
t.Fatalf("错误没有点名缺哪一项:%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// 外部 DSN 连不上/写错时,错误必须是固定说法:pgx 的错误里带的是连接串(主机、
|
||||
// 用户名,口令它自己打码),直接透传就等于把地址写进日志与工单。
|
||||
func TestGovernanceFileModeNeverLeaksPasswordInErrors(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "governance.dsn")
|
||||
// 故意用非法 sslmode + 不可达端口:无论失败在解析还是连接阶段,都不许泄露。
|
||||
if err := os.WriteFile(path, []byte("postgres://gov:LEAK-ME-PLEASE@127.0.0.1:1/gov?sslmode=bogus&connect_timeout=1\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("LIBRARY_GOVERNANCE_DSN_FILE", path)
|
||||
_, err := readRoutingBindings(context.Background(), options{governancePG: ""})
|
||||
if err == nil {
|
||||
t.Fatal("坏 DSN 竟然读成功了")
|
||||
}
|
||||
if strings.Contains(err.Error(), "LEAK-ME-PLEASE") {
|
||||
t.Fatalf("错误里漏了口令:%v", err)
|
||||
}
|
||||
}
|
||||
+86
-26
@@ -33,6 +33,7 @@ import (
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"knowledge-governance/internal/librarydeadletter"
|
||||
"knowledge-governance/internal/librarydsn"
|
||||
"knowledge-governance/internal/libraryfreeze"
|
||||
"knowledge-governance/internal/libraryhandoff"
|
||||
"knowledge-governance/internal/librarymaintenance"
|
||||
@@ -146,9 +147,13 @@ func usage() {
|
||||
|
||||
Flags (defaults target the local WKNFull test stack):
|
||||
-source-app WKNFull-app container whose env describes the source
|
||||
-source-pg WKNFull-postgres source PostgreSQL container (address resolution)
|
||||
-source-pg WKNFull-postgres source PostgreSQL container; empty = the source DB is a
|
||||
managed instance: connect with the source app env
|
||||
(DB_HOST/DB_PORT/DB_NAME/DB_USER/DB_PASSWORD), TLS mode from
|
||||
LIBRARY_SOURCE_DB_SSLMODE (default require)
|
||||
-source-es WKNFull-elasticsearch
|
||||
-governance-pg knowledge-governance-p1-postgres-1 routing bindings read here
|
||||
-governance-pg knowledge-governance-p1-postgres-1 routing bindings read here; empty = read
|
||||
the DSN from LIBRARY_GOVERNANCE_DSN_FILE (0600 file)
|
||||
-project weknora071full Compose project owning the writer group
|
||||
-writers WKNFull-app declared writer services (comma separated)
|
||||
-non-writers WKNFull-frontend=reason,... every other project service
|
||||
@@ -182,9 +187,9 @@ func parseFlags(command string, args []string) (options, []string, error) {
|
||||
set := flag.NewFlagSet(command, flag.ContinueOnError)
|
||||
set.SetOutput(os.Stderr)
|
||||
set.StringVar(&opts.sourceApp, "source-app", opts.sourceApp, "source container describing the stack")
|
||||
set.StringVar(&opts.sourcePG, "source-pg", opts.sourcePG, "source PostgreSQL container")
|
||||
set.StringVar(&opts.sourcePG, "source-pg", opts.sourcePG, "source PostgreSQL container; empty = connect to the source app's external (managed) database")
|
||||
set.StringVar(&opts.sourceES, "source-es", opts.sourceES, "source Elasticsearch container")
|
||||
set.StringVar(&opts.governancePG, "governance-pg", opts.governancePG, "governance PostgreSQL container")
|
||||
set.StringVar(&opts.governancePG, "governance-pg", opts.governancePG, "governance PostgreSQL container; empty = read the DSN from LIBRARY_GOVERNANCE_DSN_FILE")
|
||||
set.StringVar(&opts.project, "project", opts.project, "Compose project owning the writers")
|
||||
set.StringVar(&opts.writers, "writers", opts.writers, "declared writer services")
|
||||
set.StringVar(&opts.nonWriters, "non-writers", "", "service=reason for every other project service")
|
||||
@@ -285,15 +290,25 @@ func discoverSource(ctx context.Context, opts options, binding map[string]string
|
||||
if app["DB_DRIVER"] != "postgres" {
|
||||
return config, errors.New("source database driver is not postgres")
|
||||
}
|
||||
address, err := containerAddress(ctx, opts.sourcePG)
|
||||
if err != nil {
|
||||
return config, err
|
||||
// 两种来源:给了 -source-pg 就按容器地址拼(0.7.1 栈);没给说明库是托管
|
||||
// 实例(RDS 型 0.9.0 栈没有源库容器),用源应用容器 env 里的 DB_* 直连。
|
||||
if librarydsn.SourceMode(opts.sourcePG) == librarydsn.SourceExternal {
|
||||
dsn, dsnErr := librarydsn.SourceURL(app, os.Getenv)
|
||||
if dsnErr != nil {
|
||||
return config, dsnErr
|
||||
}
|
||||
config.DatabaseURL = dsn
|
||||
} else {
|
||||
address, addressErr := containerAddress(ctx, opts.sourcePG)
|
||||
if addressErr != nil {
|
||||
return config, addressErr
|
||||
}
|
||||
dsn, dsnErr := librarydsn.ContainerURL(address, app)
|
||||
if dsnErr != nil {
|
||||
return config, dsnErr
|
||||
}
|
||||
config.DatabaseURL = dsn
|
||||
}
|
||||
dsn := strings.Join([]string{
|
||||
"postgres://" + urlEscape(app["DB_USER"]) + ":" + urlEscape(app["DB_PASSWORD"]) +
|
||||
"@" + address + ":" + app["DB_PORT"] + "/" + urlEscape(app["DB_NAME"]) + "?sslmode=disable",
|
||||
}, "")
|
||||
config.DatabaseURL = dsn
|
||||
config.AppImageID, err = containerImageID(ctx, opts.sourceApp)
|
||||
if err != nil {
|
||||
return config, err
|
||||
@@ -388,17 +403,12 @@ func parseBool(value string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
func urlEscape(value string) string {
|
||||
replacer := strings.NewReplacer("%", "%25", ":", "%3A", "@", "%40", "/", "%2F", "?", "%3F", "#", "%23", "&", "%26", " ", "%20")
|
||||
return replacer.Replace(value)
|
||||
}
|
||||
|
||||
// scopeBindings 返回 scope→知识库 映射。治理 API 用的是 LIBRARY_PRODUCTION_KB_*,
|
||||
// CLI 读的是治理库里的 routing bindings;同一台机器上两份不一致时,
|
||||
// CLI 抓到的会是"配置里那个库"而不是 API 认为的目标——所以环境变量优先,
|
||||
// 读不到才退回 bindings。两边讲同一份身份,验收才可比。
|
||||
func scopeBindings(ctx context.Context, container string) (map[string]string, error) {
|
||||
binding, err := readRoutingBindings(ctx, container)
|
||||
func scopeBindings(ctx context.Context, opts options) (map[string]string, error) {
|
||||
binding, err := readRoutingBindings(ctx, opts)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -410,12 +420,57 @@ func scopeBindings(ctx context.Context, container string) (map[string]string, er
|
||||
return binding, nil
|
||||
}
|
||||
|
||||
func readRoutingBindings(ctx context.Context, container string) (map[string]string, error) {
|
||||
raw, err := dockerOutput(ctx, "exec", container, "psql", "-U", "governance", "-d", "governance", "-tAc",
|
||||
"SELECT internal_kb_id||'|'||external_kb_id FROM routing_kb_bindings WHERE id='active'")
|
||||
// readRoutingBindings 读治理库里生效的 scope 绑定。两种来源:
|
||||
// - 给了 -governance-pg:docker exec 进治理库容器跑 psql(0.7.1 本机栈);
|
||||
// - 没给:治理库是托管实例(RDS 型 0.9.0 栈没有 governance 容器),DSN 来自
|
||||
// LIBRARY_GOVERNANCE_DSN_FILE 指向的 0600 文件。
|
||||
//
|
||||
// 两条路径只共用行解析,不共用连接方式:容器里没有网络到 RDS 的凭据,宿主机也不该
|
||||
// 为了读两列去 docker exec 一个不存在的容器。
|
||||
func readRoutingBindings(ctx context.Context, opts options) (map[string]string, error) {
|
||||
if librarydsn.GovernanceMode(opts.governancePG) == librarydsn.GovernanceFile {
|
||||
dsn, err := librarydsn.GovernanceURL(os.Getenv)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return readRoutingBindingsFromDSN(ctx, dsn)
|
||||
}
|
||||
raw, err := dockerOutput(ctx, "exec", opts.governancePG, "psql", "-U", "governance", "-d", "governance", "-tAc",
|
||||
routingBindingsQuery)
|
||||
if err != nil {
|
||||
return nil, errors.New("read routing bindings failed")
|
||||
}
|
||||
return parseRoutingBindings(raw)
|
||||
}
|
||||
|
||||
const routingBindingsQuery = "SELECT internal_kb_id||'|'||external_kb_id FROM routing_kb_bindings WHERE id='active'"
|
||||
|
||||
// readRoutingBindingsFromDSN 用外部 DSN 读同一份绑定。
|
||||
//
|
||||
// 只读会话是刻意的:这条路径只是读两列,出了问题必须是"读不到",而不是在治理库上
|
||||
// 留下一次写入。错误一律收敛成固定说法:pgx 的错误里会带连接串(口令它自己打了码,
|
||||
// 但主机与用户名照旧出去),而这里连那份地址都不该出现在日志或工单里。
|
||||
func readRoutingBindingsFromDSN(ctx context.Context, dsn string) (map[string]string, error) {
|
||||
parsed, err := pgxpool.ParseConfig(dsn)
|
||||
if err != nil {
|
||||
return nil, errors.New("governance database url is invalid")
|
||||
}
|
||||
parsed.ConnConfig.RuntimeParams["default_transaction_read_only"] = "on"
|
||||
parsed.ConnConfig.RuntimeParams["statement_timeout"] = "30000"
|
||||
parsed.MaxConns = 2
|
||||
pool, err := pgxpool.NewWithConfig(ctx, parsed)
|
||||
if err != nil {
|
||||
return nil, errors.New("open governance database failed")
|
||||
}
|
||||
defer pool.Close()
|
||||
var raw string
|
||||
if err = pool.QueryRow(ctx, routingBindingsQuery).Scan(&raw); err != nil {
|
||||
return nil, errors.New("read routing bindings failed")
|
||||
}
|
||||
return parseRoutingBindings(raw)
|
||||
}
|
||||
|
||||
func parseRoutingBindings(raw string) (map[string]string, error) {
|
||||
internal, external, ok := strings.Cut(strings.TrimSpace(raw), "|")
|
||||
if !ok || internal == "" || external == "" {
|
||||
return nil, errors.New("routing bindings are incomplete")
|
||||
@@ -587,7 +642,12 @@ func openArchive(opts options) (*librarysnapshot.Archive, error) {
|
||||
|
||||
func runStatus(ctx context.Context, opts options) error {
|
||||
report := map[string]any{}
|
||||
binding, bindingErr := scopeBindings(ctx, opts.governancePG)
|
||||
// 只报来源模式:地址与凭据连 describe/status 都不出现,日志自然也不会带出去。
|
||||
report["databases"] = map[string]string{
|
||||
"source_db": librarydsn.SourceMode(opts.sourcePG),
|
||||
"governance_db": librarydsn.GovernanceMode(opts.governancePG),
|
||||
}
|
||||
binding, bindingErr := scopeBindings(ctx, opts)
|
||||
var config sourceConfig
|
||||
var err error
|
||||
if bindingErr != nil {
|
||||
@@ -680,7 +740,7 @@ func runPreflight(ctx context.Context, opts options) error {
|
||||
if opts.scope != "internal" && opts.scope != "external" {
|
||||
return errors.New("-scope must be internal or external")
|
||||
}
|
||||
binding, err := scopeBindings(ctx, opts.governancePG)
|
||||
binding, err := scopeBindings(ctx, opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -773,7 +833,7 @@ func runCapture(ctx context.Context, opts options) error {
|
||||
return err
|
||||
}
|
||||
defer archive.Close()
|
||||
binding, err := scopeBindings(ctx, opts.governancePG)
|
||||
binding, err := scopeBindings(ctx, opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -1150,7 +1210,7 @@ func runDeadLetters(ctx context.Context, opts options) error {
|
||||
if opts.scope != "internal" && opts.scope != "external" {
|
||||
return errors.New("-scope must be internal or external")
|
||||
}
|
||||
binding, err := scopeBindings(ctx, opts.governancePG)
|
||||
binding, err := scopeBindings(ctx, opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// v090 采集脚本是本轮新增的唯一入口。为什么用 bash -n 而不是"跑一遍":跑一遍要
|
||||
// 停写入者、连 RDS——那是真实采集,不是单元测试。语法与声明在这里钉住,能让
|
||||
// "标志写错一个字母"在 CI 里就红,而不是等到开维护窗口时才发现。
|
||||
func TestV090CaptureScriptIsSyntacticallyValid(t *testing.T) {
|
||||
bash, err := exec.LookPath("bash")
|
||||
if err != nil {
|
||||
t.Fatalf("需要 bash 才能检查脚本语法(CI 与测试机都有):%v", err)
|
||||
}
|
||||
path := filepath.Join("..", "..", "scripts", "library-capture-v090.sh")
|
||||
out, err := exec.Command(bash, "-n", path).CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("bash -n %s 失败:%v\n%s", path, err, out)
|
||||
}
|
||||
}
|
||||
|
||||
// 脚本内容断言:外部库模式(两个 -pg 传空)、写者声明、容器名、归档根、
|
||||
// LIBRARYCTL_BIN 缺失即失败——这些一旦漂移,真跑时会封错栈或读错库。
|
||||
func TestV090CaptureScriptDeclaresExternalDatabases(t *testing.T) {
|
||||
body, err := os.ReadFile(filepath.Join("..", "..", "scripts", "library-capture-v090.sh"))
|
||||
if err != nil {
|
||||
t.Fatalf("读脚本:%v", err)
|
||||
}
|
||||
for _, want := range []string{
|
||||
"set -euo pipefail",
|
||||
"LIBRARYCTL_MISSING",
|
||||
"-project weknora-v090",
|
||||
"-writers app",
|
||||
"-source-app weknora-v090-app",
|
||||
"-source-es weknora-v090-elasticsearch",
|
||||
`-source-pg ""`,
|
||||
`-governance-pg ""`,
|
||||
"frontend=UI only; no knowledge writes",
|
||||
"docreader=parse RPC service; app persists parse output",
|
||||
"elasticsearch=search store; not an initiator",
|
||||
"redis=queue/cache store; not an initiator",
|
||||
"/var/lib/knowledge-governance/library",
|
||||
"/var/lib/knowledge-governance/freeze",
|
||||
} {
|
||||
if !strings.Contains(string(body), want) {
|
||||
t.Fatalf("脚本缺少 %q", want)
|
||||
}
|
||||
}
|
||||
// 0.7.1 的服务名绝不能出现在 v090 入口里:那会让封存停错栈。
|
||||
for _, forbidden := range []string{"WKNFull", "weknora071full"} {
|
||||
if strings.Contains(string(body), forbidden) {
|
||||
t.Fatalf("v090 脚本里出现了 0.7.1 的 %q", forbidden)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -26,6 +26,8 @@ import (
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/librarydsn"
|
||||
)
|
||||
|
||||
// Config is fixed at start-up by the operator (systemd unit / CLI flags). It is
|
||||
@@ -429,15 +431,22 @@ func (a *Agent) health() response {
|
||||
}
|
||||
|
||||
// identity 被日志与健康应答共用一份,避免两处描述漂移。
|
||||
//
|
||||
// 数据库只报来源模式(container|external|file),不报地址:agent 的应答会被治理侧
|
||||
// 存档,DSN 里带口令,一旦写进身份就再也收不回来了。
|
||||
func (c Config) identity() map[string]any {
|
||||
identity := map[string]any{
|
||||
"project": c.Project,
|
||||
"archive_root": c.ArchiveRoot,
|
||||
"journal_root": c.JournalRoot,
|
||||
"source_app": c.SourceApp,
|
||||
"source_pg": c.SourcePG,
|
||||
"source_es": c.SourceES,
|
||||
"writers": c.Writers,
|
||||
// source_pg 仍是容器名(不是地址、不是凭据),保留它以让既有健康页/事后核对
|
||||
// 读得到同一份字段;数据库**来源**由下面两个模式字段表达,外部模式下它为空。
|
||||
"source_pg": c.SourcePG,
|
||||
"source_db": librarydsn.SourceMode(c.SourcePG),
|
||||
"governance_db": librarydsn.GovernanceMode(c.GovernancePG),
|
||||
"writers": c.Writers,
|
||||
}
|
||||
if c.SourceS3Endpoint != "" {
|
||||
identity["source_s3_endpoint"] = c.SourceS3Endpoint
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
// Package librarydsn 决定整库采集链路里两个 DSN 究竟从哪来。
|
||||
//
|
||||
// 为什么要有它:容器模式下 DSN 是"源库容器的地址 + 源应用容器的 env"拼出来的;
|
||||
// RDS 型 0.9.0 栈没有源库容器(库是托管实例),治理库同样只在外部。旧拼法在那种
|
||||
// 栈上不是权限不够,而是地址解析方式错了——库根本没有容器地址可查。把"用哪种
|
||||
// 来源"收敛成一处,libraryctl 与 libraryagent 共用同一份判定,日志/describe 只报
|
||||
// 模式(container|external|file),永远不报地址与凭据。
|
||||
package librarydsn
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/url"
|
||||
"strings"
|
||||
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
)
|
||||
|
||||
const (
|
||||
// SourceContainer:-source-pg 给了容器名,地址从容器网络解析(0.7.1 栈)。
|
||||
SourceContainer = "container"
|
||||
// SourceExternal:-source-pg 为空,直连源应用容器 env 里声明的托管库(RDS)。
|
||||
SourceExternal = "external"
|
||||
|
||||
// GovernanceContainer:-governance-pg 给了容器名,用 docker exec psql 读。
|
||||
GovernanceContainer = "container"
|
||||
// GovernanceFile:-governance-pg 为空,DSN 来自 0600 文件(LIBRARY_GOVERNANCE_DSN_FILE)。
|
||||
GovernanceFile = "file"
|
||||
)
|
||||
|
||||
// allowedSSLModes 是允许的 sslmode 集合。
|
||||
//
|
||||
// 只收这四个:prefer/allow 会在服务端不支持时**静默降级成明文**,托管实例上那等于
|
||||
// "以为加密了其实没有"——这种"看起来配好了"的失败模式比直接报错更难发现。
|
||||
var allowedSSLModes = map[string]bool{
|
||||
"disable": true, // 仅显式测试:本机回环或明确无 TLS 的隔离库
|
||||
"require": true,
|
||||
"verify-ca": true,
|
||||
"verify-full": true,
|
||||
}
|
||||
|
||||
// SourceMode 把 -source-pg 的取值翻译成来源模式。空值 = 外部(RDS)。
|
||||
func SourceMode(sourcePG string) string {
|
||||
if strings.TrimSpace(sourcePG) == "" {
|
||||
return SourceExternal
|
||||
}
|
||||
return SourceContainer
|
||||
}
|
||||
|
||||
// GovernanceMode 把 -governance-pg 的取值翻译成来源模式。空值 = DSN 文件。
|
||||
func GovernanceMode(governancePG string) string {
|
||||
if strings.TrimSpace(governancePG) == "" {
|
||||
return GovernanceFile
|
||||
}
|
||||
return GovernanceContainer
|
||||
}
|
||||
|
||||
// ContainerURL 复现容器模式原有的 DSN 形状:地址来自容器网络,sslmode=disable
|
||||
// (容器私网内没有 TLS 终结)。抽到这里只为一件事:让"模式选择"有一条可测的路径,
|
||||
// 而形状与旧实现逐字节一致(真跑过的栈不能因为重构换了连接串)。
|
||||
func ContainerURL(address string, app map[string]string) (string, error) {
|
||||
if strings.TrimSpace(address) == "" {
|
||||
return "", errors.New("source container address is empty")
|
||||
}
|
||||
for _, key := range []string{"DB_USER", "DB_PASSWORD", "DB_NAME"} {
|
||||
if app[key] == "" {
|
||||
return "", fmt.Errorf("source app environment is missing %s", key)
|
||||
}
|
||||
}
|
||||
target := url.URL{
|
||||
Scheme: "postgres",
|
||||
User: url.UserPassword(app["DB_USER"], app["DB_PASSWORD"]),
|
||||
Host: net.JoinHostPort(address, defaultPort(app["DB_PORT"])),
|
||||
Path: "/" + app["DB_NAME"],
|
||||
RawQuery: url.Values{"sslmode": {"disable"}}.Encode(),
|
||||
}
|
||||
return target.String(), nil
|
||||
}
|
||||
|
||||
// SourceURL 用源应用容器 env 里的 DB_* 直连托管库(RDS)。
|
||||
//
|
||||
// 缺哪个变量就点哪个变量名:把"连接被拒"与"配置缺一项"分开,才不用去猜着改。
|
||||
// 错误里只有变量名——DB_PASSWORD 的值绝不能出现在任何一处输出里。
|
||||
func SourceURL(app map[string]string, getenv func(string) string) (string, error) {
|
||||
missing := []string{}
|
||||
for _, key := range []string{"DB_HOST", "DB_USER", "DB_NAME"} {
|
||||
if strings.TrimSpace(app[key]) == "" {
|
||||
missing = append(missing, key)
|
||||
}
|
||||
}
|
||||
// 口令不 TrimSpace:前后空格也是口令的一部分,改掉就成另一个口令了。
|
||||
if app["DB_PASSWORD"] == "" {
|
||||
missing = append(missing, "DB_PASSWORD")
|
||||
}
|
||||
if len(missing) > 0 {
|
||||
return "", fmt.Errorf("source app environment is missing %s", strings.Join(missing, ", "))
|
||||
}
|
||||
sslmode := strings.TrimSpace(getenv("LIBRARY_SOURCE_DB_SSLMODE"))
|
||||
if sslmode == "" {
|
||||
sslmode = "require"
|
||||
}
|
||||
if !allowedSSLModes[sslmode] {
|
||||
return "", errors.New("LIBRARY_SOURCE_DB_SSLMODE must be one of disable, require, verify-ca, verify-full")
|
||||
}
|
||||
target := url.URL{
|
||||
Scheme: "postgres",
|
||||
User: url.UserPassword(app["DB_USER"], app["DB_PASSWORD"]),
|
||||
Host: net.JoinHostPort(strings.TrimSpace(app["DB_HOST"]), defaultPort(app["DB_PORT"])),
|
||||
Path: "/" + strings.TrimSpace(app["DB_NAME"]),
|
||||
RawQuery: url.Values{"sslmode": {sslmode}}.Encode(),
|
||||
}
|
||||
return target.String(), nil
|
||||
}
|
||||
|
||||
// GovernanceURL 从 0600 文件读治理库 DSN。
|
||||
//
|
||||
// 文件读法复用 dobotaiops.ReadDSNFile:那里已经把"必须是普通文件、必须 0600、
|
||||
// 错误只报路径与原因"钉住并测过——再写一份同义实现只会让两条路径的规则慢慢分叉。
|
||||
func GovernanceURL(getenv func(string) string) (string, error) {
|
||||
path := strings.TrimSpace(getenv("LIBRARY_GOVERNANCE_DSN_FILE"))
|
||||
if path == "" {
|
||||
return "", errors.New("LIBRARY_GOVERNANCE_DSN_FILE is not set: the governance DSN file is required when -governance-pg is empty")
|
||||
}
|
||||
dsn, err := dobotaiops.ReadDSNFile(path)
|
||||
if err != nil {
|
||||
// ReadDSNFile 的错误只含路径与原因,不含 DSN/口令。
|
||||
return "", fmt.Errorf("governance DSN file unusable: %w", err)
|
||||
}
|
||||
return dsn, nil
|
||||
}
|
||||
|
||||
// defaultPort:托管实例的连接信息里端口常常单独给;缺了就按 PostgreSQL 默认。
|
||||
// 端口不是凭据,给默认值不会把"配错了"藏起来。
|
||||
func defaultPort(port string) string {
|
||||
if trimmed := strings.TrimSpace(port); trimmed != "" {
|
||||
return trimmed
|
||||
}
|
||||
return "5432"
|
||||
}
|
||||
@@ -0,0 +1,174 @@
|
||||
package librarydsn
|
||||
|
||||
import (
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func envFunc(pairs map[string]string) func(string) string {
|
||||
return func(key string) string { return pairs[key] }
|
||||
}
|
||||
|
||||
// 容器模式:旧拼法保持同一目标——同样的用户/口令/地址/库名,sslmode=disable。
|
||||
// 真跑过的 0.7.1 栈不能因为这次接入换了连接目标(换 sslmode 或换转义方式都会让
|
||||
// "同一个库"连不上或连成别的身份)。断言用解析后的分量,不用字面转义:
|
||||
// userinfo 里 ':' 保不保留不影响解析结果,钉字面只会把重构误判成回归。
|
||||
func TestContainerURLKeepsLegacyShape(t *testing.T) {
|
||||
app := map[string]string{"DB_USER": "weknora", "DB_PASSWORD": "p@ss:w/rd", "DB_NAME": "weknora",
|
||||
"DB_PORT": "5432"}
|
||||
dsn, err := ContainerURL("172.18.0.5", app)
|
||||
if err != nil {
|
||||
t.Fatalf("container url: %v", err)
|
||||
}
|
||||
parsed, err := url.Parse(dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("生成的连接串自己都解析不了:%v", err)
|
||||
}
|
||||
if parsed.Scheme != "postgres" || parsed.Host != "172.18.0.5:5432" || parsed.Path != "/weknora" {
|
||||
t.Fatalf("容器模式连接目标变了:%s", dsn)
|
||||
}
|
||||
if parsed.User.Username() != "weknora" {
|
||||
t.Fatalf("用户名变了:%s", dsn)
|
||||
}
|
||||
if password, _ := parsed.User.Password(); password != "p@ss:w/rd" {
|
||||
t.Fatalf("口令转义后取不回来:%s", dsn)
|
||||
}
|
||||
if parsed.Query().Get("sslmode") != "disable" {
|
||||
t.Fatalf("容器模式不该改 sslmode:%s", dsn)
|
||||
}
|
||||
if _, err = ContainerURL("", app); err == nil {
|
||||
t.Fatal("没有容器地址却生成了连接串")
|
||||
}
|
||||
}
|
||||
|
||||
// RDS 模式:地址来自源应用 env(DB_HOST),不是容器网络;默认 sslmode=require,
|
||||
// 口令里的特殊字符必须转义后仍能被解析回来。
|
||||
func TestSourceURLUsesRDSHostAndRequiresTLSByDefault(t *testing.T) {
|
||||
app := map[string]string{
|
||||
"DB_HOST": "pgm-2ze0abcd.pg.rds.aliyuncs.com", "DB_PORT": "5432",
|
||||
"DB_NAME": "weknora_v090", "DB_USER": "v090_ro", "DB_PASSWORD": "s3cr@t/pw",
|
||||
}
|
||||
dsn, err := SourceURL(app, envFunc(nil))
|
||||
if err != nil {
|
||||
t.Fatalf("rds url: %v", err)
|
||||
}
|
||||
parsed, err := url.Parse(dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("生成的连接串自己都解析不了:%v", err)
|
||||
}
|
||||
if parsed.Host != "pgm-2ze0abcd.pg.rds.aliyuncs.com:5432" {
|
||||
t.Fatalf("host 不是 RDS 地址:%q", parsed.Host)
|
||||
}
|
||||
if got := parsed.Query().Get("sslmode"); got != "require" {
|
||||
t.Fatalf("默认 sslmode 不是 require:%q", got)
|
||||
}
|
||||
if password, _ := parsed.User.Password(); password != "s3cr@t/pw" {
|
||||
t.Fatal("口令转义后取不回来")
|
||||
}
|
||||
// 显式 disable 只用于本机隔离测试;它是允许的,但必须由调用方写清楚。
|
||||
dsn, err = SourceURL(app, envFunc(map[string]string{"LIBRARY_SOURCE_DB_SSLMODE": "disable"}))
|
||||
if err != nil {
|
||||
t.Fatalf("explicit disable: %v", err)
|
||||
}
|
||||
if !strings.HasSuffix(dsn, "sslmode=disable") {
|
||||
t.Fatalf("显式 disable 没被采纳:%s", dsn)
|
||||
}
|
||||
// prefer 会静默降级成明文:那不是"配好了 TLS",直接拒。
|
||||
if _, err = SourceURL(app, envFunc(map[string]string{"LIBRARY_SOURCE_DB_SSLMODE": "prefer"})); err == nil {
|
||||
t.Fatal("sslmode=prefer 被接受了")
|
||||
}
|
||||
}
|
||||
|
||||
// 缺变量必须失败,且点名缺的那一项;错误里不许出现口令。
|
||||
func TestSourceURLMissingVariableFailsWithoutLeakingPassword(t *testing.T) {
|
||||
app := map[string]string{"DB_HOST": "pgm-x.pg.rds.aliyuncs.com", "DB_NAME": "kb",
|
||||
"DB_USER": "u", "DB_PASSWORD": "TOP-SECRET-VALUE"}
|
||||
delete(app, "DB_HOST")
|
||||
_, err := SourceURL(app, envFunc(nil))
|
||||
if err == nil {
|
||||
t.Fatal("缺 DB_HOST 竟然成功了")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "DB_HOST") {
|
||||
t.Fatalf("错误没有点名缺哪一项:%v", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), "TOP-SECRET-VALUE") {
|
||||
t.Fatalf("错误里漏了口令:%v", err)
|
||||
}
|
||||
app["DB_HOST"] = "pgm-x.pg.rds.aliyuncs.com"
|
||||
delete(app, "DB_PASSWORD")
|
||||
_, err = SourceURL(app, envFunc(nil))
|
||||
if err == nil || !strings.Contains(err.Error(), "DB_PASSWORD") {
|
||||
t.Fatalf("缺 DB_PASSWORD 没有点名:%v", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), "TOP-SECRET-VALUE") {
|
||||
t.Fatalf("错误里漏了口令:%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// 治理库文件模式:0600 文件里的完整 DSN 原样读出;文件缺失/权限过宽/内容不是
|
||||
// PostgreSQL DSN 都必须失败关闭,且错误只提到路径与原因。
|
||||
func TestGovernanceURLReadsDSNFileAndFailsClosed(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "governance.dsn")
|
||||
const password = "GOV-PASSWORD-VALUE"
|
||||
dsn := "postgres://governance:" + password + "@pgm-gov.pg.rds.aliyuncs.com:5432/knowledge_governance_test?sslmode=require"
|
||||
if err := os.WriteFile(path, []byte(dsn+"\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := GovernanceURL(envFunc(map[string]string{"LIBRARY_GOVERNANCE_DSN_FILE": path}))
|
||||
if err != nil {
|
||||
t.Fatalf("file mode: %v", err)
|
||||
}
|
||||
if got != dsn {
|
||||
t.Fatalf("读出的 DSN 与文件内容不一致(不想在这里做任何改写)")
|
||||
}
|
||||
|
||||
// 没声明文件:失败并点名变量。
|
||||
if _, err = GovernanceURL(envFunc(nil)); err == nil || !strings.Contains(err.Error(), "LIBRARY_GOVERNANCE_DSN_FILE") {
|
||||
t.Fatalf("没配置文件时没有点名变量:%v", err)
|
||||
}
|
||||
|
||||
// 文件不存在:点名路径,不泄漏任何内容。
|
||||
missing := filepath.Join(dir, "absent.dsn")
|
||||
if _, err = GovernanceURL(envFunc(map[string]string{"LIBRARY_GOVERNANCE_DSN_FILE": missing})); err == nil ||
|
||||
!strings.Contains(err.Error(), missing) {
|
||||
t.Fatalf("文件缺失没有点名路径:%v", err)
|
||||
}
|
||||
|
||||
// 组/其他可读:拒绝(这条规则由 dobotaiops.ReadDSNFile 判定,这里只钉住接线)。
|
||||
wide := filepath.Join(dir, "wide.dsn")
|
||||
if err = os.WriteFile(wide, []byte(dsn+"\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = os.Chmod(wide, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = GovernanceURL(envFunc(map[string]string{"LIBRARY_GOVERNANCE_DSN_FILE": wide})); err == nil {
|
||||
t.Fatal("0644 的 DSN 文件被接受了")
|
||||
}
|
||||
|
||||
// 内容不是 PostgreSQL DSN:失败,且不要把内容回显进错误。
|
||||
bogus := filepath.Join(dir, "bogus.dsn")
|
||||
if err = os.WriteFile(bogus, []byte("mysql://u:"+password+"@host/db\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, err = GovernanceURL(envFunc(map[string]string{"LIBRARY_GOVERNANCE_DSN_FILE": bogus}))
|
||||
if err == nil {
|
||||
t.Fatal("非 PostgreSQL DSN 被接受了")
|
||||
}
|
||||
if strings.Contains(err.Error(), password) {
|
||||
t.Fatalf("错误里漏了口令:%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestModesDeriveFromEmptyFlag(t *testing.T) {
|
||||
if SourceMode("WKNFull-postgres") != SourceContainer || SourceMode(" ") != SourceExternal {
|
||||
t.Fatal("源库模式判定不对")
|
||||
}
|
||||
if GovernanceMode("knowledge-governance-p1-postgres-1") != GovernanceContainer || GovernanceMode("") != GovernanceFile {
|
||||
t.Fatal("治理库模式判定不对")
|
||||
}
|
||||
}
|
||||
Executable
+56
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env bash
|
||||
# Whole-library capture entry point for the RDS-backed WeKnora 0.9.0 test stack
|
||||
# (Compose project weknora-v090).
|
||||
#
|
||||
# Why a separate file instead of a parameterised one: this stack has no source
|
||||
# PostgreSQL container and no governance PostgreSQL container. Both databases are
|
||||
# managed instances (RDS). -source-pg / -governance-pg are therefore passed empty,
|
||||
# which is libraryctl's documented signal to resolve them from the source app
|
||||
# container env (DB_HOST/DB_PORT/DB_NAME/DB_USER/DB_PASSWORD + LIBRARY_SOURCE_DB_SSLMODE)
|
||||
# and from LIBRARY_GOVERNANCE_DSN_FILE (0600 file). Keeping the 0.7.1 entry point
|
||||
# untouched means a change here can never alter a stack that already ran.
|
||||
#
|
||||
# Every service of the Compose project is declared below: exactly one writer (app)
|
||||
# and a reason for each sibling which must NOT be stopped. ProjectController
|
||||
# refuses to run if the project gains a service that is not declared here, so a
|
||||
# new container cannot be silently left outside the maintenance window.
|
||||
#
|
||||
# Credentials are read at call time (container env / 0600 file) and are never
|
||||
# printed. Nothing here writes to the source stack except SIGTERM/SIGSTART of the
|
||||
# declared writer.
|
||||
set -euo pipefail
|
||||
|
||||
# The binary is pinned by the caller on purpose: there is no "whatever is in this
|
||||
# directory" fallback, so a stale build next to the script cannot be executed by
|
||||
# accident against a live stack.
|
||||
BIN="${LIBRARYCTL_BIN:-}"
|
||||
if [ -z "$BIN" ]; then
|
||||
echo "LIBRARYCTL_MISSING: set LIBRARYCTL_BIN to the libraryctl binary built for this run" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ ! -x "$BIN" ]; then
|
||||
echo "LIBRARYCTL_NOT_EXECUTABLE: $BIN is not an executable file" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Archives/journals live outside the home directory on this host (they survive the
|
||||
# operator account); both roots stay overridable for an isolated rehearsal.
|
||||
ARCHIVE_ROOT="${LIBRARY_ARCHIVE_ROOT:-/var/lib/knowledge-governance/library}"
|
||||
JOURNAL_ROOT="${LIBRARY_JOURNAL_ROOT:-/var/lib/knowledge-governance/freeze}"
|
||||
|
||||
# Reasons must not contain commas: the flag is comma-separated.
|
||||
NON_WRITERS="frontend=UI only; no knowledge writes"
|
||||
NON_WRITERS+=",docreader=parse RPC service; app persists parse output"
|
||||
NON_WRITERS+=",elasticsearch=search store; not an initiator"
|
||||
NON_WRITERS+=",redis=queue/cache store; not an initiator"
|
||||
|
||||
exec "$BIN" "$@" \
|
||||
-project weknora-v090 \
|
||||
-writers app \
|
||||
-source-app weknora-v090-app \
|
||||
-source-es weknora-v090-elasticsearch \
|
||||
-source-pg "" \
|
||||
-governance-pg "" \
|
||||
-non-writers "$NON_WRITERS" \
|
||||
-archive "$ARCHIVE_ROOT" \
|
||||
-journal "$JOURNAL_ROOT"
|
||||
Reference in New Issue
Block a user