Files
Enterprise-Knowledge-Base/internal/datasource/connector/feishu/connector.go
T
qilifan 9afb8b7b68 feat(im,datasource): support Lark, Feishu's international edition
Lark and Feishu are the same product deployed on two isolated clouds
(open.larksuite.com and open.feishu.cn). WeKnora touches both from two
places — the IM bot and the Wiki data-source connector — and this brings
Lark to parity across both. Neither API surface differs, so rather than
duplicate either implementation, each is parametrized by a Region and
registered twice.

IM (internal/im/feishu):
  Every API call routes through a.api(), which prefixes the region's
  host, and the WebSocket client passes larkws.WithDomain — the SDK
  defaults to open.feishu.cn, so Lark long connections would otherwise
  fail to authenticate.

Data source (internal/datasource/connector/feishu):
  Lark was half-supported: the client honoured a base_url override and a
  LarkBaseURL constant existed, but the connector registered only as
  "feishu" and the UI never exposed base_url, so Lark wikis could not be
  configured at all. Lark is now its own connector type. base_url is kept
  as an explicit override so data sources that pointed a "feishu"
  connector at open.larksuite.com keep working.

Bugs found and fixed along the way:

- Wiki resource links were hardcoded to https://feishu.cn/wiki/{token} in
  four places, so a Lark wiki linked into the wrong cloud entirely.
- Bot identity is now platform-prefixed (feishu:cli_x vs lark:cli_x).
  BotIdentity carries a unique index, so without this a Lark channel
  would be rejected as a duplicate of an unrelated Feishu bot sharing an
  app_id.
- The card image cache is now keyed by app_id. image_keys are issued per
  app on one cloud; a Feishu key handed to a Lark app fails the card
  update with code=200570.
- The datasource doc claimed Lark "adapts automatically" — it could not,
  since the UI had no way to reach base_url.

Permissions are where the clouds genuinely diverge. Identical APIs do not
mean identical permission catalogues: aily:file:read, aily:file:write and
corehr:file:download exist on Feishu but not on Lark, so pasting Feishu's
scope JSON into Lark fails to import. That JSON is also over-broad (it
covers products WeKnora never calls) and simultaneously incomplete (it
omits the drive:*/docx:* scopes the connector needs). Lark therefore gets
its own list, derived from the 17 endpoints the code actually calls and
checked identifier-by-identifier against Lark's own API reference, plus a
full Feishu→Lark mapping table marking each scope verified, unverified,
or absent. Both features use tenant_access_token only — no user_access_
token exists anywhere in the repo — so the user dimension stays empty.

Known limitation: Region.ThinkingText makes Lark's initial card
placeholder English, but once streaming starts the think-block headers
come from the shared MarkdownThinkStyle and are Chinese on every platform
(Lark, Slack, Telegram, Mattermost alike). Fixing that means threading a
locale through FormatIMDisplayContent, which has no platform context — a
cross-platform change, not a Lark one. Documented as such.

The channel-creation 400 message is now derived from validIMPlatforms so
the list cannot drift as platforms are added.

Tests: region invariants and per-region URL construction on both sides,
the Feishu/Lark bot-identity split, platform-to-channel mapping, base_url
override precedence, an end-to-end SendReply against a fake Open Platform
(token fetch, reply, thread-unsupported fallback, hard-error surfacing),
and a ListResources drive asserting wiki links never leak the other
cloud's host.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 11:05:54 +08:00

573 lines
18 KiB
Go

package feishu
import (
"context"
"encoding/json"
"errors"
"fmt"
"strconv"
"strings"
"time"
"unicode/utf8"
"github.com/Tencent/WeKnora/internal/datasource"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
)
// Connector implements the datasource.Connector interface for Feishu and, with
// the same code, for Lark: the two clouds expose an identical wiki/docx/drive
// API surface. A Region picks the cloud — see region.go.
type Connector struct {
region Region
}
// NewConnector creates a connector for the given region (RegionFeishu or RegionLark).
func NewConnector(region Region) *Connector {
return &Connector{region: region}
}
// Type returns the connector type identifier.
func (c *Connector) Type() string {
return c.region.ConnectorType
}
const feishuWikiNodeResourceSeparator = ":"
// Validate verifies that the Feishu configuration is valid by testing connectivity.
func (c *Connector) Validate(ctx context.Context, config *types.DataSourceConfig) error {
feishuConfig, err := parseFeishuConfig(config, c.region)
if err != nil {
return err
}
client := NewClient(feishuConfig)
if err := client.Ping(ctx); err != nil {
return fmt.Errorf("feishu connection failed: %w", err)
}
return nil
}
// ListResources lists Feishu Wiki resources for selection, loading the tree
// lazily one level at a time to avoid traversing the entire wiki up front.
//
// - parentID == "" → list all accessible wiki spaces.
// - parentID == spaceID → list the top-level nodes of that space.
// - parentID == "spaceID:nodeToken" → list the direct children of that node.
//
// Eagerly recursing the whole tree here used to time out for large wikis
// (Tencent/WeKnora#1672); the recursive walk now happens only at sync time.
func (c *Connector) ListResources(
ctx context.Context, config *types.DataSourceConfig, parentID string,
) ([]types.Resource, error) {
feishuConfig, err := parseFeishuConfig(config, c.region)
if err != nil {
return nil, err
}
client := NewClient(feishuConfig)
if parentID == "" {
spaces, err := client.ListWikiSpaces(ctx)
if err != nil {
return nil, fmt.Errorf("list feishu wiki spaces: %w", err)
}
resources := make([]types.Resource, 0, len(spaces))
for _, space := range spaces {
resources = append(resources, types.Resource{
ExternalID: space.SpaceID,
Name: space.Name,
Type: "wiki_space",
Description: space.Description,
URL: c.region.wikiURL(space.SpaceID),
HasChildren: true,
Metadata: map[string]interface{}{
"visibility": space.Visibility,
"space_id": space.SpaceID,
},
})
}
return resources, nil
}
// Lazy load: list only the direct children of the given space / node.
spaceID, nodeToken := parseWikiResourceID(parentID)
nodes, err := client.ListWikiNodes(ctx, spaceID, nodeToken)
if err != nil {
return nil, fmt.Errorf("list feishu wiki nodes under %s: %w", parentID, err)
}
resources := make([]types.Resource, 0, len(nodes))
for _, node := range nodes {
resources = append(resources, c.wikiNodeToResource(spaceID, node))
}
return resources, nil
}
// ResolveResourceAncestors returns the resource IDs of every parent that has to
// be expanded so the lazily-loaded picker can reveal each given selection. For a
// selected node "spaceID:nodeToken" that is its space plus every intermediate
// node up the tree; the walk uses GetWikiNode (parent_node_token) and is O(depth)
// per selection, so it never re-traverses the whole wiki.
func (c *Connector) ResolveResourceAncestors(
ctx context.Context, config *types.DataSourceConfig, resourceIDs []string,
) ([]string, error) {
feishuConfig, err := parseFeishuConfig(config, c.region)
if err != nil {
return nil, err
}
client := NewClient(feishuConfig)
seen := make(map[string]bool)
ancestors := make([]string, 0)
add := func(id string) {
if id != "" && !seen[id] {
seen[id] = true
ancestors = append(ancestors, id)
}
}
for _, rid := range resourceIDs {
spaceID, nodeToken := parseWikiResourceID(rid)
if spaceID == "" || nodeToken == "" {
// A space-level selection is already a top-level node in the picker;
// there is nothing above it to reveal.
continue
}
// The space's direct children must be loaded to reveal the top-level node.
add(spaceID)
// Walk up from the selection to the top, loading each intermediate
// parent so the path down to the selection becomes visible.
current := nodeToken
for current != "" {
node, err := client.GetWikiNode(ctx, spaceID, current)
if err != nil {
// Best-effort: a broken path just stays collapsed, the rest of
// the selections are still revealed.
logger.Warnf(ctx, "[Feishu] resolve ancestors: get node %s:%s: %v", spaceID, current, err)
break
}
if node.ParentNodeID == "" {
break
}
add(makeWikiNodeResourceID(spaceID, node.ParentNodeID))
current = node.ParentNodeID
}
}
return ancestors, nil
}
// FetchAll performs a full sync of all documents from the specified wiki spaces.
func (c *Connector) FetchAll(ctx context.Context, config *types.DataSourceConfig, resourceIDs []string) ([]types.FetchedItem, error) {
feishuConfig, err := parseFeishuConfig(config, c.region)
if err != nil {
return nil, err
}
client := NewClient(feishuConfig)
var allItems []types.FetchedItem
for _, resourceID := range resourceIDs {
spaceID, nodeToken := parseWikiResourceID(resourceID)
// List all nodes in this wiki space or selected node subtree recursively
nodes, err := client.ListWikiNodesRecursiveFrom(ctx, spaceID, nodeToken)
if err != nil {
var partialErr *partialWikiNodeListError
if !errors.As(err, &partialErr) {
return nil, fmt.Errorf("list nodes for resource %s: %w", resourceID, err)
}
allItems = appendWikiNodeListFailureItems(allItems, spaceID, resourceID, partialErr.Failures)
}
// Fetch content for each document node
for _, node := range nodes {
item, err := c.fetchNodeContent(ctx, client, node, spaceID, resourceID)
if err != nil {
// Log error but continue with other nodes
allItems = append(allItems, types.FetchedItem{
ExternalID: node.NodeToken,
Title: node.Title,
SourceResourceID: resourceID,
Metadata: map[string]string{
"error": err.Error(),
},
})
continue
}
if item != nil {
allItems = append(allItems, *item)
}
}
}
return allItems, nil
}
// FetchIncremental performs an incremental sync by comparing node edit times
// against the previously recorded state.
func (c *Connector) FetchIncremental(ctx context.Context, config *types.DataSourceConfig, cursor *types.SyncCursor) ([]types.FetchedItem, *types.SyncCursor, error) {
feishuConfig, err := parseFeishuConfig(config, c.region)
if err != nil {
return nil, nil, err
}
client := NewClient(feishuConfig)
// Parse the previous cursor state
var prevCursor feishuCursor
if cursor != nil && cursor.ConnectorCursor != nil {
cursorBytes, _ := json.Marshal(cursor.ConnectorCursor)
_ = json.Unmarshal(cursorBytes, &prevCursor)
}
// Build new cursor to track current state
newCursor := feishuCursor{
LastSyncTime: time.Now(),
SpaceNodeTimes: make(map[string]map[string]string),
}
var changedItems []types.FetchedItem
// Get resource IDs from config
resourceIDs := config.ResourceIDs
if len(resourceIDs) == 0 {
return nil, nil, fmt.Errorf("no resource IDs (wiki space IDs or wiki node IDs) configured")
}
for _, resourceID := range resourceIDs {
spaceID, nodeToken := parseWikiResourceID(resourceID)
// List all nodes in this wiki space or selected node subtree
nodes, err := client.ListWikiNodesRecursiveFrom(ctx, spaceID, nodeToken)
var partialErr *partialWikiNodeListError
if err != nil {
if !errors.As(err, &partialErr) {
return nil, nil, fmt.Errorf("list nodes for resource %s: %w", resourceID, err)
}
changedItems = appendWikiNodeListFailureItems(changedItems, spaceID, resourceID, partialErr.Failures)
}
newCursor.SpaceNodeTimes[resourceID] = make(map[string]string)
if partialErr != nil && prevCursor.SpaceNodeTimes != nil {
if prevTimes, ok := prevCursor.SpaceNodeTimes[resourceID]; ok {
for nodeToken, editTime := range prevTimes {
newCursor.SpaceNodeTimes[resourceID][nodeToken] = editTime
}
}
}
// Build a set of current node tokens for deletion detection
currentNodes := make(map[string]bool)
for _, node := range nodes {
currentNodes[node.NodeToken] = true
// Use ObjEditTime (document content edit time) for change detection,
// NOT NodeEditTime which only tracks node attribute changes (title, position).
editTimeStr := node.ObjEditTime
if editTimeStr == "" {
editTimeStr = node.NodeEditTime // fallback for nodes that don't have obj_edit_time
}
newCursor.SpaceNodeTimes[resourceID][node.NodeToken] = editTimeStr
// Check if node has changed since last sync
if prevCursor.SpaceNodeTimes != nil {
if prevTimes, ok := prevCursor.SpaceNodeTimes[resourceID]; ok {
if prevEditTime, exists := prevTimes[node.NodeToken]; exists {
if prevEditTime == editTimeStr {
// Node unchanged, skip
continue
}
}
}
}
// Node is new or changed — fetch its content
item, err := c.fetchNodeContent(ctx, client, node, spaceID, resourceID)
if err != nil {
// Record failed items
changedItems = append(changedItems, types.FetchedItem{
ExternalID: node.NodeToken,
Title: node.Title,
SourceResourceID: resourceID,
Metadata: map[string]string{
"error": err.Error(),
},
})
continue
}
if item != nil {
changedItems = append(changedItems, *item)
}
}
// Detect deleted nodes
if partialErr == nil && prevCursor.SpaceNodeTimes != nil {
if prevTimes, ok := prevCursor.SpaceNodeTimes[resourceID]; ok {
for nodeToken := range prevTimes {
if !currentNodes[nodeToken] {
// Node was deleted
changedItems = append(changedItems, types.FetchedItem{
ExternalID: nodeToken,
IsDeleted: true,
SourceResourceID: resourceID,
})
}
}
}
}
}
// Build next sync cursor
nextCursorMap := make(map[string]interface{})
cursorBytes, _ := json.Marshal(newCursor)
_ = json.Unmarshal(cursorBytes, &nextCursorMap)
nextSyncCursor := &types.SyncCursor{
LastSyncTime: time.Now(),
ConnectorCursor: nextCursorMap,
}
return changedItems, nextSyncCursor, nil
}
func appendWikiNodeListFailureItems(items []types.FetchedItem, spaceID string, resourceID string, failures []wikiNodeListFailure) []types.FetchedItem {
for _, failure := range failures {
node := failure.Node
title := node.Title
if title == "" {
title = node.NodeToken
}
items = append(items, types.FetchedItem{
ExternalID: node.NodeToken,
Title: title,
SourceResourceID: resourceID,
Metadata: map[string]string{
"error": failure.Err.Error(),
"channel": types.ChannelFeishu,
"node_token": node.NodeToken,
"space_id": spaceID,
"failure_stage": "list_children",
},
})
}
return items
}
// fetchNodeContent fetches the content of a single wiki node and converts it to FetchedItem.
// Dispatches to different retrieval strategies based on obj_type:
// - docx/doc → export API → .docx file
// - sheet → export API → .xlsx file
// - bitable → export API → .xlsx file
// - file → drive download → original file (PDF/Word/image/etc.)
// - mindnote → skip (no API)
// - slides → skip (no API)
func (c *Connector) fetchNodeContent(ctx context.Context, client *Client, node wikiNode, spaceID string, resourceID string) (*types.FetchedItem, error) {
if !isSupportedDocType(node.ObjType) {
return nil, nil
}
editTime := parseFeishuTimestamp(node.NodeEditTime)
baseMeta := map[string]string{
"obj_token": node.ObjToken,
"obj_type": node.ObjType,
"node_token": node.NodeToken,
"space_id": spaceID,
"creator": node.Creator,
"owner": node.Owner,
"channel": types.ChannelFeishu,
}
switch node.ObjType {
case "docx", "doc", "sheet", "bitable":
// Export as a file via the async export API
data, fileName, err := client.ExportAndDownload(ctx, node.ObjToken, node.ObjType)
if err != nil {
return nil, fmt.Errorf("export %s (%s): %w", node.Title, node.ObjType, err)
}
// Ensure a reasonable file name with correct extension
ext := exportFileExtToSuffix[objTypeToExportFileExtension[node.ObjType]]
if fileName == "" {
fileName = sanitizeFileName(node.Title) + ext
} else if !strings.HasSuffix(strings.ToLower(fileName), ext) {
// Feishu often returns the doc title without extension — append it
fileName = sanitizeFileName(fileName) + ext
}
return &types.FetchedItem{
ExternalID: node.NodeToken,
Title: node.Title,
Content: data,
ContentType: "application/octet-stream",
FileName: fileName,
URL: c.region.wikiURL(node.NodeToken),
UpdatedAt: editTime,
SourceResourceID: resourceID,
Metadata: baseMeta,
}, nil
case "file":
// Download the original uploaded file from Drive
data, err := client.DownloadDriveFile(ctx, node.ObjToken)
if err != nil {
return nil, fmt.Errorf("download file %s (%s): %w", node.Title, node.ObjToken, err)
}
// Use the node title as file name; it usually preserves the original extension
fileName := node.Title
if fileName == "" {
fileName = node.ObjToken
}
return &types.FetchedItem{
ExternalID: node.NodeToken,
Title: node.Title,
Content: data,
ContentType: "application/octet-stream",
FileName: fileName,
URL: c.region.wikiURL(node.NodeToken),
UpdatedAt: editTime,
SourceResourceID: resourceID,
Metadata: baseMeta,
}, nil
default:
return nil, nil
}
}
// --- Helper functions ---
func makeWikiNodeResourceID(spaceID, nodeToken string) string {
return spaceID + feishuWikiNodeResourceSeparator + nodeToken
}
func parseWikiResourceID(resourceID string) (spaceID string, nodeToken string) {
spaceID, nodeToken, _ = strings.Cut(resourceID, feishuWikiNodeResourceSeparator)
return spaceID, nodeToken
}
func (c *Connector) wikiNodeToResource(spaceID string, node wikiNode) types.Resource {
parentID := spaceID
if node.ParentNodeID != "" {
parentID = makeWikiNodeResourceID(spaceID, node.ParentNodeID)
}
name := node.Title
if name == "" {
name = node.NodeToken
}
modifiedAt := parseFeishuTimestamp(node.ObjEditTime)
if modifiedAt.IsZero() {
modifiedAt = parseFeishuTimestamp(node.NodeEditTime)
}
return types.Resource{
ExternalID: makeWikiNodeResourceID(spaceID, node.NodeToken),
Name: name,
Type: "wiki_node",
URL: c.region.wikiURL(node.NodeToken),
ParentID: parentID,
HasChildren: node.HasChild,
ModifiedAt: modifiedAt,
Metadata: map[string]interface{}{
"space_id": spaceID,
"node_token": node.NodeToken,
"obj_token": node.ObjToken,
"obj_type": node.ObjType,
},
}
}
// parseFeishuConfig extracts and validates Feishu/Lark-specific configuration.
//
// base_url stays an explicit override so existing data sources that pointed a
// "feishu" connector at open.larksuite.com keep working; when it is unset the
// region's own host is filled in, making the resolved Config.BaseURL concrete
// for everything downstream.
func parseFeishuConfig(config *types.DataSourceConfig, region Region) (*Config, error) {
if config == nil {
return nil, fmt.Errorf("config is nil")
}
credBytes, err := json.Marshal(config.Credentials)
if err != nil {
return nil, fmt.Errorf("marshal credentials: %w", err)
}
var feishuConfig Config
if err := json.Unmarshal(credBytes, &feishuConfig); err != nil {
return nil, fmt.Errorf("parse %s credentials: %w", region.ConnectorType, err)
}
if feishuConfig.AppID == "" || feishuConfig.AppSecret == "" {
return nil, fmt.Errorf("%s app_id and app_secret are required", region.ConnectorType)
}
if feishuConfig.BaseURL == "" {
feishuConfig.BaseURL = region.OpenBaseURL
}
if err := datasource.ValidateConnectorBaseURL(feishuConfig.GetBaseURL()); err != nil {
return nil, err
}
return &feishuConfig, nil
}
// isSupportedDocType checks if a Feishu document type can be synced.
// mindnote and slides have no content read API and are skipped.
func isSupportedDocType(objType string) bool {
switch objType {
case "docx", "doc", "sheet", "bitable", "file":
return true
default:
// mindnote, slides — no content retrieval API available
return false
}
}
// parseFeishuTimestamp parses a Feishu unix timestamp string (seconds) into time.Time.
func parseFeishuTimestamp(ts string) time.Time {
if ts == "" {
return time.Time{}
}
sec, err := strconv.ParseInt(ts, 10, 64)
if err != nil {
return time.Time{}
}
return time.Unix(sec, 0)
}
// sanitizeFileName removes characters that are invalid in filenames and
// truncates at a UTF-8 rune boundary. Raw byte truncation would split a
// multi-byte codepoint (Chinese chars are 3 bytes) and produce invalid UTF-8
// that downstream validation (utf8.ValidString) rejects.
func sanitizeFileName(name string) string {
if name == "" {
return "untitled"
}
replacer := strings.NewReplacer(
"/", "_", "\\", "_", ":", "_", "*", "_",
"?", "_", "\"", "_", "<", "_", ">", "_", "|", "_",
)
result := replacer.Replace(name)
const maxBytes = 200
if len(result) > maxBytes {
result = result[:maxBytes]
for len(result) > 0 {
r, size := utf8.DecodeLastRuneInString(result)
if r != utf8.RuneError || size != 1 {
break
}
result = result[:len(result)-1]
}
}
return result
}