Skip to content

Commit 813f17d

Browse files
mayoalexanderclaude
andcommitted
fix(data-sources): s3 was ingestable all along, and list was reading one side of a join
#182734, measured with the CLI's own survey: 16 source(s) · 1 bulk-importable 8 connected source(s) are hidden from discovery (incl. google-drive) TWO DIFFERENT DEFECTS. 1. UNDER-REPORTING, not a missing feature. fl-api's BloqIngestionController::startFolderIngestion validates `in:dropbox,google_drive,s3`, and FileIngestionService switches on all three with each implemented. This CLI advertised two. So `sync` could always take s3 and nothing said so. I went looking to widen this list much further — the catalog offers 49 source types — and the server says no. Three is the real ceiling, and adding a fourth here would turn a truthful "read-only" into a promise that fails at run time. Widening it is a SERVER change. The constant now says that, and a test pins it to exactly the server's rule in both directions: adding a type the server rejects fails, and dropping one we ship fails. 2. `list` IS A DISCOVERY SURFACE AND IT READ ONE SIDE. `survey` already joins the availability list against real connections and reports the delta — that join is the whole reason survey exists. `list` did not do it, so 8 connected, executable sources were absent from the surface that answers "what can I use", including google-drive with three working accounts: simultaneously the only bulk-importable source we have and invisible in the list. `list` now runs the same join and shows what it finds, labelled, with the command that works. --json carries them too, flagged hidden_but_connected, because omitting a real source is the bug rather than the tidy answer. A failed join degrades to the old behaviour instead of dropping sources. Does not touch the LINK half — entity resolution and typed relations are bloq anomalyco#629, and this is only about gather. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_019wF7NrsCHAhXRn3ZddtQv5
1 parent 0a754f2 commit 813f17d

2 files changed

Lines changed: 136 additions & 3 deletions

File tree

‎packages/opencode/src/cli/cmd/platform-data-sources.test.ts‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ import {
1212
pickEnumerator,
1313
surveySources,
1414
countItems,
15+
BULK_INGESTABLE_TYPES,
1516
} from "./platform-data-sources"
1617

1718
// ---------------------------------------------------------------------------
@@ -304,3 +305,68 @@ test("countItems: returns null (not 0) when there is no array to count", () => {
304305
expect(countItems(null)).toBeNull()
305306
expect(countItems("a string")).toBeNull()
306307
})
308+
309+
310+
// ---------------------------------------------------------------------------
311+
// #182734 — bulk-ingest ceiling, and the join `list` was not doing
312+
// ---------------------------------------------------------------------------
313+
314+
/**
315+
* Measured 2026-08-28 with the CLI's own survey:
316+
*
317+
* 16 source(s) · 1 bulk-importable
318+
* 8 connected source(s) are hidden from discovery (incl. google-drive)
319+
*
320+
* Two different defects. The first was UNDER-reporting: fl-api validates
321+
* `in:dropbox,google_drive,s3` and FileIngestionService implements all three, while this
322+
* CLI advertised two. The second was a discovery surface reading one side of a join that
323+
* `survey` already knew how to do.
324+
*/
325+
326+
test("BULK_INGESTABLE_TYPES equals the server's validation rule — no more, no fewer", () => {
327+
// More would promise a source fl-api rejects at run time; fewer hides one we ship.
328+
expect([...BULK_INGESTABLE_TYPES].sort()).toEqual(["dropbox", "google_drive", "s3"])
329+
})
330+
331+
test("s3 is bulk-ingestable — it was missing, and that was the under-report", () => {
332+
expect(isBulkIngestable("s3")).toBe(true)
333+
})
334+
335+
test("bulk-ingestable matches across the spellings the CLI disagrees with itself about", () => {
336+
expect(isBulkIngestable("google_drive")).toBe(true)
337+
expect(isBulkIngestable("google-drive")).toBe(true)
338+
expect(isBulkIngestable("Google Drive")).toBe(true)
339+
expect(normalizeSourceType("Google_Drive")).toBe("google-drive")
340+
})
341+
342+
test("a source the server would reject is never advertised as importable", () => {
343+
for (const t of ["slack", "notion", "obsidian", "imessage-bridge", "youtube"]) {
344+
expect(isBulkIngestable(t)).toBe(false)
345+
}
346+
})
347+
348+
test("surveySources flags a connected source the availability list omits", () => {
349+
const available = [{ type: "obsidian", name: "Obsidian", functions: ["list_files"] }]
350+
const connections = [
351+
{ type: "google-drive", name: "Google Drive", account_email: "[email protected]" },
352+
{ type: "obsidian", name: "Obsidian" },
353+
]
354+
const hidden = surveySources(available, connections).filter((s) => s.hiddenButConnected)
355+
expect(hidden.map((s) => s.type)).toEqual(["google-drive"])
356+
expect(hidden[0].accounts).toEqual(["[email protected]"])
357+
// A hidden google-drive is still correctly reported as importable.
358+
expect(hidden[0].bulkIngestable).toBe(true)
359+
})
360+
361+
test("a source present on both sides is connected but NOT flagged hidden", () => {
362+
const ob = surveySources(
363+
[{ type: "obsidian", name: "Obsidian" }],
364+
[{ type: "obsidian", name: "Obsidian" }],
365+
).find((s) => s.type === "obsidian")!
366+
expect(ob.connected).toBe(true)
367+
expect(ob.hiddenButConnected).toBe(false)
368+
})
369+
370+
test("no connections means nothing is hidden, not everything", () => {
371+
expect(surveySources([{ type: "obsidian" }], []).some((s) => s.hiddenButConnected)).toBe(false)
372+
})

‎packages/opencode/src/cli/cmd/platform-data-sources.ts‎

Lines changed: 70 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -156,7 +156,22 @@ export function stringifySource(result: any, maxChars = 12000): string {
156156
* are different questions, and conflating them is the mental-model error survey
157157
* exists to prevent.
158158
*/
159-
export const BULK_INGESTABLE_TYPES = ["dropbox", "google_drive"] as const
159+
export const BULK_INGESTABLE_TYPES = ["dropbox", "google_drive", "s3"] as const
160+
161+
/**
162+
* THIS LIST MIRRORS A SERVER VALIDATION RULE. Keep them equal.
163+
*
164+
* fl-api BloqIngestionController::startFolderIngestion validates
165+
* 'source' => 'required|in:dropbox,google_drive,s3'
166+
* and FileIngestionService switches on exactly those three, each implemented.
167+
*
168+
* s3 was missing here, so `survey` reported the CLI could bulk-ingest two sources when
169+
* the platform could do three — under-reporting a capability we already shipped.
170+
*
171+
* Widening this further is a SERVER change, not an edit here. Adding a type the server
172+
* rejects would turn a truthful "read-only" into a promise that fails at run time, which
173+
* is the failure mode this file exists to prevent.
174+
*/
160175

161176
/**
162177
* Fold `google_drive` / `google-drive` / `Google Drive` onto one key.
@@ -314,12 +329,47 @@ const ListCommand = cmd({
314329
}
315330
const data = (await res.json()) as any
316331
const sources: any[] = data?.data?.sources ?? data?.sources ?? []
332+
333+
// #182734: THIS LIST IS A DISCOVERY SURFACE, AND IT WAS READING ONE SIDE.
334+
//
335+
// `survey` already joins the availability list against actual connections and reports
336+
// the delta. Measured 2026-08-28: 8 of 16 connected sources were absent here —
337+
// including google-drive with three working accounts, the one source most people want.
338+
// A source that is connected and executable must never be missing from the list that
339+
// tells you what you can use, so the same join runs here.
340+
let hiddenConnected: SurveyedSource[] = []
341+
try {
342+
const userId = await requireUserId()
343+
if (userId) {
344+
const connRes = await irisFetch(`/api/v1/users/${userId}/integrations`)
345+
if (connRes.ok) {
346+
const cj = (await connRes.json()) as any
347+
const connections: any[] = cj?.connections ?? cj?.data ?? (Array.isArray(cj) ? cj : [])
348+
hiddenConnected = surveySources(sources, connections).filter((x) => x.hiddenButConnected)
349+
}
350+
}
351+
} catch {
352+
// Non-fatal. A failed join must not remove sources we DID resolve.
353+
}
354+
317355
if (json) {
318-
await writeJson(sources)
356+
// Hidden sources are real and executable; omitting them is the bug. Flagged so a
357+
// caller can tell where each came from.
358+
await writeJson([
359+
...sources,
360+
...hiddenConnected.map((h) => ({
361+
type: h.type,
362+
name: h.name,
363+
connected: true,
364+
listed_in_availability: false,
365+
hidden_but_connected: true,
366+
accounts: h.accounts,
367+
})),
368+
])
319369
return
320370
}
321371
printDivider()
322-
if (sources.length === 0) {
372+
if (sources.length === 0 && hiddenConnected.length === 0) {
323373
console.log(` ${dim("(no connected sources — add one with:")} ${highlight("iris integrations connect <type>")}${dim(")")}`)
324374
} else {
325375
for (const s of sources) {
@@ -352,7 +402,24 @@ const ListCommand = cmd({
352402
if (fns.length) console.log(` ${dim("functions:")} ${fns.join(", ")}`)
353403
}
354404
}
405+
406+
// Connected, executable, and absent from the availability list. Shown rather than
407+
// dropped, and labelled so the omission is visible instead of silent.
408+
for (const h of hiddenConnected) {
409+
console.log(
410+
` ${bold(h.type)} ${dim(h.name ?? "")} ${UI.Style.TEXT_WARNING}⚠ connected, not in the availability list${UI.Style.TEXT_NORMAL}`,
411+
)
412+
if (h.accounts.length) console.log(` ${dim(h.accounts.join(", "))}`)
413+
console.log(` ${dim("usable with:")} iris data-sources read ${h.type}`)
414+
}
415+
355416
printDivider()
417+
if (hiddenConnected.length) {
418+
console.log(
419+
` ${UI.Style.TEXT_WARNING}${hiddenConnected.length} source(s) are connected but missing from the availability list${UI.Style.TEXT_NORMAL}`,
420+
)
421+
console.log(` ${dim("Look closer:")} ${highlight("iris data-sources survey")}`)
422+
}
356423
console.log(` ${dim("read a source:")} ${highlight("iris data-sources read <type> -f <function> -p key=value")}`)
357424
prompts.outro("Done")
358425
},

0 commit comments

Comments
 (0)