Skip to content

feat: Pdf doc build fix#1104

Merged
Wallgau merged 55 commits into
mainfrom
pdf-doc-build-fix
Mar 13, 2026
Merged

feat: Pdf doc build fix#1104
Wallgau merged 55 commits into
mainfrom
pdf-doc-build-fix

Conversation

@Wallgau

@Wallgau Wallgau commented Mar 12, 2026

Copy link
Copy Markdown
Collaborator

Olfa Maslah and others added 30 commits March 3, 2026 15:25
Switch default OpenRAG docs onboarding ingestion to URL crawling and align frontend/backend filters to target system_default content. Add guardrails for missing URL flow configuration and missing JWT to prevent onboarding failures and ensure graceful fallback behavior.

Made-with: Cursor
Track the OpenRAG version used for default docs ingestion and reingest system docs on application upgrade. Add safe cleanup and startup handling so upgrades refresh docs without failing when optional session context is unavailable.

Made-with: Cursor
…tartup reingestion.

Centralize URL-ingest eligibility checks, persist remote signature consistently across onboarding and upgrades, add a manual refresh endpoint/UI action, and include targeted signature fallback tests.

Made-with: Cursor
Emit start/skip/complete/fail refresh events with non-PII metadata and persist remote signature consistently after refresh to avoid stale state.

Made-with: Cursor
Normalize frontend filter payloads to omit wildcard dimensions and stabilize remote docs signatures by preferring ETag over Last-Modified fallback formatting.

Made-with: Cursor
Delete both URL-ingested system_default docs and legacy local sample-data docs during refresh/upgrade to avoid stale default content.

Made-with: Cursor
Replace component-id tweaks with X-Langflow-Global-Var passthrough for docs URL and crawl settings so the flow remains stable when component IDs change.

Made-with: Cursor
Move URL ingestion request construction/execution into LangflowFileService so main startup code stays orchestration-focused and reuses consistent Langflow header/error handling.

Made-with: Cursor
This updates OpenSearch/Langflow fallback retrieval to handle mapping edge-cases and score thresholds consistently, and hardens document deletion UX/API behavior to avoid false-success signals.

Made-with: Cursor
Revert frontend, flow-component, and unrelated backend adjustments so this branch focuses on the URL-first default docs ingestion and refresh metadata path.

Made-with: Cursor
This ensures empty-query fallback returns no results and that RequestError after num_candidates retry degrades to keyword fallback instead of hard-failing.

Made-with: Cursor
Handle 404 responses from document delete as a non-throwing zero-deletion result and update the integration test to account for ingestion runs that complete without indexed chunks.

Made-with: Cursor
Restore legacy vector fallback for KNN field errors in Langflow component code and make backend search gracefully degrade to legacy or keyword fallback paths instead of surfacing HTTP 500s when dynamic KNN fields are not queryable.

Made-with: Cursor
Keep semantic fallback constrained to documents that contain the legacy embedding field so KNN fallback does not return non-embedded documents, and sync the updated component code into all Langflow flow JSONs.

Made-with: Cursor
Made-with: Cursor

# Conflicts:
#	flows/ingestion_flow.json
#	flows/openrag_agent.json
#	flows/openrag_nudges.json
#	flows/openrag_url_mcp.json
#	src/main.py
#	src/services/search_service.py
Ensure fallback warning logs in the multimodal OpenSearch component always include reason/error text directly in the message string so the context is reliably emitted across logger implementations.

Made-with: Cursor
Restore the retrieval flow Agent node to the expected model-based template so non-streaming chat calls no longer fail at runtime, and harden OpenSearch legacy fallback filtering in search service.

Made-with: Cursor
Handle backend 404 responses as non-fatal in the Python documents client and align integration assertions with environments where ingestion may complete with zero successful files.

Made-with: Cursor
@github-actions github-actions Bot added enhancement 🔵 New feature or request and removed enhancement 🔵 New feature or request labels Mar 12, 2026
@mpawlow

mpawlow commented Mar 12, 2026

Copy link
Copy Markdown
Collaborator

Test Scenario: Fetch non-existent default document

  • ✅ PASS

Details

DEFAULT_DOCS_URL=https://alkdjklfsdjklfsd
image

@mpawlow

mpawlow commented Mar 12, 2026

Copy link
Copy Markdown
Collaborator

Test Scenario: Chat with filter on URL ingested OpenRAG documentation

  • ✅ PASS

Details

image

@mpawlow

mpawlow commented Mar 12, 2026

Copy link
Copy Markdown
Collaborator

UI / UX Review 1

image

(ux 1a) [Question]: Just to confirm that the UI / UX changes were reviewed / approved by @Analuisa1a

(ux 1b) [Minor]: The "Fetch latest docs" button seems to prominent in the "Product Knowledge" page

  • Concern: Assuming it may be an action that the user may invoke very rarely
  • Concern: "Fetch latest docs" button action may be confusing compared to the "Sync" button action
    • Assumption: "Sync" operation would also fetch the latest documents
  • Suggestion: (for now or the future,) downgrade the prominent of the "Fetch latest docs" operation to a 3 meatballs menu or button drop-down selection somewhere (there is no obvious / existing place to place this operation atm)
  • Please feel free to ignore these suggestions based on what Design feedback indicates

@Wallgau Wallgau closed this Mar 12, 2026
@Wallgau Wallgau reopened this Mar 12, 2026
Retry URL ingest flow validation/import once on transient Langflow errors so short outages do not leave URL ingestion broken for the process lifetime.

Made-with: Cursor
@github-actions github-actions Bot removed the enhancement 🔵 New feature or request label Mar 12, 2026
@github-actions github-actions Bot added enhancement 🔵 New feature or request and removed enhancement 🔵 New feature or request labels Mar 12, 2026
Add defensive shape checks for legacy vector fallback query rewriting and gracefully fall back to keyword retrieval when expected bool/dis_max paths are missing.

Made-with: Cursor
@github-actions github-actions Bot added enhancement 🔵 New feature or request and removed enhancement 🔵 New feature or request labels Mar 12, 2026
@Analuisa1a

Analuisa1a commented Mar 13, 2026

Copy link
Copy Markdown
Collaborator

Reviewed with @Wallgau

Design recommendation is that the icon should stay with sync, but be removed from the fetch, as we don't want to confuse users with the same item. Also, that the buttons should flip as the syncing has priority over the fetching element to the user

@mpawlow I would replace fetch with the menu button if we had more items to list. I don't want to hide the button and give the user more clicks if there is only one item hidden. For now, we can show both, but have the Sync take more prominence.

Screenshot 2026-03-12 at 4 55 58 PM

Move the Fetch latest docs action after Sync and keep its loading state aligned with the existing row-level refresh cue behavior.

Made-with: Cursor
@github-actions github-actions Bot added enhancement 🔵 New feature or request and removed enhancement 🔵 New feature or request labels Mar 13, 2026
Return a generic message for non-auth delete failures while preserving detailed server-side logging to avoid leaking internal exception details.

Made-with: Cursor
@github-actions github-actions Bot added enhancement 🔵 New feature or request and removed enhancement 🔵 New feature or request labels Mar 13, 2026
@mpawlow

mpawlow commented Mar 13, 2026

Copy link
Copy Markdown
Collaborator

Sounds good. Thanks @Analuisa1a 🙏

@mpawlow mpawlow left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Wallgau

Code Review 2

  • Approved / LGTM 👍

@github-actions github-actions Bot added the lgtm label Mar 13, 2026
@Wallgau Wallgau merged commit a930481 into main Mar 13, 2026
11 checks passed
@github-actions github-actions Bot deleted the pdf-doc-build-fix branch March 13, 2026 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend 🔷 Issues related to backend services (OpenSearch, Langflow, APIs) enhancement 🔵 New feature or request frontend 🟨 Issues related to the UI/UX lgtm tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants