Skip to content

fix: .txt and .md filenames should treated as duplicates.#1190

Merged
Wallgau merged 5 commits into
release-0.4.0from
overwrite-txt-file
Mar 21, 2026
Merged

fix: .txt and .md filenames should treated as duplicates.#1190
Wallgau merged 5 commits into
release-0.4.0from
overwrite-txt-file

Conversation

@Wallgau

@Wallgau Wallgau commented Mar 18, 2026

Copy link
Copy Markdown
Collaborator

issue: #1174
this issue happened because .txt uploads were internally renamed and indexed as .md, the duplicate check looked for filename.txt while existing chunks were stored under filename.md, so it missed the match and appended chunks instead of triggering overwrite.

Before:
https://github.com/user-attachments/assets/4a9168fd-fac5-44b5-b92b-d84bbb1412e8

Now:
https://github.com/user-attachments/assets/5e4e5009-2990-4518-ae48-55efb9766dc4

Ensure duplicate checks and replacement deletions consider .txt/.md aliases so re-uploading text files triggers overwrite behavior instead of appending chunks.

Made-with: Cursor
@Wallgau Wallgau requested a review from lucaseduoli March 18, 2026 21:52
@github-actions github-actions Bot added the backend 🔷 Issues related to backend services (OpenSearch, Langflow, APIs) label Mar 18, 2026
@Wallgau Wallgau changed the title Treat .txt and .md filenames as duplicates. fix: .txt and .md filenames should treated as duplicates. Mar 18, 2026
@github-actions github-actions Bot added the bug 🔴 Something isn't working. label Mar 18, 2026
@Wallgau Wallgau requested a review from ricofurtado March 18, 2026 23:30
@github-actions github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Mar 19, 2026
@ricofurtado ricofurtado requested a review from Copilot March 19, 2026 16:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes duplicate/overwrite detection for .txt uploads that were historically indexed under a .md filename variant, ensuring overwrite prompts and deletion behave consistently with what’s stored in OpenSearch.

Changes:

  • Added get_filename_aliases() helper to generate .txt/.md equivalent filename candidates.
  • Updated OpenSearch duplicate-check logic to search across filename aliases.
  • Updated OpenSearch delete-by-filename logic (processor path) to delete across filename aliases.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
src/utils/file_utils.py Introduces .txt/.md alias helper for consistent duplicate detection.
src/models/processors.py Checks/deletes documents by iterating over filename aliases during ingestion/overwrite.
src/api/documents.py Updates filename existence endpoint to consider alias variants.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/models/processors.py Outdated
Comment thread src/api/documents.py Outdated
Comment thread src/utils/file_utils.py
Comment on lines +104 to +123
def get_filename_aliases(filename: str) -> list[str]:
"""Return equivalent filename variants used by ingestion/indexing.

Legacy Langflow ingest may index `.txt` uploads as `.md`. This helper keeps
duplicate detection/deletion consistent by checking both forms.
"""
normalized = (filename or "").strip()
if not normalized:
return []

aliases = [normalized]
lower_name = normalized.lower()

if lower_name.endswith(".txt"):
aliases.append(normalized[:-4] + ".md")
elif lower_name.endswith(".md"):
aliases.append(normalized[:-3] + ".txt")

# Keep order stable while removing duplicates.
return list(dict.fromkeys(aliases))
Comment thread src/utils/file_utils.py
Comment thread src/utils/file_utils.py
return clean_name


def get_filename_aliases(filename: str) -> list[str]:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Empty filename fallback inconsistency (processors.py:79, processors.py:134, documents.py:126)

All three callers use the same pattern:
candidate_filenames = get_filename_aliases(filename) or [filename]

When filename="", get_filename_aliases("") returns [], so the or kicks in and produces [""] — a search/delete for an empty-string filename. The function's [] return for empty input was meant as an early-exit guard, but the callers undo it.

Suggested fix — remove the early-exit in file_utils.py:110-112 and instead handle it at the function level by always including the original:

  def get_filename_aliases(filename: str) -> list[str]:                                                                                                                                                                                          
      normalized = (filename or "").strip()                                  
      if not normalized:                                                                                                                                                                                                                         
          return [filename] if filename else []  # caller gets original back, not []
                                                                                                                                                                                                                                                 
      aliases = [normalized]                                                 
      ...                 

Or, simpler — document that callers should never pass an empty filename and add an assert in dev mode.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I addressed the empty filename issue, but at the call sites rather than by changing get filename aliases return semantics.

get filename aliases still returns [] for empty or whitespace input (guard remains).
The problematic behavior came from callers doing get filename aliases(filename) or [filename], which could turn empty input into [""].

I removed that fallback in the affected paths and now handle empty alias lists explicitly:

TaskProcessor.check filename exists() returns False when aliases are empty.
TaskProcessor.delete document by filename() no ops (with log) when aliases are empty.
api/documents.check filename exists() returns { "exists": false } early when aliases are empty.
This avoids empty string OpenSearch queries while keeping get filename aliases() focused on normalized alias generation.

Comment thread src/api/documents.py Outdated
Comment thread src/models/processors.py Outdated
@github-actions github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Mar 19, 2026
@mpawlow mpawlow removed their request for review March 19, 2026 20:56
@github-actions github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Mar 19, 2026
Avoid empty-string filename queries and reduce redundant alias rechecks across retries, while clarifying alias behavior for .txt/.md ingestion compatibility.

Made-with: Cursor
@github-actions github-actions Bot added tests bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Mar 20, 2026
@Wallgau Wallgau requested a review from ricofurtado March 20, 2026 00:20
@github-actions github-actions Bot added the lgtm label Mar 21, 2026
@Wallgau Wallgau merged commit 05f22b7 into release-0.4.0 Mar 21, 2026
8 checks passed
@github-actions github-actions Bot deleted the overwrite-txt-file branch March 21, 2026 01:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend 🔷 Issues related to backend services (OpenSearch, Langflow, APIs) bug 🔴 Something isn't working. lgtm tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants