fix: .txt and .md filenames should treated as duplicates.#1190
Conversation
Ensure duplicate checks and replacement deletions consider .txt/.md aliases so re-uploading text files triggers overwrite behavior instead of appending chunks. Made-with: Cursor
There was a problem hiding this comment.
Pull request overview
Fixes duplicate/overwrite detection for .txt uploads that were historically indexed under a .md filename variant, ensuring overwrite prompts and deletion behave consistently with what’s stored in OpenSearch.
Changes:
- Added
get_filename_aliases()helper to generate.txt/.mdequivalent filename candidates. - Updated OpenSearch duplicate-check logic to search across filename aliases.
- Updated OpenSearch delete-by-filename logic (processor path) to delete across filename aliases.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
src/utils/file_utils.py |
Introduces .txt/.md alias helper for consistent duplicate detection. |
src/models/processors.py |
Checks/deletes documents by iterating over filename aliases during ingestion/overwrite. |
src/api/documents.py |
Updates filename existence endpoint to consider alias variants. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| def get_filename_aliases(filename: str) -> list[str]: | ||
| """Return equivalent filename variants used by ingestion/indexing. | ||
|
|
||
| Legacy Langflow ingest may index `.txt` uploads as `.md`. This helper keeps | ||
| duplicate detection/deletion consistent by checking both forms. | ||
| """ | ||
| normalized = (filename or "").strip() | ||
| if not normalized: | ||
| return [] | ||
|
|
||
| aliases = [normalized] | ||
| lower_name = normalized.lower() | ||
|
|
||
| if lower_name.endswith(".txt"): | ||
| aliases.append(normalized[:-4] + ".md") | ||
| elif lower_name.endswith(".md"): | ||
| aliases.append(normalized[:-3] + ".txt") | ||
|
|
||
| # Keep order stable while removing duplicates. | ||
| return list(dict.fromkeys(aliases)) |
| return clean_name | ||
|
|
||
|
|
||
| def get_filename_aliases(filename: str) -> list[str]: |
There was a problem hiding this comment.
Empty filename fallback inconsistency (processors.py:79, processors.py:134, documents.py:126)
All three callers use the same pattern:
candidate_filenames = get_filename_aliases(filename) or [filename]
When filename="", get_filename_aliases("") returns [], so the or kicks in and produces [""] — a search/delete for an empty-string filename. The function's [] return for empty input was meant as an early-exit guard, but the callers undo it.
Suggested fix — remove the early-exit in file_utils.py:110-112 and instead handle it at the function level by always including the original:
def get_filename_aliases(filename: str) -> list[str]:
normalized = (filename or "").strip()
if not normalized:
return [filename] if filename else [] # caller gets original back, not []
aliases = [normalized]
...
Or, simpler — document that callers should never pass an empty filename and add an assert in dev mode.
There was a problem hiding this comment.
Good catch. I addressed the empty filename issue, but at the call sites rather than by changing get filename aliases return semantics.
get filename aliases still returns [] for empty or whitespace input (guard remains).
The problematic behavior came from callers doing get filename aliases(filename) or [filename], which could turn empty input into [""].
I removed that fallback in the affected paths and now handle empty alias lists explicitly:
TaskProcessor.check filename exists() returns False when aliases are empty.
TaskProcessor.delete document by filename() no ops (with log) when aliases are empty.
api/documents.check filename exists() returns { "exists": false } early when aliases are empty.
This avoids empty string OpenSearch queries while keeping get filename aliases() focused on normalized alias generation.
Avoid empty-string filename queries and reduce redundant alias rechecks across retries, while clarifying alias behavior for .txt/.md ingestion compatibility. Made-with: Cursor
issue: #1174
this issue happened because .txt uploads were internally renamed and indexed as .md, the duplicate check looked for filename.txt while existing chunks were stored under filename.md, so it missed the match and appended chunks instead of triggering overwrite.
Before:
https://github.com/user-attachments/assets/4a9168fd-fac5-44b5-b92b-d84bbb1412e8
Now:
https://github.com/user-attachments/assets/5e4e5009-2990-4518-ae48-55efb9766dc4