Fall back to lazy index when repo exceeds max file limit by SagarSDagdu · Pull Request #10234 · warpdotdev/warp

SagarSDagdu · 2026-05-06T04:52:16Z

Description

Fixes #10207. When initial repo indexing exceeded MAX_FILES_PER_REPO (100k), the repo landed in IndexedRepoState::Failed and the file tree silently rendered an empty entry, hiding populated folders like .agents in monorepos (the reporter has a 153k-file workspace).

Loom for before and after

https://www.loom.com/share/660730d889b6466798b52210ca6f9049

What: On ExceededMaxFileLimit during the full-depth scan in LocalRepoMetadataModel, retry once with max_depth=1 and install the repo as Indexed. The existing per-directory lazy-load path (LAZY_LOAD_FILE_LIMIT=5000) handles expansion. A new RepositoryIndexedWithLimit event surfaces a toast in the file tree view explaining the degraded mode.

Why: The current behavior gives the user no way to browse a too-large repo and no signal that anything went wrong — the folder just looks empty.

How:

crates/repo_metadata/src/local_model.rs: in the build-tree spawn, retry with max_depth=1 on ExceededMaxFileLimit. New event variant RepositoryIndexedWithLimit { path }. Telemetry still records BuildTreeFailed.
crates/repo_metadata/src/wrapper_model.rs: forward the new variant.
app/src/code/file_tree/view.rs: handle the new event with a toast — "Repository is too large to fully index. Subfolders load when expanded." Mirrors the existing show_exceeded_file_limit_toast for individual oversized subfolders.
app/src/{search/files/model.rs, remote_server/server_model.rs, ai/skills/file_watchers/skill_watcher.rs}: add the new variant to existing no-op match arms.

Linked Issue

Closes #10207

The linked issue is labeled ready-to-spec or ready-to-implement.

Screenshots / Videos

N/A — no UI layout changes; only a toast and behavior fix that is hard to demo without a 100k-file repo.

Testing

cargo test -p repo_metadata --lib — 46 passed
cargo test -p warp --lib code::file_tree:: — 41 passed
cargo clippy -p repo_metadata -p warp --lib --tests -- -D warnings — clean
cargo fmt --check — clean

Regression test gap: A direct unit test for the >100k fallback would require parameterizing MAX_FILES_PER_REPO (currently a const) so a test can use a small limit, or creating 100k+ files at test time (impractical). Happy to add a test-only override if reviewer prefers — left out to keep this fix minimal.

Agent Mode

Warp Agent Mode - This PR was created via Warp's AI Agent Mode

When initial repo indexing exceeded MAX_FILES_PER_REPO (100k), the repo landed in IndexedRepoState::Failed and the file tree silently rendered an empty entry, hiding populated folders like .agents in monorepos (reported in warpdotdev#10207 with a 153k-file workspace). Now, on ExceededMaxFileLimit during the full-depth scan, fall back to a single-level scan so the user can still browse the tree — subdirectories load on expand via the existing 5k-per-directory lazy-load path. A new RepositoryIndexedWithLimit event surfaces a toast in the file tree view explaining the degraded mode. Closes warpdotdev#10207

…ded-max-file-limit-fallback

oz-for-oss · 2026-05-06T04:52:29Z

@SagarSDagdu

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss

Overview

This PR falls back to a shallow repository tree when full indexing exceeds the max-file limit and surfaces a degraded-mode toast to the file tree. The core fallback path updates the repository model, but one subscriber now explicitly drops the degraded-mode signal in a way that leaves repo-local skills undiscovered for the oversized repositories this change targets.

Concerns

Project skill discovery scans on RepositoryUpdated, but the fallback tree only contains first-level lazy directories. Provider directories such as .agents/skills are not present until expansion, and the new degraded-mode event is ignored while later lazy-load updates only process queued creations. Oversized repos can therefore become browseable while their repo-local skills remain unloaded.

Verdict

Found: 0 critical, 1 important, 0 suggestions

Request changes

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

In degraded mode the metadata tree is shallow (depth=1), so find_skill_directories_in_tree never sees provider paths like .agents/skills, leaving repo-local skills undiscovered. Add find_top_level_skill_directories_in_filesystem that probes the known skill provider paths directly under the repo root, and handle RepositoryIndexedWithLimit in SkillWatcher: watch the repo and feed the probed directories into spawn_read_skills_from_directories. The repository file watcher catches deeper/nested skill directories as they are added or as subtrees are lazy-loaded. Addresses review feedback on warpdotdev#10234.

SagarSDagdu · 2026-05-06T05:05:39Z

/oz-review Fixed the previously mentioned issue, please re-review

oz-for-oss · 2026-05-06T05:05:52Z

@SagarSDagdu

I'm re-reviewing this pull request in response to a review request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss

Overview

This PR retries oversized local repo indexing with a shallow tree, forwards a degraded-index event, and surfaces the degraded mode through file-tree/skill-watcher consumers.

Concerns

The new file-tree toast handler discards the repository path, so a global repo metadata event can show the toast in file tree views or windows that are not displaying the large repo.
This PR adds user-visible toast copy but does not include screenshots or video; for faster review, please upload visual evidence of the toast/behavior.

Verdict

Found: 0 critical, 2 important, 0 suggestions

Request changes

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

RepoMetadataModel events are global and fire in every file-tree view, so unconditionally showing the toast popped it in unrelated windows. Only toast when the affected repo is one of this view's roots (either directly or as the backing entry of an absorbed root). Addresses review feedback on warpdotdev#10234.

SagarSDagdu · 2026-05-06T06:01:12Z

/oz-review Fixed all the mentioned issues and resolved comments, please re-review

oz-for-oss · 2026-05-06T06:01:26Z

@SagarSDagdu

I'm re-reviewing this pull request in response to a review request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss

Overview

This PR falls back to a shallow repo metadata index when the full scan exceeds the file limit, forwards a new degraded-index event, surfaces a file-tree toast, and adds filesystem probing for top-level skill provider directories.

Concerns

The new filesystem skill probe is only wired into the async RepositoryIndexedWithLimit event path; the synchronous SkillWatcher::read_skills_for_repos path still reads only the shallow metadata tree, so cloud-environment initial skill loading can still miss repo skills in degraded mode.
This is user-visible behavior and copy, but the PR has no screenshot or video evidence. For faster review, please upload screenshots or a video of the feature working end to end.

Verdict

Found: 0 critical, 2 important, 0 suggestions

Request changes

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

`SkillWatcher::read_skills_for_repos` is the synchronous entry point used by cloud-environment skill loading; it relied solely on `find_skill_directories_in_tree`, so degraded-indexed repos missed their skills there too. Also probe top-level skill provider paths via the filesystem and dedupe before reading. Addresses review feedback on warpdotdev#10234.

SagarSDagdu · 2026-05-06T06:52:35Z

/oz-review Fixed all the mentioned issues and resolved comments, also attached a loom video for the demo.
please re-review

oz-for-oss · 2026-05-06T06:52:49Z

@SagarSDagdu

I'm re-reviewing this pull request in response to a review request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss

Overview

This PR adds a degraded indexing fallback for repositories that exceed the full-tree file limit, forwards a new metadata event, and surfaces a file-tree toast while preserving lazy directory expansion.

Concerns

The new direct filesystem probe for top-level skill provider directories bypasses the metadata tree filtering path, so ignored .agents/skills or .claude/skills directories can now be loaded as project skills.

Security

Ignored repo-local skill provider directories can now be activated without the gitignore filtering used by the existing metadata-tree discovery path.

Verdict

Found: 0 critical, 1 important, 0 suggestions

Request changes

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

The fallback probe accepted any `<repo>/<provider>/skills` directory that existed on disk, even if the repo's gitignore excluded it (e.g. a project that vendors a third-party `.claude/skills`). The tree-based path always honored gitignore via `get_repo_contents`'s `include_ignored=false` default, so degraded mode was strictly more permissive than full-depth mode. Load `gitignores_for_directory(repo_path)` and reject candidates that match (with `check_ancestors=true`, mirroring `Entry::build_tree`'s filter) before treating them as project skills. Addresses review feedback on warpdotdev#10234.

SagarSDagdu · 2026-05-06T07:05:07Z

/oz-review

oz-for-oss · 2026-05-06T07:05:22Z

@SagarSDagdu

I'm re-reviewing this pull request in response to a review request.

You can view the conversation on Warp.

I reviewed this pull request and requested human review from: @moirahuang.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

Oz no longer requests changes for this pull request after the latest automated review.

oz-for-oss

Overview

This PR adds a degraded indexing fallback for repositories that exceed the full-tree file limit, forwards the new metadata event, shows a file-tree toast, and adds a direct root skill-directory probe for degraded repos.

Concerns

Non-blocking: the toast visibility check can miss views rooted at deeper descendants of the oversized repo when those descendants are not present in the shallow fallback tree.

Verdict

Found: 0 critical, 0 important, 1 suggestions

Approve with nits

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss · 2026-05-06T07:14:23Z

+                id: RepositoryIdentifier::Local(std_path),
+            } => {
+                // RepoMetadataModel is global; only toast in views that display this repo.
+                let displays_repo = self.root_directories.contains_key(std_path)


💡 [SUGGESTION] Include displayed descendant roots in this check; if the file tree is rooted inside a too-large repo but that path is not present in the shallow fallback tree, this currently skips the degraded-mode toast.

moirahuang · 2026-05-06T17:10:27Z

HI @SagarSDagdu, we really appreciate the initiative on the newly filed issue. Per the contribution guidelines, the Warp team will manually triage and add a ready-to-spec tag when the issue is ready for a spec.

SagarSDagdu added 2 commits May 6, 2026 10:18

Merge remote-tracking branch 'origin/master' into fix/file-tree-excee…

882b2c9

…ded-max-file-limit-fallback

cla-bot Bot added the cla-signed label May 6, 2026

github-actions Bot added the external-contributor Indicates that a PR has been opened by someone outside the Warp team. label May 6, 2026

oz-for-oss Bot previously requested changes May 6, 2026

View reviewed changes

Comment thread app/src/ai/skills/file_watchers/skill_watcher.rs

oz-for-oss Bot previously requested changes May 6, 2026

View reviewed changes

Comment thread app/src/code/file_tree/view.rs Outdated

Comment thread app/src/code/file_tree/view.rs

oz-for-oss Bot previously requested changes May 6, 2026

View reviewed changes

Comment thread app/src/ai/skills/file_watchers/utils.rs

Comment thread app/src/code/file_tree/view.rs

oz-for-oss Bot previously requested changes May 6, 2026

View reviewed changes

Comment thread app/src/ai/skills/file_watchers/utils.rs

oz-for-oss Bot reviewed May 6, 2026

View reviewed changes

oz-for-oss Bot requested a review from moirahuang May 6, 2026 07:14

Conversation

SagarSDagdu commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Loom for before and after

Linked Issue

Screenshots / Videos

Testing

Agent Mode

Uh oh!

oz-for-oss Bot commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot left a comment

Choose a reason for hiding this comment

Overview

Concerns

Verdict

Uh oh!

Uh oh!

SagarSDagdu commented May 6, 2026

Uh oh!

oz-for-oss Bot commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot left a comment

Choose a reason for hiding this comment

Overview

Concerns

Verdict

Uh oh!

Uh oh!

Uh oh!

SagarSDagdu commented May 6, 2026

Uh oh!

oz-for-oss Bot commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot left a comment

Choose a reason for hiding this comment

Overview

Concerns

Verdict

Uh oh!

Uh oh!

Uh oh!

SagarSDagdu commented May 6, 2026

Uh oh!

oz-for-oss Bot commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot left a comment

Choose a reason for hiding this comment

Overview

Concerns

Security

Verdict

Uh oh!

Uh oh!

SagarSDagdu commented May 6, 2026

Uh oh!

oz-for-oss Bot commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot left a comment

Choose a reason for hiding this comment

Overview

Concerns

Verdict

Uh oh!

oz-for-oss Bot May 6, 2026

Choose a reason for hiding this comment

Uh oh!

moirahuang commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

SagarSDagdu commented May 6, 2026 •

edited

Loading

oz-for-oss Bot commented May 6, 2026 •

edited

Loading

oz-for-oss Bot commented May 6, 2026 •

edited

Loading

oz-for-oss Bot commented May 6, 2026 •

edited

Loading

oz-for-oss Bot commented May 6, 2026 •

edited

Loading

oz-for-oss Bot commented May 6, 2026 •

edited

Loading

moirahuang commented May 6, 2026 •

edited

Loading