Repository navigation
Commit 8569c13
committed
perf(adapter): parallelize copilot OTEL and database loaders
Adopt read_files_parallel in the remaining loaders that iterate over
multiple sources: copilot (OTEL files) and the SQLite-backed goose,
hermes, and kilo loaders, which can each see more than one database when
multiple data dirs / homes / channels are configured.
Each database is opened on its own read-only connection inside the
worker, and the sequential dedup pass runs over the parallel results in
their original path order, so the surviving record per key is identical
to the single-threaded read. Within a single database the row scan stays
sequential (inherent to one SQLite connection); the win is overlapping
multiple sources and keeping all loaders on one shared code path.
Output verified byte-identical to the previous release across report
modes on synthesized multi-database fixtures with overlapping ids, with
single-thread and multi-thread runs matching.1 parent 1dce684 commit 8569c13
4 files changed
Lines changed: 63 additions & 16 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
| 11 | + | |
| 12 | + | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
| |||
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
30 | 46 | | |
31 | | - | |
32 | | - | |
| 47 | + | |
| 48 | + | |
33 | 49 | | |
34 | 50 | | |
35 | 51 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
6 | 8 | | |
7 | 9 | | |
8 | 10 | | |
| |||
31 | 33 | | |
32 | 34 | | |
33 | 35 | | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
34 | 52 | | |
35 | 53 | | |
36 | | - | |
37 | | - | |
| 54 | + | |
| 55 | + | |
38 | 56 | | |
39 | 57 | | |
40 | 58 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
18 | 25 | | |
19 | 26 | | |
20 | | - | |
21 | | - | |
| 27 | + | |
| 28 | + | |
22 | 29 | | |
23 | 30 | | |
24 | 31 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
6 | 8 | | |
7 | 9 | | |
8 | 10 | | |
| |||
17 | 19 | | |
18 | 20 | | |
19 | 21 | | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
20 | 29 | | |
21 | 30 | | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
| 31 | + | |
| 32 | + | |
27 | 33 | | |
28 | 34 | | |
29 | 35 | | |
| |||
0 commit comments