Skip to content

Commit e14a19e

Browse files
committed
chore(gates): simplify native duplication scope and prune removed clones
1 parent e091b4b commit e14a19e

10 files changed

Lines changed: 128 additions & 271 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -246,13 +246,14 @@ Useful direct entry points:
246246
and `linux/`, including tests. No Xcode or JVM is needed.
247247
- `pnpm native-dupes:report` — inspect all native clones without failing on new findings;
248248
writes `.tmp/native-dupes/jscpd-report.json` with both locations and `isNew` markers.
249-
- `pnpm native-dupes:prototype` — compare detection settings and the Duo fixture pair;
250-
see [measurements and baseline review](docs/experiments/native-duplication.md).
251249
- `pnpm native-dupes:baseline` — update accepted clones only after reviewing their source;
252250
do not use a baseline refresh merely to clear a failure.
253251
- `pnpm fallow:all` — full-tree audit, including grandfathered baseline findings
254252
- `pnpm fallow:baseline` — intentionally regenerate the reviewed Fallow health baseline
255253

254+
For native duplication settings, reproduction commands, and baseline pruning, see
255+
[the experiment notes](docs/experiments/native-duplication.md).
256+
256257
See [`docs/agents/testing.md`](docs/agents/testing.md) for check selection and blocked validation; see
257258
[`test authoring`](docs/agents/test-authoring.md) for shared test utilities and test-speed rules. See [gate maintenance](docs/agents/gates.md) for mutation and fuzz harnesses. For real devices, follow
258259
[`docs/agents/device-verification.md`](docs/agents/device-verification.md); a fixture-backed test does

‎docs/experiments/native-duplication.md‎

Lines changed: 72 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -6,16 +6,27 @@ Swift build caches, Android build output, and packaged helpers. No source exclus
66

77
## Reproduce the comparison
88

9+
The table records source commit `a16d4372376bf18eea3dff4cbe85e15fc68e7fc0`.
10+
It is an experiment record, not a target for later revisions. Files below the token minimum do
11+
not contribute to jscpd's analyzed-file count. Percentages are its duplicated-line metric,
12+
not unique lines to remove.
13+
14+
Use jscpd 5.4.2. To replay the recorded inputs without changing this checkout, extract only
15+
tracked source from that commit:
16+
917
```sh
1018
pnpm install --frozen-lockfile
11-
pnpm native-dupes:prototype
19+
pnpm exec jscpd --version
20+
native_dupes_source="$(mktemp -d "${TMPDIR:-/tmp}/agent-device-native-dupes.XXXXXX")"
21+
git archive a16d4372376bf18eea3dff4cbe85e15fc68e7fc0 -- apple android linux \
22+
packages/platform-apple/src/core/__tests__/display-inventory.test.ts \
23+
packages/platform-apple/src/__tests__/simctl-facade.test.ts | tar -x -C "$native_dupes_source"
1224
```
1325

14-
Each variant scans `apple/`, `android/`, and `linux/`, then scans the two TypeScript tests named in
15-
the original report. JSON reports and `summary.json` go to `.tmp/native-dupes-prototypes/`.
16-
The prototype fixes the worker count at two. The table records this source revision, not a target
17-
for later revisions. Percentages are jscpd's duplicated-line metric, not unique lines to remove.
18-
Files below the token minimum do not contribute to its analyzed-file count.
26+
The version command must report `jscpd 5.4.2`. The commands below fix the worker count at two.
27+
`--no-gitignore` lets the tracked-source archive be scanned independently of its temporary path;
28+
the archive contains no ignored build output. Each row has a native scan and a separate scan of
29+
the two TypeScript tests. Reports go to `.tmp/native-dupes-prototypes/`.
1930

2031
| Mode | Tokens / lines | Native clones | Duplicated lines | Duplication | Duo pair clones | Duo panel fragments |
2132
| --- | --- | ---: | ---: | ---: | ---: | ---: |
@@ -28,6 +39,44 @@ Files below the token minimum do not contribute to its analyzed-file count.
2839
| weak + consistent identifier normalization | 50 / 5 | 196 | 2150 | 4.44% | 5 | 0 |
2940
| weak + gaps up to 8 lines | 30 / 3 | 384 | 2960 | 6.10% | 13 | 0 |
3041

42+
Exact commands, in table order:
43+
44+
```sh
45+
# mild-50-5
46+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode mild --min-tokens 50 --min-lines 5 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/mild-50-5-native
47+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode mild --min-tokens 50 --min-lines 5 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/mild-50-5-duo
48+
49+
# weak-50-5
50+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode weak --min-tokens 50 --min-lines 5 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-50-5-native
51+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode weak --min-tokens 50 --min-lines 5 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-50-5-duo
52+
53+
# weak-70-5
54+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode weak --min-tokens 70 --min-lines 5 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-70-5-native
55+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode weak --min-tokens 70 --min-lines 5 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-70-5-duo
56+
57+
# weak-100-8
58+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode weak --min-tokens 100 --min-lines 8 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-100-8-native
59+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode weak --min-tokens 100 --min-lines 8 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-100-8-duo
60+
61+
# weak-30-3
62+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode weak --min-tokens 30 --min-lines 3 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-30-3-native
63+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode weak --min-tokens 30 --min-lines 3 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-30-3-duo
64+
65+
# weak-20-3
66+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode weak --min-tokens 20 --min-lines 3 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-20-3-native
67+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode weak --min-tokens 20 --min-lines 3 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-20-3-duo
68+
69+
# renamed-50-5
70+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode weak --min-tokens 50 --min-lines 5 --ignore-identifiers=consistent --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/renamed-50-5-native
71+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode weak --min-tokens 50 --min-lines 5 --ignore-identifiers=consistent --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/renamed-50-5-duo
72+
73+
# gap-30-3
74+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode weak --min-tokens 30 --min-lines 3 --max-gap-lines 8 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/gap-30-3-native
75+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode weak --min-tokens 30 --min-lines 3 --max-gap-lines 8 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/gap-30-3-duo
76+
77+
rm -rf "$native_dupes_source"
78+
```
79+
3180
The measured native runs took about 170–195 ms each, including Node startup, using two workers.
3281
This is a single timing sample per variant, not a performance benchmark. Linux execution belongs
3382
to the Repo Guards & Compatibility CI job.
@@ -53,16 +102,23 @@ in 5.1.0. A custom comparison script or percentage ceiling is unnecessary. The b
53102
content fingerprints and clone multiplicities. It tolerates line shifts and fails on a third copy
54103
of an accepted pair. A new clone fails even if another clone is removed and the total stays equal.
55104
The executable policy tests cover each language, headers, line shifts, extra copies, replacement
56-
clones, missing baselines, and empty scans.
105+
clones, missing baselines, empty scans, and removed-clone pruning.
57106

58107
The rollout starts with a report-only measurement and saves that result in
59108
`native-dupes-baseline.json`. `check:native-dupes` then rejects new fingerprints or increased
60109
multiplicities. It scans all native files rather than filtering by changed files. This keeps
61110
unchanged files available as comparison targets and needs no Git history in CI. `check:affected`
62-
selects it for native source changes; configuration and baseline changes select the full check set.
111+
selects it for non-documentation changes under the roots in `.jscpd-native.json`. It does not
112+
filter by extension, so new scanner formats cannot escape selection. Some metadata changes also
113+
select this inexpensive scan. Configuration and baseline changes select the full check set.
63114
`--changed` is unsuitable for PR enforcement because it filters by working-tree status, which is
64115
empty after a commit. The normal check never updates the baseline.
65116

117+
After rebasing onto `21f3f50a90f21bf8f994e742cbb7dd295f255d6c`, the initial baseline was
118+
refreshed from main's native source: 82 clones, 854 duplicated lines (1.78%), and 218 analyzed
119+
files. Three fingerprints were replaced by upstream native changes. The table above remains the
120+
historical comparison at `a16d4372376bf18eea3dff4cbe85e15fc68e7fc0`.
121+
66122
## Motivating cases and limits
67123

68124
The closed-Duo fixtures differ at the measured revision. `display-inventory.test.ts` includes
@@ -91,3 +147,11 @@ share an owner. If retaining duplication is intentional, review its source befor
91147
`pnpm native-dupes:baseline`. Review the baseline diff together with the source change. Do not
92148
regenerate it merely to clear a failing check. A detector upgrade or a change to matching options
93149
also needs a fresh comparison and baseline review. Missing baselines and empty scans fail closed.
150+
151+
The normal check does not prune stale fingerprints. If a clone disappears while its fingerprint
152+
remains in the baseline, the same block can be reintroduced up to its saved multiplicity without
153+
failing. When removing a clone, run `pnpm native-dupes:baseline` in the same PR. jscpd's
154+
`--update-baseline` replaces the saved fingerprints with those found by that scan, removing absent
155+
fingerprints and reducing obsolete multiplicities. Review that the refresh removes or reduces
156+
entries; investigate any additions. The executable pruning test proves that removal passes,
157+
an unpruned allowance accepts reintroduction, and a refreshed baseline rejects reintroduction.

‎native-dupes-baseline.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,6 @@
77
"16f64c13af3b5133": 1,
88
"177dd03fcad181e3": 1,
99
"1820c3730c884e03": 1,
10-
"1bcd955ed0310803": 1,
1110
"1e78746beb116a42": 1,
1211
"20171c882520cfa8": 1,
1312
"227ed1e368c386a3": 1,
@@ -18,8 +17,8 @@
1817
"3a4e282cb1e5c9e5": 1,
1918
"3cbf0776f9254f0d": 1,
2019
"3d12b710a26fd356": 1,
20+
"3e739c27a227ed2c": 1,
2121
"411c3f06f49cd0a2": 1,
22-
"524d0e8ddb986bee": 1,
2322
"52af4ac9900c2849": 1,
2423
"54b3cba0fee01e85": 1,
2524
"5642c43795f3f29c": 1,
@@ -47,6 +46,7 @@
4746
"8f44d31bc1be550a": 1,
4847
"93fdf4830756d189": 1,
4948
"950d64ca7c88e560": 1,
49+
"96d008f544ddde7e": 1,
5050
"9cac643baca35232": 1,
5151
"9db36d2dd0a613e1": 1,
5252
"a1e36b17c095ff5d": 1,
@@ -76,11 +76,11 @@
7676
"e88f11085e4e61d5": 1,
7777
"e8984499b65f90d8": 1,
7878
"edaddb383eec4cc2": 1,
79+
"edcf0986a0221682": 1,
7980
"f2cf05f6de02ad76": 1,
8081
"f65edc1e8000a681": 1,
8182
"f901d457d8368e97": 1,
8283
"fc5de2790407f117": 1,
83-
"fd3d1f4c504aa1b4": 1,
8484
"fed244c725eb70f6": 1,
8585
"ff2cfa62ae0daa36": 1
8686
}

‎package.json‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -147,10 +147,9 @@
147147
"fallow:baseline": "fallow health --report-only --save-baseline fallow-baselines/health.json --summary",
148148
"check:fallow": "fallow audit",
149149
"check:native-dupes": "jscpd --config .jscpd-native.json --baseline native-dupes-baseline.json --fail-on-new-clones --fail-on-empty",
150-
"check:native-dupes:test": "node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test scripts/native-dupes/*.test.ts",
150+
"check:native-dupes:test": "node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test scripts/native-dupes/check.test.ts",
151151
"native-dupes:report": "jscpd --config .jscpd-native.json --baseline native-dupes-baseline.json --fail-on-empty",
152152
"native-dupes:baseline": "jscpd --config .jscpd-native.json --baseline native-dupes-baseline.json --update-baseline --fail-on-empty",
153-
"native-dupes:prototype": "node --experimental-strip-types scripts/native-dupes/prototype.ts",
154153
"check:affected": "node --experimental-strip-types scripts/check-affected/run.ts",
155154
"check:affected:test": "node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test scripts/check-affected/*.test.ts",
156155
"gate": "node --experimental-strip-types scripts/gate/run.ts",
@@ -329,7 +328,6 @@
329328
"@agent-device/selectors": "workspace:*",
330329
"@agent-device/session-journal": "workspace:*",
331330
"@agent-device/test-kit": "workspace:*",
332-
"@agent-device/testmu": "workspace:*",
333331
"@agent-device/xml": "workspace:*",
334332
"@arethetypeswrong/cli": "^0.18.5",
335333
"@chenglou/freerange": "^0.0.4",
@@ -367,6 +365,7 @@
367365
"vite": "^8.3.5",
368366
"vitest": "5.0.4",
369367
"yaml": "^2.9.0",
370-
"yauzl": "^3.4.0"
368+
"yauzl": "^3.4.0",
369+
"@agent-device/testmu": "workspace:*"
371370
}
372371
}

‎scripts/check-affected/model.test.ts‎

Lines changed: 18 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,7 @@ test('Android helper change selects the android-helpers build', () => {
156156
'replay-android',
157157
]);
158158
assert.deepEqual(ids(['android/ime-helper/AndroidManifest.xml']), [
159+
'native-dupes',
159160
'android-helpers',
160161
'replay-android',
161162
]);
@@ -454,54 +455,26 @@ test('vitest-related delegates changed paths to Vitest instead of modeling proje
454455

455456
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..');
456457

457-
test('native duplication covers language inputs without selecting metadata or TypeScript', () => {
458-
for (const file of [
459-
'apple/fold-helper/Fold.m',
460-
'apple/snapshot-bridge/Bridge.h',
461-
'android/ime-helper/Input.kt',
462-
'android/ime-helper/Build.kts',
463-
'android/snapshot-helper/Touch.java',
464-
'linux/atspi-dump.py',
465-
]) {
466-
assert.ok(ids([file]).includes('native-dupes'), file);
467-
}
468-
for (const file of [
469-
'apple/runner/README.md',
470-
'android/ime-helper/AndroidManifest.xml',
471-
'packages/platform-apple/src/core/app-resolution.ts',
472-
]) {
473-
assert.ok(!ids([file]).includes('native-dupes'), file);
458+
test('every configured scan root selects native duplication regardless of extension', () => {
459+
const config: { path: string[] } = JSON.parse(
460+
fs.readFileSync(path.join(repoRoot, '.jscpd-native.json'), 'utf8'),
461+
);
462+
for (const root of config.path) {
463+
for (const extension of ['mm', 'future']) {
464+
const file = path.posix.join(root, 'duplication-probe', `Source.${extension}`);
465+
assert.ok(
466+
plan([file]).reasons.some(
467+
(entry) => entry.check === 'native-dupes' && entry.rule === 'own:native-dupes',
468+
),
469+
file,
470+
);
471+
}
472+
const neighbor = path.posix.join(`${root}-neighbor`, 'Source.mm');
473+
assert.ok(!plan([neighbor]).reasons.some((entry) => entry.rule === 'own:native-dupes'));
474474
}
475+
assert.ok(!ids(['packages/platform-apple/src/core/app-resolution.ts']).includes('native-dupes'));
475476
for (const file of ['.jscpd-native.json', 'native-dupes-baseline.json']) {
476477
assert.equal(plan([file]).failOpen, true);
477478
assert.ok(ids([file]).includes('native-dupes-model'), file);
478479
}
479480
});
480-
481-
test('native duplication selection covers every configured root, format, and custom extension', () => {
482-
const config: { path: string[]; format: string[]; formatsExts: Record<string, string[]> } =
483-
JSON.parse(fs.readFileSync(path.join(repoRoot, '.jscpd-native.json'), 'utf8'));
484-
const defaultExtensions: Record<string, string[]> = {
485-
swift: ['swift'],
486-
objectivec: ['m'],
487-
kotlin: ['kt', 'kts'],
488-
java: ['java'],
489-
python: ['py'],
490-
};
491-
for (const format of config.format) {
492-
const defaults = defaultExtensions[format];
493-
assert.ok(defaults, `Declare the scanner extensions for the added ${format} format`);
494-
const extensions = new Set([...defaults, ...(config.formatsExts[format] ?? [])]);
495-
for (const root of config.path) {
496-
for (const extension of extensions) {
497-
const file = `${root}/duplication-probe/Source.${extension}`;
498-
assert.ok(
499-
plan([file]).reasons.some(
500-
(entry) => entry.check === 'native-dupes' && entry.rule === 'own:native-dupes',
501-
),
502-
file,
503-
);
504-
}
505-
}
506-
}
507-
});

‎scripts/check-affected/model.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
// non-.ts fixture whose owning suite cannot be derived). Existing GitHub CI
2020
// remains authoritative; this only optimizes local/agent feedback.
2121

22+
import nativeDupesConfig from '../../.jscpd-native.json' with { type: 'json' };
2223
import { WIRE_SURFACE_FILES } from '../../test/wire-compat/surface.ts';
2324
import { deviceLaneOwnership } from './device-lanes.ts';
2425

@@ -396,7 +397,7 @@ const ownsAppleRunnerBuildSource = (file: string): boolean =>
396397
file.endsWith('.swift');
397398

398399
const nativeDupesOwnership: OwnershipRule = ({ file }) =>
399-
/^(?:apple|android|linux)\/.*\.(?:swift|m|h|kt|kts|java|py)$/.test(file)
400+
nativeDupesConfig.path.some((root) => file.startsWith(`${root}/`))
400401
? [
401402
reason(
402403
'native-dupes',

0 commit comments

Comments
 (0)