Skip to content

Commit 6f5b86a

Browse files
authored
chore(gates): detect new native duplication with jscpd (#3980)
* chore(gates): detect new native clones against a reviewed baseline * chore(gates): preserve native duplication report artifacts * chore(gates): guard native scan scope and portable prototype counts * chore(gates): simplify native duplication scope and prune removed clones
1 parent 518b827 commit 6f5b86a

11 files changed

Lines changed: 668 additions & 1 deletion

File tree

‎.github/workflows/ci.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -164,6 +164,23 @@ jobs:
164164
--base
165165
${{ env.FALLOW_BASE }}
166166
167+
- name: Test native clone detection and baseline policy
168+
uses: ./.github/actions/run-gate
169+
with: { gate: native-dupes-model }
170+
171+
- name: Check new native code clones
172+
uses: ./.github/actions/run-gate
173+
with: { gate: native-dupes }
174+
175+
- name: Upload native duplication report
176+
if: always()
177+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
178+
with:
179+
name: native-dupes-report
180+
include-hidden-files: true
181+
path: .tmp/native-dupes/jscpd-report.json
182+
if-no-files-found: ignore
183+
167184
- name: Check import-direction DAG
168185
# Structured import-direction lint over the resolved graph. See
169186
# scripts/layering/check.ts and CONTEXT.md (Architecture: folder DAG +

‎.jscpd-native.json‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"path": ["apple", "android", "linux"],
3+
"format": ["swift", "objectivec", "kotlin", "java", "python"],
4+
"formatsExts": { "objectivec": ["m", "h"] },
5+
"mode": "weak",
6+
"minTokens": 50,
7+
"minLines": 5,
8+
"reporters": ["console", "json"],
9+
"output": ".tmp/native-dupes",
10+
"noColors": true,
11+
"noTips": true
12+
}

‎CONTRIBUTING.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -241,9 +241,19 @@ Useful direct entry points:
241241
fails when `coverage/` was not written by that command on this commit. `--all` reports every
242242
function and never fails: that is `main`'s backlog. Both list the per-function CRAP ceilings from
243243
`.fallowrc.json` they honoured. `check:fallow` leaves CRAP off without coverage.
244+
- `pnpm check:native-dupes` — reject new native code clones against the reviewed baseline;
245+
scans Swift, Objective-C (including headers), Kotlin, Java, and Python in `apple/`, `android/`,
246+
and `linux/`, including tests. No Xcode or JVM is needed.
247+
- `pnpm native-dupes:report` — inspect all native clones without failing on new findings;
248+
writes `.tmp/native-dupes/jscpd-report.json` with both locations and `isNew` markers.
249+
- `pnpm native-dupes:baseline` — update accepted clones only after reviewing their source;
250+
do not use a baseline refresh merely to clear a failure.
244251
- `pnpm fallow:all` — full-tree audit, including grandfathered baseline findings
245252
- `pnpm fallow:baseline` — intentionally regenerate the reviewed Fallow health baseline
246253

254+
For native duplication settings, reproduction commands, and baseline pruning, see
255+
[the experiment notes](docs/experiments/native-duplication.md).
256+
247257
See [`docs/agents/testing.md`](docs/agents/testing.md) for check selection and blocked validation; see
248258
[`test authoring`](docs/agents/test-authoring.md) for shared test utilities and test-speed rules. See [gate maintenance](docs/agents/gates.md) for mutation and fuzz harnesses. For real devices, follow
249259
[`docs/agents/device-verification.md`](docs/agents/device-verification.md); a fixture-backed test does
Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
# Native duplication defaults
2+
3+
Measured at `a16d4372376bf18eea3dff4cbe85e15fc68e7fc0` with jscpd 5.4.2 on macOS arm64.
4+
The scans include native tests and Objective-C headers. They respect `.gitignore`, which excludes
5+
Swift build caches, Android build output, and packaged helpers. No source exclusions were added.
6+
7+
## Reproduce the comparison
8+
9+
The table records source commit `a16d4372376bf18eea3dff4cbe85e15fc68e7fc0`.
10+
It is an experiment record, not a target for later revisions. Files below the token minimum do
11+
not contribute to jscpd's analyzed-file count. Percentages are its duplicated-line metric,
12+
not unique lines to remove.
13+
14+
Use jscpd 5.4.2. To replay the recorded inputs without changing this checkout, extract only
15+
tracked source from that commit:
16+
17+
```sh
18+
pnpm install --frozen-lockfile
19+
pnpm exec jscpd --version
20+
native_dupes_source="$(mktemp -d "${TMPDIR:-/tmp}/agent-device-native-dupes.XXXXXX")"
21+
git archive a16d4372376bf18eea3dff4cbe85e15fc68e7fc0 -- apple android linux \
22+
packages/platform-apple/src/core/__tests__/display-inventory.test.ts \
23+
packages/platform-apple/src/__tests__/simctl-facade.test.ts | tar -x -C "$native_dupes_source"
24+
```
25+
26+
The version command must report `jscpd 5.4.2`. The commands below fix the worker count at two.
27+
`--no-gitignore` lets the tracked-source archive be scanned independently of its temporary path;
28+
the archive contains no ignored build output. Each row has a native scan and a separate scan of
29+
the two TypeScript tests. Reports go to `.tmp/native-dupes-prototypes/`.
30+
31+
| Mode | Tokens / lines | Native clones | Duplicated lines | Duplication | Duo pair clones | Duo panel fragments |
32+
| --- | --- | ---: | ---: | ---: | ---: | ---: |
33+
| mild | 50 / 5 | 85 | 873 | 1.80% | 4 | 0 |
34+
| weak | 50 / 5 | 82 | 857 | 1.77% | 4 | 0 |
35+
| weak | 70 / 5 | 18 | 228 | 0.47% | 2 | 0 |
36+
| weak | 100 / 8 | 0 | 0 | 0.00% | 0 | 0 |
37+
| weak | 30 / 3 | 401 | 2971 | 6.12% | 16 | 0 |
38+
| weak | 20 / 3 | 860 | 5435 | 11.20% | 31 | 2 |
39+
| weak + consistent identifier normalization | 50 / 5 | 196 | 2150 | 4.44% | 5 | 0 |
40+
| weak + gaps up to 8 lines | 30 / 3 | 384 | 2960 | 6.10% | 13 | 0 |
41+
42+
Exact commands, in table order:
43+
44+
```sh
45+
# mild-50-5
46+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode mild --min-tokens 50 --min-lines 5 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/mild-50-5-native
47+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode mild --min-tokens 50 --min-lines 5 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/mild-50-5-duo
48+
49+
# weak-50-5
50+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode weak --min-tokens 50 --min-lines 5 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-50-5-native
51+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode weak --min-tokens 50 --min-lines 5 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-50-5-duo
52+
53+
# weak-70-5
54+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode weak --min-tokens 70 --min-lines 5 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-70-5-native
55+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode weak --min-tokens 70 --min-lines 5 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-70-5-duo
56+
57+
# weak-100-8
58+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode weak --min-tokens 100 --min-lines 8 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-100-8-native
59+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode weak --min-tokens 100 --min-lines 8 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-100-8-duo
60+
61+
# weak-30-3
62+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode weak --min-tokens 30 --min-lines 3 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-30-3-native
63+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode weak --min-tokens 30 --min-lines 3 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-30-3-duo
64+
65+
# weak-20-3
66+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode weak --min-tokens 20 --min-lines 3 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-20-3-native
67+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode weak --min-tokens 20 --min-lines 3 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/weak-20-3-duo
68+
69+
# renamed-50-5
70+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode weak --min-tokens 50 --min-lines 5 --ignore-identifiers=consistent --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/renamed-50-5-native
71+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode weak --min-tokens 50 --min-lines 5 --ignore-identifiers=consistent --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/renamed-50-5-duo
72+
73+
# gap-30-3
74+
pnpm exec jscpd "$native_dupes_source/apple" "$native_dupes_source/android" "$native_dupes_source/linux" --format swift,objectivec,kotlin,java,python --formats-exts objectivec:m,h --mode weak --min-tokens 30 --min-lines 3 --max-gap-lines 8 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/gap-30-3-native
75+
pnpm exec jscpd "$native_dupes_source/packages/platform-apple/src/core/__tests__/display-inventory.test.ts" "$native_dupes_source/packages/platform-apple/src/__tests__/simctl-facade.test.ts" --format typescript --mode weak --min-tokens 30 --min-lines 3 --max-gap-lines 8 --workers 2 --no-gitignore --absolute --reporters json --output .tmp/native-dupes-prototypes/gap-30-3-duo
76+
77+
rm -rf "$native_dupes_source"
78+
```
79+
80+
The measured native runs took about 170–195 ms each, including Node startup, using two workers.
81+
This is a single timing sample per variant, not a performance benchmark. Linux execution belongs
82+
to the Repo Guards & Compatibility CI job.
83+
84+
## Selected policy
85+
86+
Use weak mode with a minimum of 50 tokens and 5 lines. Weak mode removes whitespace and comments
87+
from matching. The default mild mode adds three findings without improving either motivating
88+
example. At 70 tokens, all seven Java findings disappear, including repeated snapshot capture and
89+
error response handling. At 100 tokens and 8 lines, the native report is empty. At 30 tokens and
90+
3 lines, imports, small guards, and repeated test setup dominate a much larger review backlog.
91+
Identifier normalization also expands that backlog and still misses the Duo panels.
92+
93+
The selected scan analyzes 218 files and reports 82 clones: 68 Swift, 7 Java, 7 Objective-C, and
94+
0 Python. There are no Kotlin source files in the measured trees; an executable fixture proves
95+
Kotlin detection. The Objective-C extension mapping includes `.h`, because those headers contain
96+
executable geometry code as well as declarations. Five header findings are declaration/implementation
97+
pairs. Existing findings also include deliberately similar tests and Swift package manifests.
98+
The baseline records these findings without declaring them defects or requiring extraction.
99+
100+
jscpd 5.4.2 has a [built-in fingerprint baseline](https://jscpd.dev/guides/baseline), introduced
101+
in 5.1.0. A custom comparison script or percentage ceiling is unnecessary. The baseline records
102+
content fingerprints and clone multiplicities. It tolerates line shifts and fails on a third copy
103+
of an accepted pair. A new clone fails even if another clone is removed and the total stays equal.
104+
The executable policy tests cover each language, headers, line shifts, extra copies, replacement
105+
clones, missing baselines, empty scans, and removed-clone pruning.
106+
107+
The rollout starts with a report-only measurement and saves that result in
108+
`native-dupes-baseline.json`. `check:native-dupes` then rejects new fingerprints or increased
109+
multiplicities. It scans all native files rather than filtering by changed files. This keeps
110+
unchanged files available as comparison targets and needs no Git history in CI. `check:affected`
111+
selects it for non-documentation changes under the roots in `.jscpd-native.json`. It does not
112+
filter by extension, so new scanner formats cannot escape selection. Some metadata changes also
113+
select this inexpensive scan. Configuration and baseline changes select the full check set.
114+
`--changed` is unsuitable for PR enforcement because it filters by working-tree status, which is
115+
empty after a commit. The normal check never updates the baseline.
116+
117+
After rebasing onto `21f3f50a90f21bf8f994e742cbb7dd295f255d6c`, the initial baseline was
118+
refreshed from main's native source: 82 clones, 854 duplicated lines (1.78%), and 218 analyzed
119+
files. Three fingerprints were replaced by upstream native changes. The table above remains the
120+
historical comparison at `a16d4372376bf18eea3dff4cbe85e15fc68e7fc0`.
121+
122+
## Motivating cases and limits
123+
124+
The closed-Duo fixtures differ at the measured revision. `display-inventory.test.ts` includes
125+
bounds, orientation, and unique IDs absent from `simctl-facade.test.ts`. At 50 / 5, jscpd finds
126+
shared imports and the separate single-panel fixture, but no shared closed-Duo panel fragment.
127+
At 20 / 3, it finds two 26-token panel fragments, at `simctl-facade.test.ts:36–40` and `46–50`
128+
against `display-inventory.test.ts:76–80` and `93–97`. It also reports 860 native clones. Enabling
129+
gap merging cannot recover fragments below its seed token minimum. TypeScript remains owned by
130+
Fallow; this experiment does not add a second TypeScript enforcement policy.
131+
132+
No matrix variant finds a clone in the checked-in `Fold.m`. A separate 10-token / 2-line probe
133+
finds two tiny repeated fragments (lines 165–167 against 230–232, and 214–216 against 218–220).
134+
Those are not evidence that it would catch the historical seven-line case; that original duplicated
135+
revision was not replayed. Keep small or reimplemented logic in code review. Renamed or partially
136+
edited copies can evade the selected exact-token policy. Reported blocks need human assessment,
137+
especially test setup and Objective-C declarations.
138+
139+
## Inspect or change the baseline
140+
141+
Run `pnpm native-dupes:report` to inspect findings without failing on new clones. Both report and
142+
check write `.tmp/native-dupes/jscpd-report.json`; each clone has an `isNew` field. CI uploads the
143+
check report as `native-dupes-report` even when the check fails.
144+
145+
When a check reports a new clone, compare both locations and remove the repeated block when they
146+
share an owner. If retaining duplication is intentional, review its source before running
147+
`pnpm native-dupes:baseline`. Review the baseline diff together with the source change. Do not
148+
regenerate it merely to clear a failing check. A detector upgrade or a change to matching options
149+
also needs a fresh comparison and baseline review. Missing baselines and empty scans fail closed.
150+
151+
The normal check does not prune stale fingerprints. If a clone disappears while its fingerprint
152+
remains in the baseline, the same block can be reintroduced up to its saved multiplicity without
153+
failing. When removing a clone, run `pnpm native-dupes:baseline` in the same PR. jscpd's
154+
`--update-baseline` replaces the saved fingerprints with those found by that scan, removing absent
155+
fingerprints and reducing obsolete multiplicities. Review that the refresh removes or reduces
156+
entries; investigate any additions. The executable pruning test proves that removal passes,
157+
an unpruned allowance accepts reintroduction, and a refreshed baseline rejects reintroduction.

‎native-dupes-baseline.json‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
{
2+
"version": 1,
3+
"fingerprints": {
4+
"08241ae1747c89ca": 1,
5+
"0c4f1a12fc6266db": 1,
6+
"13e6519917e53aab": 1,
7+
"16f64c13af3b5133": 1,
8+
"177dd03fcad181e3": 1,
9+
"1820c3730c884e03": 1,
10+
"1e78746beb116a42": 1,
11+
"20171c882520cfa8": 1,
12+
"227ed1e368c386a3": 1,
13+
"27eb86edc47396c2": 1,
14+
"2bb560fa44bc0244": 1,
15+
"2ebd28fb2fcbff63": 1,
16+
"320b08ffc8a201a2": 1,
17+
"3a4e282cb1e5c9e5": 1,
18+
"3cbf0776f9254f0d": 1,
19+
"3d12b710a26fd356": 1,
20+
"3e739c27a227ed2c": 1,
21+
"411c3f06f49cd0a2": 1,
22+
"52af4ac9900c2849": 1,
23+
"54b3cba0fee01e85": 1,
24+
"5642c43795f3f29c": 1,
25+
"5fa328e6b670b2b2": 1,
26+
"6102c64469fb953e": 1,
27+
"616ccefa6658d7d2": 1,
28+
"65cd2c6cb3d72ad2": 1,
29+
"65cd54e3cc7ae70b": 1,
30+
"65e1ce685b7d8fec": 1,
31+
"680fc5b248058fe3": 1,
32+
"681fecffbcf9ab84": 1,
33+
"6bd1eae209b54d99": 1,
34+
"6bf827b16ebbcdf9": 1,
35+
"707b34f8e5cb3dec": 1,
36+
"734566f43c6eeead": 1,
37+
"73ae12f493ebe4ea": 1,
38+
"764a7f81b4003392": 1,
39+
"7aa11fc25854039d": 1,
40+
"7ba42cc0416d2c0b": 1,
41+
"7cb430dbae9d7106": 1,
42+
"8054ef6981eda322": 1,
43+
"851cc95e5042cc92": 1,
44+
"8792a45dec33a4ff": 1,
45+
"89044cc861f67793": 1,
46+
"8f44d31bc1be550a": 1,
47+
"93fdf4830756d189": 1,
48+
"950d64ca7c88e560": 1,
49+
"96d008f544ddde7e": 1,
50+
"9cac643baca35232": 1,
51+
"9db36d2dd0a613e1": 1,
52+
"a1e36b17c095ff5d": 1,
53+
"a579c1b5b09a0ee4": 1,
54+
"ab2e4a3493b14737": 1,
55+
"abb89aabf30035db": 1,
56+
"af0b0f46b52c4b61": 1,
57+
"b407499133c2b5a4": 1,
58+
"b559829f47faaf3e": 1,
59+
"b7db5591789370e5": 1,
60+
"bbb5e2b410e79619": 1,
61+
"bce350f674af29b1": 1,
62+
"c1bd7ebca2d95036": 1,
63+
"c64b76f41c6604bd": 1,
64+
"cf6e4f135bedee4a": 1,
65+
"d2fc7241663fd1da": 1,
66+
"d52956d1f66b005e": 1,
67+
"d65d2069d5c1284c": 1,
68+
"d891ba5c8127fe48": 1,
69+
"db5122c549f1ddec": 1,
70+
"de2f09c7e4f92a19": 1,
71+
"dff4eab801b90d8f": 1,
72+
"e1956821626a2308": 1,
73+
"e4ae6f992718e4bd": 1,
74+
"e7f1b8db1990c7e0": 1,
75+
"e7fc04b26c3b63d2": 1,
76+
"e88f11085e4e61d5": 1,
77+
"e8984499b65f90d8": 1,
78+
"edaddb383eec4cc2": 1,
79+
"edcf0986a0221682": 1,
80+
"f2cf05f6de02ad76": 1,
81+
"f65edc1e8000a681": 1,
82+
"f901d457d8368e97": 1,
83+
"fc5de2790407f117": 1,
84+
"fed244c725eb70f6": 1,
85+
"ff2cfa62ae0daa36": 1
86+
}
87+
}

0 commit comments

Comments
 (0)