Skip to content

Commit 70a79ef

Browse files
authored
Validate precache_manifest.json (F-07, F-08) and --source-maps in S12/S15 (#8)
* Validate `precache_manifest.json` (`F-07`, `F-08`) and `--source-maps` in S12/S15 - Add `F-07` (`precache_manifest.json` revalidation check) and `F-08` (`precache_manifest.json` entry `urlHashed` / hash integrity, missing hashed entrypoints, and `.map` / `.symbols` / `.info.json` exclusion) to `UrlChecker`. - Add `sourceMaps` (`--source-maps`) to `BuildOptions` in `redteam_harness` and enable it in `S12` (`PrecacheManifestScenario`) and `S15` (`CacheCheckDogfoodScenario`) to catch `--web-content-hash` + `--source-maps` hash corruption (flutter/flutter#193601, flutter/flutter#193614). * Use Dart 3 map pattern matching in `UrlChecker` precache manifest validation
1 parent fde7757 commit 70a79ef

7 files changed

Lines changed: 486 additions & 53 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
## 0.0.2-wip
22

3+
- Add `precache_manifest.json` probing (`F-07` revalidation + `F-08` content-hash / `urlHashed` consistency check) in `UrlChecker`, and add `BuildOptions(sourceMaps: true)` (`--source-maps`) + `sourceMappingURL` resolution checks to `S12` (`PrecacheManifestScenario`) in `tool/redteam_harness` (`flutter/flutter#193601`).
34
- Add `--backend=firebase-emulator` support in `tool/redteam_harness` (`HostingServer(useFirebaseEmulator: true)`) and recognize local Firebase `superstatic` emulator signatures (`127.0.0.1` / `localhost` 404 header middleware order, unquoted MD5 ETags, and uncompressed local responses) in `UrlChecker` (`E-01`, `E-02`, `E-03`).
45
- Add first-class WebAssembly (`dart2wasm`) target detection (`W-01`, `--[no-]wasm`) in `UrlChecker` and `check` CLI.
56
- Probe synthetic missing `.wasm` hashed entrypoints (`/main.dart.00000000.wasm`) for SPA HTML rewrite traps (`F-05`) and negative-cache `404` poisoning (`F-06`) (`W-02`).

‎docs/design.md‎

Lines changed: 18 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -103,23 +103,24 @@ flowchart TD
103103

104104
### 3.2 Rule Catalog (`CheckFinding` IDs)
105105

106-
| Rule ID | Category | Severity | Verification Condition |
107-
| :------ | :----------------------------------------------------------------- | :--------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
108-
| `F-01` | Root HTML (`index.html` / `/`) | `fail` / `ok` | Browser `max-age <= 0` (or `no-cache`/`no-store`), not `immutable`, and shared CDN TTL (`CDN-Cache-Control` / `Surrogate-Control` / `s-maxage`) `<= 0`. |
109-
| `F-02` | Bootloader (`flutter_bootstrap.js`) | `fail` / `ok` | Must revalidate (`max-age=0, must-revalidate` or `no-cache`) with shared CDN TTL `<= 0`. |
110-
| `F-03` | App Bundles (`mainJsPath`, `mainWasmPath`, `jsSupportRuntimePath`) | `fail` / `warn` / `ok` | Hashed bundles require `max-age >= 31536000`, `immutable`, and not `private`/`no-cache`/`no-store` (`warn` if missing `immutable`). Unhashed bundles must revalidate (`max-age <= 0`). |
111-
| `F-04` | Hashed Manifests (`assetManifest`, `fontManifest`) | `fail` / `warn` / `ok` | When `buildConfig` specifies content-hashed manifest paths, enforces `max-age >= 31536000, immutable`. |
112-
| `F-05` | SPA Catch-All Rewrite Poisoning | `fail` / `ok` | Probes `/main.dart.00000000.js`, `/main.dart.00000000.wasm`, and `/assets/__cache_check_missing__.00000000.png` (plus guards `/flutter_bootstrap.js` and manifest probes). Fails if any returns `200` / HTML. |
113-
| `F-06` | Negative-Cache (`404`) TTL | `fail` / `warn` / `ok` | Inspects `404` responses on missing hashed paths. Fails if `immutable` or effective TTL `> 60s`; warns if `0 < TTL <= 60s`. |
114-
| `F-07` | Unhashed Manifests (`assets/AssetManifest.bin.json`) | `fail` / `ok` | Probes `assets/AssetManifest.bin.json` when unhashed and fails if `max-age > 0` or missing explicit revalidation. |
115-
| `F-12` | Firebase Fastly `return(pass)` | `warn` | Emitted when `HostPlatform.firebase` is detected (`Vary: x-fh-requested-host`) and root/bootloader uses `no-cache`, `no-store`, or `private` instead of `max-age=0, must-revalidate`. |
116-
| `W-01` | WebAssembly (`dart2wasm`) Target Detection | `warn` / `ok` | Verifies `_flutter.buildConfig` in `flutter_bootstrap.js` includes a `dart2wasm` (`mainWasmPath`) target alongside `dart2js` (warns when missing unless `--no-wasm` is passed). |
117-
| `M-01` | WebAssembly MIME Type | `fail` / `ok` | `.wasm` responses must carry `Content-Type: application/wasm` (required by `WebAssembly.instantiateStreaming`). |
118-
| `M-02` | JavaScript Module MIME Type | `fail` / `ok` | `.mjs` and `.js` responses must carry `text/javascript` or `application/javascript`. |
119-
| `C-01` | Payload Compression | `warn` / `ok` | Payloads exceeding `minCompressionBytes` (`1024` bytes) should be compressed with `br`, `gzip`, or `zstd`. |
120-
| `C-02` | Compression Cache Key (`Vary`) | `warn` | Compressed payloads must include `Vary: Accept-Encoding` so shared caches do not serve mismatched encodings. |
121-
| `R-01` | Revalidation Validators | `warn` | `index.html` should provide `ETag` or `Last-Modified` headers for conditional requests. |
122-
| `R-02` | Conditional `304 Not Modified` | `warn` / `ok` | Sends follow-up `GET` with `If-None-Match: <etag>` and verifies `304 Not Modified` (warns if origin returns `200 OK`). |
106+
| Rule ID | Category | Severity | Verification Condition |
107+
| :------ | :----------------------------------------------------------------------------- | :--------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
108+
| `F-01` | Root HTML (`index.html` / `/`) | `fail` / `ok` | Browser `max-age <= 0` (or `no-cache`/`no-store`), not `immutable`, and shared CDN TTL (`CDN-Cache-Control` / `Surrogate-Control` / `s-maxage`) `<= 0`. |
109+
| `F-02` | Bootloader (`flutter_bootstrap.js`) | `fail` / `ok` | Must revalidate (`max-age=0, must-revalidate` or `no-cache`) with shared CDN TTL `<= 0`. |
110+
| `F-03` | App Bundles (`mainJsPath`, `mainWasmPath`, `jsSupportRuntimePath`) | `fail` / `warn` / `ok` | Hashed bundles require `max-age >= 31536000`, `immutable`, and not `private`/`no-cache`/`no-store` (`warn` if missing `immutable`). Unhashed bundles must revalidate (`max-age <= 0`). |
111+
| `F-04` | Hashed Manifests (`assetManifest`, `fontManifest`) | `fail` / `warn` / `ok` | When `buildConfig` specifies content-hashed manifest paths, enforces `max-age >= 31536000, immutable`. |
112+
| `F-05` | SPA Catch-All Rewrite Poisoning | `fail` / `ok` | Probes `/main.dart.00000000.js`, `/main.dart.00000000.wasm`, and `/assets/__cache_check_missing__.00000000.png` (plus guards `/flutter_bootstrap.js` and manifest probes). Fails if any returns `200` / HTML. |
113+
| `F-06` | Negative-Cache (`404`) TTL | `fail` / `warn` / `ok` | Inspects `404` responses on missing hashed paths. Fails if `immutable` or effective TTL `> 60s`; warns if `0 < TTL <= 60s`. |
114+
| `F-07` | Unhashed Manifests (`assets/AssetManifest.bin.json`, `precache_manifest.json`) | `fail` / `ok` | Probes `assets/AssetManifest.bin.json` (when unhashed) and `/precache_manifest.json` (when present) and fails if `max-age > 0` or missing explicit revalidation. |
115+
| `F-08` | Precache Manifest Content-Hash Integrity (`precache_manifest.json`) | `fail` / `ok` | When `/precache_manifest.json` is present, verifies every hashed entry has `urlHashed: true` matching its filename hash, no non-runtime `.map`/`.symbols`/`.info.json` files are included, and hashed entrypoints are listed. |
116+
| `F-12` | Firebase Fastly `return(pass)` | `warn` | Emitted when `HostPlatform.firebase` is detected (`Vary: x-fh-requested-host`) and root/bootloader uses `no-cache`, `no-store`, or `private` instead of `max-age=0, must-revalidate`. |
117+
| `W-01` | WebAssembly (`dart2wasm`) Target Detection | `warn` / `ok` | Verifies `_flutter.buildConfig` in `flutter_bootstrap.js` includes a `dart2wasm` (`mainWasmPath`) target alongside `dart2js` (warns when missing unless `--no-wasm` is passed). |
118+
| `M-01` | WebAssembly MIME Type | `fail` / `ok` | `.wasm` responses must carry `Content-Type: application/wasm` (required by `WebAssembly.instantiateStreaming`). |
119+
| `M-02` | JavaScript Module MIME Type | `fail` / `ok` | `.mjs` and `.js` responses must carry `text/javascript` or `application/javascript`. |
120+
| `C-01` | Payload Compression | `warn` / `ok` | Payloads exceeding `minCompressionBytes` (`1024` bytes) should be compressed with `br`, `gzip`, or `zstd`. |
121+
| `C-02` | Compression Cache Key (`Vary`) | `warn` | Compressed payloads must include `Vary: Accept-Encoding` so shared caches do not serve mismatched encodings. |
122+
| `R-01` | Revalidation Validators | `warn` | `index.html` should provide `ETag` or `Last-Modified` headers for conditional requests. |
123+
| `R-02` | Conditional `304 Not Modified` | `warn` / `ok` | Sends follow-up `GET` with `If-None-Match: <etag>` and verifies `304 Not Modified` (warns if origin returns `200 OK`). |
123124

124125
---
125126

‎lib/src/url_checker.dart‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import 'dart:convert';
12
import 'dart:io';
23

34
import 'package:http/http.dart' as http;
@@ -193,6 +194,9 @@ class UrlChecker {
193194
final bool verbose;
194195

195196
static final RegExp _hashedPattern = RegExp(r'\.[0-9a-f]{8,64}\.');
197+
static final RegExp _hashedEntryFilenamePattern = RegExp(
198+
r'\.[0-9a-f]{8,64}(\.|$)',
199+
);
196200

197201
static Future<void> checkUrl(
198202
String targetUrl, {
@@ -488,6 +492,13 @@ class UrlChecker {
488492
);
489493
}
490494

495+
detected = await _probePrecacheManifest(
496+
baseUri: baseUri,
497+
targets: targets,
498+
currentPlatform: detected,
499+
findings: findings,
500+
);
501+
491502
return detected;
492503
}
493504

@@ -668,6 +679,131 @@ class UrlChecker {
668679
return detected;
669680
}
670681

682+
Future<HostPlatform> _probePrecacheManifest({
683+
required Uri baseUri,
684+
required List<_AssetTarget> targets,
685+
required HostPlatform currentPlatform,
686+
required List<CheckFinding> findings,
687+
}) async {
688+
const String precacheManifestPath = 'precache_manifest.json';
689+
final http.Response manifestResp = await _client.get(
690+
baseUri.resolve(precacheManifestPath),
691+
);
692+
final HostPlatform detected = _detectPlatform(
693+
currentPlatform,
694+
manifestResp.headers,
695+
);
696+
if (manifestResp.statusCode != 200 || _isHtmlSpaFallback(manifestResp)) {
697+
return detected;
698+
}
699+
700+
final Object? decoded;
701+
try {
702+
decoded = json.decode(manifestResp.body);
703+
} on FormatException {
704+
return detected;
705+
}
706+
if (decoded case {'version': 1, 'entries': final List<Object?> entries}) {
707+
_evaluateUnhashedAssetResponse(
708+
resp: manifestResp,
709+
path: precacheManifestPath,
710+
label: 'precache manifest',
711+
ruleId: 'F-07',
712+
findings: findings,
713+
);
714+
715+
_evaluatePrecacheManifestEntries(
716+
entries: entries,
717+
targets: targets,
718+
findings: findings,
719+
);
720+
}
721+
return detected;
722+
}
723+
724+
static void _evaluatePrecacheManifestEntries({
725+
required List<Object?> entries,
726+
required List<_AssetTarget> targets,
727+
required List<CheckFinding> findings,
728+
}) {
729+
final List<String> issues = <String>[];
730+
final Set<String> manifestUrls = <String>{};
731+
732+
for (final Object? rawEntry in entries) {
733+
if (rawEntry is! Map<String, Object?>) continue;
734+
final String? issue = _validatePrecacheManifestEntry(
735+
rawEntry,
736+
manifestUrls,
737+
);
738+
if (issue != null) {
739+
issues.add(issue);
740+
}
741+
}
742+
743+
for (final _AssetTarget target in targets) {
744+
if (target.isHashed &&
745+
!target.isManifest &&
746+
!manifestUrls.contains(target.path)) {
747+
issues.add('${target.path} missing from precache_manifest.json');
748+
}
749+
}
750+
751+
if (issues.isNotEmpty) {
752+
findings.add(
753+
CheckFinding(
754+
ruleId: 'F-08',
755+
severity: Severity.fail,
756+
path: 'precache_manifest.json',
757+
message:
758+
'Invalid precache_manifest.json entries: ${issues.take(5).join(', ')}.',
759+
),
760+
);
761+
} else {
762+
findings.add(
763+
const CheckFinding(
764+
ruleId: 'F-08',
765+
severity: Severity.ok,
766+
path: 'precache_manifest.json',
767+
message: 'All entries in precache_manifest.json have consistent content hashes and urlHashed flags.',
768+
),
769+
);
770+
}
771+
}
772+
773+
static String? _validatePrecacheManifestEntry(
774+
Map<String, Object?> entry,
775+
Set<String> manifestUrls,
776+
) {
777+
if (entry case {
778+
'url': final String rawUrl,
779+
'hash': final String rawHash,
780+
'urlHashed': final bool rawUrlHashed,
781+
}) {
782+
final String cleanUrl = _normalizeRelativePath(rawUrl);
783+
manifestUrls.add(cleanUrl);
784+
if (_isExcludedPrecacheArtifact(cleanUrl)) {
785+
return '$cleanUrl (non-runtime artifact must be excluded)';
786+
}
787+
final String basename = cleanUrl.split('/').last;
788+
final bool hashMatchesFilename =
789+
basename.contains('.$rawHash.') || basename.endsWith('.$rawHash');
790+
final bool hasFilenameHash = _hashedEntryFilenamePattern.hasMatch(
791+
basename,
792+
);
793+
if (rawUrlHashed != hashMatchesFilename ||
794+
(hasFilenameHash && !rawUrlHashed)) {
795+
return '$cleanUrl (hash=$rawHash, urlHashed=$rawUrlHashed)';
796+
}
797+
return null;
798+
}
799+
return 'malformed entry ($entry)';
800+
}
801+
802+
static bool _isExcludedPrecacheArtifact(String url) =>
803+
url.endsWith('.map') ||
804+
url.endsWith('.symbols') ||
805+
url.endsWith('.info.json');
806+
671807
Future<HostPlatform> _probeMissingHashedAssets({
672808
required Uri baseUri,
673809
required HostPlatform currentPlatform,

0 commit comments

Comments
 (0)