Skip to content

Commit 65f518c

Browse files
authored
module: add a read-only mode to the compile cache
A compile cache generated ahead of time and shipped inside an application package should only ever be read: the package may be immutable or covered by an integrity check, and a cache directory that appears at run time would be a surprise. Add readOnly to module.enableCompileCache() and NODE_COMPILE_CACHE_READONLY=1: existing entries are loaded as before, nothing is serialized or persisted, flushCompileCache() is a no-op, and the cache directory is used as found rather than created, so enabling against a missing directory fails instead of making one. Signed-off-by: Shelley Vohr <[email protected]> PR-URL: #65302 Reviewed-By: James M Snell <[email protected]> Reviewed-By: René <[email protected]>
1 parent eae0ca3 commit 65f518c

9 files changed

Lines changed: 189 additions & 26 deletions

File tree

‎doc/api/cli.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3740,6 +3740,15 @@ When set to 1, the [module compile cache][] can be reused across different direc
37403740
locations as long as the module layout relative to the cache directory remains the same,
37413741
and by any user (the cache subdirectory is not suffixed with the creating user's uid).
37423742

3743+
### `NODE_COMPILE_CACHE_READONLY=1`
3744+
3745+
<!-- YAML
3746+
added: REPLACEME
3747+
-->
3748+
3749+
When set to 1, the [module compile cache][] only reads existing entries from
3750+
its directory: nothing is written to it and it is not created if missing.
3751+
37433752
### `NODE_DEBUG=module[,…]`
37443753

37453754
<!-- YAML

‎doc/api/module.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -429,6 +429,15 @@ There are two ways to enable the portable mode:
429429
430430
2. Setting the environment variable: [`NODE_COMPILE_CACHE_PORTABLE=1`][]
431431
432+
### Read-only compile cache
433+
434+
A cache that was generated ahead of time, for example at build time to be
435+
shipped inside an application package, can be enabled with `readOnly: true`
436+
(or [`NODE_COMPILE_CACHE_READONLY=1`][]). Node.js then loads whatever entries
437+
the directory holds and never writes to it: modules without a usable entry are
438+
compiled as usual but not persisted, [`module.flushCompileCache()`][] is a
439+
no-op, and the directory is not created if it is missing.
440+
432441
### Limitations of the compile cache
433442
434443
Currently when using the compile cache with [V8 JavaScript code coverage][], the
@@ -502,6 +511,9 @@ The following constants are returned as the `status` field in the object returne
502511
<!-- YAML
503512
added: v22.8.0
504513
changes:
514+
- version: REPLACEME
515+
pr-url: https://github.com/nodejs/node/pull/65302
516+
description: Add the `readOnly` option.
505517
- version:
506518
- v25.4.0
507519
- v24.15.0
@@ -528,6 +540,11 @@ changes:
528540
the cache can be reused even if the project directory is moved. This is a best-effort
529541
feature. If not specified, it will depend on whether the environment variable
530542
[`NODE_COMPILE_CACHE_PORTABLE=1`][] is set.
543+
* `readOnly` {boolean} Optional. If `true`, existing cache entries in `directory` are
544+
used but nothing is ever written to it, and the directory is not created when it does
545+
not exist (enabling then fails). Meant for caches generated ahead of time and shipped
546+
with an application. If not specified, it will depend on whether the environment
547+
variable [`NODE_COMPILE_CACHE_READONLY=1`][] is set.
531548
* Returns: {Object}
532549
* `status` {integer} One of the [`module.constants.compileCacheStatus`][]
533550
* `message` {string|undefined} If Node.js cannot enable the compile cache, this contains
@@ -2067,6 +2084,7 @@ returned object contains the following keys:
20672084
[`--require`]: cli.md#-r---require-module
20682085
[`NODE_COMPILE_CACHE=dir`]: cli.md#node_compile_cachedir
20692086
[`NODE_COMPILE_CACHE_PORTABLE=1`]: cli.md#node_compile_cache_portable1
2087+
[`NODE_COMPILE_CACHE_READONLY=1`]: cli.md#node_compile_cache_readonly1
20702088
[`NODE_DISABLE_COMPILE_CACHE=1`]: cli.md#node_disable_compile_cache1
20712089
[`NODE_V8_COVERAGE=dir`]: cli.md#node_v8_coveragedir
20722090
[`Object.freeze()`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/freeze

‎doc/node.1‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1864,6 +1864,10 @@ When set to 1, the module compile cache can be reused across different directory
18641864
locations as long as the module layout relative to the cache directory remains the same,
18651865
and by any user (the cache subdirectory is not suffixed with the creating user's uid).
18661866
.
1867+
.It Ev NODE_COMPILE_CACHE_READONLY Ar 1
1868+
When set to 1, the module compile cache only reads existing entries from
1869+
its directory: nothing is written to it and it is not created if missing.
1870+
.
18671871
.It Ev NODE_DEBUG Ar module[,…]
18681872
\fB','\fR-separated list of core modules that should print debug information.
18691873
.

‎lib/internal/modules/helpers.js‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -462,21 +462,25 @@ function stringify(body) {
462462
* after this method is called.
463463
* This method accepts either:
464464
* - A string: path to the cache directory.
465-
* - An options object `{directory?: string, portable?: boolean}`:
465+
* - An options object `{directory?: string, portable?: boolean, readOnly?: boolean}`:
466466
* - `directory`: A string path to the cache directory.
467467
* - `portable`: If `portable` is true, the cache directory will be considered relative.
468468
* Defaults to `NODE_COMPILE_CACHE_PORTABLE === '1'`.
469+
* - `readOnly`: If `readOnly` is true, existing cache entries are used but nothing is
470+
* written, and the cache directory is not created. Defaults to
471+
* `NODE_COMPILE_CACHE_READONLY === '1'`.
469472
* If cache directory is undefined, it defaults to the `NODE_COMPILE_CACHE` environment variable.
470473
* If `NODE_COMPILE_CACHE` isn't set, it defaults to `path.join(os.tmpdir(), 'node-compile-cache')`.
471-
* @param {string | { directory?: string, portable?: boolean } | undefined} options
474+
* @param {string | { directory?: string, portable?: boolean, readOnly?: boolean } | undefined} options
472475
* @returns {{status: number, message?: string, directory?: string}}
473476
*/
474477
function enableCompileCache(options) {
475478
let portable;
479+
let readOnly;
476480
let directory;
477481

478482
if (typeof options === 'object' && options !== null) {
479-
({ directory, portable } = options);
483+
({ directory, portable, readOnly } = options);
480484
} else {
481485
directory = options;
482486
}
@@ -486,7 +490,10 @@ function enableCompileCache(options) {
486490
if (portable === undefined) {
487491
portable = process.env.NODE_COMPILE_CACHE_PORTABLE === '1';
488492
}
489-
const nativeResult = _enableCompileCache(directory, portable);
493+
if (readOnly === undefined) {
494+
readOnly = process.env.NODE_COMPILE_CACHE_READONLY === '1';
495+
}
496+
const nativeResult = _enableCompileCache(directory, portable, readOnly);
490497
const result = { status: nativeResult[0] };
491498
if (nativeResult[1]) {
492499
result.message = nativeResult[1];

‎src/compile_cache.cc‎

Lines changed: 52 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -257,7 +257,8 @@ CompileCacheEntry* CompileCacheHandler::GetOrInsert(Local<String> code,
257257
// If the portable cache is enabled and it seems possible to compute the
258258
// relative position from an absolute path, we use the relative position
259259
// in the cache key.
260-
if (portable_ == EnableOption::PORTABLE && IsAbsoluteFilePath(file_path)) {
260+
if (HasOption(portable_, EnableOption::PORTABLE) &&
261+
IsAbsoluteFilePath(file_path)) {
261262
// Normalize the path to ensure it is consistent.
262263
std::string normalized_file_path = NormalizeFileURLOrPath(env, file_path);
263264
if (normalized_file_path.empty()) {
@@ -332,6 +333,10 @@ void CompileCacheHandler::MaybeSaveImpl(CompileCacheEntry* entry,
332333
Debug("keeping the in-memory entry\n");
333334
return;
334335
}
336+
if (read_only_) {
337+
Debug("read-only, not serializing\n");
338+
return;
339+
}
335340
Debug("%s the in-memory entry\n",
336341
entry->cache == nullptr ? "initializing" : "refreshing");
337342

@@ -356,6 +361,9 @@ void CompileCacheHandler::MaybeSave(CompileCacheEntry* entry,
356361

357362
void CompileCacheHandler::MaybeSave(CompileCacheEntry* entry,
358363
std::string_view transpiled) {
364+
if (read_only_) {
365+
return;
366+
}
359367
CHECK(entry->type == CachedCodeType::kStrippedTypeScript);
360368
Debug("[compile cache] saving transpilation cache for %s %s\n",
361369
entry->type_name(),
@@ -388,6 +396,10 @@ void CompileCacheHandler::MaybeSave(CompileCacheEntry* entry,
388396
*/
389397
void CompileCacheHandler::Persist() {
390398
DCHECK(!compile_cache_dir_.empty());
399+
if (read_only_) {
400+
Debug("[compile cache] read-only, skipping persistence\n");
401+
return;
402+
}
391403

392404
// TODO(joyeecheung): do this using a separate event loop to utilize the
393405
// libuv thread pool and do the file system operations concurrently.
@@ -554,10 +566,11 @@ CompileCacheEnableResult CompileCacheHandler::Enable(Environment* env,
554566
cache_tag,
555567
cache_dir_with_tag);
556568

557-
if (!env->permission()->is_granted(
558-
env,
559-
permission::PermissionScope::kFileSystemWrite,
560-
cache_dir_with_tag)) [[unlikely]] {
569+
const bool read_only = HasOption(option, EnableOption::READ_ONLY);
570+
if (!read_only && !env->permission()->is_granted(
571+
env,
572+
permission::PermissionScope::kFileSystemWrite,
573+
cache_dir_with_tag)) [[unlikely]] {
561574
result.message = "Skipping compile cache because write permission for " +
562575
cache_dir_with_tag + " is not granted";
563576
result.status = CompileCacheEnableStatus::FAILED;
@@ -574,25 +587,46 @@ CompileCacheEnableResult CompileCacheHandler::Enable(Environment* env,
574587
return result;
575588
}
576589

577-
fs::FSReqWrapSync req_wrap;
578-
int err = fs::MKDirpSync(
579-
nullptr, &(req_wrap.req), cache_dir_with_tag, 0777, nullptr);
580-
if (is_debug_) {
581-
Debug("[compile cache] creating cache directory %s...%s\n",
590+
if (read_only) {
591+
// A read-only cache is used as found and never created: without the
592+
// directory there is nothing to read.
593+
uv_fs_t stat_req;
594+
int err =
595+
uv_fs_stat(nullptr, &stat_req, cache_dir_with_tag.c_str(), nullptr);
596+
// libuv normalizes st_mode across platforms; MSVC has no S_ISDIR macro.
597+
bool is_dir = err == 0 && (stat_req.statbuf.st_mode & S_IFMT) == S_IFDIR;
598+
uv_fs_req_cleanup(&stat_req);
599+
Debug("[compile cache] read-only cache directory %s...%s\n",
582600
cache_dir_with_tag,
583-
err < 0 ? uv_strerror(err) : "success");
584-
}
585-
if (err != 0 && err != UV_EEXIST) {
586-
result.message =
587-
"Cannot create cache directory: " + std::string(uv_strerror(err));
588-
result.status = CompileCacheEnableStatus::FAILED;
589-
return result;
601+
is_dir ? "found" : "not found");
602+
if (!is_dir) {
603+
result.message =
604+
"Cache directory does not exist (read-only): " + cache_dir_with_tag;
605+
result.status = CompileCacheEnableStatus::FAILED;
606+
return result;
607+
}
608+
} else {
609+
fs::FSReqWrapSync req_wrap;
610+
int err = fs::MKDirpSync(
611+
nullptr, &(req_wrap.req), cache_dir_with_tag, 0777, nullptr);
612+
if (is_debug_) {
613+
Debug("[compile cache] creating cache directory %s...%s\n",
614+
cache_dir_with_tag,
615+
err < 0 ? uv_strerror(err) : "success");
616+
}
617+
if (err != 0 && err != UV_EEXIST) {
618+
result.message =
619+
"Cannot create cache directory: " + std::string(uv_strerror(err));
620+
result.status = CompileCacheEnableStatus::FAILED;
621+
return result;
622+
}
590623
}
591624

592625
result.cache_directory = absolute_cache_dir_base;
593626
compile_cache_dir_ = cache_dir_with_tag;
594627
portable_ = option;
595-
if (option == EnableOption::PORTABLE) {
628+
read_only_ = read_only;
629+
if (HasOption(option, EnableOption::PORTABLE)) {
596630
normalized_compile_cache_dir_ =
597631
NormalizeFileURLOrPath(env, compile_cache_dir_);
598632
}

‎src/compile_cache.h‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,22 @@ struct CompileCacheEnableResult {
6060
std::string message; // Set in case of failure.
6161
};
6262

63-
enum class EnableOption : uint8_t { DEFAULT, PORTABLE };
63+
enum class EnableOption : uint8_t {
64+
DEFAULT = 0,
65+
PORTABLE = 1 << 0,
66+
// Only read existing cache entries: nothing is compiled into the in-memory
67+
// store for persisting, nothing is written to disk, and the cache directory
68+
// is not created if it does not exist.
69+
READ_ONLY = 1 << 1,
70+
};
71+
72+
inline constexpr EnableOption operator|(EnableOption a, EnableOption b) {
73+
return static_cast<EnableOption>(static_cast<uint8_t>(a) |
74+
static_cast<uint8_t>(b));
75+
}
76+
inline constexpr bool HasOption(EnableOption value, EnableOption flag) {
77+
return (static_cast<uint8_t>(value) & static_cast<uint8_t>(flag)) != 0;
78+
}
6479

6580
class CompileCacheHandler {
6681
public:
@@ -82,6 +97,7 @@ class CompileCacheHandler {
8297
bool rejected);
8398
void MaybeSave(CompileCacheEntry* entry, std::string_view transpiled);
8499
std::string_view cache_dir() { return compile_cache_dir_; }
100+
bool read_only() const { return read_only_; }
85101

86102
private:
87103
void ReadCacheFile(CompileCacheEntry* entry);
@@ -107,6 +123,7 @@ class CompileCacheHandler {
107123
std::string compile_cache_dir_;
108124
std::string normalized_compile_cache_dir_;
109125
EnableOption portable_ = EnableOption::DEFAULT;
126+
bool read_only_ = false;
110127
std::unordered_map<uint32_t, std::unique_ptr<CompileCacheEntry>>
111128
compiler_cache_store_;
112129
};

‎src/env.cc‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1194,8 +1194,14 @@ void Environment::InitializeCompileCache() {
11941194
DebugCategory::COMPILE_CACHE,
11951195
"[compile cache] using relative path\n");
11961196
}
1197-
EnableCompileCache(dir_from_env,
1198-
portable ? EnableOption::PORTABLE : EnableOption::DEFAULT);
1197+
std::string read_only_env;
1198+
bool read_only = credentials::SafeGetenv(
1199+
"NODE_COMPILE_CACHE_READONLY", &read_only_env, this) &&
1200+
read_only_env == "1";
1201+
EnableOption option = EnableOption::DEFAULT;
1202+
if (portable) option = option | EnableOption::PORTABLE;
1203+
if (read_only) option = option | EnableOption::READ_ONLY;
1204+
EnableCompileCache(dir_from_env, option);
11991205
}
12001206

12011207
CompileCacheEnableResult Environment::EnableCompileCache(

‎src/node_modules.cc‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -524,7 +524,10 @@ void EnableCompileCache(const FunctionCallbackInfo<Value>& args) {
524524

525525
EnableOption option = EnableOption::DEFAULT;
526526
if (args.Length() > 1 && args[1]->IsTrue()) {
527-
option = EnableOption::PORTABLE;
527+
option = option | EnableOption::PORTABLE;
528+
}
529+
if (args.Length() > 2 && args[2]->IsTrue()) {
530+
option = option | EnableOption::READ_ONLY;
528531
}
529532

530533
Utf8Value value(isolate, args[0]);
Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
'use strict';
2+
3+
// This tests module.enableCompileCache({ directory, readOnly: true }): existing
4+
// entries are read, nothing is written, and a missing directory is not created.
5+
6+
const common = require('../common');
7+
const { spawnSyncAndAssert } = require('../common/child_process');
8+
const assert = require('assert');
9+
const fs = require('fs');
10+
const path = require('path');
11+
const tmpdir = require('../common/tmpdir');
12+
const fixtures = require('../common/fixtures');
13+
14+
tmpdir.refresh();
15+
const wrapper = fixtures.path('compile-cache-wrapper-options.js');
16+
const target = path.join(tmpdir.path, 'target.js');
17+
fs.writeFileSync(target, 'module.exports = 1;');
18+
const other = path.join(tmpdir.path, 'other.js');
19+
fs.writeFileSync(other, 'module.exports = 2;');
20+
const directory = path.join(tmpdir.path, 'cache');
21+
const list = () => fs.readdirSync(directory, { recursive: true }).sort();
22+
const run = (options, extraEnv, requires, check) => spawnSyncAndAssert(
23+
process.execPath,
24+
[...requires.flatMap((r) => ['-r', r]), target],
25+
{
26+
env: {
27+
...process.env,
28+
NODE_DEBUG_NATIVE: 'COMPILE_CACHE',
29+
NODE_TEST_COMPILE_CACHE_OPTIONS: JSON.stringify(options),
30+
...extraEnv,
31+
},
32+
},
33+
check);
34+
35+
// Read-only against a directory that does not exist: enabling fails and
36+
// nothing is created.
37+
run({ directory, readOnly: true }, {}, [wrapper], {
38+
stderr: /read-only cache directory .*\.\.\.not found/,
39+
});
40+
assert(!fs.existsSync(directory));
41+
42+
// A normal run generates the cache for target.js.
43+
run({ directory }, {}, [wrapper], {
44+
stderr: /writing cache for .*target\.js.*success/,
45+
});
46+
const generated = list();
47+
assert.notStrictEqual(generated.length, 0);
48+
49+
// Read-only against it: target.js's entry is accepted, other.js is compiled
50+
// but not written, and persistence is skipped.
51+
run({ directory, readOnly: true }, {}, [wrapper, other], {
52+
stderr: common.mustCall((output) => {
53+
assert.match(output, /cache for .*target\.js was accepted/);
54+
assert.match(output, /read-only, skipping persistence/);
55+
assert.doesNotMatch(output, /writing cache for/);
56+
return true;
57+
}),
58+
});
59+
assert.deepStrictEqual(list(), generated);
60+
61+
// The environment variable form.
62+
run({ directory }, { NODE_COMPILE_CACHE_READONLY: '1' }, [wrapper, other], {
63+
stderr: /read-only, skipping persistence/,
64+
});
65+
assert.deepStrictEqual(list(), generated);

0 commit comments

Comments
 (0)