Skip to content

Commit 69c6070

Browse files
committed
deps: V8: backport 5ba8aab1ec1b
Original commit message: Reland "[parser] Create debug magic comment values as source substrings" This is a reland of commit 8dcf14f490c20b23b90f9f4af56d52abdd933a0c Changes since the original: The original made every value with at least SlicedString::kMinLength characters a SlicedString of the source, including usual source URLs like //# sourceURL=inside-direct-eval.js. Heap snapshots name sliced strings "(sliced string)" instead of by their contents, so DevTools' heap snapshot ContextAnalyzer, which reads the name of the script's source_url node, showed "(sliced string)" instead of the source URL. The source URL is now always a sequential copy of its value, as before the original (which internalized it), since it is used as the script name (stack traces, profiles, heap snapshots). The debug ID is also copied, since it is short. Only the sourceMappingURL value is a substring of the source, which covers the case the original was about (huge inline source maps). A huge sourceURL or debugId value is copied onto the heap as before the original. Added a regression test that a source URL longer than SlicedString::kMinLength is sequential (script, streaming and direct eval) and that heap snapshots name the source_url node by it, and updated the existing tests accordingly. Original change's description: > [parser] Create debug magic comment values as source substrings > > The values of the //# sourceURL, sourceMappingURL and debugId magic > comments were copied into LiteralBuffers while scanning and then > internalized. An inline source map (sourceMappingURL=data:...) can be > tens of MiB, so this allocated a string of that size on the heap in a > non-throwing allocation, e.g. aborting a Node.js worker with a small > old-generation limit even though the (external) source fits. > > The scanner now only records the source locations of these values, and > the values are created as substrings of the script source: > - On the main thread, when parsing finishes. > - For background and streaming compiles, where the source is not > available while parsing, the locations are kept in the > BackgroundCompileTask and the substrings are created in > FinalizeScript after the source is set. Locations outside of the > source provided by the embedder are ignored. > > The semantics are unchanged: the last comment wins, trailing garbage or > an empty value removes the value, and a source map URL provided through > the API takes precedence. > > Since these values are now usually SlicedStrings of the script source, > the code serializer serializes all sliced strings as sequential > strings (one-byte if their contents are, even if their parent is > two-byte). The script source is an attached object, and off-thread > deserialization attaches a placeholder for it, so a serialized slice > of it would not be valid. > > Consuming a code cache still materializes these values as sequential > strings, e.g. on Node.js's NODE_COMPILE_CACHE path, so a huge cached > value can still be large on the heap there. Keeping them as substrings > of the source when deserializing requires a code cache format change > and is left for a follow-up. > > This also removes the LiteralBuffer copies of the values. > > [email protected], [email protected] > > Bug: 566467883 > Refs: nodejs#64155 > Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/8472768 > Reviewed-by: Leszek Swirski <[email protected]> > Reviewed-by: Patrick Thier <[email protected]> > Cr-Commit-Position: refs/heads/main@{#110612} [email protected], [email protected], [email protected] Bug: 566467883 Change-Id: I76eec1fdaf71526afdd881d0d4a2b117c32d3ba8 Refs: nodejs#64155 Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/8579167 Reviewed-by: Patrick Thier <[email protected]> Commit-Queue: Patrick Thier <[email protected]> Reviewed-by: Toon Verwaest <[email protected]> Cr-Commit-Position: refs/heads/main@{#111132} Adapted to deps/v8: this also includes the upstream refactor b0f6ad031f03 ("[parser] Factor magic comment value scanning into a helper"), which the reland is based on. src/base/unique-array.h does not exist yet, so base::OwnedVector is used in the serializer, the serializer keeps the 15.6 external string handling (without the later SBXCHECK hardening), and the CodeSerializer override is placed for the older class layout. The inspector test is not included because deps/v8 does not ship test/inspector. Refs: v8/v8@5ba8aab Refs: v8/v8@b0f6ad0 Signed-off-by: eliau2005 <[email protected]>
1 parent ee982db commit 69c6070

16 files changed

Lines changed: 802 additions & 134 deletions

‎common.gypi‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@
5050

5151
# Reset this number to 0 on major V8 upgrades.
5252
# Increment by one for each non-official patch applied to deps/v8.
53-
'v8_embedder_string': '-node.10',
53+
'v8_embedder_string': '-node.11',
5454

5555
##### V8 defaults for Node.js #####
5656

‎deps/v8/src/codegen/compiler.cc‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2024,6 +2024,9 @@ void BackgroundCompileTask::Run(
20242024
parser.ParseOnBackground(isolate, &info, script_, start_position_,
20252025
end_position_, function_literal_id_);
20262026
parser.UpdateStatistics(script_, &use_counts_, &total_preparse_skipped_);
2027+
if (toplevel_script_compilation) {
2028+
debug_magic_comment_locations_ = parser.debug_magic_comment_locations();
2029+
}
20272030

20282031
TRACE_EVENT(TRACE_DISABLED_BY_DEFAULT("v8.compile"),
20292032
"V8.CompileCodeBackground");
@@ -2767,8 +2770,14 @@ MaybeHandle<SharedFunctionInfo> BackgroundCompileTask::FinalizeScript(
27672770
{
27682771
DisallowGarbageCollection no_gc;
27692772
SetScriptFieldsFromDetails(isolate, *script, script_details, &no_gc);
2770-
LOG(isolate, ScriptDetails(*script));
27712773
}
2774+
// After SetScriptFieldsFromDetails, so that a source map URL provided
2775+
// through the API takes precedence. The locations come from the streamed
2776+
// source; the embedder is expected to pass the same source here, but an
2777+
// out-of-bounds location is ignored.
2778+
Parser::SetDebugMagicCommentFields(isolate, script, source,
2779+
debug_magic_comment_locations_);
2780+
LOG(isolate, ScriptDetails(*script));
27722781
}
27732782

27742783
ReportStatistics(isolate);

‎deps/v8/src/codegen/compiler.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
#include "src/objects/debug-objects.h"
2525
#include "src/parsing/parse-info.h"
2626
#include "src/parsing/pending-compilation-error-handler.h"
27+
#include "src/parsing/scanner.h" // For Scanner::DebugMagicCommentLocations.
2728
#include "src/snapshot/code-serializer.h"
2829
#include "src/utils/allocation.h"
2930
#include "src/zone/zone.h"
@@ -649,6 +650,9 @@ class V8_EXPORT_PRIVATE BackgroundCompileTask {
649650
DeferredFinalizationJobDataList jobs_to_retry_finalization_on_main_thread_;
650651
base::SmallVector<v8::Isolate::UseCounterFeature, 8> use_counts_;
651652
int total_preparse_skipped_ = 0;
653+
// The source is not available on the background thread, so the debug magic
654+
// comment values are created from these locations when finalizing.
655+
Scanner::DebugMagicCommentLocations debug_magic_comment_locations_;
652656

653657
// Single function data for top-level function compilation.
654658
MaybeIndirectHandle<SharedFunctionInfo> input_shared_info_;

‎deps/v8/src/parsing/literal-buffer.cc‎

Lines changed: 0 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -5,29 +5,11 @@
55
#include "src/parsing/literal-buffer.h"
66

77
#include "src/base/strings.h"
8-
#include "src/execution/isolate.h"
9-
#include "src/execution/local-isolate.h"
10-
#include "src/heap/factory.h"
11-
#include "src/objects/objects.h"
12-
#include "src/objects/string.h"
138
#include "src/utils/memcopy.h"
149

1510
namespace v8 {
1611
namespace internal {
1712

18-
template <typename IsolateT>
19-
DirectHandle<String> LiteralBuffer::Internalize(IsolateT* isolate) const {
20-
if (is_one_byte()) {
21-
return isolate->factory()->InternalizeString(one_byte_literal());
22-
}
23-
return isolate->factory()->InternalizeString(two_byte_literal());
24-
}
25-
26-
template DirectHandle<String> LiteralBuffer::Internalize(
27-
Isolate* isolate) const;
28-
template DirectHandle<String> LiteralBuffer::Internalize(
29-
LocalIsolate* isolate) const;
30-
3113
size_t LiteralBuffer::NewCapacity(size_t min_capacity) {
3214
return min_capacity < (kMaxGrowth / (kGrowthFactor - 1))
3315
? min_capacity * kGrowthFactor

‎deps/v8/src/parsing/literal-buffer.h‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -98,9 +98,6 @@ class LiteralBuffer final {
9898
is_one_byte_ = true;
9999
}
100100

101-
template <typename IsolateT>
102-
DirectHandle<String> Internalize(IsolateT* isolate) const;
103-
104101
private:
105102
static constexpr size_t kInitialCapacity = 256;
106103
static constexpr size_t kGrowthFactor = 4;

‎deps/v8/src/parsing/parser.cc‎

Lines changed: 34 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
#include "src/common/message-template.h"
2121
#include "src/common/scoped-modification.h"
2222
#include "src/compiler-dispatcher/lazy-compile-dispatcher.h"
23+
#include "src/heap/factory-inl.h"
2324
#include "src/heap/local-heap-inl.h"
2425
#include "src/heap/parked-scope.h"
2526
#include "src/logging/counters.h"
@@ -3572,32 +3573,50 @@ void Parser::InsertSloppyBlockFunctionVarBindings(DeclarationScope* scope) {
35723573
// ----------------------------------------------------------------------------
35733574
// Parser support
35743575

3575-
template <typename IsolateT>
3576-
void Parser::HandleDebugMagicComments(IsolateT* isolate,
3576+
void Parser::HandleDebugMagicComments(Isolate* isolate,
35773577
DirectHandle<Script> script) {
3578-
DirectHandle<String> source_url = scanner_.SourceUrl(isolate);
3579-
if (!source_url.is_null()) {
3578+
// On the main thread, the scanned source is the script source.
3579+
DirectHandle<String> source(Cast<String>(script->source()), isolate);
3580+
SetDebugMagicCommentFields(isolate, script, source,
3581+
scanner_.debug_magic_comment_locations());
3582+
}
3583+
3584+
// static
3585+
void Parser::SetDebugMagicCommentFields(
3586+
Isolate* isolate, DirectHandle<Script> script, DirectHandle<String> source,
3587+
const Scanner::DebugMagicCommentLocations& locations) {
3588+
source = String::Flatten(isolate, source);
3589+
auto is_valid = [&](Scanner::Location location) {
3590+
return location.IsValid() && location.length() != 0 &&
3591+
static_cast<uint32_t>(location.end_pos) <= source->length();
3592+
};
3593+
if (is_valid(locations.source_url)) {
3594+
// The source URL is used as the script name (e.g. in stack traces,
3595+
// profiles and heap snapshots), so it is a sequential copy rather than a
3596+
// substring of the source.
3597+
DirectHandle<String> source_url = isolate->factory()->NewCopiedSubstring(
3598+
source, locations.source_url.beg_pos, locations.source_url.length());
35803599
script->set_source_url(*source_url);
35813600
}
3582-
DirectHandle<String> source_mapping_url = scanner_.SourceMappingUrl(isolate);
35833601
// The API can provide a source map URL and the API should take precedence.
35843602
// Let's make sure we do not override the API with the magic comment.
3585-
if (!source_mapping_url.is_null() &&
3586-
IsUndefined(script->source_mapping_url())) {
3603+
if (IsUndefined(script->source_mapping_url()) &&
3604+
is_valid(locations.source_mapping_url)) {
3605+
// The source map URL can be arbitrarily large (e.g. an inline source map),
3606+
// so it is a substring of the source rather than a copy.
3607+
DirectHandle<String> source_mapping_url = isolate->factory()->NewSubString(
3608+
source, locations.source_mapping_url.beg_pos,
3609+
locations.source_mapping_url.end_pos);
35873610
script->set_source_mapping_url(*source_mapping_url);
35883611
}
3589-
3590-
DirectHandle<String> debug_id = scanner_.DebugId(isolate);
3591-
if (!debug_id.is_null()) {
3612+
if (is_valid(locations.debug_id)) {
3613+
// Debug IDs are short, so they are copied like the source URL.
3614+
DirectHandle<String> debug_id = isolate->factory()->NewCopiedSubstring(
3615+
source, locations.debug_id.beg_pos, locations.debug_id.length());
35923616
script->set_debug_id(*debug_id);
35933617
}
35943618
}
35953619

3596-
template void Parser::HandleDebugMagicComments(Isolate* isolate,
3597-
DirectHandle<Script> script);
3598-
template void Parser::HandleDebugMagicComments(LocalIsolate* isolate,
3599-
DirectHandle<Script> script);
3600-
36013620
void Parser::UpdateStatistics(Isolate* isolate, DirectHandle<Script> script) {
36023621
CHECK_NOT_NULL(isolate);
36033622

@@ -3700,9 +3719,6 @@ void Parser::ParseOnBackground(LocalIsolate* isolate, ParseInfo* info,
37003719
MaybeProcessSourceRanges(info, result, stack_limit_);
37013720
});
37023721
// We need to unpark by now though, to be able to internalize.
3703-
if (flags().is_toplevel()) {
3704-
HandleDebugMagicComments(isolate, script);
3705-
}
37063722
if (result == nullptr) return;
37073723
PostProcessParseResult(isolate, info, result);
37083724
}

‎deps/v8/src/parsing/parser.h‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -170,8 +170,19 @@ class V8_EXPORT_PRIVATE Parser : public NON_EXPORTED_BASE(ParserBase<Parser>) {
170170
DirectHandle<Script> script,
171171
base::SmallVector<v8::Isolate::UseCounterFeature, 8>* use_counters,
172172
int* preparse_skipped);
173-
template <typename IsolateT>
174-
void HandleDebugMagicComments(IsolateT* isolate, DirectHandle<Script> script);
173+
void HandleDebugMagicComments(Isolate* isolate, DirectHandle<Script> script);
174+
const Scanner::DebugMagicCommentLocations& debug_magic_comment_locations()
175+
const {
176+
return scanner_.debug_magic_comment_locations();
177+
}
178+
// Sets the debug magic comment fields of {script} to the values in {source}
179+
// at the given locations: the source map URL is a substring of {source}, the
180+
// source URL and debug ID are sequential copies. Locations that are empty,
181+
// invalid or out of bounds are ignored.
182+
static void SetDebugMagicCommentFields(
183+
Isolate* isolate, DirectHandle<Script> script,
184+
DirectHandle<String> source,
185+
const Scanner::DebugMagicCommentLocations& locations);
175186

176187
private:
177188
friend class ParserBase<Parser>;

‎deps/v8/src/parsing/scanner.cc‎

Lines changed: 45 additions & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -277,6 +277,27 @@ void ProcessPerFunctionCompileHints(const base::Vector<const uint8_t>& data,
277277

278278
} // namespace
279279

280+
template <typename CharCallback>
281+
Scanner::Location Scanner::ScanMagicCommentValue(CharCallback on_char) {
282+
DCHECK_EQ(c0_, '=');
283+
Advance();
284+
while (IsWhiteSpace(c0_)) {
285+
Advance();
286+
}
287+
int value_beg_pos = source_pos();
288+
while (c0_ != kEndOfInput && !IsWhiteSpaceOrLineTerminator(c0_)) {
289+
on_char(c0_);
290+
Advance();
291+
}
292+
Location value_location(value_beg_pos, source_pos());
293+
// Allow whitespace at the end.
294+
while (c0_ != kEndOfInput && !unibrow::IsLineTerminator(c0_)) {
295+
if (!IsWhiteSpace(c0_)) return Location::invalid();
296+
Advance();
297+
}
298+
return value_location;
299+
}
300+
280301
void Scanner::TryToParseMagicComment(base::uc32 hash_or_at_sign) {
281302
// Magic comments are of the form: //[#@]\s<name>=\s*<value>\s*.* and this
282303
// function will just return if it cannot parse a magic comment.
@@ -293,12 +314,14 @@ void Scanner::TryToParseMagicComment(base::uc32 hash_or_at_sign) {
293314
}
294315
if (!name.is_one_byte()) return;
295316
base::Vector<const uint8_t> name_literal = name.one_byte_literal();
296-
LiteralBuffer* value;
297-
LiteralBuffer per_function_compile_hints_value;
317+
// The debug magic comment values are recorded as source locations, so that
318+
// they can be created as substrings of the source instead of being copied.
319+
Location* value_location = nullptr;
320+
bool is_compile_hints = false;
298321
if (name_literal == base::StaticOneByteVector("sourceURL")) {
299-
value = &source_url_;
322+
value_location = &debug_magic_comment_locations_.source_url;
300323
} else if (name_literal == base::StaticOneByteVector("sourceMappingURL")) {
301-
value = &source_mapping_url_;
324+
value_location = &debug_magic_comment_locations_.source_mapping_url;
302325
DCHECK(hash_or_at_sign == '#' || hash_or_at_sign == '@');
303326
saw_source_mapping_url_magic_comment_at_sign_ = hash_or_at_sign == '@';
304327
} else if (!saw_non_comment_ &&
@@ -309,45 +332,33 @@ void Scanner::TryToParseMagicComment(base::uc32 hash_or_at_sign) {
309332
} else if (name_literal ==
310333
base::StaticOneByteVector("functionsCalledOnLoad") &&
311334
hash_or_at_sign == '#') {
312-
value = &per_function_compile_hints_value;
335+
is_compile_hints = true;
313336
} else if (name_literal == base::StaticOneByteVector("debugId") &&
314337
hash_or_at_sign == '#') {
315-
value = &debug_id_;
338+
value_location = &debug_magic_comment_locations_.debug_id;
316339
} else {
317340
return;
318341
}
319342
if (c0_ != '=') {
320343
return;
321344
}
322-
value->Start();
323-
Advance();
324-
while (IsWhiteSpace(c0_)) {
325-
Advance();
326-
}
327-
while (c0_ != kEndOfInput && !unibrow::IsLineTerminator(c0_)) {
328-
if (IsWhiteSpace(c0_)) {
329-
break;
330-
}
331-
value->AddChar(c0_);
332-
Advance();
333-
}
334-
// Allow whitespace at the end.
335-
while (c0_ != kEndOfInput && !unibrow::IsLineTerminator(c0_)) {
336-
if (!IsWhiteSpace(c0_)) {
337-
value->Start();
338-
break;
339-
}
340-
Advance();
341-
}
342-
if (value == &per_function_compile_hints_value &&
343-
per_function_compile_hints_value.is_one_byte()) {
344-
base::Vector<const uint8_t> value_literal =
345-
per_function_compile_hints_value.one_byte_literal();
346-
per_function_compile_hint_positions_.clear();
347-
per_function_compile_hint_positions_idx_ = 0;
348-
ProcessPerFunctionCompileHints(value_literal, source_pos(),
349-
per_function_compile_hint_positions_);
345+
if (!is_compile_hints) {
346+
DCHECK_NOT_NULL(value_location);
347+
*value_location = ScanMagicCommentValue([](base::uc32) {});
348+
return;
350349
}
350+
LiteralBuffer value;
351+
value.Start();
352+
Location value_location_in_source =
353+
ScanMagicCommentValue([&value](base::uc32 c) { value.AddChar(c); });
354+
// A value followed by trailing garbage is treated as empty, which clears
355+
// the previously recorded compile hints.
356+
if (!value_location_in_source.IsValid()) value.Start();
357+
if (!value.is_one_byte()) return;
358+
per_function_compile_hint_positions_.clear();
359+
per_function_compile_hint_positions_idx_ = 0;
360+
ProcessPerFunctionCompileHints(value.one_byte_literal(), source_pos(),
361+
per_function_compile_hint_positions_);
351362
}
352363

353364
bool Scanner::HasPerFunctionCompileHint(int position) {
@@ -689,43 +700,6 @@ Token::Value Scanner::ScanTemplateSpan() {
689700
return result;
690701
}
691702

692-
template <typename IsolateT>
693-
DirectHandle<String> Scanner::SourceUrl(IsolateT* isolate) const {
694-
DirectHandle<String> tmp;
695-
if (source_url_.length() > 0) {
696-
tmp = source_url_.Internalize(isolate);
697-
}
698-
return tmp;
699-
}
700-
701-
template DirectHandle<String> Scanner::SourceUrl(Isolate* isolate) const;
702-
template DirectHandle<String> Scanner::SourceUrl(LocalIsolate* isolate) const;
703-
704-
template <typename IsolateT>
705-
DirectHandle<String> Scanner::SourceMappingUrl(IsolateT* isolate) const {
706-
DirectHandle<String> tmp;
707-
if (source_mapping_url_.length() > 0) {
708-
tmp = source_mapping_url_.Internalize(isolate);
709-
}
710-
return tmp;
711-
}
712-
713-
template DirectHandle<String> Scanner::SourceMappingUrl(Isolate* isolate) const;
714-
template DirectHandle<String> Scanner::SourceMappingUrl(
715-
LocalIsolate* isolate) const;
716-
717-
template <typename IsolateT>
718-
DirectHandle<String> Scanner::DebugId(IsolateT* isolate) const {
719-
DirectHandle<String> tmp;
720-
if (debug_id_.length() > 0) {
721-
tmp = debug_id_.Internalize(isolate);
722-
}
723-
return tmp;
724-
}
725-
726-
template DirectHandle<String> Scanner::DebugId(Isolate* isolate) const;
727-
template DirectHandle<String> Scanner::DebugId(LocalIsolate* isolate) const;
728-
729703
bool Scanner::ScanDigitsWithNumericSeparators(bool (*predicate)(base::uc32 ch),
730704
bool is_check_first_digit) {
731705
// we must have at least one digit after 'x'/'b'/'o'

0 commit comments

Comments
 (0)