Don't count time spent checkpointed in CLOCK_MONOTONIC by aaltinaydev · Pull Request #16733 · google/gvisor

aaltinaydev · 2026-10-30T06:30:39Z

The idea

On restore, gVisor advances CLOCK_MONOTONIC by the time that the sandbox spent checkpointed. In a sandbox restored minutes later, every pending timer and timeout therefore expires at once, right when the application wakes up to serve a request.

This PR makes the clocks behave across checkpoint/restore as they do across a Linux suspend:

CLOCK_MONOTONIC continues from its value at checkpoint time when the restored kernel resumes.
CLOCK_BOOTTIME, until now an alias of CLOCK_MONOTONIC, becomes a separate clock that counts the checkpointed time, as it counts suspend time on Linux. Applications that need the real elapsed time, e.g. to expire idle connections, can use it. sysinfo(2) uptime uses it too.
Relative CLOCK_REALTIME sleeps and timers, which glibc uses for sleep(3) and nanosleep(3), are measured with CLOCK_MONOTONIC, as on Linux.

The restore also takes the first clock calibration sample early. Without it, the VDSO cannot compute the time for about 1 s after each restore, and every application clock read is a system call.

Unlike the autoresearch patch in #16135, this PR does not freeze CLOCK_BOOTTIME, also moves relative CLOCK_REALTIME timers (timerfd, POSIX timers) to CLOCK_MONOTONIC, does not count the restore itself in CLOCK_MONOTONIC, and adds the early calibration.

Fixes #16135

COMPATIBILITY:NEW-APP-CONFIG: in sandboxes restored after a delay, applications ran all pending timers at once, and relative CLOCK_REALTIME sleeps and timers changed with the realtime clock, unlike on Linux.

Performance impact

After restoring a 2-minute-old checkpoint at 0.25 CPU with 176 sandboxes: time to app readiness 6.28 → 1.46 ms, first request 2.48 → 0.95 ms, warm resume -12%. The gain holds at 1 CPU and with no CPU limit.
Restoring a fresh checkpoint: first request -7.5% to -8% and readiness -5% to -6.5%, from the early calibration. The restore itself is unchanged.
Against the autoresearch patch: first request -6.5% to -8.3% and readiness -5% to -8% in every configuration (geomean -3.6%). The restore itself and warm resume are within ±0.6%.
//test/benchmarks/base: no change overall (geomean +0.6%).

Benchmarks: master vs this PR, and the autoresearch patch vs this PR

goos: linux
goarch: amd64
pkg: sandbox checkpoint/restore churn harness (c4-standard-48, COS 129)
                                             │    before     │                 after                  │
                                             │    sec/op     │   sec/op     vs base                   │
WarmResume/sandboxes=176/quota=0.25/age=2m      51.00m ±  0%   44.94m ± 0%  -11.89% (n=2640)
WakeupProbe/sandboxes=176/quota=0.25/age=2m     6.282m ±  1%   1.462m ± 2%  -76.73% (n=2640)
FirstRequest/sandboxes=176/quota=0.25/age=2m   2480.2µ ±  4%   947.1µ ± 0%  -61.82% (n=2640)
Restore/sandboxes=176/quota=0.25/age=2m         36.14m ±  0%   36.17m ± 0%        ~ (p=0.122 n=2640)
WarmResume/sandboxes=18/quota=0.25/age=2m       51.44m ±  3%   44.12m ± 0%  -14.23% (n=270)
WakeupProbe/sandboxes=18/quota=0.25/age=2m      6.468m ±  2%   1.530m ± 9%  -76.34% (n=270)
FirstRequest/sandboxes=18/quota=0.25/age=2m    2627.7µ ± 47%   962.1µ ± 1%  -63.39% (n=270)
Restore/sandboxes=18/quota=0.25/age=2m          36.09m ±  0%   36.07m ± 0%        ~ (p=0.773 n=270)
WarmResume/sandboxes=18/quota=1/age=2m          39.74m ±  1%   30.33m ± 0%  -23.69% (n=270)
WakeupProbe/sandboxes=18/quota=1/age=2m        10.218m ±  3%   1.260m ± 3%  -87.67% (n=270)
FirstRequest/sandboxes=18/quota=1/age=2m        978.1µ ±  1%   955.1µ ± 1%   -2.34% (p=0.000 n=270)
Restore/sandboxes=18/quota=1/age=2m             22.74m ±  1%   22.57m ± 1%   -0.75% (p=0.014 n=270)
WarmResume/sandboxes=18/quota=none/age=2m       39.96m ±  0%   36.50m ± 1%   -8.67% (n=270)
WakeupProbe/sandboxes=18/quota=none/age=2m      1.856m ±  4%   1.421m ± 2%  -23.42% (n=270)
FirstRequest/sandboxes=18/quota=none/age=2m     1.265m ±  3%   1.013m ± 2%  -19.92% (n=270)
Restore/sandboxes=18/quota=none/age=2m          31.17m ±  1%   28.37m ± 1%   -9.01% (n=270)
WarmResume/sandboxes=1/quota=0.25/age=0         57.94m ±  3%   57.41m ± 2%        ~ (p=0.567 n=320)
WakeupProbe/sandboxes=1/quota=0.25/age=0        2.148m ±  4%   2.008m ± 3%   -6.53% (p=0.009 n=320)
FirstRequest/sandboxes=1/quota=0.25/age=0      1024.1µ ±  3%   946.9µ ± 3%   -7.54% (p=0.000 n=320)
Restore/sandboxes=1/quota=0.25/age=0            48.73m ±  3%   48.29m ± 2%        ~ (p=0.780 n=320)
WarmResume/sandboxes=44/quota=0.25/age=0        43.58m ±  0%   43.45m ± 0%   -0.31% (p=0.000 n=14080)
WakeupProbe/sandboxes=44/quota=0.25/age=0       1.433m ±  0%   1.360m ± 0%   -5.06% (n=14080)
FirstRequest/sandboxes=44/quota=0.25/age=0      937.8µ ±  0%   862.2µ ± 0%   -8.07% (n=14080)
Restore/sandboxes=44/quota=0.25/age=0           36.29m ±  0%   36.26m ± 0%        ~ (p=0.977 n=14080)
geomean                                         9.531m         6.683m       -29.88%

goos: linux
goarch: amd64
pkg: sandbox checkpoint/restore churn harness (c4-standard-48, COS 129)
                                             │ autoresearch │                 after                 │
                                             │    sec/op    │   sec/op     vs base                  │
WarmResume/sandboxes=176/quota=0.25/age=2m      45.08m ± 0%   44.94m ± 0%  -0.33% (p=0.000 n=2640)
WakeupProbe/sandboxes=176/quota=0.25/age=2m     1.589m ± 3%   1.462m ± 2%  -8.00% (n=2640)
FirstRequest/sandboxes=176/quota=0.25/age=2m   1027.8µ ± 0%   947.1µ ± 0%  -7.85% (n=2640)
Restore/sandboxes=176/quota=0.25/age=2m         36.16m ± 0%   36.17m ± 0%       ~ (p=0.570 n=2640)
WarmResume/sandboxes=18/quota=0.25/age=2m       44.32m ± 0%   44.12m ± 0%  -0.45% (p=0.011 n=270)
WakeupProbe/sandboxes=18/quota=0.25/age=2m      1.613m ± 9%   1.530m ± 9%  -5.16% (p=0.003 n=270)
FirstRequest/sandboxes=18/quota=0.25/age=2m    1032.7µ ± 1%   962.1µ ± 1%  -6.84% (n=270)
Restore/sandboxes=18/quota=0.25/age=2m          36.03m ± 0%   36.07m ± 0%       ~ (p=0.930 n=270)
WarmResume/sandboxes=18/quota=1/age=2m          30.51m ± 0%   30.33m ± 0%  -0.59% (p=0.016 n=270)
WakeupProbe/sandboxes=18/quota=1/age=2m         1.337m ± 6%   1.260m ± 3%  -5.76% (p=0.000 n=270)
FirstRequest/sandboxes=18/quota=1/age=2m       1030.8µ ± 1%   955.1µ ± 1%  -7.34% (n=270)
Restore/sandboxes=18/quota=1/age=2m             22.67m ± 1%   22.57m ± 1%       ~ (p=0.122 n=270)
WarmResume/sandboxes=18/quota=none/age=2m       36.61m ± 1%   36.50m ± 1%       ~ (p=0.393 n=270)
WakeupProbe/sandboxes=18/quota=none/age=2m      1.506m ± 4%   1.421m ± 2%  -5.63% (p=0.000 n=270)
FirstRequest/sandboxes=18/quota=none/age=2m     1.084m ± 1%   1.013m ± 2%  -6.50% (n=270)
Restore/sandboxes=18/quota=none/age=2m          28.20m ± 1%   28.37m ± 1%  +0.60% (p=0.017 n=270)
WarmResume/sandboxes=1/quota=0.25/age=0         58.72m ± 2%   57.41m ± 2%       ~ (p=0.109 n=320)
WakeupProbe/sandboxes=1/quota=0.25/age=0        2.148m ± 5%   2.008m ± 3%  -6.54% (p=0.008 n=320)
FirstRequest/sandboxes=1/quota=0.25/age=0      1016.2µ ± 2%   946.9µ ± 3%  -6.81% (p=0.000 n=320)
Restore/sandboxes=1/quota=0.25/age=0            49.32m ± 2%   48.29m ± 2%       ~ (p=0.309 n=320)
WarmResume/sandboxes=44/quota=0.25/age=0        43.46m ± 0%   43.45m ± 0%       ~ (p=0.932 n=14080)
WakeupProbe/sandboxes=44/quota=0.25/age=0       1.435m ± 0%   1.360m ± 0%  -5.19% (n=14080)
FirstRequest/sandboxes=44/quota=0.25/age=0      940.3µ ± 0%   862.2µ ± 0%  -8.31% (n=14080)
Restore/sandboxes=44/quota=0.25/age=0           36.18m ± 0%   36.26m ± 0%  +0.25% (p=0.000 n=14080)
geomean                                         6.934m        6.683m       -3.61%

goos: linux
goarch: amd64
cpu: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz
                                        │   before    │               after                │
                                        │   sec/op    │   sec/op     vs base               │
SizeNginx-48                               1.425 ± 4%    1.412 ± 2%       ~ (p=0.838 n=16)
SizeNode-48                                2.198 ± 3%    2.235 ± 2%       ~ (p=0.086 n=16)
StartupEmpty-48                           192.6m ± 3%   195.0m ± 3%  +1.25% (p=0.039 n=16)
StartupNginx-48                           535.1m ± 2%   538.7m ± 3%       ~ (p=0.539 n=16)
StartupNode-48                            691.3m ± 2%   695.3m ± 2%       ~ (p=0.468 n=16)
CheckpointEmpty-48                        65.36m ± 3%   67.93m ± 2%  +3.93% (p=0.006 n=16)
RestoreEmpty-48                           194.3m ± 1%   193.5m ± 1%       ~ (p=0.445 n=16)
CheckpointNginx-48                        104.2m ± 2%   105.2m ± 9%       ~ (p=0.239 n=16)
RestoreNginx-48                           258.6m ± 2%   258.5m ± 2%       ~ (p=0.897 n=16)
Sysbench/testname.CPU-48                  457.4µ ± 3%   454.0µ ± 1%       ~ (p=0.149 n=16)
Sysbench/testname.CPU/threads.16-48       33.74µ ± 2%   34.15µ ± 2%       ~ (p=0.131 n=16)
Sysbench/testname.Memory-48                1.335 ± 0%    1.337 ± 0%       ~ (p=0.642 n=16)
Sysbench/testname.Memory/threads.16-48     1.302 ± 1%    1.295 ± 1%  -0.51% (p=0.039 n=16)
Sysbench/testname.Mutex-48                4.635µ ± 2%   4.567µ ± 1%       ~ (p=0.180 n=16)
Syscallbench/syscall.getpid-48            3.140µ ± 1%   3.149µ ± 1%       ~ (p=1.000 n=16)
Syscallbench/syscall.getpidopt-48         1.066µ ± 3%   1.123µ ± 1%  +5.44% (p=0.000 n=16)
SyscallUnderSeccomp/cacheable.false-48    1.113µ ± 1%   1.117µ ± 1%       ~ (p=0.622 n=16)
SyscallUnderSeccomp/cacheable.true-48     1.094µ ± 2%   1.103µ ± 1%       ~ (p=0.116 n=16)
ForkUnderSeccomp/seccomp.none-48          351.8µ ± 0%   352.4µ ± 1%       ~ (p=0.564 n=16)
ForkUnderSeccomp/seccomp.cacheable-48     357.0µ ± 1%   357.4µ ± 1%       ~ (p=0.926 n=16)
ForkUnderSeccomp/seccomp.uncacheable-48   357.9µ ± 0%   357.3µ ± 1%       ~ (p=0.224 n=16)
geomean                                   3.827m        3.851m       +0.62%

             │            before            │                        after                        │
             │ average_container_size.bytes │ average_container_size.bytes  vs base               │
SizeNginx-48                   47.53M ± 33%                   43.17M ± 25%       ~ (p=0.696 n=16)
SizeNode-48                    64.36M ± 41%                   72.88M ± 33%       ~ (p=0.985 n=16)
geomean                        55.31M                         56.09M        +1.42%

                                    │            before            │                        after                        │
                                    │ cpu_events.events_per_second │ cpu_events.events_per_second  vs base               │
Sysbench/testname.CPU-48                               3.053k ± 1%                    3.072k ± 1%  +0.64% (p=0.018 n=16)
Sysbench/testname.CPU/threads.16-48                    41.42k ± 1%                    41.20k ± 1%       ~ (p=0.220 n=16)
geomean                                                11.25k                         11.25k       +0.04%

                                       │              before              │                          after                          │
                                       │ memory_operations.ops_per_second │ memory_operations.ops_per_second  vs base               │
Sysbench/testname.Memory-48                                   1.023M ± 1%                        1.021M ± 0%       ~ (p=0.539 n=16)
Sysbench/testname.Memory/threads.16-48                        1.072M ± 1%                        1.073M ± 1%       ~ (p=0.564 n=16)
geomean                                                       1.048M                             1.046M       -0.11%

                           │          before          │                      after                      │
                           │ average_execution_time.s │ average_execution_time.s  vs base               │
Sysbench/testname.Mutex-48                739.8m ± 2%                761.7m ± 1%  +2.96% (p=0.015 n=16)

                           │      before       │                  after                   │
                           │ average_latency.s │ average_latency.s  vs base               │
Sysbench/testname.Mutex-48         739.8m ± 2%         761.8m ± 1%  +2.97% (p=0.015 n=16)

                           │         before          │                   after                   │
                           │ stddev_execution_time.s │ stddev_execution_time.s  vs base          │
Sysbench/testname.Mutex-48                0.000 ± 0%                0.000 ± 0%  ~ (p=0.484 n=16)

Benchmark environment

GCE c4-standard-48, COS cos-129-19506-448-8 (kernel 6.12.105+), hyperdisk-balanced, runsc on systrap. before is master at cd90d7a74, after is eeb83261a. The two later commits come from review: they change Timekeeper.endRestore, which runs once per restore, and add an error check. autoresearch is the issue's patch rebased onto cd90d7a74.

The churn harness follows the issue's description, as the original is not public. Each sandbox runs a small Go HTTP app with a 1 s periodic job and 10 housekeeping timers, has its own veth with --network=sandbox, and is checkpointed and restored 2 minutes later (age=2m) or at once (age=0). Samples are activations: Restore is runsc restore --detach, WakeupProbe the time from then until GET /healthz answers, FirstRequest the next GET /work, and WarmResume the time from runsc create to the end of the first request. Each configuration ran on its own identical VM, each build alone, in an order that rotates between configurations. Activations within a run are not independent, so the per-activation p-values overstate significance. The sandboxes=44/age=0 configuration ran 4 rounds; their per-run medians give FirstRequest -8.05%, WakeupProbe -4.66% and WarmResume -0.28% (p=0.029, n=4). An earlier single run of it showed WarmResume +6.5%, which these rounds did not reproduce.

The base benchmarks are the real //test/benchmarks/base targets, run against a docker runtime for each build, in the order before, after, after, before, with -test.count=8. None of them exercises code that this PR changes outside of restore. Run order alone moves some microbenchmarks by several percent: Syscallbench/getpidopt measured +9.7% with after second and -4.1% with after first. SizeEmpty fails with both builds.

Testing

New syscall tests, which pass on native and on systrap with and without save/restore, for relative CLOCK_REALTIME sleeps and timers across a checkpoint, CLOCK_BOOTTIME, the VDSO after restore, and sysinfo(2) uptime.
New container test TestCheckpointRestoreClocks (systrap, KVM) and Timekeeper unit tests.
No regressions in 27 timing-related syscall tests, with and without save/restore, or in the TestCheckpoint* container tests.

google-cla · 2026-10-30T06:31:05Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

milantracy · 2026-10-30T06:53:39Z

no 'Generated with Claude Code' please. Claude code has never signed CLA

Linux arms the relative CLOCK_REALTIME timers of clock_nanosleep(2), timerfd_settime(2) and timer_settime(2) on CLOCK_MONOTONIC (see __hrtimer_setup), so that they measure elapsed time, and discontinuous changes to the realtime clock don't affect them. gVisor instead waited for the realtime clock to reach the deadline, so such sleeps and timers were cut short or extended if the realtime clock jumped, for example when the host clock is stepped, or when a checkpointed sandbox is restored. This matters in practice because glibc implements nanosleep(3), and thus sleep(3) and usleep(3), with relative CLOCK_REALTIME clock_nanosleep(2). CLOCK_REALTIME timers move between the two clocks as they are armed with relative and absolute times, with the new SampledTimer.SetWithClock.

After a restore, CLOCK_MONOTONIC jumped forward by the time the sandbox spent checkpointed, to make it look as if the application had not been scheduled for that long. For a sandbox restored minutes or hours after its checkpoint, this means that every CLOCK_MONOTONIC timer and timeout that was pending at checkpoint time (sleeps, poll and epoll_wait timeouts, language runtimes' periodic GC and housekeeping, heartbeats) expires at once right after restore, however long it had left to run. The resulting burst of work competes with whatever the restored application is woken up to do, which is especially costly under a small CPU quota. Instead, make the clocks behave as they do across a Linux system suspend: - CLOCK_MONOTONIC continues from its value at checkpoint time when the restored kernel resumes. Until then, it stays at that value, so that the time that the restore takes, which includes loading memory unless restoring in the background, does not count either. - CLOCK_BOOTTIME, previously an alias of CLOCK_MONOTONIC, becomes a distinct clock that counts the time spent checkpointed and restoring, like it counts the time spent suspended on Linux. It keeps the behavior that CLOCK_MONOTONIC had, so applications that want to know how much time really passed, such as to expire idle connections, can still do so. sysinfo(2) uptime is based on it, rounded up to the second as on Linux, and so stays consistent with /proc/uptime. - CLOCK_REALTIME is unchanged and still reflects the time of day. The VDSO serves CLOCK_BOOTTIME with an offset from CLOCK_MONOTONIC published in its parameter page.

Calibrating the clocks takes two samples, which the Timekeeper takes when it updates the clocks, at most every ApproxUpdateInterval (1 second). Until then, the VDSO cannot compute the time, and falls back to system calls. A restored sandbox gets new clocks, so for about a second after every restore, all application clock reads are system calls: a Go application's time.Now() measured 4.6-11.4µs, instead of 25-70ns. This is when restored applications are woken up to serve requests, and when most clock reads happen, so take the first sample early in the restore. The Timekeeper then takes the second sample when the kernel is loaded, which completes the calibration before the application resumes. The first frequency estimate then comes from the tens of milliseconds between the two samples, rather than a second. Its error, at most the samples' overhead divided by that interval (typically well under 0.1%), is in the range that the following updates correct without a time jump, as they do for any calibration error.

kerumeto · 2026-10-30T23:06:07Z

@@ -76,6 +79,24 @@ type Timekeeper struct {
 	// It is set only once, by SetClocks.


End restore now sets this too right?

Yes, thanks. Updated the comment in 4f5ce20: SetClocks sets it in a fresh sentry, and endRestore sets it when a restored sentry resumes.

kerumeto · 2026-10-30T23:07:37Z

+	newNow := now
+	if oldC := t.clock; oldC != c {
+		oldC.EventUnregister(&t.entry)
+		c.EventRegister(&t.entry)


Why do we ignore the error here?

No good reason, I copied it from the existing SetClock. The SampledClocks that exist today (timekeeperClock and the one in contexttest) never fail EventRegister, since they embed ktime.NoClockEvents. But SampledTimer.init panics if it fails, so SetWithClock and SetClock now do the same (4f5ce20).

kerumeto · 2026-10-30T23:09:31Z

+	if err != nil {
+		panic("Unable to get current realtime: " + err.Error())
+	}
+	t.monotonicOffset = t.saveMonotonic - nowMonotonic


AI:
In endRestore, the offset is computed from the old calibration, and then update() may replace that calibration. So the first CLOCK_MONOTONIC reads after resume could come out slightly below the checkpoint value. The safety check that would normally stop this, monotonicLowerBound, is 0 after restore.

Good catch, fixed in 1431321. endRestore now computes the offsets from the same clocks.Update result that it writes to the VDSO, inside params.Write, so monotonic time continues from exactly saveMonotonic. monotonicLowerBound also starts at saveMonotonic on restore. TestTimekeeperRestoreClockError covers it: without the fix, a 1µs clock error that a parked update corrects makes monotonic time resume 1µs below saveMonotonic, in both GetTime and the VDSO.

Timekeeper.endRestore computed the new monotonic offset from the current time of the clocks, then updated the clocks and published them to the VDSO. If the updater was parked, as it is after 2 seconds without clock reads during a long restore, the update recalibrates the clocks at once, and so corrects the error that their estimate had built up. Monotonic time then continued from saveMonotonic minus this error, both in GetTime and in the VDSO, and could go backwards across the checkpoint. Compute the offsets from the parameters of the update that is written to the VDSO instead. Also start the lower bound of monotonic time at saveMonotonic.

manninglucas

LGTM. Clean, well-sequenced fix that aligns gVisor's CLOCK_MONOTONIC, CLOCK_BOOTTIME, sysinfo(2) uptime, and relative CLOCK_REALTIME timers/sleeps with Linux behavior across suspend/resume, while pre-calibrating the VDSO during restore so resumed workloads don't fall back to clock_gettime syscalls.

Security: This PR modifies Sentry timekeeping state and appends an 8-byte boottimeOffset field to the VDSO parameter page (vdsoParams / struct params, growing the structure from 104 to 112 bytes within the 4096-byte page). It introduces no new host syscalls, seccomp rules, or untrusted input paths, preserves t.mu -> t.updateMu and clockSeq synchronization, and bounds boottimeOffset updates to nonnegative nowRealtime - saveRealtime deltas during restore.

[🤖] This review is part of an experimental auto-review process to help speed up gVisor review velocity. It is a work in progress. Assisted-By: Claude

manninglucas · 2026-10-30T23:21:27Z

 	// It is set only once, by SetClocks.
 	monotonicOffset int64 `state:"nosave"`


Minor nit: this comment on monotonicOffset (// It is set only once, by SetClocks.) is now slightly stale since SetClocks only sets it on fresh boot (t.restored == nil) and endRestore() sets it in Resume after restore.

Updated in 4f5ce20, thanks.

SampledTimer.init panics if the clock's EventRegister fails, but SetClock and the new SetWithClock ignored its error. No SampledClock returns an error today, but check it in the same way. Also update the comment of Timekeeper.monotonicOffset, which endRestore sets after restore.

### The idea On restore, gVisor advances `CLOCK_MONOTONIC` by the time that the sandbox spent checkpointed. In a sandbox restored minutes later, every pending timer and timeout therefore expires at once, right when the application wakes up to serve a request. This PR makes the clocks behave across checkpoint/restore as they do across a Linux suspend: - `CLOCK_MONOTONIC` continues from its value at checkpoint time when the restored kernel resumes. - `CLOCK_BOOTTIME`, until now an alias of `CLOCK_MONOTONIC`, becomes a separate clock that counts the checkpointed time, as it counts suspend time on Linux. Applications that need the real elapsed time, e.g. to expire idle connections, can use it. `sysinfo(2)` uptime uses it too. - Relative `CLOCK_REALTIME` sleeps and timers, which glibc uses for `sleep(3)` and `nanosleep(3)`, are measured with `CLOCK_MONOTONIC`, as on Linux. The restore also takes the first clock calibration sample early. Without it, the VDSO cannot compute the time for about 1 s after each restore, and every application clock read is a system call. Unlike the autoresearch patch in #16135, this PR does not freeze `CLOCK_BOOTTIME`, also moves relative `CLOCK_REALTIME` timers (`timerfd`, POSIX timers) to `CLOCK_MONOTONIC`, does not count the restore itself in `CLOCK_MONOTONIC`, and adds the early calibration. Fixes #16135 `COMPATIBILITY:NEW-APP-CONFIG`: in sandboxes restored after a delay, applications ran all pending timers at once, and relative `CLOCK_REALTIME` sleeps and timers changed with the realtime clock, unlike on Linux. ### Performance impact - After restoring a 2-minute-old checkpoint at 0.25 CPU with 176 sandboxes: time to app readiness 6.28 → 1.46 ms, first request 2.48 → 0.95 ms, warm resume -12%. The gain holds at 1 CPU and with no CPU limit. - Restoring a fresh checkpoint: first request -7.5% to -8% and readiness -5% to -6.5%, from the early calibration. The restore itself is unchanged. - Against the autoresearch patch: first request -6.5% to -8.3% and readiness -5% to -8% in every configuration (geomean -3.6%). The restore itself and warm resume are within ±0.6%. - `//test/benchmarks/base`: no change overall (geomean +0.6%). <details> <summary>Benchmarks: master vs this PR, and the autoresearch patch vs this PR</summary> ``` goos: linux goarch: amd64 pkg: sandbox checkpoint/restore churn harness (c4-standard-48, COS 129) │ before │ after │ │ sec/op │ sec/op vs base │ WarmResume/sandboxes=176/quota=0.25/age=2m 51.00m ± 0% 44.94m ± 0% -11.89% (n=2640) WakeupProbe/sandboxes=176/quota=0.25/age=2m 6.282m ± 1% 1.462m ± 2% -76.73% (n=2640) FirstRequest/sandboxes=176/quota=0.25/age=2m 2480.2µ ± 4% 947.1µ ± 0% -61.82% (n=2640) Restore/sandboxes=176/quota=0.25/age=2m 36.14m ± 0% 36.17m ± 0% ~ (p=0.122 n=2640) WarmResume/sandboxes=18/quota=0.25/age=2m 51.44m ± 3% 44.12m ± 0% -14.23% (n=270) WakeupProbe/sandboxes=18/quota=0.25/age=2m 6.468m ± 2% 1.530m ± 9% -76.34% (n=270) FirstRequest/sandboxes=18/quota=0.25/age=2m 2627.7µ ± 47% 962.1µ ± 1% -63.39% (n=270) Restore/sandboxes=18/quota=0.25/age=2m 36.09m ± 0% 36.07m ± 0% ~ (p=0.773 n=270) WarmResume/sandboxes=18/quota=1/age=2m 39.74m ± 1% 30.33m ± 0% -23.69% (n=270) WakeupProbe/sandboxes=18/quota=1/age=2m 10.218m ± 3% 1.260m ± 3% -87.67% (n=270) FirstRequest/sandboxes=18/quota=1/age=2m 978.1µ ± 1% 955.1µ ± 1% -2.34% (p=0.000 n=270) Restore/sandboxes=18/quota=1/age=2m 22.74m ± 1% 22.57m ± 1% -0.75% (p=0.014 n=270) WarmResume/sandboxes=18/quota=none/age=2m 39.96m ± 0% 36.50m ± 1% -8.67% (n=270) WakeupProbe/sandboxes=18/quota=none/age=2m 1.856m ± 4% 1.421m ± 2% -23.42% (n=270) FirstRequest/sandboxes=18/quota=none/age=2m 1.265m ± 3% 1.013m ± 2% -19.92% (n=270) Restore/sandboxes=18/quota=none/age=2m 31.17m ± 1% 28.37m ± 1% -9.01% (n=270) WarmResume/sandboxes=1/quota=0.25/age=0 57.94m ± 3% 57.41m ± 2% ~ (p=0.567 n=320) WakeupProbe/sandboxes=1/quota=0.25/age=0 2.148m ± 4% 2.008m ± 3% -6.53% (p=0.009 n=320) FirstRequest/sandboxes=1/quota=0.25/age=0 1024.1µ ± 3% 946.9µ ± 3% -7.54% (p=0.000 n=320) Restore/sandboxes=1/quota=0.25/age=0 48.73m ± 3% 48.29m ± 2% ~ (p=0.780 n=320) WarmResume/sandboxes=44/quota=0.25/age=0 43.58m ± 0% 43.45m ± 0% -0.31% (p=0.000 n=14080) WakeupProbe/sandboxes=44/quota=0.25/age=0 1.433m ± 0% 1.360m ± 0% -5.06% (n=14080) FirstRequest/sandboxes=44/quota=0.25/age=0 937.8µ ± 0% 862.2µ ± 0% -8.07% (n=14080) Restore/sandboxes=44/quota=0.25/age=0 36.29m ± 0% 36.26m ± 0% ~ (p=0.977 n=14080) geomean 9.531m 6.683m -29.88% ``` ``` goos: linux goarch: amd64 pkg: sandbox checkpoint/restore churn harness (c4-standard-48, COS 129) │ autoresearch │ after │ │ sec/op │ sec/op vs base │ WarmResume/sandboxes=176/quota=0.25/age=2m 45.08m ± 0% 44.94m ± 0% -0.33% (p=0.000 n=2640) WakeupProbe/sandboxes=176/quota=0.25/age=2m 1.589m ± 3% 1.462m ± 2% -8.00% (n=2640) FirstRequest/sandboxes=176/quota=0.25/age=2m 1027.8µ ± 0% 947.1µ ± 0% -7.85% (n=2640) Restore/sandboxes=176/quota=0.25/age=2m 36.16m ± 0% 36.17m ± 0% ~ (p=0.570 n=2640) WarmResume/sandboxes=18/quota=0.25/age=2m 44.32m ± 0% 44.12m ± 0% -0.45% (p=0.011 n=270) WakeupProbe/sandboxes=18/quota=0.25/age=2m 1.613m ± 9% 1.530m ± 9% -5.16% (p=0.003 n=270) FirstRequest/sandboxes=18/quota=0.25/age=2m 1032.7µ ± 1% 962.1µ ± 1% -6.84% (n=270) Restore/sandboxes=18/quota=0.25/age=2m 36.03m ± 0% 36.07m ± 0% ~ (p=0.930 n=270) WarmResume/sandboxes=18/quota=1/age=2m 30.51m ± 0% 30.33m ± 0% -0.59% (p=0.016 n=270) WakeupProbe/sandboxes=18/quota=1/age=2m 1.337m ± 6% 1.260m ± 3% -5.76% (p=0.000 n=270) FirstRequest/sandboxes=18/quota=1/age=2m 1030.8µ ± 1% 955.1µ ± 1% -7.34% (n=270) Restore/sandboxes=18/quota=1/age=2m 22.67m ± 1% 22.57m ± 1% ~ (p=0.122 n=270) WarmResume/sandboxes=18/quota=none/age=2m 36.61m ± 1% 36.50m ± 1% ~ (p=0.393 n=270) WakeupProbe/sandboxes=18/quota=none/age=2m 1.506m ± 4% 1.421m ± 2% -5.63% (p=0.000 n=270) FirstRequest/sandboxes=18/quota=none/age=2m 1.084m ± 1% 1.013m ± 2% -6.50% (n=270) Restore/sandboxes=18/quota=none/age=2m 28.20m ± 1% 28.37m ± 1% +0.60% (p=0.017 n=270) WarmResume/sandboxes=1/quota=0.25/age=0 58.72m ± 2% 57.41m ± 2% ~ (p=0.109 n=320) WakeupProbe/sandboxes=1/quota=0.25/age=0 2.148m ± 5% 2.008m ± 3% -6.54% (p=0.008 n=320) FirstRequest/sandboxes=1/quota=0.25/age=0 1016.2µ ± 2% 946.9µ ± 3% -6.81% (p=0.000 n=320) Restore/sandboxes=1/quota=0.25/age=0 49.32m ± 2% 48.29m ± 2% ~ (p=0.309 n=320) WarmResume/sandboxes=44/quota=0.25/age=0 43.46m ± 0% 43.45m ± 0% ~ (p=0.932 n=14080) WakeupProbe/sandboxes=44/quota=0.25/age=0 1.435m ± 0% 1.360m ± 0% -5.19% (n=14080) FirstRequest/sandboxes=44/quota=0.25/age=0 940.3µ ± 0% 862.2µ ± 0% -8.31% (n=14080) Restore/sandboxes=44/quota=0.25/age=0 36.18m ± 0% 36.26m ± 0% +0.25% (p=0.000 n=14080) geomean 6.934m 6.683m -3.61% ``` ``` goos: linux goarch: amd64 cpu: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz │ before │ after │ │ sec/op │ sec/op vs base │ SizeNginx-48 1.425 ± 4% 1.412 ± 2% ~ (p=0.838 n=16) SizeNode-48 2.198 ± 3% 2.235 ± 2% ~ (p=0.086 n=16) StartupEmpty-48 192.6m ± 3% 195.0m ± 3% +1.25% (p=0.039 n=16) StartupNginx-48 535.1m ± 2% 538.7m ± 3% ~ (p=0.539 n=16) StartupNode-48 691.3m ± 2% 695.3m ± 2% ~ (p=0.468 n=16) CheckpointEmpty-48 65.36m ± 3% 67.93m ± 2% +3.93% (p=0.006 n=16) RestoreEmpty-48 194.3m ± 1% 193.5m ± 1% ~ (p=0.445 n=16) CheckpointNginx-48 104.2m ± 2% 105.2m ± 9% ~ (p=0.239 n=16) RestoreNginx-48 258.6m ± 2% 258.5m ± 2% ~ (p=0.897 n=16) Sysbench/testname.CPU-48 457.4µ ± 3% 454.0µ ± 1% ~ (p=0.149 n=16) Sysbench/testname.CPU/threads.16-48 33.74µ ± 2% 34.15µ ± 2% ~ (p=0.131 n=16) Sysbench/testname.Memory-48 1.335 ± 0% 1.337 ± 0% ~ (p=0.642 n=16) Sysbench/testname.Memory/threads.16-48 1.302 ± 1% 1.295 ± 1% -0.51% (p=0.039 n=16) Sysbench/testname.Mutex-48 4.635µ ± 2% 4.567µ ± 1% ~ (p=0.180 n=16) Syscallbench/syscall.getpid-48 3.140µ ± 1% 3.149µ ± 1% ~ (p=1.000 n=16) Syscallbench/syscall.getpidopt-48 1.066µ ± 3% 1.123µ ± 1% +5.44% (p=0.000 n=16) SyscallUnderSeccomp/cacheable.false-48 1.113µ ± 1% 1.117µ ± 1% ~ (p=0.622 n=16) SyscallUnderSeccomp/cacheable.true-48 1.094µ ± 2% 1.103µ ± 1% ~ (p=0.116 n=16) ForkUnderSeccomp/seccomp.none-48 351.8µ ± 0% 352.4µ ± 1% ~ (p=0.564 n=16) ForkUnderSeccomp/seccomp.cacheable-48 357.0µ ± 1% 357.4µ ± 1% ~ (p=0.926 n=16) ForkUnderSeccomp/seccomp.uncacheable-48 357.9µ ± 0% 357.3µ ± 1% ~ (p=0.224 n=16) geomean 3.827m 3.851m +0.62% │ before │ after │ │ average_container_size.bytes │ average_container_size.bytes vs base │ SizeNginx-48 47.53M ± 33% 43.17M ± 25% ~ (p=0.696 n=16) SizeNode-48 64.36M ± 41% 72.88M ± 33% ~ (p=0.985 n=16) geomean 55.31M 56.09M +1.42% │ before │ after │ │ cpu_events.events_per_second │ cpu_events.events_per_second vs base │ Sysbench/testname.CPU-48 3.053k ± 1% 3.072k ± 1% +0.64% (p=0.018 n=16) Sysbench/testname.CPU/threads.16-48 41.42k ± 1% 41.20k ± 1% ~ (p=0.220 n=16) geomean 11.25k 11.25k +0.04% │ before │ after │ │ memory_operations.ops_per_second │ memory_operations.ops_per_second vs base │ Sysbench/testname.Memory-48 1.023M ± 1% 1.021M ± 0% ~ (p=0.539 n=16) Sysbench/testname.Memory/threads.16-48 1.072M ± 1% 1.073M ± 1% ~ (p=0.564 n=16) geomean 1.048M 1.046M -0.11% │ before │ after │ │ average_execution_time.s │ average_execution_time.s vs base │ Sysbench/testname.Mutex-48 739.8m ± 2% 761.7m ± 1% +2.96% (p=0.015 n=16) │ before │ after │ │ average_latency.s │ average_latency.s vs base │ Sysbench/testname.Mutex-48 739.8m ± 2% 761.8m ± 1% +2.97% (p=0.015 n=16) │ before │ after │ │ stddev_execution_time.s │ stddev_execution_time.s vs base │ Sysbench/testname.Mutex-48 0.000 ± 0% 0.000 ± 0% ~ (p=0.484 n=16) ``` </details> <details> <summary>Benchmark environment</summary> GCE c4-standard-48, COS `cos-129-19506-448-8` (kernel 6.12.105+), hyperdisk-balanced, runsc on systrap. `before` is master at `cd90d7a74`, `after` is `eeb83261a`. The two later commits come from review: they change `Timekeeper.endRestore`, which runs once per restore, and add an error check. `autoresearch` is the issue's patch rebased onto `cd90d7a74`. The churn harness follows the issue's description, as the original is not public. Each sandbox runs a small Go HTTP app with a 1 s periodic job and 10 housekeeping timers, has its own veth with `--network=sandbox`, and is checkpointed and restored 2 minutes later (`age=2m`) or at once (`age=0`). Samples are activations: `Restore` is `runsc restore --detach`, `WakeupProbe` the time from then until `GET /healthz` answers, `FirstRequest` the next `GET /work`, and `WarmResume` the time from `runsc create` to the end of the first request. Each configuration ran on its own identical VM, each build alone, in an order that rotates between configurations. Activations within a run are not independent, so the per-activation p-values overstate significance. The `sandboxes=44/age=0` configuration ran 4 rounds; their per-run medians give FirstRequest -8.05%, WakeupProbe -4.66% and WarmResume -0.28% (p=0.029, n=4). An earlier single run of it showed WarmResume +6.5%, which these rounds did not reproduce. The base benchmarks are the real `//test/benchmarks/base` targets, run against a docker runtime for each build, in the order before, after, after, before, with `-test.count=8`. None of them exercises code that this PR changes outside of restore. Run order alone moves some microbenchmarks by several percent: `Syscallbench/getpidopt` measured +9.7% with `after` second and -4.1% with `after` first. `SizeEmpty` fails with both builds. </details> ### Testing - New syscall tests, which pass on native and on systrap with and without save/restore, for relative `CLOCK_REALTIME` sleeps and timers across a checkpoint, `CLOCK_BOOTTIME`, the VDSO after restore, and `sysinfo(2)` uptime. - New container test `TestCheckpointRestoreClocks` (systrap, KVM) and Timekeeper unit tests. - No regressions in 27 timing-related syscall tests, with and without save/restore, or in the `TestCheckpoint*` container tests. FUTURE_COPYBARA_INTEGRATE_REVIEW=#16733 from aaltinaydev:freeze-monotonic 4f5ce20 PiperOrigin-RevId: 1010903892

github-actions Bot requested review from manninglucas and relkochta October 30, 2026 06:35

aaltinaydev removed request for manninglucas and relkochta October 30, 2026 06:39

Anil Altinay added 3 commits October 30, 2026 01:12

aaltinaydev force-pushed the freeze-monotonic branch from 3ffb5ff to eeb8326 Compare October 30, 2026 08:45

aaltinaydev requested review from kerumeto and manninglucas October 30, 2026 22:57

kerumeto requested changes Oct 30, 2026

View reviewed changes

manninglucas approved these changes Oct 30, 2026

View reviewed changes

kerumeto approved these changes Oct 31, 2026

View reviewed changes

kerumeto added the ready to pull label Oct 31, 2026

copybara-service Bot mentioned this pull request Oct 31, 2026

Don't count time spent checkpointed in CLOCK_MONOTONIC #16773

Merged

copybara-service Bot merged commit 25172e5 into google:master Oct 31, 2026
14 checks passed

tamird mentioned this pull request Nov 13, 2026

Exercise the IPv6 multicast device binding before binding an address #16925

Open

Navigation Menu

Repository navigation

Don't count time spent checkpointed in CLOCK_MONOTONIC - #16733

Don't count time spent checkpointed in CLOCK_MONOTONIC#16733
copybara-service[bot] merged 5 commits into
google:masterfrom
aaltinaydev:freeze-monotonic

aaltinaydev commented Oct 30, 2026 •

edited

Loading

Uh oh!

google-cla Bot commented Oct 30, 2026

Uh oh!

milantracy commented Oct 30, 2026

Uh oh!

kerumeto Oct 30, 2026

Uh oh!

aaltinaydev Oct 30, 2026

Uh oh!

kerumeto Oct 30, 2026

Uh oh!

aaltinaydev Oct 30, 2026

Uh oh!

kerumeto Oct 30, 2026

Uh oh!

aaltinaydev Oct 30, 2026

Uh oh!

manninglucas left a comment

Uh oh!

manninglucas Oct 30, 2026

Uh oh!

aaltinaydev Oct 30, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -76,6 +79,24 @@ type Timekeeper struct {
		// It is set only once, by SetClocks.

		// It is set only once, by SetClocks.
		monotonicOffset int64 `state:"nosave"`

Conversation

aaltinaydev commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The idea

Performance impact

Testing

Uh oh!

google-cla Bot commented Oct 30, 2026

Uh oh!

milantracy commented Oct 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manninglucas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

aaltinaydev commented Oct 30, 2026 •

edited

Loading