Skip to content

Commit 06b444e

Browse files
fix(site-router): seed a PKCS#8 key, and read the failure out of the journal
Three rounds of this were spent reading the wrong source. Every check went to raw.githubusercontent.com/vyos/vyos-1x/current/..., and vyos-1x has no branch called `current` — GitHub silently serves the default branch, `rolling`, which is six months ahead of the 2026.03 circinus image the appliance boots. So each "upstream says" in the last three commit messages described code the guest does not run. The key. On the shipped image wrap_private_key has exactly one armour, PKCS#8 `-----BEGIN PRIVATE KEY-----`. The two-armour PKCS#8-or-SEC1 loop that the previous commit called measured fact is rolling-only, so switching from RSA PKCS#1 to EC SEC1 exchanged one rejected format for another and cost a second boot. genPrivateKey "ed25519" is the one sprig generator that emits PKCS#8; verified against the rendered value, and PKCS#8 is the first armour on rolling too, so it holds whichever version the appliance ends up carrying. The reporter. It could not have worked either. vyos-router.service is Type=simple, so systemd calls it started when the process forks and an `After=` on it is satisfied before the configuration is loaded — visible in the boot log, where the report printed seconds ahead of "Configuration error". The file it read is written by nothing at all: the loader's own log is written only on the success path, after the sys.exit(1) that a rejected config takes. What does persist is the journal, because conf_mode/pki.py and conf_mode/service_https.py are dispatched through vyos-configd, which logs the ConfigError it caught. The script now waits on /tmp/vyos-config-status the way upstream's own vyos-config init does, and prints that journal when the status is non-zero. No kernel argument needed, which was the other half of the plan this replaces. `system login operator-group` is removed from the chart and the baked default. The node does not exist in the circinus line — it is in rolling and absent from circinus-public-unmaintained, checked against both real refs — and cli-shell-api loadFile prints "Set failed" and continues, so it was silently dropped on every boot rather than doing anything. The version footer keeps a note rather than an edit. It is missing components the image declares, so migration_needed() is true every boot and the "captured verbatim" claim above it is no longer fully true. Naming the missing entries by hand is exactly how the earlier pki@1 mistake happened, so the note says to re-capture from a save on the image instead. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 75c8eb5 commit 06b444e

5 files changed

Lines changed: 106 additions & 53 deletions

File tree

‎packages/apps/site-router/templates/_helpers.tpl‎

Lines changed: 33 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -161,16 +161,28 @@ mode of drift here is a TLS error nobody expects.
161161
{{- if and $existing (hasKey $existing "data") (hasKey $existing.data "tls.crt") (hasKey $existing.data "tls.key") -}}
162162
{{- dict "cert" (index $existing.data "tls.crt" | b64dec) "key" (index $existing.data "tls.key" | b64dec) "serverName" $serverName | toYaml -}}
163163
{{- else -}}
164-
{{- /* EC, and not by preference. VyOS stores a private key as bare base64 and
165-
rebuilds the PEM itself on read (vyos/pki.py load_private_key), trying
166-
exactly two armours: PKCS#8 `PRIVATE KEY` and SEC1 `EC PRIVATE KEY`.
167-
sprig's genSelfSignedCert is RSA and emits PKCS#1 `RSA PRIVATE KEY`, which
168-
neither wrap parses, so the commit dies at src/conf_mode/pki.py with
169-
"Invalid private key on certificate" and the gateway boots with no
170-
configuration at all. Measured on a real appliance boot, not deduced.
171-
genPrivateKey "ecdsa" emits SEC1, which is the second armour VyOS tries,
172-
and genSelfSignedCertWithKey issues a matching certificate for it. */ -}}
173-
{{- $key := genPrivateKey "ecdsa" -}}
164+
{{- /* Ed25519, and the reason is the format rather than the curve. VyOS stores
165+
a private key as bare base64 and rebuilds the PEM itself on read
166+
(vyos/pki.py wrap_private_key), and on the SHIPPED 2026.03 image that
167+
function has exactly one armour: PKCS#8 `-----BEGIN PRIVATE KEY-----`.
168+
Anything else fails load_private_key, conf_mode/pki.py raises "Invalid
169+
private key on certificate", and the gateway boots with no configuration
170+
at all.
171+
172+
sprig gives one generator that emits PKCS#8, and this is it. genPrivateKey
173+
"rsa" emits PKCS#1 and genPrivateKey "ecdsa" emits SEC1; both are rejected,
174+
and both have been shipped here and failed a real boot.
175+
176+
Two earlier revisions of this comment cited a two-armour PKCS#8-or-SEC1
177+
loop as measured fact. That loop is real but lives on vyos-1x `rolling`,
178+
which is six months ahead of this image — and it was read from a URL
179+
naming a branch that does not exist (`.../vyos-1x/current/...`), which
180+
GitHub silently serves from the default branch. Check upstream behaviour
181+
against a ref that resolves, or against the image itself.
182+
183+
PKCS#8 is also the first armour tried on `rolling`, so this choice holds
184+
whichever version the appliance ends up carrying. */ -}}
185+
{{- $key := genPrivateKey "ed25519" -}}
174186
{{- $gen := genSelfSignedCertWithKey $serverName nil (list $serverName) 3650 $key -}}
175187
{{- dict "cert" $gen.Cert "key" $key "serverName" $serverName | toYaml -}}
176188
{{- end -}}
@@ -326,6 +338,17 @@ lockstep with the pinned image. This is the strongest form of the
326338
the whole config.boot, not just a seed, now lives beside the image. When the
327339
image bumps, re-capture config.boot from the new image's `save`.
328340
341+
KNOWN DRIFT, do not read the footer as authoritative: it is missing at least
342+
three components the 2026.03 image declares, so the version map here and the
343+
image's own do not match and `migration_needed()` is true on every boot. The
344+
practical effect is small, because VyOS prunes footer entries the system does
345+
not know and the seed rewrites config.boot from the NoCloud disk each boot
346+
anyway — but "captured verbatim" above is a claim this file no longer fully
347+
supports, and the fix is a re-capture from a `save` on the image rather than
348+
hand-editing entries in. Adding an entry by hand is specifically what NOT to
349+
do: a component named at a version the image does not carry is how a migration
350+
gets triggered rather than avoided.
351+
329352
WHY cloud-init at all (and not controller-only over the API): the controller's
330353
channel is the HTTPS REST API, which can only answer once the gateway already
331354
has an IP, the API running, AND this per-instance api-key installed — it cannot
@@ -533,11 +556,6 @@ system {
533556
}
534557
host-name "{{ $ctx.Release.Name }}"
535558
login {
536-
operator-group default {
537-
command-policy {
538-
allow "*"
539-
}
540-
}
541559
user vyos {
542560
authentication {
543561
encrypted-password "*"

‎packages/apps/site-router/tests/secret_cloudinit_test.yaml‎

Lines changed: 15 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -360,18 +360,23 @@ tests:
360360
# The format of the seeded key, pinned because getting it wrong costs the whole
361361
# gateway and does so silently at boot rather than at render.
362362
#
363-
# VyOS stores a private key as bare base64 and rebuilds the PEM itself on read
364-
# (vyos/pki.py load_private_key), trying exactly two armours: PKCS#8 `PRIVATE
365-
# KEY` and SEC1 `EC PRIVATE KEY`. sprig's genSelfSignedCert is RSA and emits
366-
# PKCS#1, which neither wrap parses, so src/conf_mode/pki.py refuses the commit
367-
# with "Invalid private key on certificate" and the guest comes up with no
368-
# configuration at all. That shipped once and was found on a live boot, not in
369-
# any test — hence this one.
370-
- it: seeds an EC private key, the one shape VyOS can read back
363+
# VyOS stores a private key as bare base64 and rebuilds the PEM itself on read,
364+
# and on the shipped 2026.03 image wrap_private_key has exactly ONE armour:
365+
# PKCS#8 `-----BEGIN PRIVATE KEY-----`. Anything else fails load_private_key,
366+
# conf_mode/pki.py refuses the commit with "Invalid private key on
367+
# certificate", and the guest comes up with no configuration at all.
368+
#
369+
# Two wrong formats have shipped here and each cost a live boot: PKCS#1 from
370+
# genSelfSignedCert, then SEC1 from genPrivateKey "ecdsa". The second was
371+
# chosen from a two-armour loop that exists only on vyos-1x `rolling`, read
372+
# from a branch name that does not resolve. genPrivateKey "ed25519" is the one
373+
# sprig generator that emits PKCS#8, and PKCS#8 is the first armour on
374+
# `rolling` too, so it holds either way.
375+
- it: seeds a PKCS#8 private key, the one shape this image can read back
371376
asserts:
372377
- matchRegex:
373378
path: stringData["tls.key"]
374-
pattern: '\A-----BEGIN EC PRIVATE KEY-----'
379+
pattern: '\A-----BEGIN PRIVATE KEY-----'
375380
documentIndex: 0
376381
# And it reaches config.boot as bare base64 under pki, with no armour and
377382
# no newlines, which is what the base64 validator on that node requires.
@@ -385,5 +390,5 @@ tests:
385390
documentIndex: 1
386391
- notMatchRegex:
387392
path: stringData.userdata
388-
pattern: 'BEGIN (RSA |EC )?PRIVATE KEY'
393+
pattern: 'BEGIN [A-Z0-9 ]*PRIVATE KEY'
389394
documentIndex: 1

‎packages/system/vyos-router-image/overlay/config.boot.default‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -37,11 +37,6 @@ system {
3737
}
3838
host-name "vyos"
3939
login {
40-
operator-group default {
41-
command-policy {
42-
allow "*"
43-
}
44-
}
4540
user vyos {
4641
authentication {
4742
encrypted-password "*"
Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,20 @@
11
[Unit]
22
Description=Report the VyOS configuration load result to the console
3-
# After, not Before: the point is to read what vyos-router left behind. It runs
4-
# whether that succeeded or failed, because "which line did it object to" is only
5-
# answerable from the log and the console is the only way off this appliance.
3+
# After= is a hint here and nothing more: vyos-router.service is Type=simple, so
4+
# systemd calls it started the instant the process forks, which happens long
5+
# before the configuration is loaded. The script waits on /tmp/vyos-config-status
6+
# the way upstream's own vyos-config init does; that is what actually orders it.
67
After=vyos-router.service
78
Wants=vyos-router.service
89

910
[Service]
1011
Type=oneshot
1112
RemainAfterExit=yes
1213
ExecStart=/usr/local/sbin/vyos-appliance-config-report.sh
14+
# The script bounds its own wait at 240s and always exits 0. This is the backstop
15+
# for a hang inside journalctl, and it is longer than that wait so a timeout here
16+
# means something other than the wait went wrong.
17+
TimeoutStartSec=300
1318

1419
[Install]
1520
WantedBy=vyos.target
Lines changed: 50 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,62 @@
11
#!/bin/sh
2-
# Print why the configuration failed, to the console, after vyos-router has tried
3-
# to load it.
2+
# Say what vyos-router made of the configuration, on the console.
43
#
5-
# vyos-router reports a rejected config.boot as the single line "Configuration
6-
# error" and puts the reason in /var/log/vyatta/vyos-config.log inside the guest.
7-
# On this appliance nothing can go and read that: there is no SSH service, the
8-
# `vyos` login is locked, and the bootloader is password-protected, so the
9-
# console is the only channel and it carries the one useless line. The bring-up
10-
# emitter does not help either, because it is installed by the configuration that
11-
# just failed to load.
4+
# A rejected config.boot reaches the console as the single line "Configuration
5+
# error". On this appliance nothing can go and read the reason: there is no SSH
6+
# service, the `vyos` login is locked, and the bootloader is password-protected.
7+
# The bring-up emitter does not help either, because cron installs it from the
8+
# configuration that just failed to load. So the console is the only channel and
9+
# it carries one useless line.
1210
#
13-
# So this runs after vyos-router regardless of outcome and puts the tail on the
14-
# console, where logSerialConsole and the e2e collector already pick it up. It is
15-
# a few lines on a successful boot and the whole answer on a failed one.
11+
# WHERE THE REASON ACTUALLY IS. Not in a file. vyos-boot-config-loader.py catches
12+
# ConfigSessionError, writes the status file and exits before the block that
13+
# writes its own log, so that log exists only on success. What does persist is
14+
# the journal: conf_mode/pki.py and conf_mode/service_https.py are both dispatched
15+
# through vyos-configd, which logs the ConfigError it caught. That is readable
16+
# today with no kernel argument and no image debug flag.
17+
#
18+
# WHY IT POLLS. vyos-router.service is Type=simple, so systemd treats it as
19+
# started the moment the process forks and an `After=` on it is satisfied long
20+
# before the configuration is loaded — measured: an earlier revision of this
21+
# script printed its line at boot and the "Configuration error" arrived seconds
22+
# later. Waiting on the status file is what upstream's own vyos-config init does.
1623
set -u
1724

18-
LOG=/var/log/vyatta/vyos-config.log
19-
MAX=25
25+
STATUS_FILE=/tmp/vyos-config-status
26+
WAIT=240
27+
MAX=40
2028

2129
log() { echo "vyos-appliance-config-report: $*" > /dev/console 2>&1; }
2230

23-
if [ ! -s "$LOG" ]; then
24-
log "no $LOG; vyos-router either did not run or wrote nothing"
31+
dump() {
32+
# -o cat drops the syslog preamble; the message is the whole point and the
33+
# console is narrow.
34+
journalctl -b -u "$1" -p err --no-pager -o cat 2>/dev/null | tail -n "$MAX" \
35+
| while IFS= read -r line; do
36+
[ -n "$line" ] && log "| $1: $line"
37+
done
38+
}
39+
40+
i=0
41+
while [ ! -f "$STATUS_FILE" ] && [ "$i" -lt "$WAIT" ]; do
42+
i=$((i + 1))
43+
sleep 1
44+
done
45+
46+
if [ ! -f "$STATUS_FILE" ]; then
47+
log "no $STATUS_FILE after ${WAIT}s; the config loader never reported. Journal follows."
48+
dump vyos-configd
49+
dump vyos-router
2550
exit 0
2651
fi
2752

28-
log "last $MAX lines of $LOG:"
29-
tail -n "$MAX" "$LOG" 2>/dev/null | while IFS= read -r line; do
30-
log "| $line"
31-
done
53+
status="$(cat "$STATUS_FILE" 2>/dev/null | tr -d '[:space:]')"
54+
if [ "$status" = "0" ]; then
55+
log "configuration loaded and committed"
56+
exit 0
57+
fi
58+
59+
log "configuration REJECTED (status ${status:-unknown}); reason follows"
60+
dump vyos-configd
61+
dump vyos-router
3262
exit 0

0 commit comments

Comments
 (0)