Skip to content

reboot cause test failed if warm-reboot follows any kind of hardware-caused reboot #3871

@stephenxs

Description

@stephenxs

Description
If switch has rebooted due to some hardware cause (power, watchdog) and then is warm-rebooted, its hardware reboot causes are not cleared, which fails the reboot cause handling by platform API.
For example, if the system is rebooted by power cycle (cutting down power supply) then its reboot files representing a power-cycle reboot. after that, a warm reboot is executed and after system starts up, the file representing power-cycle reboot still contains "1".
One possible reason of this issue is that warm reboot is essentially a kernel reload rather than a hardware-caused reboot, which hardware probably isn’t aware of.
In this sense, it’s better to provide a way to distinguish the kernel-reload scenario from the real hardware-caused reboot.

Steps to reproduce the issue:

  1. cut off the SWITCH's power supply.
  2. wait for the SWITCH comes back.
  3. perform warm-reboot
  4. check the reboot cause after system comes back.

Describe the results you received:
the reboot cause "power off" is displayed.

Describe the results you expected:
the reboot cause should be "warm-reboot"

Preliminary analysis
The reboot cause handling logic in SONiC
For platform API, the reboot cause after fast/warm reboot should be “non hardware reboot”.
For the SONiC, the reboot-cause handling logic is like:

  1. If user issues a command of any kinds of reboot (like reboot/fast-reboot/warm-reboot), the system will record this action in the "/host/reboot-cause/reboot-cause.txt".
  2. When system boots, regarding the reboot cause handling,
    a. it will first call the platform API to detect whether there is a hardware reboot cause and if yes, it will take it as the reboot cause
    b. if not, it will check whether the "/host/reboot-cause/reboot-cause.txt" contains a valid reboot cause. if yes, it will take it as the reboot cause.
    c. If both checking above failed, it will return “reboot cause unknown”

To reverse the order of parsing software reboot cause and hardware reboot cause, which means parsing software reboot cause and then hardware reboot cause, is able to handle the issue. But it will introduce another kind of issue, like the following scenario:

  1. User issues warm-reboot
  2. Somehow there is a power outage that causes a power shutdown from the switch’s POV before the system handling the software reboot cause file. This can also cause a longer traffic loss which is not expected during a normal warm reboot.
  3. after the system starts up the reboot cause will be warm-reboot, which is not correct. (Furthermore, the user may waste time to investigate why a warm-reboot causes traffic loss more than expected.)

The root cause of returning the wrong reboot cause in the above scenario, I believe, is that the software reboot cause is not volatile across all kinds of reboots and reflects the penultimate reboot but not the last one. But the hardware one is volatile so that it’s able to always reflect the reboot cause of the last one.
In this sense, any attempt to handle the issue via recording something in non-volatile media can fall into this trap because all of this kind of attempt require to record something across reboots.

Conditions a good solution must satisfy:

  1. Before reloading the kernel, recording something like a flag indicating this action in the memory (which IS volatile)
  2. After kernel reloaded, parse that flag and reflect it in some way which is available from userspace
  3. Platform API can take advantage of the result of 2 to test whether it underwent a warm/fast reboot and if yes return the “reboot cause none”

So far I notice that the /proc/cmdline satisfies the above requirement because of its work flow:

  1. loading the kernel to be rebooted to into memory, and
  2. actually rebooting to the pre-loaded kernel.
    To load a kernel, the syntax is as follows:
    kexec -l kernel-image --append=command-line-options --initrd=initrd-image
    the command line is passed to the new kernel without using some non-volatile memory so that it satisfies the above requirement.

In this sense, the flow of reboot cause handling should be:

  1. test whether /proc/cmdline indicating a fast/warm reboot, if yes, return it as the reboot cause
  2. test the /host/reboot-cause.txt
  3. call the platform API.

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**

```
(paste your output here)
```

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions