May I ask you if the overall concept is acceptable and heading in the right direction, @kaspar030?

I would like to enhance the new ztimer regarding power saving features. Its concepts are really well-fitting to my application and would remove some quirks I was forced to introduce ;)

@jue89, thanks for working on this!

May I ask you if the overall concept is acceptable and heading in the right direction, @kaspar030?

It might, but I'd like to get some evaluation before pursuing this.

I think, as a first step, we should consider the HF clock running while not in deep sleep and stopped while in deep sleep. On most platforms, that's the case anyways. In those cases, we just need to add blocking a PM mode, in ztimer_periph_timer, as soon as a time is set (and unblocking when the llist becomes empty). This shouldn't be more than a couple of lines.

For platforms that don't deep-sleep as long as the HF timer is running (I think that would be at least nrf52), I'd rather, as first step, add pm_layered hooks and use them stop the timer on entering sleep, and restore it on waeking up. That is probably also quite easy to implement in a generic way.

At that point, we should have deep sleep power consumption down quite a lot. This could be the most important metric. Also, this would be doable without requiring any user API changes, with almost no added complexity, without changing on how timers can be used and most of it contained to ztimer or the pm layer.
It is also easy to grasp: use ZTIMER_MSEC if possible, allowing deep sleep if nothing else is running. When using ZTIMER_USEC, be aware that it prevents deep sleep while a timer is active. ztimer_now() counts active time for USEC, keeps counting in deep sleep for MSEC.

At that point, we need proper measurements on the extra power use of a permanently active HF clock vs. one that's off, on multiple platforms. If we're lucky, it doesn't really matter, as in active mode, the CPU clock is running full-tilt anyways, and another running counter might be barely noticable. My second preferred outcome would be a clear sign that most HF timer hardware uses substantial power when activated, so we know for sure that we need to disable them as much as possible.

It might turn out to be a tough call on whether any added complexity is worth it for saving a couple of percent of power in active mode, especially as it depends on the application how much of the time a node is sleeping anyways.

But I really think we need numbers to decide this.

Thank you for your reply!

I think, as a first step, we should consider the HF clock running while not in deep sleep and stopped while in deep sleep. On most platforms, that's the case anyways. In those cases, we just need to add blocking a PM mode, in ztimer_periph_timer, as soon as a time is set (and unblocking when the llist becomes empty). This shouldn't be more than a couple of lines.

I tried to implement this PR by touching just ztimer/periph_timer.c, but I failed. The problem is ztimer_periph_timer doesn't know if it is still required or not if ztimer_extend is used to enlarge a non-32bit periph_timer to 32bit (i.e. clock->ops->cancel() is never called if clock->max_value < UINT32_MAX is true). Or am I wrong?

This forced me to extend ztimer_ops_t and implemented the tracking, whether the clock is still required, inside of ztimer/core.c.

For platforms that don't deep-sleep as long as the HF timer is running (I think that would be at least nrf52), I'd rather, as first step, add pm_layered hooks and use them stop the timer on entering sleep, and restore it on waeking up. That is probably also quite easy to implement in a generic way.

Yip, full ack.

It is also easy to grasp: use ZTIMER_MSEC if possible, allowing deep sleep if nothing else is running. When using ZTIMER_USEC, be aware that it prevents deep sleep while a timer is active. ztimer_now() counts active time for USEC, keeps counting in deep sleep for MSEC.

That is the reason why I started working on evtimer_on_ztimer in #13661. This PR already removes some dependencies on ztimer_now64() for long-running timers and allows for the GNRC network stack running on ztimer_now(ZTIMER_MSEC) without knowing of its existence.

As a bonus of the assumption you described above, a device can check how long it has been in deep-sleep by comparing ZTIMER_MSEC and ZTIMER_USEC :) This is really helpful during development since the developer will be warned if the device wasn't sleeping at all.

At that point, we need proper measurements on the extra power use of a permanently active HF clock vs. one that's off, on multiple platforms. If we're lucky, it doesn't really matter, as in active mode, the CPU clock is running full-tilt anyways, and another running counter might be barely noticable. My second preferred outcome would be a clear sign that most HF timer hardware uses substantial power when activated, so we know for sure that we need to disable them as much as possible.

I had a deep-dive into the SAML21 family and the difference between the running and the stopped HF clock is a current reduction from 400µA to 3µA. So for my use-case (it's a battery-driven device) switching off the HF clock is substantial for sufficient battery life.

A colleague of mine has experiences with the nRF52 family and he told me about similar power savings.

It might turn out to be a tough call on whether any added complexity is worth it for saving a couple of percent of power in active mode, especially as it depends on the application how much of the time a node is sleeping anyways.

Sure. But I haven't seen a IoT battery-driven application that isn't sleeping most of the day, yet ;)

I tried to implement this PR by touching just ztimer/periph_timer.c, but I failed. The problem is ztimer_periph_timer doesn't know if it is still required or not if ztimer_extend is used to enlarge a non-32bit periph_timer to 32bit (i.e. clock->ops->cancel() is never called if clock->max_value < UINT32_MAX is true). Or am I wrong?

No, not at all. I understood that the acquire / release is used to enable/disable the clock whenever it is not needed by applications (as in, acquire / release by them), instead of by ztimer itself.

Can we get around that by, instead of doing acquire/release in all the hot code paths, adding ztimer_pause()/ztimer_resume(), once, to the generic ztimer_clock?
That could then be called by our pm hooks.
It could be as simple as:

0. add stopped state flag to ztimer_t
1. on stop, set stopped state, remove possible upper timer, store current time (re-using checkpoint variable)
2. on resume, do the opposite.

ztimer_stop() would then be called by pm hooks on deep sleep enter, for specific clocks only.
ztimer_resume() would be called on wakeup.

Hm, that still requires clock specific ops. And it turns the supposed-to-be-easy stuff into ztimer's design.

I had a deep-dive into the SAML21 family and the difference between the running and the stopped HF clock is a current reduction from 400µA to 3µA. So for my use-case (it's a battery-driven device) switching off the HF clock is substantial for sufficient battery life.

Yeah, but the HF clock is also the one that drives the CPU. So while that is active anyways, running the timer hardware on top should be cheap. But I think turning off HF timer hardware in deep sleep was actually your intention, right?

BTW, let's use "ztimer clock" if referring to that, and "clock" when referring to MCU clocks (which might drive timer hardware). I've used only "clock" before for both, I guess that causes confusion.

edit clarified

Can we get around that by, instead of doing acquire/release in all the hot code paths, adding ztimer_pause()/ztimer_resume(), once, to the generic ztimer_clock?
That could then be called by our pm hooks.

To be honest, I don't like that idea. The pm hooks are called, once the user application tells the pm to go into deep sleep. So the user application (and all other parts of RIOT) must be aware that a certain pm state must be blocked or unblocked to use ZTIMER_USEC.

I would state the following logical chains:

setting a ztimer clock

The user requires a high precision ztimer clock (i.e. ZTIMER_USEC) => ztimer requires the underlying hardware clock (i.e. periph_timer) => the underlying hardware clock requires the CPU to be within a certain pm state => block(certain pm state)

ztimer reaches the target of the last ztimer clock

The hardware timer fires an interrupt => periph_timer tells ztimer the target has been reached => ztimer realizes that no other ztimer clocks are depending on periph_timer => ztimer releases periph_timer => unblock(certain pm state)

Yeah, but the HF clock is also the one that drives the CPU. So while that is active anyways, running the timer hardware on top should be cheap. But I think turning off HF timer hardware in deep sleep was actually your intention, right?

The CPU core isn't clocked at all during deep-sleep. The only living clock is the RTC/RTT. It will wake the whole system (including the HF clock), once it fires an interrupt. (The neat part of ZTIMER_MSEC sourced by the RTC/RTT. It just works ;))

To be clear: I want to have an easy-to-use OS that aims for the lowest power consumption as possible without requiring the user to understand every little aspect of the power management system.
I would state that the majority of present and future IoT applications are and will be battery-driven. Hence, power management that just works is an essential feature of an operating system.

To be honest, I don't like that idea at all. The pm hooks are called, once the user application tells the pm to go into deep sleep. So the user application (and all other parts of RIOT) must be aware that a certain pm state must be blocked or unblocked to use ZTIMER_USEC.

I was thinking that in addition to pm_layered ztimer_pause()/ztimer_resume(), the actual ztimer clocks block/unblock their power mode. If we move that code to the generic ztimer_clock code, it can be done in the extension clock, so the extended clock doesn't even care.

That way we accomplish automatic power down (if necessary) for HF ztimer clocks if the platform doesn't support it does it automatically.

I'm thinking this, concretely:

Case 1: e.g., samr21-xpro, where the HF timer clock stops when entering deep sleep. No extension.
So the ztimer_periph_timer_t's "pm_mode" would be set to the lowest one that keeps the thing running. ztimer_set() would block that PM when there's a timer active, and unblock when not.

Case 2: e.g., nrf51, HF needs special handling (disabling), ztimer_extend extends 24bit to 32bit.
That ztimer_extend's "pm_block" would be set to the correct pm mode. Extend handles blocking the power mode if there's no active timer but (only the extension timer). the lower ztimer_periph_pm is oblivious of that, (it always has a timer running anyways). But now the pm hooks get configured to call that ztimer_periph_timer's ztimer_pause() on entering deep sleep, and ztimer_resume() on wakeup. That pm mode can only be selected if there's no timer active, as the upper ztimer_extend would otherwise block that pm mode entirely.

The application is completely unaware. As long as it has an active timer, ztimer_extend would block the right power mode. If there's no active timer, the ztimer clock would stop counting on entering deep sleep, but continue counting once back to active mode.

Oh I got you wrong, sry. It's a little bit hard to understand implementation ideas just by reading text that is not code :D

Hopefully, I got you right this time. I try to describe what I understood regarding the central logic of the interaction with pm_layered:

Once a ztimer_t is added to the linked-list of a ztimer_clock_t and that ztimer_clock_t is only active in a certain pm mode, pm_block(certain_pm_mode) is called.

Once a ztimer_t is removed from the linked-list of a ztimer_clock_t (due to an expired ztimer_t or a ztimer_remove() call) and that ztimer_clock_t is only active in a certain pm mode, pm_unblock(certain_pm_mode) is called.

Am I right?

Am I right?

Yes! (Only the first added or last removed trigger pm_block()/pm_unblock(), but I think that is clear).

Does that make sense?

Yip, I think I can come up with an implementation of your proposed approach.

I think it makes sense to open another PR and close this one?! So both implementations still can be compared and tested.

Let's move the discussion to #13722.