Restorable apps by mike-sul · Pull Request #429 · foundriesio/meta-lmp

mike-sul · 2021-09-22T13:08:28Z

Changes required for the "restorable" apps foundriesio/aktualizr-lite#120.

Add skopeo as dependency for aklite;
Add the auth config env variable to the aklite systemd service, skopeo uses it to get auth for Registry.

doanac · 2021-09-22T21:59:34Z

 FILES_${PN}-get = "${bindir}/aktualizr-get"
 FILES_${PN}-lite = "${bindir}/aktualizr-lite"
+FILES_${PN}-lite-lib = "${libdir}/libaktualizr_lite.so"
+FILES_${PN}-dev += "${includedir}/${PN}-lite"


once foundriesio/aktualizr-lite#121 is merged, this will need to be:

FILES_${PN}-dev += "${includedir}/${PN}-lite ${includedir}/json"

If you want I can split this patch out and try and get it merged independently of this.

If you want I can split this patch out and try and get it merged independently of this.

Please, do it. I still need to put the finishing touches to the "restorable apps" and it makes sense to add another commit to the given PR that bumps the final version of it (aklite), so this PR is not ready to be merged, while the "libapi" related changes are needed now.

mike-sul · 2021-09-27T10:46:00Z

I verified it for HEAD of the lmp-manifest master and foundriesio/aktualizr-lite#120. https://ci.foundries.io/projects/msul-dev01/lmp/builds/1008/

mike-sul · 2021-09-28T15:39:12Z

@ricardosalveti @doanac I think this PR is ready to be merged now.

mike-sul · 2021-09-29T08:41:37Z

@ricardosalveti So, can I merge it so we start dogfooding of the restorable/resetable apps?

ricardosalveti · 2021-10-07T19:41:14Z

 SYSTEMD_SERVICE_${PN}-lite = "aktualizr-lite.service"

 COMPOSE_HTTP_TIMEOUT ?= "60"
+DOCKER_CRED_HELPER_CFG ?= "${libdir}/docker/config.json"


I'm generally in favor of having a variable flexible enough for allowing it to be customized, but in this case shouldn't we just assume /usr/lib/docker/config.json in aktualizr-lite.service.in?

This won't change in docker itself.

I just wanted to make it consistent with this

meta-lmp/meta-lmp-base/recipes-support/docker-cli-config/docker-cli-config_0.1.bb

Line 15 in c1efc55

install -m 0644 ${WORKDIR}/config.json ${D}${libdir}/docker/config.json

ricardosalveti · 2021-10-07T19:43:22Z

 DEPENDS = "boost curl glib-2.0"

-SRCREV = "dd8fe0a9741ef05951b3fb213fb64f5896001726"
+SRCREV = "b0dfd09e8d7bab8e154d2778a8131ba60c082525"


This is not yet merged in master.

Yes, now it became dependent on foundriesio/lmp-device-register#28.

The new version of lmp-dev-reg is merged to master, now.

mike-sul · 2021-10-11T09:04:02Z

This is the LmP build that incorporates lmp-manifest master's head and the given PR. https://ci.foundries.io/projects/msul-dev01/lmp/builds/1042/.
I tested the main use-cases, it works as expected.
I think it's time to merge it so we start dogfooding as well as e2e testing @mwasilew.
The next step for me is to update the doc with info on "restorable" apps, how to enable, configure and preload.

mwasilew · 2021-10-11T10:21:03Z

@mike-sul do you have a description of how this is intended to work and how to test it?

mike-sul · 2021-10-11T10:49:36Z

do you have a description of how this is intended to work and how to test it?

It's spread across a few jira tickets. https://foundriesio.atlassian.net/browse/FFTK-780, https://foundriesio.atlassian.net/browse/FFTK-911.

I just put a summary here, and try to produce a doc this or next week.

Control flow

Apps&images are pulled into /var/sota/reset-apps during download phase by using skopeo;
Apps&images are installed/copied into /var/sota/compose-apps and /var/lib/docker at the beginning of an install phase by using skopeo and C++ business logic in aklite;
Compose Apps are started at the end of an install phase or containers are recreated before reboot and started after reboot

How to enable

a) At device registration time, use/add --restorable-apps option of lmp-device-register;

b) It's automatically enabled if "restorable" apps are preloaded. In order, to preload "restorable" apps the following should be specified in the factory-config.yml

containers:
  preloaded_images:
    enabled: true
    restorable: true
    shortlist: <> # optional

c) manually by setting the following param in the *.toml config

[pacman]
reset_apps = ""

mwasilew · 2021-10-11T10:56:17Z

OK, IIUC the goal is to restore the app in case containers get corrupted. Skopeo acts as local image registry. So when running app gets corrupted, aklite prunes it and uses skopeo to restore to the 'default' state from local registry. Is this the intention or I'm totally missing the idea?

mike-sul · 2021-10-11T11:06:15Z

So when running app gets corrupted, aklite prunes it and uses skopeo to restore to the 'default' state from local registry. Is this the intention or I'm totally missing the idea?

The primary goal is to support "factory reset" feature. "factory reset" for Apps means removing /var/lib/docker and /var/sota/compose-apps and maybe an overall /var/sota/ folder, so device re-registration is required. @ricardosalveti
The given approach allows resetting/restoring of the configured Apps without a need to re-download all Apps&images.

The secondary goal is to recover from corruptions of the docker daemon image&layer store (container corruptions). The flow is exactly what you described. It allows removing an overall /var/lib/docker folder or some parts of it and then recovering with minimum or zero downloads.
As a matter of fact, the given approach itself significantly decreases possibility of the container corruption, at least I couldn't reproduce the issue in the case of the "restorable apps".

mwasilew · 2021-10-11T11:21:39Z

How do I force "factory reset"?

mike-sul · 2021-10-11T11:38:24Z

How do I force "factory reset"?

I think a proper (user facing) "factory reset" of an overall device is not implemented, @ricardosalveti do you know how it's going to look like?

But, underneath, when it comes down to apps resetting in the scope of an overall "factory reset", it means just rm -rf /var/lib/docker and rm -rf /var/lib/compose-apps. So, this is the way to emulate the Apps part of "factory reset".

skopeo can use docker cred helpers. The env. var. REGISTRY_AUTH_FILE should be set and refers a json file with a list of docker cred helpers. ``` { "credHelpers": { "hub.foundries.io": "fio-helper" } } ``` Signed-off-by: Mike Sul <[email protected]>

Relevant changes: - 42c5863..aa7dbfd Restorable/resetable App implementation Signed-off-by: Mike Sul <[email protected]>

Relevant changes: - 9c33377 Add setting conf for Restorable Apps Signed-off-by: Mike Sul <[email protected]>

doanac · 2021-11-24T04:13:48Z

@mike-sul @ricardosalveti - this one slipped through the cracks for v84. just noticed it when I couldn't run skopeo.

mike-sul · 2021-11-24T07:57:37Z

this one slipped through the cracks for v84. just noticed it when I couldn't run skopeo

How come, the PR is not merged? Anyway, I'll double check.

mike-sul · 2021-11-24T09:20:34Z

@doanac I double checked it, LmP v84 is fine, it refers to meta-lmp 1462471 which in turn refers to aklite 5e2f6da2 which does NOT include any "restorable apps" source code nor skope. (https://ci.foundries.io/projects/msul-dev02/lmp/builds/370/).
So, an update from v83 to v84 works fine.

You might ran into the issue because you were running the newer aklite version that includes the restorable apps against/on the older LmP version (it can be done by overriding aklite version in conf/local.conf or meta-subs or manual aklite installation) that does not include skopeo.
To overcome the issue you can add skopeo_bin = "/bin/true" into sota.toml::[pacman] config.

ricardosalveti · 2021-11-26T00:16:37Z

@mwasilew @mike-sul factory reset is currently implemented via https://github.com/foundriesio/meta-lmp/blob/master/meta-lmp-base/recipes-core/initrdscripts/initramfs-framework/ostree_factory_reset (not yet documented), but after merging support for restorable apps I'm going to update to make sure it is covering this use case properly as well.

ricardosalveti · 2021-11-26T00:17:35Z

@doanac I double checked it, LmP v84 is fine, it refers to meta-lmp 1462471 which in turn refers to aklite 5e2f6da2 which does NOT include any "restorable apps" source code nor skope. (https://ci.foundries.io/projects/msul-dev02/lmp/builds/370/). So, an update from v83 to v84 works fine.

Yes, we didn't want to include in v84 as a way to separate major changes in different releases.

We're good now, will merge it.

mike-sul · 2021-11-26T08:47:35Z

@ricardosalveti The ostree_factory_reset script should work in terms of compose apps resetting. Effectively, apps reset implies

rm -rf /var/lib/docker && rm -rf /var/sota/compose-apps while aklite is stopped/not running;
an aklite restart what is supposed to restore apps with minimum requests to cloud/Registry (unfortunately it requires doing one request for an image manifest);

Changelog: - c01c39a Merge pull request foundriesio#425 from foundriesio/detsch-add-cli-doc - 5082335 README: Restructure and extend architecture section - 5c32785 docs: Add docs for key components, using content from docs.foundries.io - cd63527 Merge pull request foundriesio#432 from foundriesio/check-if-app-is-in-store - 8d014b8 appengine: Check if app is in store before other checks - 8204f3d Merge pull request foundriesio#431 from foundriesio/stream-child-output-to-stdout - 74acd8e appengine: Set timeout for long running app commands - 38eb5cc appengine: Print app installation progress - f4811be exec: Add child output printing to stdout - 4da2503 exec: Add optional param for timeout setting - a7da093 exec: Make exec non-template function - fde00f0 api: Make clang-format-20 happy - 1520ccc Merge pull request foundriesio#429 from foundriesio/detsch-fill-install-description - 213bf84 api: Fill install result description - a1bbc4c Merge pull request foundriesio#428 from foundriesio/detsch-test-pull-fail-install - 451c098 format: Some white-space fixes detected by clang-format-20 - c672b69 e2e-tests: Test pull + install cli operations whith different targets - 274565e cli: Rename InstallAppPullFailure to InstallTargetPullFailure - c90d3c5 Merge pull request foundriesio#426 from foundriesio/detsch-improve-e2e-tests - 27f4bfd e2e-tests: Re-order if conditions to improve organization - e4bb8f7 e2e-tests: Add test for random sequence of updates - f7cee2a e2e-tests: Indent log messages for internal operations - 28947ce e2e-tests: Add wrapper for running commands in targets generation script - 66c2e7c Merge pull request foundriesio#427 from foundriesio/detsch-pin-composeapp-version - 7dade6d e2e-tests: Pin composeapp version to v95.1 - a2e73c7 Merge pull request foundriesio#422 from foundriesio/detsch-avoid-deprecated-api - e5e404e Avoid use of boost process API - 7384ffc liteclient: Avoid using boost::process::system - 8042e12 Merge pull request foundriesio#423 from foundriesio/detsch-update-containers - b034728 docker: Update containers to ubuntu 25.04 - 768c9fe aktualizr: Bump to version 3d25b0304 - bdb0ba6 Merge pull request foundriesio#424 from foundriesio/detsch-tag-container - 6d47008 e2e-tests: Use tag for referencing aklite-dev image Signed-off-by: Jose Quaresma <[email protected]>

Changelog: - c01c39a Merge pull request #425 from foundriesio/detsch-add-cli-doc - 5082335 README: Restructure and extend architecture section - 5c32785 docs: Add docs for key components, using content from docs.foundries.io - cd63527 Merge pull request #432 from foundriesio/check-if-app-is-in-store - 8d014b8 appengine: Check if app is in store before other checks - 8204f3d Merge pull request #431 from foundriesio/stream-child-output-to-stdout - 74acd8e appengine: Set timeout for long running app commands - 38eb5cc appengine: Print app installation progress - f4811be exec: Add child output printing to stdout - 4da2503 exec: Add optional param for timeout setting - a7da093 exec: Make exec non-template function - fde00f0 api: Make clang-format-20 happy - 1520ccc Merge pull request #429 from foundriesio/detsch-fill-install-description - 213bf84 api: Fill install result description - a1bbc4c Merge pull request #428 from foundriesio/detsch-test-pull-fail-install - 451c098 format: Some white-space fixes detected by clang-format-20 - c672b69 e2e-tests: Test pull + install cli operations whith different targets - 274565e cli: Rename InstallAppPullFailure to InstallTargetPullFailure - c90d3c5 Merge pull request #426 from foundriesio/detsch-improve-e2e-tests - 27f4bfd e2e-tests: Re-order if conditions to improve organization - e4bb8f7 e2e-tests: Add test for random sequence of updates - f7cee2a e2e-tests: Indent log messages for internal operations - 28947ce e2e-tests: Add wrapper for running commands in targets generation script - 66c2e7c Merge pull request #427 from foundriesio/detsch-pin-composeapp-version - 7dade6d e2e-tests: Pin composeapp version to v95.1 - a2e73c7 Merge pull request #422 from foundriesio/detsch-avoid-deprecated-api - e5e404e Avoid use of boost process API - 7384ffc liteclient: Avoid using boost::process::system - 8042e12 Merge pull request #423 from foundriesio/detsch-update-containers - b034728 docker: Update containers to ubuntu 25.04 - 768c9fe aktualizr: Bump to version 3d25b0304 - bdb0ba6 Merge pull request #424 from foundriesio/detsch-tag-container - 6d47008 e2e-tests: Use tag for referencing aklite-dev image Signed-off-by: Jose Quaresma <[email protected]>

mike-sul requested review from doanac and ricardosalveti September 22, 2021 13:08

mike-sul mentioned this pull request Sep 22, 2021

Restorable apps foundriesio/aktualizr-lite#120

Merged

doanac reviewed Sep 22, 2021

View reviewed changes

mike-sul force-pushed the restorable-apps branch 2 times, most recently from 1329133 to e67d001 Compare September 24, 2021 17:25

mike-sul changed the title ~~WIP: Restorable apps~~ Restorable apps Sep 24, 2021

mike-sul marked this pull request as ready for review September 24, 2021 17:26

mike-sul force-pushed the restorable-apps branch from e67d001 to 0701ac5 Compare September 27, 2021 08:51

doanac reviewed Sep 27, 2021

View reviewed changes

Comment thread meta-lmp-base/recipes-sota/aktualizr/aktualizr_%.bbappend Outdated

mike-sul force-pushed the restorable-apps branch from 0701ac5 to f0ff22d Compare September 28, 2021 15:37

doanac approved these changes Sep 28, 2021

View reviewed changes

doanac mentioned this pull request Oct 5, 2021

recipes-sota: Add example recipe for custom C++ aklite client #437

Merged

mike-sul force-pushed the restorable-apps branch from f0ff22d to 5589926 Compare October 7, 2021 11:57

ricardosalveti reviewed Oct 7, 2021

View reviewed changes

mike-sul force-pushed the restorable-apps branch 5 times, most recently from fdbce5d to 10ee3f0 Compare October 11, 2021 07:23

mike-sul mentioned this pull request Oct 25, 2021

Guide on Restorable Apps foundriesio/docs#287

Merged

mike-sul force-pushed the restorable-apps branch from 10ee3f0 to 9f17d74 Compare November 4, 2021 12:56

mike-sul mentioned this pull request Nov 16, 2021

Switch to new aktualizr #460

Merged

mike-sul added 3 commits November 23, 2021 11:07

base: aktualizr-lite: bump to d2232e2

e2d923a

Relevant changes: - 42c5863..aa7dbfd Restorable/resetable App implementation Signed-off-by: Mike Sul <[email protected]>

base: lmp-device-register: bump rev to 9c33377

0f6c7d3

Relevant changes: - 9c33377 Add setting conf for Restorable Apps Signed-off-by: Mike Sul <[email protected]>

mike-sul force-pushed the restorable-apps branch from 9f17d74 to 0f6c7d3 Compare November 23, 2021 09:07

ricardosalveti merged commit 3ca4f94 into master Nov 26, 2021

doanac deleted the restorable-apps branch November 29, 2021 15:25

Conversation

mike-sul commented Sep 22, 2021

Uh oh!

doanac Sep 22, 2021

Choose a reason for hiding this comment

Uh oh!

mike-sul Sep 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

doanac Sep 23, 2021

Choose a reason for hiding this comment

Uh oh!

mike-sul commented Sep 27, 2021

Uh oh!

Uh oh!

mike-sul commented Sep 28, 2021

Uh oh!

mike-sul commented Sep 29, 2021

Uh oh!

ricardosalveti Oct 7, 2021

Choose a reason for hiding this comment

Uh oh!

mike-sul Oct 8, 2021

Choose a reason for hiding this comment

Uh oh!

ricardosalveti Oct 7, 2021

Choose a reason for hiding this comment

Uh oh!

mike-sul Oct 8, 2021

Choose a reason for hiding this comment

Uh oh!

mike-sul Oct 8, 2021

Choose a reason for hiding this comment

Uh oh!

mike-sul commented Oct 11, 2021

Uh oh!

mwasilew commented Oct 11, 2021

Uh oh!

mike-sul commented Oct 11, 2021

Uh oh!

mwasilew commented Oct 11, 2021

Uh oh!

mike-sul commented Oct 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mwasilew commented Oct 11, 2021

Uh oh!

mike-sul commented Oct 11, 2021

Uh oh!

doanac commented Nov 24, 2021

Uh oh!

mike-sul commented Nov 24, 2021

Uh oh!

mike-sul commented Nov 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ricardosalveti commented Nov 26, 2021

Uh oh!

ricardosalveti commented Nov 26, 2021

Uh oh!

mike-sul commented Nov 26, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mike-sul Sep 23, 2021 •

edited

Loading

mike-sul commented Oct 11, 2021 •

edited

Loading

mike-sul commented Nov 24, 2021 •

edited

Loading