pkg: add FlashDB by benpicco · Pull Request #17612 · RIOT-OS/RIOT

benpicco · 2022-02-03T17:44:03Z

Contribution description

This adds a package for FlashDB, an embedded time series and key-value data base.

It can either operate directly on a raw MTD device (flashdb_mtd) or use the VFS layer for file based operation.

Testing procedure

I added a test that uses the examples provided by FlashDB.

tests/pkg_flashdb

2022-02-05 00:21:09,418 # main(): This is RIOT! (Version: 2022.04-devel-117-gda35d-pkg/flashdb)
2022-02-05 00:21:09,591 # ==================== kvdb_basic_sample ====================
2022-02-05 00:21:09,601 # get the 'boot_count' value is 1
2022-02-05 00:21:11,210 # set the 'boot_count' value to 2
2022-02-05 00:21:11,215 # ===========================================================
2022-02-05 00:21:11,220 # ================= kvdb_type_string_sample =================
2022-02-05 00:21:12,636 # create the 'temp' string KV, value is: 36C
2022-02-05 00:21:12,642 # get the 'temp' value is: 36C
2022-02-05 00:21:14,057 # set 'temp' value to 38C
2022-02-05 00:21:14,340 # delete the 'temp' finish
2022-02-05 00:21:14,346 # ===========================================================
2022-02-05 00:21:14,351 # ================== kvdb_type_blob_sample ==================
2022-02-05 00:21:15,241 # create the 'temp' blob KV, value is: 36
2022-02-05 00:21:15,272 # get the 'temp' value is: 36
2022-02-05 00:21:16,689 # set 'temp' value to 38
2022-02-05 00:21:16,972 # delete the 'temp' finish
2022-02-05 00:21:16,977 # ===========================================================
2022-02-05 00:21:17,013 # ======================= tsdb_sample =======================
2022-02-05 00:21:17,017 # append the new status.temp (36) and status.humi (85)
2022-02-05 00:21:17,022 # append the new status.temp (38) and status.humi (90)
2022-02-05 00:21:17,028 # [query_cb] queried a TSL: time: 19, temp: 36, humi: 85
2022-02-05 00:21:17,034 # [query_cb] queried a TSL: time: 21, temp: 38, humi: 90
2022-02-05 00:21:17,060 # [query_by_time_cb] queried a TSL: time: 19, temp: 36, humi: 85
2022-02-05 00:21:17,066 # [query_by_time_cb] queried a TSL: time: 21, temp: 38, humi: 90
2022-02-05 00:21:17,084 # query count is: 0
2022-02-05 00:21:17,093 # set the TSL (time 19) status from 3 to 3
2022-02-05 00:21:17,374 # set the TSL (time 21) status from 3 to 3
2022-02-05 00:21:17,673 # ===========================================================

tests/pkg_flashdb_mtd

2022-02-05 00:12:48,097 # main(): This is RIOT! (Version: 2022.04-devel-118-g80d9a-pkg/flashdb)
2022-02-05 00:12:48,104 # [I/FAL] Flash Abstraction Layer (V0.5.0) initialize success.
2022-02-05 00:12:48,117 # ==================== kvdb_basic_sample ====================
2022-02-05 00:12:48,120 # get the 'boot_count' value is 12
2022-02-05 00:12:48,132 # set the 'boot_count' value to 13
2022-02-05 00:12:48,137 # ===========================================================
2022-02-05 00:12:48,143 # ================= kvdb_type_string_sample =================
2022-02-05 00:12:48,152 # create the 'temp' string KV, value is: 36C
2022-02-05 00:12:48,155 # get the 'temp' value is: 36C
2022-02-05 00:12:48,157 # set 'temp' value to 38C
2022-02-05 00:12:48,160 # delete the 'temp' finish
2022-02-05 00:12:48,165 # ===========================================================
2022-02-05 00:12:48,170 # ================== kvdb_type_blob_sample ==================
2022-02-05 00:12:48,180 # create the 'temp' blob KV, value is: 36
2022-02-05 00:12:48,182 # get the 'temp' value is: 36
2022-02-05 00:12:48,185 # set 'temp' value to 38
2022-02-05 00:12:48,187 # delete the 'temp' finish
2022-02-05 00:12:48,192 # ===========================================================
2022-02-05 00:12:48,198 # ======================= tsdb_sample =======================
2022-02-05 00:12:48,203 # append the new status.temp (36) and status.humi (85)
2022-02-05 00:12:48,207 # append the new status.temp (38) and status.humi (90)
2022-02-05 00:12:48,212 # [query_cb] queried a TSL: time: 1, temp: 36, humi: 85
2022-02-05 00:12:48,217 # [query_cb] queried a TSL: time: 1, temp: 38, humi: 90
2022-02-05 00:12:48,222 # [query_by_time_cb] queried a TSL: time: 1, temp: 36, humi: 85
2022-02-05 00:12:48,228 # [query_by_time_cb] queried a TSL: time: 1, temp: 38, humi: 90
2022-02-05 00:12:48,229 # query count is: 0
2022-02-05 00:12:48,233 # set the TSL (time 1) status from 3 to 3
2022-02-05 00:12:48,236 # set the TSL (time 1) status from 3 to 3
2022-02-05 00:12:48,242 # ===========================================================

Issues/PRs references

includes #17341 for convenience
repo of FlashDB

maribu · 2022-02-03T18:37:13Z

I like this one :) The CI has some comments.

fjmolinas · 2022-02-08T08:03:19Z

One question out of curiosity, the GitHub page sys its supports wear balance, do you know how that is done? I was trying to find some information to estimate flash wear out when using the API but didn't find a section on that immediately.

kfessel

There is no reason to further hold this back (review stages 1 to 5 are already green by kaspar) i did run the test on native and read this.

Issues raised earlier are addressed.

Please squash and activate CI.

I would like to see the pseudomodules listed in Doxygen.
~~I think there should be some kind of warning that this will not do wear leveling and advising a solution like littlefs2.~~
I also would like to see the title of the Doxygen group changed a little for more userfullnes of that.

Please squash these changes right in.

pkg/flashdb/doc.txt

benpicco · 2023-02-23T21:37:07Z

I think there should be some kind of warning that this will not do wear leveling and advising a solution like littlefs2.

Whaat? I thought that was one of it's selling points.

kfessel · 2023-02-23T21:48:22Z

I think there should be some kind of warning that this will not do wear leveling and advising a solution like littlefs2.

Whaat? I thought that was one of it's selling points.

maybe i missed something or the wearleveling is done another way than in littlefs2.
The memory-file for (mtd) on native has one instance of the datebase after multiple runs.
The littlefs2 memoryfile (vfs version) has the database multiple times (every time written to another place)
Maybe this does not level wear across reboot?

kfessel · 2023-02-23T23:03:01Z

seems like the compactness of the written data (littlefs2 is more spacey when writing to different places) in the mtd threw me off -> it writes to different places (levels the wear)

pkg/flashdb/doc.txt

riot-ci · 2023-02-24T00:08:51Z

Murdock results

✔️ PASSED

674e3ad tests/pkg_flashdb: add test for FlashDB

Success	Failures	Total	Runtime
6887	0	6888	09m:31s

Artifacts

Documentation preview

kfessel · 2023-02-24T10:22:29Z

I had another look at the writes (native mtd debug)-> only writes are happening no erases (i assume until rollover)

There is a huge number of small writes (sometimes to the same location but no erase) the test has a increasing number of writes with every run as the database increases with each run by some tvdb entries. I think this is ok.

Details

grep write 
mtd_native: write from page 4, offset 0xa3 count 1
mtd_native: write from page 5, offset 0x47 count 1
mtd_native: write from page 5, offset 0x4b count 20
mtd_native: write from page 5, offset 0x5f count 10
mtd_native: write from page 5, offset 0x69 count 4
mtd_native: write from page 5, offset 0x47 count 1
mtd_native: write from page 4, offset 0xa3 count 1
mtd_native: write from page 5, offset 0x6d count 1
mtd_native: write from page 5, offset 0x71 count 20
mtd_native: write from page 5, offset 0x85 count 4
mtd_native: write from page 5, offset 0x89 count 3
mtd_native: write from page 5, offset 0x6d count 1
mtd_native: write from page 5, offset 0x6d count 1
mtd_native: write from page 5, offset 0x8c count 1
mtd_native: write from page 5, offset 0x90 count 20
mtd_native: write from page 5, offset 0xa4 count 4
mtd_native: write from page 5, offset 0xa8 count 3
mtd_native: write from page 5, offset 0x8c count 1
mtd_native: write from page 5, offset 0x6d count 1
mtd_native: write from page 5, offset 0x8c count 1
mtd_native: write from page 5, offset 0xab count 1
mtd_native: write from page 5, offset 0xaf count 20
mtd_native: write from page 5, offset 0xc3 count 4
mtd_native: write from page 5, offset 0xc7 count 4
mtd_native: write from page 5, offset 0xab count 1
mtd_native: write from page 5, offset 0xab count 1
mtd_native: write from page 5, offset 0xcb count 1
mtd_native: write from page 5, offset 0xcf count 20
mtd_native: write from page 5, offset 0xe3 count 4
mtd_native: write from page 5, offset 0xe7 count 4
mtd_native: write from page 5, offset 0xcb count 1
mtd_native: write from page 5, offset 0xab count 1
mtd_native: write from page 5, offset 0xcb count 1
mtd_native: write from page 21, offset 0x8 count 1
mtd_native: write from page 21, offset 0xc count 12
mtd_native: write from page 2f, offset 0x88 count 8
mtd_native: write from page 21, offset 0x8 count 1
mtd_native: write from page 21, offset 0x18 count 1
mtd_native: write from page 21, offset 0x1c count 12
mtd_native: write from page 2f, offset 0x80 count 8
mtd_native: write from page 21, offset 0x18 count 1
mtd_native: write from page 20, offset 0x28 count 1
mtd_native: write from page 20, offset 0x38 count 1
mtd_native: write from page 20, offset 0x48 count 1
mtd_native: write from page 20, offset 0x58 count 1
mtd_native: write from page 20, offset 0x68 count 1
mtd_native: write from page 20, offset 0x78 count 1
mtd_native: write from page 20, offset 0x88 count 1
mtd_native: write from page 20, offset 0x98 count 1
mtd_native: write from page 20, offset 0xa8 count 1
mtd_native: write from page 20, offset 0xb8 count 1
mtd_native: write from page 20, offset 0xc8 count 1
mtd_native: write from page 20, offset 0xd8 count 1
mtd_native: write from page 20, offset 0xe8 count 1
mtd_native: write from page 20, offset 0xf8 count 1
mtd_native: write from page 21, offset 0x8 count 1
mtd_native: write from page 21, offset 0x18 count 1

pkg/flashdb/mtd/fal_mtd_port.c

kfessel

Squash and Go

benpicco · 2023-02-27T20:01:35Z

Thank you for the review!

bors merge

17612: pkg: add FlashDB r=benpicco a=benpicco 19247: pkg/tinydtls: drop libc_gettimeofday dependency r=benpicco a=benpicco Co-authored-by: Benjamin Valentin <[email protected]> Co-authored-by: Benjamin Valentin <[email protected]>

bors · 2023-02-27T21:45:52Z

Build failed (retrying...):

Murdock

bors · 2023-02-27T22:50:05Z

Canceled.

benpicco · 2023-02-28T00:37:01Z

bors merge

bors · 2023-02-28T00:37:05Z

🕐 Waiting for PR status (GitHub check) to be set, probably by CI. Bors will automatically try to run when all required PR statuses are set.

benpicco · 2023-02-28T00:39:04Z

bors merge

bors · 2023-02-28T08:11:15Z

Build succeeded:

benpicco requested review from MichelRottleuthner, aabadie, cgundogan, fjmolinas, jia200x, leandrolanzieri and smlng as code owners February 3, 2022 17:44

benpicco requested a review from maribu February 3, 2022 17:44

benpicco force-pushed the pkg/flashdb branch 2 times, most recently from b403eb9 to da35dd2 Compare February 4, 2022 23:15

benpicco mentioned this pull request Feb 8, 2022

support for storage devices > 2 GiB armink/FlashDB#132

Open

benpicco force-pushed the pkg/flashdb branch from 9bd2144 to 97667d6 Compare February 9, 2022 20:52

benpicco added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Feb 9, 2022

github-actions bot removed Area: sys Area: System Area: build system Area: Build system Area: cpu Area: CPU/MCU ports Area: drivers Area: Device drivers labels Feb 9, 2022

kfessel reviewed Feb 23, 2023

View reviewed changes

pkg/flashdb/doc.txt Show resolved Hide resolved

pkg/flashdb/doc.txt Outdated Show resolved Hide resolved

benpicco force-pushed the pkg/flashdb branch from 0fe81e6 to f776978 Compare February 23, 2023 22:08

benpicco added CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR and removed State: waiting for maintainer State: Action by a maintainer is required labels Feb 23, 2023

benpicco force-pushed the pkg/flashdb branch 2 times, most recently from 9a5f61f to 0fa2253 Compare February 23, 2023 22:27

kfessel reviewed Feb 23, 2023

View reviewed changes

pkg/flashdb/doc.txt Show resolved Hide resolved

kfessel reviewed Feb 24, 2023

View reviewed changes

pkg/flashdb/mtd/fal_mtd_port.c Outdated Show resolved Hide resolved

kfessel approved these changes Feb 27, 2023

View reviewed changes

pkg: add FlashDB

8cc689a

benpicco force-pushed the pkg/flashdb branch from 863e9f1 to 753eb68 Compare February 27, 2023 17:12

tests/pkg_flashdb: add test for FlashDB

674e3ad

benpicco force-pushed the pkg/flashdb branch from 753eb68 to 674e3ad Compare February 27, 2023 22:50

benpicco removed the Area: build system Area: Build system label Feb 28, 2023

bors bot merged commit 2c1bd90 into RIOT-OS:master Feb 28, 2023

benpicco deleted the pkg/flashdb branch February 28, 2023 08:26

MrKevinWeiss added this to the Release 2023.04 milestone Apr 25, 2023

Conversation

benpicco commented Feb 3, 2022 • edited by kfessel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution description

Testing procedure

Issues/PRs references

Uh oh!

maribu commented Feb 3, 2022

Uh oh!

fjmolinas commented Feb 8, 2022

Uh oh!

kfessel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

benpicco commented Feb 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kfessel commented Feb 23, 2023

Uh oh!

kfessel commented Feb 23, 2023

Uh oh!

Uh oh!

riot-ci commented Feb 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Murdock results

Artifacts

Uh oh!

kfessel commented Feb 24, 2023

Uh oh!

Uh oh!

kfessel left a comment

Choose a reason for hiding this comment

Uh oh!

benpicco commented Feb 27, 2023

Uh oh!

bors bot commented Feb 27, 2023

Uh oh!

bors bot commented Feb 27, 2023

Uh oh!

benpicco commented Feb 28, 2023

Uh oh!

bors bot commented Feb 28, 2023

Uh oh!

benpicco commented Feb 28, 2023

Uh oh!

bors bot commented Feb 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

benpicco commented Feb 3, 2022 •

edited by kfessel

Loading

kfessel left a comment •

edited

Loading

benpicco commented Feb 23, 2023 •

edited

Loading

riot-ci commented Feb 24, 2023 •

edited

Loading