build: create distinct history db for each store#48565
Conversation
tonistiigi
left a comment
tonistiigi
left a comment
There was a problem hiding this comment.
I don't think we should do this migration. Nothing else is doing the migration. For switching to the containerd storage, migrating images and layers should be the priority, not build history records or even build cache. As that can be easily recreated.
What should we do for people that have unreachable build records that can't be opened or removed? Should this be part of history API GC instead? |
GC may remove records that have missing blobs indeed.
This is only issue with people switching from containerd storage to graphdriver storage, what should not really happen. Containerd storage is experimental, weird things may happen. |
crazy-max
force-pushed
the
build-split-history-db
branch
2 times, most recently
from
dee11af to
5f7c398
Compare
|
Updated to just split history db, will do the follow-up in BuildKit to prune erroneous records. |
crazy-max
force-pushed
the
build-split-history-db
branch
from
5f7c398 to
ee23b04
Compare
thaJeztah
added
area/builder
| } | ||
|
|
||
| historyDB, historyConf, err := openHistoryDB(opt.Root, opt.BuilderConfig.History) | ||
| historyDB, historyConf, err := openHistoryDB(opt.Root, "history_ctd.db", opt.BuilderConfig.History) |
There was a problem hiding this comment.
Slight nit; I think we more commonly use c8d as short form for containerd (isnteead of ctd)
Does the cache.db below also depend on which backend is used? I'm wondering if the alternative would be to pass a different opt.Root when initialising? Looks like we hard-code root to be /buildkit (which probably is fine), but perhaps it could use a subdirectory based on what driver is used?
Line 430 in 83a1693
Perhaps even buildkit/<snapshotterName> (of this information is tied to the snapshotter used)
There was a problem hiding this comment.
Slight nit; I think we more commonly use
c8das short form for containerd (isnteead ofctd)
Sure! Better to be consistent.
Does the
cache.dbbelow also depend on which backend is used? I'm wondering if the alternative would be to pass a differentopt.Rootwhen initialising? Looks like we hard-coderootto be/buildkit(which probably is fine), but perhaps it could use a subdirectory based on what driver is used?
For cache blobs this is not needed afaik (cc @tonistiigi)
crazy-max
force-pushed
the
build-split-history-db
branch
from
ee23b04 to
d497c21
Compare
Signed-off-by: CrazyMax <[email protected]>
crazy-max
force-pushed
the
build-split-history-db
branch
from
d497c21 to
58f4e91
Compare
4 participants
- What I did
Update build controller logic so it's using a distinct db for each type of store:
history.dbfor graphdriverhistory_c8d.dbfor containerd snapshotterAs follow-up in BuildKit we should prune records that have missing blobs.
- Description for the changelog
- A picture of a cute animal (not mandatory but encouraged)