Skip to content

Switch all branches to track the Yocto 5.0 scarthgap release#474

Merged
ricardosalveti merged 16 commits intofoundriesio:scarthgapfrom
quaresmajose:scarthgap
Jan 21, 2025
Merged

Switch all branches to track the Yocto 5.0 scarthgap release#474
ricardosalveti merged 16 commits intofoundriesio:scarthgapfrom
quaresmajose:scarthgap

Conversation

@quaresmajose
Copy link
Copy Markdown
Member

@quaresmajose quaresmajose commented Jan 16, 2025

This PR will provide the base release of the Linux microPlatform v95 that will support the Yocto 5.0 scarthgap release.

We will deprecate the Xilinx and STM32 bsp layers as part of the microPlatform that will now be supported in a partner layer. The meta-lts-mixins Rust and GO layers will be fully removed as they are no longer needed.

The lmp-base distro is broken on CI and requires the foundriesio/ci-scripts#370 which still needs to be validated again and fixed. Locally outside CI builds everything works as expected.

@quaresmajose quaresmajose requested a review from a team January 16, 2025 15:53
@quaresmajose quaresmajose added the ok-to-test Requires OK for testing to proceed label Jan 16, 2025
@angolini
Copy link
Copy Markdown
Contributor

Are we going to make all changes in one commit only with no relevant changes?

Can we wait until the STM removal?

And please, provide a description to the PR

Comment thread lmp-bsp.xml Outdated
@ricardosalveti
Copy link
Copy Markdown
Member

Yeah, we need a better description in the commit log.

@quaresmajose
Copy link
Copy Markdown
Member Author

I make separate commits, one for each layer and include the entire log as we usually do

@quaresmajose
Copy link
Copy Markdown
Member Author

Are we going to make all changes in one commit only with no relevant changes?

I will fix that

Can we wait until the STM removal?

Sure, we can wait.

And please, provide a description to the PR

I will do

@quaresmajose quaresmajose removed the ok-to-test Requires OK for testing to proceed label Jan 17, 2025
@quaresmajose quaresmajose changed the base branch from main to scarthgap January 17, 2025 10:25
@quaresmajose quaresmajose marked this pull request as draft January 17, 2025 10:26
@quaresmajose
Copy link
Copy Markdown
Member Author

The lmp-base distro is broken on CI and requires the foundriesio/ci-scripts#370 which still needs to be validated again and fixed. Locally outside CI builds everything works as expected.

@foundriesio/platform should we disable the lmp-base distro and keep only the lmp on CI builds for the scarthgap job? until we get the foundriesio/ci-scripts#370 in good shape.

@quaresmajose
Copy link
Copy Markdown
Member Author

The lmp-base distro is broken on CI and requires the foundriesio/ci-scripts#370 which still needs to be validated again and fixed. Locally outside CI builds everything works as expected.

@foundriesio/platform should we disable the lmp-base distro and keep only the lmp on CI builds for the scarthgap job? until we get the foundriesio/ci-scripts#370 in good shape.

This lmp-base issue will be solved after the merge of this PR.

@quaresmajose quaresmajose added ok-to-test Requires OK for testing to proceed and removed ok-to-test Requires OK for testing to proceed labels Jan 20, 2025
@quaresmajose quaresmajose marked this pull request as ready for review January 20, 2025 18:02
@quaresmajose quaresmajose requested a review from a team January 20, 2025 18:08
@ricardosalveti ricardosalveti added the ok-to-test Requires OK for testing to proceed label Jan 20, 2025
@ricardosalveti
Copy link
Copy Markdown
Member

CI still contains stm32/xilinx, needs to be removed from ci-scripts.

Comment thread lmp-bsp.xml
@quaresmajose
Copy link
Copy Markdown
Member Author

The commit logs get truncated by github but if we clone my tree https://github.com/quaresmajose/lmp-manifest everything is good.

Relevant changes:
- 8d153f1d toastergui: verify that an existing layer path is given
- c7e094ec Fix disk space monitoring on cephfs
- 973aa73b SECURITY.md: add file
- 08033b63 runqueue: Fix deferred task/multiconfig race issue
- d97d62e2 bitbake-user-manual: add BB_HASH_CODEPARSER_VALS
- 1b74a7b7 bitbake-user-manual: add addpylib and BB_GLOBAL_PYMODULES
- fa91e1a0 bitbake-user-manual: document BB_CACHEDIR
- 66131fa6 cooker: do not abort on single ctrl-c
- e602963d npmsw fetcher: Avoid instantiating Fetch class if url list is empty
- b4763c2c runqueue: fix PSI check calculation
- bb9e5232 bitbake: ConfHandler: Allow variable flag name with a single character
- 316524ab event: add bb.event.ParseError
- ae2efb05 fetch2/crate: create versioned 'name' entries
- 46e1ea2e bitbake: Bump to version 2.4.0
- dafa07c0 fetch2: Display all missing checksum at once
- ccb28785 bitbake: bitbake-user-manual: Update EXCLUDE_FROM_WORLD
- 3f4ede2d cache: Make EXCLUDE_FROM_WORLD boolean
- 935ab36f fetch2/local: Mention the value of localpath in failure message
- a0d8108e fetch/git: Fix local clone url to make it work with repo
- 9c07e377 crate.py: make checksum verification mandatory
- 4920686c fetch2: Add checksum capability for crate fetcher
- 0f887589 fetch2/sftp: Fix fetching URIs with spaces
- 00f9ab2c ConfHandler: Allow the '@' character in variable flag names
- 887804de bitbake-user-manual: fix links to supported release manuals
- 4d9ec332 fetch2: Add autorev warning when it is set too late
- 450b4a00 fetch2: Rename __BBSEENSRCREV -> __BBSRCREV_SEEN
- 282a3d61 toaster: update fixtures for mickledore
- b01d369f toaster: update gen_fixtures.py for mickledore
- 81a58647 build: Make python output print to stdout when running with -v (verbose)
- 278bd2f1 crate.py: authorize crate url with parameters
- b29f6e04 wget.py: Combine urlopener exceptions
- 6041b347 wget.py: Add catch TimeoutError exception
- e859ce61 bitbake-user-manual: update Hello World example
- cbad8cb2 doc: ref-variables: add LAYERSERIES_COMPAT to term glossary
- ce5b65d5 server/xmlrpc: Fix after currentAsyncCommand locking changes
- 6bc65e64 documentation: bitbake: add file-checksums to varflags section
- 0ac6f6cb fetch/npmsw: add more short forms for git operations
- 47b271e6 gitsm: Fix path construction for relative submodule URI
- f9165a79 utils: Use internal fetcher function to avoid duplication
- 956128e3 fetch2/wget: Drop unused import
- 9414ba62 fetch2: Add GIT_CACHE_PATH and SSL_CERT_DIR into FETCH_EXPORT_VARS
- cada37c6 cooker: Fix memory resident cache invalidation issue
- 18d4f9e8 lib/bb/siggen: fix debug() call
- dea1b2f3 bitbake: Bump to version 2.3.1
- b7ed7e9a event/cooker/runqueue: Add ability to interrupt longer running code
- 36a231d2 fetch/git: Replace mkdtemp with TemporaryDirectory and avoid exception masking
- a64ec64c fetch/git: Removed unused variables in _contains_lfs
- d2be7f7f tests/fetch: Add real git lfs tests and decorator
- 70f84863 tests/fetch: git-lfs restore _find_git_lfs
- 358b5b02 server/process: Improve idle thread exception handling
- 8fc5c50c cooker: Ensure lock is held with changing notifier
- c27e48fa runqueue: Drop SystemExit usage
- 13279044 cookerdata: Drop dubious exception handling code
- 2365d891 cookerdata: Improve early exception handling
- 25cb4e17 action.py: add topdir in bblayers_conf path for remove-layer
- 77aa230b action.py: add topdir in bblayers_conf path for add-layer
- f4ebb276 fetch2/wget: clean up netrc usage
- 3d96c07f data: Evaluate the value of export/unexport/network flags
- aaefb43b cookerdata: Remove incorrect SystemExit usage
- 5daecc25 fetch2: Add NODE_EXTRA_CA_CERTS to export list
- 195750f2 siggen: Fix inefficient string concatenation
- 22be2dbd bitbake-user-manual: show how use BB_LOGCONFIG to log warnings
- a56cd53e bblayers/query: Adjust show-layers output layout
- 517cc3f5 bblayers/query: Replace layer directory name with layer name for show-layers
- c19035e8 bb/utils: include SSL certificate paths in export_proxies
- ee89ade5 cache/codeparser: Switch to a new BB_CACHEDIR variable for cache location
- f90c45df toaster: update fixtures
- af354e97 bitbake-getvar: Add a quiet command line argument
- 323f6ce2 bitbake: fix deprecated threading.Thread.setDaemon
- fcb64e11 data_smart: Small optimisation to _findVar()
- f24bbaad data: Add support for new BB_HASH_CODEPARSER_VALS for cache optimisation
- f45a94e6 cooker: Fix parsing race around cache handling
- abeff1f8 cache: Only write files if we have data
- 4712d623 cooker: Fix siggen recipe cache race issue
- 4abc598f server/process: Fix lockfile contents check bug
- 1a219180 fetch2/tests: Add parameter to recipe_uri
- c0e349a5 fetch2/tests: Add test for Mercurial
- 80421abd fetch2: kill parameters in uri_find_decoded
- b986eac1 server/process: Improve lockfile handling at exit
- bf681d17 persist_data: Handle sqlite error when cachefile path is too long
- 0f9a0c78 server/process: Move heartbeat to idle thread
- 412a6d87 main: Move --buildfile help at the end of "Execution" group
- 4f73c2eb cooker: Fix exit handling issues
- af38345b knotty: Improve shutdown handling
- 32da768a tinfoil: Don't wait for events indefinitely
- b5215887 process/cooker/command: Fix currentAsyncCommand locking/races
- 4a41a7d0 cooker/command: Drop async command handler indirection via cooker
- 9257c48c Group and reorder options in bitbake help
- af4ccab8 cooker: Clean up inotify idle handler
- 9ece4a2e cooker: Rework the parsing results submission
- d2a3f662 lib/bb: Update thread/process locks to use a timeout
- 367a83ed command: Ensure that failure cases call finishAsyncComand
- 36589f13 cookerdata: Fix previous commit to use a string, not a generator
- 8c405c97 cookerdata: Fix cache/reparsing issue
- 6567ad61 knotty: Ping the server/cooker periodically
- 67dd9a5e server/process: Run idle commands in a separate idle thread
- a9c63ce8 server/process: Add locking around idle functions accesses
- 645c9d3b event: Always use threadlock
- 102e8d0d server/process: Improve idle loop exit code
- 6569ab64 cooker: Ensure commands clean up any parser processes
- 89435442 command: Tweak finishAsyncCommand ordering to avoid races
- 9cf3102d server/process: Improve exception and idle function logging
- 5f8b9b9c cooker/cookerdata: Rework the way the datastores are reset
- 8c36c90a event: Add enable/disable heartbeat code
- 2d0940b9 knotty: Avoid looping with tracebacks
- aeacfd39 cache: Drop reciever side counting for SiggenRecipeInfo
- eadddd94 tinfoil: Ensure CommandExit is handled
- 26ed783c cooker: Ensure cache is cleared for partial resets
- 1e12f0a4 event: builtins fix for 'd' deletion
- cd06beb9 cache/siggen: Fix cache issues with signature handling
- b02ebbff server/process: Add bitbake.sock race handling
- 513e6c4e siggen: Minor code improvement
- 9a2b13af cache: Allow compression of the data in SiggenRecipeInfo
- fd3359de command: Add ping command
- 8f9402b2 README: add required python version for bitbake
- 50261fe0 bitbake-user-manual: remove "OEBasic" signature generator
- 4f57853f bitbake-user-manual: double colon fix
- b4999425 fetch2/git: Clarify the meaning of namespace
- 7379c285 bitbake-user-manual: Document override :append, :prepend, :remove order
- bb2ea002 bitbake: Convert to argparse
- cfd7c989 process: log odd unlink events with bitbake.sock
- 1aa672b0 codeparser/data: Add vardepsexclude support to module dependency code
- 91441e15 ast/data/codeparser: Add dependencies from python module functions
- e1c1139a siggen: Clarify which fn is meant
- c79ecec5 build/siggen: Rework stamps functions
- c6d325fc siggen: Add dummy dataCaches from task context/datastore
- 1252e5bc bitbake: siggen/runqueue: Switch to using RECIPE_SIGGEN_INFO feature for signature dumping
- 6b213590 siggen: Directly store datacaches reference
- 376a516d main/process: Add extra sockname debugging
- 744310f3 bitbake: Bump minimum python version requirement to 3.8
- ef722822 build/siggen/runqueue: Drop do_setscene references
- b36545b4 siggen: Drop non-multiconfig aware siggen support
- b6601be2 fetch2/wget.py: correctly match versioned directories
- 897f238e bitbake-user-manual: add reference to bitbake git repository
- d32e5b0e fetch2/git: Prevent git fetcher from fetching gitlab repository metadata
- 88350002 fetch2/wget: handle username/password in uri
- 22685460 main/server: Add lockfile debugging upon server retry
- a1a86f8c main: Add timestamp to server retry messages
- 6709aedc cookerdata: Ensure layers use LAYERSERIES_COMPAT fairly
- afb8478d parse: Add support for addpylib conf file directive and BB_GLOBAL_PYMODULES
- 5d27c555 bitbake-user-manual: update references to Yocto Project manual
- a4ce040a bin/utils: Ensure locale en_US.UTF-8 is available on the system
- 105f2897 cooker: Start sync thread a little earlier
- 2aed34e1 cooker: Use event to terminate parser threads
- 0736a8a0 siggen/cache: Optionally allow adding siggen hash data to the bitbake cache
- e621093a cache/siggen: Simplify passing basehash data into the cache
- fff13b1e data: Tweak code layout
- 51cf7cf1 README: Improve explanation about running the testsuite
- b4a8e507 data_smart: Small cache reuse optimization
- a4693b70 runqueue: Improve error message for missing multiconfig
- 09a4c159 BBHandler: Remove pointless global variable declarations
- 6f4ef770 data: Drop obsolete pydoc/path code
- df5b3b84 data_smart: Use regex consistently for override matching
- ef9c033b utils: Allow to_boolean to support int values
- 43035b75 data_smart: Add debugging for overrides stability issue
- d386fa81 command.py: cleanup bb.cache.parse_recipe
- 19475627 data/siggen: Switch to use frozensets and optimize
- e479d1e4 runqueue: Fix race issues around hash equivalence and sstate reuse
- 9233ad68 runqueue: Add further debug for sstate reuse issues
- 063ffe69 cache: Drop support for not saving the cache file
- 3a82acdc worker/runqueue: Reduce initial data transfer in workerdata
- c97c1f1c server: Ensure cooker profiling works
- 6b24efc0 cache: Drop unused function
- af83ee32 cache: Drop broken/unused code
- 78387931 cache/cookerdata: Move recipe parsing functions from cache to databuilder
- 58e29fb1 toaster: Add refreshed oe-core and poky fixtures
- e060daae toaster: fixtures/gen_fixtures.py: update branches
- 0dca22a1 toaster: fixtures/README: django 1.8 -> 3.2
- 0ed7c75e gitsm.py: process_submodules(): Set nobranch=1 for url
- 3ad27272 gitsm: Fix regression in gitsm submodule path parsing
- 584989ed data/data_smart/build: Clean up datastore finalize/update_data references
- 94e49b9b data_smart: allow python snippets to include a dictionary
- 60f43d04 data: drop unused __expand_var_regexp__ and __expand_python_regexp__
- d2cb388f bitbake: bitbake-layers: checkout layer(s) branch when clone exists
- 2d07f252 asyncrpc: serv: correct closed client socket detection
- 8ad31063 fetch2/git: don't set core.fsyncobjectfiles=0
- 0f3e9d87 bitbake-user-manual: details about variable flags starting with underscore
- 209f7ba3 tests/fetch: Allow handling of a file:// url within a submodule
- ee04cf09 tests: bb.tests.fetch.URLHandle: add 2 new tests
- 72e9847d doc: bitbake-user-manual: expand description of BB_PRESSURE_MAX variables
- 484ab42f utils/ply: Update md5 to better report errors with hashlib
- 0a493a77 bitbake: user-manual: inform about spaces in :remove
- 82f40261 runqueue: fix a typo
- cf23612f cooker: fix a typo
- 074da4c4 bitbake: Bump to version 2.2.0
- 0d317209 utils: Add enable_loopback_networking()
- c014281f doc: bitbake-user-manual: Add definition for BB_PRESSURE_MAX variables
- d599af48 fetch2/ssh.py: fix checkstatus
- 3f23140f bitbake-user-manual: mention pydevshell in OE
- 19f82650 fetch2/crate fixup c212b0f3 loglevel
- c1d9c1d2 lib/bb: warning when the debug message is invalid
- eb25cd4d bitbake: Fix a few more logger debug() calls
- be1ee681 fetch2: Remove unneeded conditional
- 6424f4b7 fetch2: Ensure mirror tarballs don't enforce checksum
- 04014b8b toaster: fix kirkstone version
- 5964bb67 asyncrpc/client: Fix unix domain socket chdir race issues
- 4b8de2e7 wget: Avoid bad checksum race issues
- cffcfacb Revert "fetch: use BPN instead"
- d613e48c fetch: use BPN instead
- b0506480 tests: Add test for possible gitsm deadlock
- c5fcdd80 tests: Add Timeout class
- 0361ecf7 gitsm: Error out if submodule refers to parent repo
- 2cd76e8a Fix npm to use https rather than http
- 2f6c7523 bitbake-layers: initialize tinfoil before registering command line arguments
- 7a8eb8da utils: Pass lock argument in fileslocked
- 9bdedc80 runqueue: Change pressure file warning to a note
- f33ce7e7 BBHandler/cooker: Implement recipe and global classes
- 1aa338a2 bitbake: Add copyright headers where missing
- 0f4f3af6 BBHandler: Make inherit calls more directly
- f7c55c81 BBHandler: Allow earlier exit for classes not found
- 950a2ea4 build: prefix the tasks with a timestamp in the log task_order
- 44c39543 runqueue: add memory pressure regulation
- 97955f3c siggen: Fix insufficent entropy in sigtask file names
- 8efced47 runqueue: Drop deadlock breaking force fail
- cf7f60b8 runqueue: Improve deadlock warning messages
- 27228c7f runqueue: Ensure deferred tasks are sorted by multiconfig
- 87e42f12 bitbake-user-manual: npm fetcher: improve description of SRC_URI format
- 502e05cb runqueue: add cpu/io pressure regulation
- 274b2d7a bb/utils: movefile: use the logger for printing
- 7236488b bb/utils: remove: check the path again the expand python glob
- 68f4ce66 asyncrpc: Add TCP Keep Alives
- b171aa45 bitbake: bitbake-user-manual: hashserv can be accessed on a dedicated domain
- 3e1444e5 fetch2: Drop DL_DIR fallback for local file fetcher
- eff16e47 fetch2: Ensure directory exists before creating symlink
- 5e3b2ad9 fetch: bb.fatal when trying to checksum non-existing files
- 04dc17be fetch2/git: show SRCREV and git repo in error message about fixed SRCREV
- 712da71b ConfHandler/BBHandler: Improve comment error messages and add tests
- 4a0bd3bc fetch2: gitsm: fix incorrect handling of git submodule relative urls
- d3e64f64 server/process: Fix logging issues where only the first message was displayed
- cca79995 doc: bitbake-user-manual: document npm and npmsw fetchers
- c9bab35f doc: bitbake-user-manual: add explicit target for crates fetcher
- 8189f58d bitbake-user-manual: Correct description of the ??= operator
- 9fc1bab6 ConfHandler: Remove lingering close
- c9aaca3d tests/fetch: Add test for broken mirror tarball
- 35167536 event.py: ignore exceptions from stdout and sterr operations in atexit
- 247f3536 tests/fetch: Add tests for premirror using real project
- bfed4ad2 tests/fetch: Drop unnecessary duplicated function
- 3c0cd401 cooker: Drop sre_constants usage
- 7a28ac4f server/process: Avoid tracebacks at exit
- f5ad8349 server/process: Remove daemonic thread usage
- bd12792f server/process: Avoid risk of exception deadlocks
- cd7cce4c fetch/wget: Move files into place atomically
- 773e0815 Add tests to cover BB_FETCH_PREMIRRORONLY functionality
- b47ecab3 fetch2: Honour BB_FETCH_PREMIRRORONLY option
- 928aa3dc bitbake: Bump to version 2.0.1
- 9e72a391 cache/siggen: Add unihash cache copy function
- 7912dabb runqueue: Fix unihash cache mismatch issues
- 6a346df5 osc: fix DeprecationWarning
- aa4cee1b fetch2/osc: Add support to query latest revision
- 4d671504 build: Add clean_stamp API function to allow removal of task stamps
- f5f9a7b8 data: Do not depend on vardepvalueexclude flag
- 3ec78686 fetch2/osc: Small fixes for osc fetcher
- ea6a676c bitbake-diffsigs: break on first dependent task difference
- d3d406e8 fetch/git : Use cat as pager
- a70a7376 cache: correctly handle file names containing colons
- dd2475e3 cookerdata: Change emphasis in error message to be clearer to users
- 0ed800e1 runqueue: Fix sig file location when using multiconfig
- c1c8fc67 fetch2/ssh.py: decode path back for ssh
- a23c201c fetch2/osc: Add missing parameter
- d720dfa4 buildinfohelper: Drop unused variables
- e6a3173c runqueue: Drop pointless variable assignment
- 7254eb6b persist_data: Use a valid exception for missing implementation
- 62556508 ui/knotty: Drop pointless pass statement
- 42809f6a siggen: Drop pointless break statement
- 879f17ec fetch2/crate: Drop unused import
- 3bcb20f0 buildinfohelper: Drop unused function
- 140929b4 msg: Drop unused local variable
- 1c811ad6 cooker: Drop unused loop
- aca0ff85 ui/buildinfohelper: Drop unused import
- 543315e6 server/process: Drop unused import
- da812d93 tests/parse: Fix one test overwriting another
- e8b176de bitbake-diffsigs: Make PEP8 compliant
- 98cad863 fetch2/git: Simplify the validation of SHA-1 revisions
- 07de375c providers: use local variable for packages_dynamic pattern

Signed-off-by: Jose Quaresma <[email protected]>