Makefile.include: building in docker should be done after clean by cladmi · Pull Request #10461 · RIOT-OS/RIOT

cladmi · 2018-11-23T17:33:27Z

Contribution description

When building in parallel, clean is done at the same time as the
container starts building.

So the parallel clean all handling should also be done for the
..in-docker-container target.

Testing procedure

I added a test commit to increase the probability of showing the issue and should be removed before merging. The issue is also present and was found without.

Testing should do a BUILD_IN_DOCKER=1 make -j clean all. It could be shown reliably with any example that takes more than 5 seconds to build on your machine.

BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -C examples/gnrc_networking -j clean all

With this PR, (should be run when sudo does not ask for the password anymore).

BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -C examples/gnrc_networking -j clean all                                                                                                                                                                                                  
make: Entering directory '/home/harter/work/git/RIOT/examples/gnrc_networking'                                                                                                                                                                                                            
sleep 2                                                                                                                                                                                                                                                                                   
for i in  ; do "make" -C /home/harter/work/git/RIOT/pkg/$i clean ; done                                                                                                                                                                                                                   
rm -rf /home/harter/work/git/RIOT/examples/gnrc_networking/bin/native                                                                                                                                                                                                                     
rm -rf scan-build/native
Launching build container using image "riot/riotbuild:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/RIOT:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/RIOT/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/RIOT/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/RIOT/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/RIOT:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache  \
     \
    -w '/data/riotbuild/riotproject/examples/gnrc_networking/' \
    'riot/riotbuild:latest' make all
Building application "gnrc_networking" for "native" with MCU "native".

"make" -C /data/riotbuild/riotbase/core
"make" -C /data/riotbuild/riotbase/drivers
"make" -C /data/riotbuild/riotbase/drivers/netdev_eth
"make" -C /data/riotbuild/riotbase/drivers/periph_common
"make" -C /data/riotbuild/riotbase/sys
"make" -C /data/riotbuild/riotbase/sys/auto_init
"make" -C /data/riotbuild/riotbase/sys/auto_init/netif
"make" -C /data/riotbuild/riotbase/sys/div
"make" -C /data/riotbuild/riotbase/sys/evtimer
"make" -C /data/riotbuild/riotbase/sys/fmt
"make" -C /data/riotbuild/riotbase/sys/iolist
"make" -C /data/riotbuild/riotbase/sys/luid
"make" -C /data/riotbuild/riotbase/sys/net/crosslayer/inet_csum
"make" -C /data/riotbuild/riotbase/sys/net/gnrc
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/netapi
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/netif
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/netif/hdr
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/netreg
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/network_layer/icmpv6
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/network_layer/icmpv6/echo
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/network_layer/icmpv6/error
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/network_layer/ipv6
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/network_layer/ipv6/hdr
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/network_layer/ipv6/nib
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/network_layer/ndp
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/pkt
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/pktbuf
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/pktbuf_static
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/pktdump
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/routing/rpl
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/transport_layer/udp
"make" -C /data/riotbuild/riotbase/sys/net/network_layer/icmpv6
"make" -C /data/riotbuild/riotbase/sys/net/network_layer/ipv6/addr
"make" -C /data/riotbuild/riotbase/sys/net/network_layer/ipv6/hdr
"make" -C /data/riotbuild/riotbase/sys/net/transport_layer/udp
"make" -C /data/riotbuild/riotbase/sys/od
"make" -C /data/riotbuild/riotbase/sys/ps
"make" -C /data/riotbuild/riotbase/sys/random
"make" -C /data/riotbuild/riotbase/sys/random/tinymt32
"make" -C /data/riotbuild/riotbase/sys/shell
"make" -C /data/riotbuild/riotbase/sys/shell/commands
"make" -C /data/riotbuild/riotbase/sys/trickle
"make" -C /data/riotbuild/riotbase/sys/xtimer
"make" -C /data/riotbuild/riotboard/native
"make" -C /data/riotbuild/riotboard/native/drivers
"make" -C /data/riotbuild/riotcpu/native
"make" -C /data/riotbuild/riotcpu/native/netdev_tap
"make" -C /data/riotbuild/riotcpu/native/periph
"make" -C /data/riotbuild/riotcpu/native/vfs
   text    data     bss     dec     hex filename
 215663     641  102788  319092   4de74 /data/riotbuild/riotproject/examples/gnrc_networking/bin/native/gnrc_networking.elf
make: Leaving directory '/home/harter/work/git/RIOT/examples/gnrc_networking'

With only the test commit, clean happens during the build in docker:

BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -C examples/gnrc_networking -j clean all
make: Entering directory '/home/harter/work/git/RIOT/examples/gnrc_networking'
sleep 2
Launching build container using image "riot/riotbuild:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/RIOT:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/RIOT/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/RIOT/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/RIOT/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/RIOT:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache  \
     \
    -w '/data/riotbuild/riotproject/examples/gnrc_networking/' \
    'riot/riotbuild:latest' make all 
Building application "gnrc_networking" for "native" with MCU "native".

"make" -C /data/riotbuild/riotbase/core
"make" -C /data/riotbuild/riotbase/drivers
"make" -C /data/riotbuild/riotbase/drivers/netdev_eth
"make" -C /data/riotbuild/riotbase/drivers/periph_common
"make" -C /data/riotbuild/riotbase/sys
"make" -C /data/riotbuild/riotbase/sys/auto_init
"make" -C /data/riotbuild/riotbase/sys/auto_init/netif
"make" -C /data/riotbuild/riotbase/sys/div
"make" -C /data/riotbuild/riotbase/sys/evtimer
"make" -C /data/riotbuild/riotbase/sys/fmt
"make" -C /data/riotbuild/riotbase/sys/iolist
"make" -C /data/riotbuild/riotbase/sys/luid
for i in  ; do "make" -C /home/harter/work/git/RIOT/pkg/$i clean ; done
rm -rf /home/harter/work/git/RIOT/examples/gnrc_networking/bin/native
rm -rf scan-build/native
/data/riotbuild/riotbase/sys/luid/luid.c:62:1: fatal error: opening dependency file /data/riotbuild/riotproject/examples/gnrc_networking/bin/native/luid/luid.d: No such file or directory
 }
 ^
compilation terminated.
/data/riotbuild/riotbase/Makefile.base:83: recipe for target '/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/luid/luid.o' failed
make[3]: *** [/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/luid/luid.o] Error 1
/data/riotbuild/riotbase/Makefile.base:20: recipe for target 'ALL--/data/riotbuild/riotbase/sys/luid' failed
make[2]: *** [ALL--/data/riotbuild/riotbase/sys/luid] Error 2
/data/riotbuild/riotbase/Makefile.base:20: recipe for target 'ALL--/data/riotbuild/riotbase/sys' failed
make[1]: *** [ALL--/data/riotbuild/riotbase/sys] Error 2
/data/riotbuild/riotbase/Makefile.include:449: recipe for target '/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/application_gnrc_networking.a' failed
make: *** [/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/application_gnrc_networking.a] Error 2
/home/harter/work/git/RIOT/makefiles/docker.inc.mk:105: recipe for target '..in-docker-container' failed
make: *** [..in-docker-container] Error 2
make: Leaving directory '/home/harter/work/git/RIOT/examples/gnrc_networking'

Issues/PRs references

Was found while testing #10344 and also mentioned in #10459 (comment)

jcarrano

Tested:

Without this fix:

RIOT/examples/gnrc_networking $BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -j clean all
sleep 2
Launching build container using image "riot/riotbuild:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/jcarrano/source/masterRIOT:/data/riotbuild/riotbase' \
    -v '/home/jcarrano/source/masterRIOT/cpu:/data/riotbuild/riotcpu' \
    -v '/home/jcarrano/source/masterRIOT/boards:/data/riotbuild/riotboard' \
    -v '/home/jcarrano/source/masterRIOT/makefiles:/data/riotbuild/riotmake' \
    -v '/home/jcarrano/source/masterRIOT:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/jcarrano/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache -v /home/jcarrano/source/vanillaRIOT/.git:/home/jcarrano/source/vanillaRIOT/.git \
     \
    -w '/data/riotbuild/riotproject/examples/gnrc_networking/' \
    'riot/riotbuild:latest' make all 
Building application "gnrc_networking" for "native" with MCU "native".

"make" -C /data/riotbuild/riotbase/core
"make" -C /data/riotbuild/riotbase/drivers
"make" -C /data/riotbuild/riotbase/drivers/netdev_eth
"make" -C /data/riotbuild/riotbase/drivers/periph_common
"make" -C /data/riotbuild/riotbase/sys
"make" -C /data/riotbuild/riotbase/sys/auto_init
"make" -C /data/riotbuild/riotbase/sys/auto_init/netif
"make" -C /data/riotbuild/riotbase/sys/div
"make" -C /data/riotbuild/riotbase/sys/evtimer
"make" -C /data/riotbuild/riotbase/sys/fmt
"make" -C /data/riotbuild/riotbase/sys/iolist
"make" -C /data/riotbuild/riotbase/sys/luid
"make" -C /data/riotbuild/riotbase/sys/net/crosslayer/inet_csum
"make" -C /data/riotbuild/riotbase/sys/net/gnrc
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/netapi
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/netif
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/netif/hdr
for i in  ; do "make" -C /home/jcarrano/source/masterRIOT/pkg/$i clean ; done
rm -rf /home/jcarrano/source/masterRIOT/examples/gnrc_networking/bin/native
rm -rf scan-build/native
/data/riotbuild/riotbase/sys/net/gnrc/netif/hdr/gnrc_netif_hdr_print.c:62:1: fatal error: opening dependency file /data/riotbuild/riotproject/examples/gnrc_networking/bin/native/gnrc_netif_hdr/gnrc_netif_hdr_print.d: No such file or directory
 }
 ^
compilation terminated.
/data/riotbuild/riotbase/Makefile.base:83: recipe for target '/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/gnrc_netif_hdr/gnrc_netif_hdr_print.o' failed
make[4]: *** [/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/gnrc_netif_hdr/gnrc_netif_hdr_print.o] Error 1
/data/riotbuild/riotbase/Makefile.base:20: recipe for target 'ALL--/data/riotbuild/riotbase/sys/net/gnrc/netif/hdr' failed
make[3]: *** [ALL--/data/riotbuild/riotbase/sys/net/gnrc/netif/hdr] Error 2
/data/riotbuild/riotbase/Makefile.base:20: recipe for target 'ALL--/data/riotbuild/riotbase/sys/net/gnrc' failed
make[2]: *** [ALL--/data/riotbuild/riotbase/sys/net/gnrc] Error 2
/data/riotbuild/riotbase/Makefile.base:20: recipe for target 'ALL--/data/riotbuild/riotbase/sys' failed
make[1]: *** [ALL--/data/riotbuild/riotbase/sys] Error 2
/data/riotbuild/riotbase/Makefile.include:449: recipe for target '/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/application_gnrc_networking.a' failed
make: *** [/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/application_gnrc_networking.a] Error 2
make: *** [/home/jcarrano/source/masterRIOT/makefiles/docker.inc.mk:106: ..in-docker-container] Error 2

With the fix it works. Squash and we merge.

Makefile.include

comment addressed

jcarrano · 2018-11-28T09:50:11Z

@cladmi While PRing #10492 I realized it may make more sense if you cherry-pick that commit into this PR (I couldn't figure out which PR depended on which, so for me that means they should go together.)

cladmi · 2018-12-05T16:38:50Z

Rebased now that #10492 is merged

cladmi · 2018-12-05T18:14:01Z

Build is now working in murdock, tell me when I should remove the test commits.

jcarrano · 2018-12-05T18:21:04Z

Remove the test commits and let's go forward.

cladmi · 2018-12-05T19:11:41Z

Removed the test commit.

When building in parallel, clean is done at the same time as the container starts building. So the parallel `clean all` handling should also be done for the `..in-docker-container` target.

cladmi · 2018-12-06T11:35:23Z

It failed because of the native test failure (now disabled) I just rebased to master to re-trigger.

cladmi requested review from jcarrano and jnohlgard November 23, 2018 17:33

jcarrano mentioned this pull request Nov 26, 2018

make: make clean all does not make sense and should be removed #10459

Closed

4 tasks

jcarrano approved these changes Nov 26, 2018

View reviewed changes

kaspar030 previously requested changes Nov 26, 2018

View reviewed changes

Makefile.include Outdated Show resolved Hide resolved

jcarrano mentioned this pull request Nov 28, 2018

mcuboot.mk: Add mcuboot targets to BUILDDEPS. #10492

Merged

jcarrano mentioned this pull request Nov 28, 2018

mcuboot.mk: refactor into a declarative-style makefile. #10494

Closed

cladmi force-pushed the pr/make/clean_all_docker_container branch from 2048a0a to 5e0a193 Compare December 5, 2018 16:37

cladmi force-pushed the pr/make/clean_all_docker_container branch from 5e0a193 to c0d89ec Compare December 5, 2018 19:11

cladmi removed the CI: needs squashing Commits in this PR need to be squashed; If set, CI systems will mark this PR as unmergable label Dec 5, 2018

Makefile.include: building in docker should be done after clean

135d2f7

When building in parallel, clean is done at the same time as the container starts building. So the parallel `clean all` handling should also be done for the `..in-docker-container` target.

cladmi force-pushed the pr/make/clean_all_docker_container branch from c0d89ec to 135d2f7 Compare December 6, 2018 11:34

cladmi mentioned this pull request Dec 6, 2018

Makefile.include: remove rule to remake PKGs' Makefile.include #10344

Merged

jcarrano merged commit a38cb6c into RIOT-OS:master Dec 6, 2018

cladmi deleted the pr/make/clean_all_docker_container branch December 6, 2018 15:05

aabadie added this to the Release 2019.01 milestone Dec 19, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Makefile.include: building in docker should be done after clean #10461

Makefile.include: building in docker should be done after clean #10461
jcarrano merged 1 commit intoRIOT-OS:masterfrom
cladmi:pr/make/clean_all_docker_container

cladmi commented Nov 23, 2018

Uh oh!

jcarrano left a comment

Uh oh!

Uh oh!

jcarrano commented Nov 28, 2018

Uh oh!

cladmi commented Dec 5, 2018

Uh oh!

cladmi commented Dec 5, 2018

Uh oh!

jcarrano commented Dec 5, 2018

Uh oh!

cladmi commented Dec 5, 2018

Uh oh!

cladmi commented Dec 6, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

cladmi commented Nov 23, 2018

Contribution description

Testing procedure

Issues/PRs references

Uh oh!

jcarrano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jcarrano commented Nov 28, 2018

Uh oh!

cladmi commented Dec 5, 2018

Uh oh!

cladmi commented Dec 5, 2018

Uh oh!

jcarrano commented Dec 5, 2018

Uh oh!

cladmi commented Dec 5, 2018

Uh oh!

cladmi commented Dec 6, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants