makefiles/utils: functions for lowercase and uppercase by cladmi · Pull Request #12119 · RIOT-OS/RIOT

cladmi · 2019-08-29T11:52:40Z

Contribution description

Add make only function to convert strings to lowercase and uppercase.
This can replace the $(shell echo $(var) | tr 'a-z-' 'A-Z_') pattern.
Using the 'make' implementation results in being around 100 times faster.

I was inspired by wondering how faster was the lc implementation in mips.inc.mk.

RIOT/makefiles/arch/mips.inc.mk

Line 7 in e9ea6a0

    
           lc = $(subst A,a,$(subst B,b,$(subst C,c,$(subst D,d,$(subst E,e,$(subst F,f,$(subst G,g,$(subst H,h,$(subst I,i,$(subst J,j,$(subst K,k,$(subst L,l,$(subst M,m,$(subst N,n,$(subst O,o,$(subst P,p,$(subst Q,q,$(subst R,r,$(subst S,s,$(subst T,t,$(subst U,u,$(subst V,v,$(subst W,w,$(subst X,x,$(subst Y,y,$(subst Z,z,$1))))))))))))))))))))))))))

Testing procedure

Running the test in tests/build_system_utils should work.

If you want to verify the test tests, you can also play with changing the functions implementation and replacing some characters.

Speed comparison.

I used this diff to compare the speed.

diff to get speed comparison

diff --git a/makefiles/utils/strings.mk b/makefiles/utils/strings.mk
index 2feedcff7..5c3e45b33 100644
--- a/makefiles/utils/strings.mk
+++ b/makefiles/utils/strings.mk
@@ -6,3 +6,28 @@
 lowercase = $(subst A,a,$(subst B,b,$(subst C,c,$(subst D,d,$(subst E,e,$(subst F,f,$(subst G,g,$(subst H,h,$(subst I,i,$(subst J,j,$(subst K,k,$(subst L,l,$(subst M,m,$(subst N,n,$(subst O,o,$(subst P,p,$(subst Q,q,$(subst R,r,$(subst S,s,$(subst T,t,$(subst U,u,$(subst V,v,$(subst W,w,$(subst X,x,$(subst Y,y,$(subst Z,z,$1))))))))))))))))))))))))))
 uppercase = $(subst a,A,$(subst b,B,$(subst c,C,$(subst d,D,$(subst e,E,$(subst f,F,$(subst g,G,$(subst h,H,$(subst i,I,$(subst j,J,$(subst k,K,$(subst l,L,$(subst m,M,$(subst n,N,$(subst o,O,$(subst p,P,$(subst q,Q,$(subst r,R,$(subst s,S,$(subst t,T,$(subst u,U,$(subst v,V,$(subst w,W,$(subst x,X,$(subst y,Y,$(subst z,Z,$1))))))))))))))))))))))))))
 uppercase_and_underscore = $(call uppercase,$(subst -,_,$1))
+
+
+# Local testing
+var = abcdefghijklmnopqrstuvwxyz-123456789
+VAR = ABCDEFGHIJKLMNOPQRSTUVWXYZ_123456789
+
+ifneq ($(call uppercase_and_underscore,$(var)),$(VAR))
+  $(error $(call uppercase_and_underscore,$(var)))
+endif
+
+NUM ?= 10000
+iters := $(shell seq $(NUM))
+
+d_0 := $(shell date +%s.%N)
+a := $(foreach i,$(iters),$(call uppercase_and_underscore,$(var)))
+d_end := $(shell date +%s.%N)
+d_diff := $(shell echo $(d_end) - $(d_0) | bc)
+$(warning makefile $(d_diff): $(d_0) - $(d_end))
+
+d_0 := $(shell date +%s.%N)
+$(foreach i,$(iters),$(shell echo $(var)| tr 'a-z-' 'A-Z_'>/dev/null))
+d_end := $(shell date +%s.%N)
+
+d_diff := $(shell echo $(d_end) - $(d_0) | bc)
+$(warning shell    $(d_diff): $(d_0) - $(d_end))

make --no-print-directory -C examples/hello-world/ clean
/home/harter/work/git/RIOT/makefiles/utils/strings.mk:26: makefile .165520482: 1567079757.710985849 - 1567079757.876506331
/home/harter/work/git/RIOT/makefiles/utils/strings.mk:33: shell    18.993786935: 1567079757.878182792 - 1567079776.871969727

I also ran it with NUM=100000 to get a better average.
It got sometime from 80 to 150 times faster depending on local load.

NUM=100000 make --no-print-directory -C examples/hello-world/ clean
/home/harter/work/git/RIOT/makefiles/utils/strings.mk:26: makefile 1.510057258: 1567079987.770061777 - 1567079989.280119035
/home/harter/work/git/RIOT/makefiles/utils/strings.mk:33: shell    236.004733208: 1567079989.281896031 - 1567080225.286629239

echo 236.004733208 / 1.510057258 | bc -l
156.28859896384140951561

Issues/PRs references

This work is done in the context of removing immediate variable evaluations and exports
#10850

As when changing a variable from immediate to deferred, a shell call would be made at each evaluation. So better fix this first.

Add make only function to convert strings to lowercase and uppercase. This can replace the `$(shell echo $(var) | tr 'a-z-' 'A-Z_')` pattern. Using the 'make' implementation results in being around 100 times faster.

cladmi · 2019-08-29T12:06:40Z

I miss-clicked "create pull request" before it was ready. Now the description and commit are up to date.

jcarrano · 2019-08-29T12:58:32Z

diff to get speed comparison

you are the man

cladmi · 2019-08-29T13:26:21Z

diff to get speed comparison

you are the man

How could it be otherwise ? :D

@cladmi

Add functions to convert to lowercase/uppercase and another one that also replaces dashes for underscores. This is a remake of a PR by @cladmi (pr RIOT-OS#12119) The Make version is about 20 times faster than using the shell. The following patch can be used for testing (run make --no-print-directory -C examples/hello-world/ clean) diff --git a/examples/hello-world/Makefile b/examples/hello-world/Makefile index 258d8e9..1f2adf837 100644 --- a/examples/hello-world/Makefile +++ b/examples/hello-world/Makefile @@ -16,3 +16,5 @@ DEVELHELP ?= 1 QUIET ?= 1 include $(RIOTBASE)/Makefile.include + +include $(RIOTMAKE)/utils/strings.mk diff --git a/makefiles/utils/strings.mk b/makefiles/utils/strings.mk index c571cd6..7dce7d80d 100644 --- a/makefiles/utils/strings.mk +++ b/makefiles/utils/strings.mk @@ -39,3 +39,28 @@ lowercase = $(call xlate,$(_UPPER_LETTERS),$(_LOWER_LETTERS),$(1)) uppercase = $(call xlate,$(_LOWER_LETTERS),$(_UPPER_LETTERS),$(1)) uppercase_and_underscore = $(call uppercase,$(subst -,_,$1)) + + +#~ # Local testing +#~ var = abcdefghijklmnopqrstuvwxyz-123456789 +#~ VAR = ABCDEFGHIJKLMNOPQRSTUVWXYZ_123456789 + +#~ ifneq ($(call uppercase_and_underscore,$(var)),$(VAR)) + #~ $(error $(call uppercase_and_underscore,$(var))) +#~ endif + +#~ NUM ?= 10000 +#~ iters := $(shell seq $(NUM)) + +#~ d_0 := $(shell date +%s.%N) +#~ a := $(foreach i,$(iters),$(call uppercase_and_underscore,$(var))) +#~ d_end := $(shell date +%s.%N) +#~ d_diff := $(shell echo $(d_end) - $(d_0) | bc) +#~ $(warning makefile $(d_diff): $(d_0) - $(d_end)) + +#~ d_0 := $(shell date +%s.%N) +#~ $(foreach i,$(iters),$(shell echo $(var)| tr 'a-z-' 'A-Z_'>/dev/null)) +#~ d_end := $(shell date +%s.%N) + +#~ d_diff := $(shell echo $(d_end) - $(d_0) | bc) +#~ $(warning shell $(d_diff): $(d_0) - $(d_end))

jcarrano · 2019-08-29T13:53:10Z

Check out jcarrano@620f031 the implementation there is slower but cleaner.

cladmi · 2019-08-29T14:02:05Z

With our current need, there is not need for a generic function I think.
You also need to define all letters of the alphabet so it is somehow almost as verbose, just on multiple lines.

git grep -e "tr '"  -e 'tr "'
Makefile.include:BOARDDEF := $(shell echo $(BOARD) | tr 'a-z' 'A-Z' | tr '-' '_')
Makefile.include:CPUDEF := $(shell echo $(CPU) | tr 'a-z' 'A-Z' | tr '-' '_')
Makefile.include:MCUDEF := $(shell echo $(MCU) | tr 'a-z' 'A-Z' | tr '-' '_')
cpu/efm32/Makefile.include:CFLAGS += -D$(shell echo $(CPU_MODEL) | tr 'a-z' 'A-Z')
cpu/esp32/Makefile.include:    FFLAGS += if=/dev/zero bs=1M count=4  | tr "\\000" "\\377" > tmp.bin && cat tmp.bin |
cpu/kinetis/kinetis-info.mk:KINETIS_INFO := $(shell printf '%s' '$(CPU_MODEL)' | tr 'a-z' 'A-Z' | sed -E -e 's/^(M|K|S9)K([ELMSVW]?|EA)([0-9]?)([0-9]?)([A-Z])([NX]?)([0-9][0-9M]?[0-9]?)([ABZ]?)(.*)$$/\1 \2 \3 \4 \5 \6 \7 \8:\9/' -e 's/^([^ ]*)  /\1 K /' -e 's/^([^:]*):([CMV])(..)([0-9]*).*$$/\1 \2 \3 \4/' -e 's/  / _ /g' -e 's/  / _ /g')
cpu/msp430_common/Makefile.include:MODEL = $(shell echo $(CPU_MODEL) | tr 'a-z' 'A-Z')
cpu/nrf5x_common/Makefile.include:FAM = $(shell echo $(CPU_FAM) | tr 'a-z-' 'A-Z_')
cpu/sam0_common/Makefile.include:CFLAGS += -DCPU_FAM_$(shell echo $(CPU_FAM) | tr 'a-z-' 'A-Z_')
cpu/sam_common/Makefile.include:CFLAGS += -DCPU_FAM_$(shell echo $(CPU_FAM) | tr 'a-z-' 'A-Z_')
cpu/stm32_common/Makefile.include:FAM = $(shell echo $(CPU_FAM) | tr 'a-z-' 'A-Z_')
cpu/stm32_common/Makefile.include:CPU_LINE ?= $(shell echo $(CPU_MODEL) | cut -c -9 | tr 'a-z-' 'A-Z_')xx
cpu/stm32_common/stm32_mem_lengths.mk:STM32_INFO := $(shell printf '%s' '$(CPU_MODEL)' | tr 'a-z' 'A-Z' | sed -E -e 's/^STM32(F|L)(0|1|2|3|4|7)([A-Z0-9])([0-9])(.)(.)(_A)?/\1 \2 \2\3\4 \3 \4 \5 \6 \7/')
cpu/stm32f0/stm32_line.mk:LINE := $(shell echo $(CPU_MODEL) | tr 'a-z-' 'A-Z_' | sed -E -e 's/^STM32F([0-9][0-9][0-9])(.)(.)/\1 \2 \3/')
cpu/stm32f1/stm32_line.mk:LINE := $(shell echo $(CPU_MODEL) | tr 'a-z-' 'A-Z_' | sed -E -e 's/^STM32F([0-9][0-9][0-9])(.)(.)/\1 \2 \3/')
cpu/stm32f3/stm32_line.mk:LINE := $(shell echo $(CPU_MODEL) | tr 'a-z-' 'A-Z_' | sed -E -e 's/^STM32F([0-9][0-9][0-9])(.)(.)/\1 \2 \3/')
cpu/stm32f4/stm32_line.mk:LINE := $(shell echo $(CPU_MODEL) | tr 'a-z-' 'A-Z_' | sed -E -e 's/^STM32F([0-9][0-9][0-9])(.)(.)/\1 \2 \3/')
makefiles/arch/cortexm.inc.mk:MODEL = $(shell echo $(CPU_MODEL) | tr 'a-z' 'A-Z')
makefiles/arch/cortexm.inc.mk:ARCH = $(shell echo $(CPU_ARCH) | tr 'a-z-' 'A-Z_')
makefiles/info.inc.mk:    echo "$(abspath $(shell echo "$(MAKEFILE_LIST)"))" | tr ' ' '\n'; \
makefiles/modules.inc.mk:EXTDEFINES = $(addprefix -D,$(shell echo '$(ED)' | tr 'a-z-' 'A-Z_'))
tests/pkg_fatfs/create_fat_image_file.sh:echo "the test file content 123 abc" | tr '\n' '\0' >> /media/riot_fatfs_disk/test.txt
tests/pkg_fatfs_vfs/create_fat_image_file.sh:echo "the test file content 123 abc" | tr '\n' '\0' >> /media/riot_fatfs_disk/test.txt

jcarrano

Ok, tested.

cladmi · 2019-08-29T14:08:18Z

Thank you for the review. I will do PRs to start using it.

makefiles/utils: functions for lowercase and uppercase

765f3e9

Add make only function to convert strings to lowercase and uppercase. This can replace the `$(shell echo $(var) | tr 'a-z-' 'A-Z_')` pattern. Using the 'make' implementation results in being around 100 times faster.

cladmi force-pushed the pr/makefiles/utils/uppercase_function branch from 2b999f5 to 765f3e9 Compare August 29, 2019 12:06

jcarrano self-requested a review August 29, 2019 12:35

jcarrano added Area: build system Area: Build system CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Type: new feature The issue requests / The PR implemements a new feature for RIOT labels Aug 29, 2019

jcarrano approved these changes Aug 29, 2019

View reviewed changes

jcarrano merged commit 866b126 into RIOT-OS:master Aug 29, 2019

cladmi deleted the pr/makefiles/utils/uppercase_function branch August 29, 2019 14:07

kb2ma added this to the Release 2019.10 milestone Sep 16, 2019

This was referenced Sep 30, 2019

makefiles: CFLAGS convert to the uppercase function instead of using the shell #12340

Merged

makefiles/arch/mips.inc.mk: use makefiles/utils lowercase function #12341

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

makefiles/utils: functions for lowercase and uppercase#12119

makefiles/utils: functions for lowercase and uppercase#12119
jcarrano merged 1 commit intoRIOT-OS:masterfrom
cladmi:pr/makefiles/utils/uppercase_function

cladmi commented Aug 29, 2019 •

edited

Loading

Uh oh!

cladmi commented Aug 29, 2019

Uh oh!

jcarrano commented Aug 29, 2019

Uh oh!

cladmi commented Aug 29, 2019

Uh oh!

jcarrano commented Aug 29, 2019

Uh oh!

cladmi commented Aug 29, 2019

Uh oh!

jcarrano left a comment

Uh oh!

cladmi commented Aug 29, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

cladmi commented Aug 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution description

Testing procedure

Speed comparison.

Issues/PRs references

Uh oh!

cladmi commented Aug 29, 2019

Uh oh!

jcarrano commented Aug 29, 2019

Uh oh!

cladmi commented Aug 29, 2019

Uh oh!

jcarrano commented Aug 29, 2019

Uh oh!

cladmi commented Aug 29, 2019

Uh oh!

jcarrano left a comment

Choose a reason for hiding this comment

Uh oh!

cladmi commented Aug 29, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cladmi commented Aug 29, 2019 •

edited

Loading