Add realpath support to pkg #7981

zrhoffman · 2024-04-03T19:08:21Z

The pkg script (along with our build scripts) makes use of realpath. But as noticed in #7958, pkg does not have a backup realpath function if the host OS does not have realpath. #7981 adds a backup realpath function to the pkg script.

Which Traffic Control components are affected by this PR?

Build system
Automation
unknown

What is the best way to verify this PR?

Try building ATS or Traffic Portal v2
./pkg vbo ats on macOS or another platform with realpath` at least temporarily removed

PR submission checklist

This PR has tests
This PR has documentation
This PR has a CHANGELOG.md entry
This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY (see the Apache Software Foundation's security guidelines for details)

zrhoffman · 2024-04-03T19:09:03Z

r? @ntheanh201, @limited

ntheanh201 · 2024-04-04T15:34:26Z

I still see that issue

./pkg -o -v -b ats

/bin/realpath
Building ats.
[+] Building 0.0s (17/17) FINISHED                                                                                                                                                                                 docker:orbstack
 => [ats internal] load build definition from Dockerfile-tsb                                                                                                                                                                  0.0s
 => => transferring dockerfile: 2.63kB                                                                                                                                                                                        0.0s
 => [ats internal] load metadata for docker.io/library/rockylinux:8                                                                                                                                                           0.0s
 => [ats internal] load .dockerignore                                                                                                                                                                                         0.0s
 => => transferring context: 2B                                                                                                                                                                                               0.0s
 => [ats internal] load build context                                                                                                                                                                                         0.0s
 => => transferring context: 31B                                                                                                                                                                                              0.0s
 => [ats tsb-build 1/6] FROM docker.io/library/rockylinux:8                                                                                                                                                                   0.0s
 => CACHED [ats stage-1 2/7] RUN YUM -y install git sudo                                                                                                                                                                      0.0s
 => CACHED [ats stage-1 3/7] RUN ECHO -e "#!/usr/bin/env bash\necho docker-compose is not supported in this container and the build command will not work.\nexit 1" > /usr/bin/docker-compose                                 0.0s
 => CACHED [ats stage-1 4/7] RUN CHMOD +x /usr/bin/docker-compose                                                                                                                                                             0.0s
 => CACHED [ats tsb-build 2/6] RUN yum clean all && yum -y install git &&  git config --global --add safe.directory '*'                                                                                                       0.0s
 => CACHED [ats tsb-build 3/6] COPY GO_VERSION /                                                                                                                                                                              0.0s
 => CACHED [ats tsb-build 4/6] RUN curl -L https://dl.google.com/go/go$(cat /GO_VERSION).linux-amd64.tar.gz | tar -C /usr/local -xz && ln -s /usr/local/go/bin/go /usr/bin/go                                                 0.0s
 => CACHED [ats tsb-build 5/6] RUN mkdir -p /go/src/github.com/comcast/tsb && cd /go/src/github.com/comcast/tsb && git init && git remote add origin https://github.com/comcast/tsb && git fetch --depth=1 origin 851be3e137  0.0s
 => CACHED [ats tsb-build 6/6] RUN cd /go/src/github.com/comcast/tsb && CGOENABLED=0 go build -o /usr/bin/tsb                                                                                                                 0.0s
 => CACHED [ats stage-1 5/7] COPY --FROM=TSB-BUILD /usr/bin/tsb /usr/bin/tsb                                                                                                                                                  0.0s
 => CACHED [ats stage-1 6/7] RUN GIT config --global user.email "[email protected]"                                                                                                                               0.0s
 => CACHED [ats stage-1 7/7] RUN GIT config --global user.name "Apache Traffic Control"                                                                                                                                       0.0s
 => [ats] exporting to image                                                                                                                                                                                                  0.0s
 => => exporting layers                                                                                                                                                                                                       0.0s
 => => writing image sha256:4ee81397cf7c5f790366fb08de0cdb72fff07b1419dfbcd0bef95e3868d56991                                                                                                                                  0.0s
 => => naming to docker.io/library/build-ats                                                                                                                                                                                  0.0s
WARN[0000] Found orphan containers ([build-traffic_stats_build-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. 
realpath: illegal option -- e
usage: realpath [-q] [path ...]
realpath: illegal option -- e
usage: realpath [-q] [path ...]
unknown flag: --services
/bin/realpath
Top-level object must be a mapping
Top-level object must be a mapping
Building /bin/realpath.
Failed to build /bin/realpath.
Results in 'dist':
total 0
Failed to build ats.
Results in 'dist':
total 0

zrhoffman · 2024-04-04T15:40:24Z

Thanks! Removed the -e from realpath in commands in e39d13a.

limited · 2024-04-04T18:56:28Z

I still saw numerous failures on this

/bin/realpath
Building /bin/realpath.
Building weasel.
Building cache-config_build.
Building docs.
Building grove_build.
Building grovetccfg_build.
Building source.
Building tc-health-client_build.
Building traffic_monitor_build.
Building traffic_ops_build.
Building traffic_portal_build.
Building traffic_router_build.
Building traffic_stats_build.
Failed to build /bin/realpath traffic_ops_build traffic_portal_build traffic_router_build traffic_stats_build.
Results in 'dist':

zrhoffman · 2024-04-06T05:44:41Z

I still saw numerous failures on this

@limited Thanks for testing, I think it is fixed as of 7d82265. Care to retest?

ntheanh201 · 2024-04-06T13:07:26Z

your new realpath of Docker compose is different than the old realpath, and I think that cause this problem:

Building ats_build.
[+] Building 0.0s (1/1) FINISHED                                                                                                                                                                                   docker:orbstack
 => [ats_build internal] load build definition from Dockerfile                                                                                                                                                                0.0s
 => => transferring dockerfile: 2B                                                                                                                                                                                            0.0s
failed to solve: failed to read dockerfile: open Dockerfile: no such file or directory
Failed to build ats_build.
Results in 'dist':
total 0
Failed to build ats.
Results in 'dist':
total 0

ntheanh201 · 2024-04-06T17:32:33Z

pkg

 		# compose file, in which case the symlink will be followed before it is processed.
 		if [ -e "$COMPOSE_FILE.$1" ] ; then
-			$SELF -f $(realpath -e "$COMPOSE_FILE.$1") $([ "$verbose" == 0 ] || echo "-v") $([ "$quiet" == 0 ] || echo "-q") $([ "$debug" == 0 ] || echo "-d") $([ "$build" == 0 ] || echo "-b") $("${COMPOSECMD[@]}" -f $(realpath -e "$COMPOSE_FILE.$1") config --services)
+			$SELF -f $(realpath "$COMPOSE_FILE.$1") $([ "$verbose" == 0 ] || echo "-v") $([ "$quiet" == 0 ] || echo "-q") $([ "$debug" == 0 ] || echo "-d") $([ "$build" == 0 ] || echo "-b") $("${COMPOSECMD[@]}" -f $(realpath "$COMPOSE_FILE.$1") config --services)


I see the different with the result of realpath $COMPOSE_FILE.$1 (./infrastructure/docker/build/docker-compose-opt.yml.ats) when you add realpath function is:
/Users/ntheanh201/Workspace/apache/trafficcontrol/infrastructure/docker/build/docker-compose-opt.yml.ats

vs. the result of realpath in master branch:
/Users/ntheanh201/Workspace/apache/trafficcontrol/traffic_server/_tsb/docker-compose.yml

This should be fixed in 7d9273d.

I see if you still use -e argument with realpath, it won't work

Fixed it with zrhoffman#124

I see if you still use -e argument with realpath, it won't work

Going back to realpath and looping through possible realpath executables in the path to find GNU realpath in ff3c456.

The `pkg` script (along with our build scripts) makes use of `realpath`. But unlike the build scripts, `pkg` does not have a backup `realpath` function if the host OS does not have `realpath`. apache#7981 adds a backup `realpath` function to the `pkg` script.

If realpath -e oes not work, the `realpath` binary is probably BSD realpath.

build/functions.sh

ntheanh201 · 2024-06-04T04:22:54Z

LGTM

zrhoffman · 2024-06-04T04:23:43Z

Thanks for reviewing! :)

zrhoffman added build related to the build process improvement The functionality exists but it could be improved in some way. labels Apr 3, 2024

This was referenced Apr 3, 2024

Updated build ATS to 9.2.4 #7958

Merged

Store Apache Traffic Server and grove logs in /var/log/trafficserver #7980

Merged

ntheanh201 reviewed Apr 6, 2024

View reviewed changes

zrhoffman added 3 commits May 30, 2024 11:03

realpath -e -> realpath

17b71e3

Improve realpath function and detection

a30432f

zrhoffman force-pushed the pkg-realpath branch 2 times, most recently from 4178b59 to 61935e1 Compare May 30, 2024 17:10

zrhoffman mentioned this pull request May 30, 2024

use GNU readlink instead of BSD readlink zrhoffman/trafficcontrol#124

Closed

zrhoffman force-pushed the pkg-realpath branch 2 times, most recently from 6a4a68b to c6e6589 Compare May 30, 2024 17:20

Use GNU realpath if realpath -e does not work

ff3c456

If realpath -e oes not work, the `realpath` binary is probably BSD realpath.

zrhoffman force-pushed the pkg-realpath branch from c6e6589 to ff3c456 Compare May 30, 2024 17:25

zrhoffman requested a review from ntheanh201 May 30, 2024 17:28

ntheanh201 reviewed May 31, 2024

View reviewed changes

build/functions.sh Show resolved Hide resolved

Add grealpath to paths to check for macOS/FreeBSD support

e298e4a

zrhoffman force-pushed the pkg-realpath branch from 3df1fe5 to e298e4a Compare May 31, 2024 11:34

zrhoffman requested a review from ntheanh201 May 31, 2024 11:35

ntheanh201 approved these changes Jun 4, 2024

View reviewed changes

zrhoffman merged commit 98687d5 into apache:master Jun 4, 2024

Add realpath support to pkg #7981

Add realpath support to pkg #7981

Uh oh!

Conversation

zrhoffman commented Apr 3, 2024

Which Traffic Control components are affected by this PR?

What is the best way to verify this PR?

PR submission checklist

Uh oh!

zrhoffman commented Apr 3, 2024

Uh oh!

ntheanh201 commented Apr 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zrhoffman commented Apr 4, 2024

Uh oh!

limited commented Apr 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zrhoffman commented Apr 6, 2024

Uh oh!

ntheanh201 commented Apr 6, 2024

Uh oh!

ntheanh201 Apr 6, 2024

Choose a reason for hiding this comment

Uh oh!

zrhoffman Apr 15, 2024

Choose a reason for hiding this comment

Uh oh!

ntheanh201 Apr 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ntheanh201 May 21, 2024

Choose a reason for hiding this comment

Uh oh!

zrhoffman May 30, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ntheanh201 commented Jun 4, 2024

Uh oh!

zrhoffman commented Jun 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ntheanh201 commented Apr 4, 2024 •

edited

Loading

limited commented Apr 4, 2024 •

edited

Loading

ntheanh201 Apr 17, 2024 •

edited

Loading