what is this file with .template-e suffix?

I'm not sure... probably created a copy with a typo. I'm removing this file.

I understand the challenge here, but this downloads the right version of boringssl_prefix_symbols (depending on boringssl's SHA) from master of our repository. That means we'd have to keep an ever-growing list of boringssl shadow headers in master of our repository forever (one version for each SHA we use), and that's not a good solution.
Also I don't like that the download command can fail and that can lead to random errors when building.
Would inlining a base64 gzip compressed version of boringssl_prefix_symbols.h fit in the podspec? I know it's dirty but it's better than the challenges of this approach.
Checking in a generated version of boringssl_prefix_symbols.h into our repo is fine, but then regenerating project could inline the base64 compressed version of it into the podspec (which make the podspec standalone).

I tried this and
gzip -c boringssl_prefix_symbols.h | base64 >boringssl_prefix_symbols.h.gz.base64 gives you a file 37KB in size.
That should be good enough for us.

$ ls -lh
total 320K
-rw-r----- 1 jtattermusch primarygroup 278K Nov 21 08:34 boringssl_prefix_symbols-7f02881e96e51f1873afcf384d02f782b48967ca.h
-rw-r----- 1 jtattermusch primarygroup  37K Nov 21 08:36 boringssl_prefix_symbols.h.gz.base64

I'll give it a try

seems dirty, can we avoid this line?

I don't think so because if users build their projects as Framework, Apple does require Framework name to be the first segment in the include statement (see Apple's doc). The only possible way to avoid it is to hack the "Header search path" of the generated gRPC-Core and BoringSSL-GRPC targets pointing to the downloaded BoringSSL repo. Not sure if it'll work for Framework (because in that case the two targets are including a header outside of the Framework's Header directory). And if it works, IMHO it's an even more dirty hack.

Ok, so just to make sure: does this basically replace all occurrences of #include <boringssl_prefix_symbols.h> with #include <openssl_grpc/boringssl_prefix_symbols.h> in all *.h, *.cc and *.c files?
If so, then fair, it's not super clean, but acceptable.

do we really need docker for this, it seems that generating the prefix header should work just fine on people's workstation?
It look like you need 3 non-trivial scripts upgrade_boringssl_objc.sh, Dockerfile and generate_boringssl_prefix_header.sh to update the prefix header, so I'm not sure this actually simplifies things?

I am using Docker for a few reasons:

• People also use Mac for the same purpose too (this process is for ObjC). Installing dependencies on Mac is not as trivial as on Linux.
• I tend to not mess with user's workspace. As you see the process involves building BoringSSL and creating a symbol list file. The BoringSSL in the user's workspace may have their own changes. And I do need to rm -rf the artifacts which makes me a bit uncomfortable. Docker just get rid of all those matters.
• We are already using Docker in tools/distrib/clang_format_code.sh.

As to your question, even if we do not use Docker, it would be pretty much combining the 2 shell scripts into one; I don't think there's much to remove there other than the Docker related lines. So I don't think Docker complicate things too.

I'd argue most people don't have docker installed on their macs, so arguably this doesn't really make things easier (you'd need to install docker on mac and build the images first which takes time) but it does make all the scripts more complicated.
I'd prefer if we provided a robust script for regenerating the .h file (that assumes golang a cmake are installed, both are easy to install on mac). We can look into using docker in a followup PR and see if it could further simplify things.
Also, for now just supporting regenerating the prefix .h on linux is completely acceptable - later we can make improvements.

seems completely unnecessary. You have the right commit of boringssl in third_party/boringssl already so why this?

Part of the reason was I did not find a way to cleanly copy both /tools/dockerfile/grpc_objc/generate_boringssl_prefix_header/generate_boringssl_prefix_header.sh and /third_party/boringssl into the container. To make it possible I probably need to set the Docker build context to be gRPC's root dir, but that means transferring the entire grpc directory to Docker when running docker build, which takes much more time than git clone when I tried it.

And also for the same sanity reason as above, it does not cost too much to get a clean version of BoringSSL.

nit: s/correcty/the correct/ ?

Ok, so just to make sure: does this basically replace all occurrences of #include <boringssl_prefix_symbols.h> with #include <openssl_grpc/boringssl_prefix_symbols.h> in all *.h, *.cc and *.c files?
If so, then fair, it's not super clean, but acceptable.

For better readability (being able to see diffs over versions etc., perhaps it would be better to check-in only the boringssl_prefix_symbols.h file as (src/boringssl/boringssl_prefix_symbols.h) and only perform the gzip compression and base64 encoding in the .template. That way the generated podspec will have the gz.b64 version of the headers, but we won't have a cryptic encoded file checked into our repository.
One would also be able to see the diffs in .h over versions, which can come very handy when trying to troubleshoot boringssl upgrades in the future (one can clearly see which symbols were added etc - very useful).

It should be pretty easy to perform gz and b64 with python code inside the template.

This should work, but the problem I meet right now is that python 2's gzip library has a bug that using gzip.compress(...) will report AttributeError: 'module' object has no attribute 'compress'. This does not happen on python 3. It happens on both my mac and my Linux. I'll check if Kokoro has the same error (still running as for now), but I guess a lot of the team's machines could have the same problem.

@jtattermusch - did we switch to python 3? Somehow I remember we dropped python 2 last year.

ignore, this is resolved

nit: the comment is out of date - looks like this is intended to be a script to perform the full update of boringssl version, not just updating the shadow list.

qq: where are the steps to obtain boringssl_prefix_symbols.h coming from? Are they extracted from the boringssl build? If so, please add a note and link to the original location.

Are they extracted from the boringssl build?

Yes

I think a cleaner and faster solution would be if boringssl_prefix_symbols.h would start with
// generated by generate_boringssl_prefix_header.sh from boringssl [THE-COMMIT-SHA] (just add that line when generating the file)
and sanity check would just check that boringssl_prefix_symbols.h has the right header.
That way you could make the check in seconds and you wouldn't need to add cmake to the sanity docker file.

I don't like that this script is not idempotent - if you run it multiple times, it will keep increasing the version number.
IMHO just having a robust .sh that regenerates the boringssl_prefix_symbols.h is enough for this script.

If you want to add a script that automates the whole boringssl update process, we can look into that but there are questions to be answered first and it doesn't belong to this PR. I'd recommend turning this script into something that just generates the header file (with use of docker if you will).

I suspect without changing the UID, the generated file will end up being owned by root.

stale

stale

I'd argue most people don't have docker installed on their macs, so arguably this doesn't really make things easier (you'd need to install docker on mac and build the images first which takes time) but it does make all the scripts more complicated.
I'd prefer if we provided a robust script for regenerating the .h file (that assumes golang a cmake are installed, both are easy to install on mac). We can look into using docker in a followup PR and see if it could further simplify things.
Also, for now just supporting regenerating the prefix .h on linux is completely acceptable - later we can make improvements.

please add a note where this magic is coming from. Without more context, this seems very arbitrary and hacky.

What magic do you mean? Do you mean the path where libssl.a is in?

sorry for not being clearer. by "magic" I meant the sequence of commands (build, call read_symbols.go, build again, ...) that someone without context will have no idea where they came from. If you just mention you got the commands from the boringssl docs for prefixing symbols then all is fine I think.

does boringssl provide the make boringssl_prefix_symbols? please link to boringssl docs if this is documented somewhere.

Looks like this is based on https://github.com/google/boringssl/blob/master/BUILDING.md#building-with-prefixed-symbols

please do include this link as explanation of what's happening.

please include some explanation of what's the point of this snippet (I see there's some explanation below, but without more context it's not really clear that this snippet is related to that).

This line has an unfinished sentence, probably a typo?

nit: instead of using a "global" variable in the prologue, would it be more readable to actually define a function and then invoke it from here?

nit: I looked at the generated file and it looks like it generates one extremely long line with base64.
It would be nicer if you wrapped the base64 string at 80 characters, which should still be parseable by the base64 command, but the generated file would look that ugly?

qq: have you checked that the symbols are actually getting prefixed when you build like this? I don't see the boringssl_prefix_symbols.h being used anywhere in our build files so I'm just wondering what the trick is. Perhaps boringssl_prefix_symbols.h is being included by boringssl headers, which makes it transparent for grpc usage of boringssl?

Yes I did check the symbols are prefixed, and correctly used by gRPC. The boringssl_prefix_symbols.h is used in BoringSSL headers. We just need to define the BORINGSSL_PREFIX macro when building gRPC-Core and BoringSSL-GRPC.