zscaler and some VPNs mess up ssl certs and bazel will fail fetching. Similar deal if you have your own repo with self-signed certs, I reckon.
So, as you run into errors, note the server bazel is trying to download from e.g., here it's mirror.bazel.build
ERROR: /private/var/tmp/_bazel_jwattenbarger/a2f146ad396f505902e2eb9d5d7b725a/external/bazel_tools/tools/test/BUILD:36:6: @bazel_tools//tools/test:coverage_report_generator depends on @remote_coverage_tools//:coverage_report_generator in repository @remote_coverage_tools which failed to fetch. no such package '@remote_coverage_tools//': java.io.IOException: Error downloading [https://mirror.bazel.build/bazel_coverage_output_generator/releases/coverage_output_generator-v2.5.zip] to /private/var/tmp/_bazel_jwattenbarger/a2f146ad396f505902e2eb9d5d7b725a/external/remote_coverage_tools/temp8333830970334961723/coverage_output_generator-v2.5.zip: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
ERROR: Analysis of target '//qa-lib/src/test:VaultTests' failed; build aborted:
INFO: Elapsed time: 13.496s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
So, we'll download what our macOS thinks the cert is (the evil man-in-the-middle work VPN cert).
Then, we'll put it in our own JAVA_HOME path's security/cacerts. You are using your own version of Java, right? Don't rely on the macOS installed Java. This method is likely possible with just the macOS Java, but you'd need sudo access because it lives in /Library. In fact, don't rely on system versions of anything if you're doing development, even if you're on Linux (<-- bonus advice, no cap). I have many bad memories of fooling with Debian's system Python and version conflicts, packages, etc. Not worth it. People use "dev containers" but you don't even need containers for these portable languages if you outline your deps in your dev environment.
BTW, in case it isn't obvious, all the files we write/generate go into the root of your project/repo directory, not your system-wide or home directory root or anything like that. Although, asdf will install Java under ~/.asdf/ we'll still consider that just for bazel (I mean it's JDK 11, if you need another version without molested certs, then install a different one from asdf).
Instead of macOS system Java, use asdf or some other home-based personal/user package manager. This might work with Homebrew, but the paths would be different. I use stuff like asdf or pyenv or nvm because I work in so many different projects and repos that have their own expectations about tool versions.
My setup is a little complicated because of how flexible my asdf setup is. I also use direnv for this.
Basically, direnv helps me set my JAVA_HOME based on what version of Java asdf sets up, and I also use it to dynamically write my bazelrc to point its server base dir to that JAVA_HOME because I don't think that .bazelrc expands paths.
So, install asdf or a similar tool. Or, install a version of Java JDK/SDK into your home directory.
Then, install direnv if you choose to. You could skip this and just make sure that JAVA_HOME in your shell environment points to the same JAVA_HOME that we're going to import keys into.
With or without direnv, put the functions from javacerts.sh into your environment; either source the file or add them to your shell init.
This might be the main part of what you need from this tutorial if your setup is different or you don't want to use asdf and direnv.
getsslcert() { echo -n | openssl s_client -connect "${1}":443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ./"${1}".crt }
java_addcert() { keytool -import -v -trustcacerts -alias "${2}" -file ./"${1}" -keystore "${JAVA_HOME}"/lib/security/cacerts -keypass changeit -storepass changeit }Next is my .envrc which is for direnv. So, you don't need to do this if you don't want to use asdf, direnv, dynamic JAVA_HOME, etc. I'd only put these in your project directory. I wouldn't try to generate or change your $HOME/.bazelrc or anything! Bazel will read .bazelrc from the project directory automatically.
asdf install
export JAVA_HOME="$(dirname $(dirname $(realpath $(asdf which javac))))"
echo "startup --server_javabase=${JAVA_HOME}" > ./.bazelrc.javahome
echo "try-import %workspace%/.bazelrc.javahome" > ./.bazelrcSince these are generated, if you want anything else in them, add it to the .envrc because they get overridden every time direnv runs in your project dir (i.e. every time you cd to it).
For educational purposes, my .bazelrc.javahome and .bazelrc files end up looking like this:
bazelrc.javahome
startup --server_javabase=/Users/jwattenbarger/.asdf/installs/java/adoptopenjdk-11.0.0+28
bazelrc
try-import %workspace%/.bazelrc.javahome
Since my setup is dynamic and nothing runs right away, you could have probably done everything up to here with no Java installed yet. For reference, here's the Java I installed for bazel.
My Java install with asdf looked like this:
$ asdf plugin-add java
$ asdf install java adoptopenjdk-11.0.0+28See top of file for an example error. From there, we see mirror.bazel.build doesn't have a cert or has the "wrong" one (one that doesn't match). I guess we'll have to trust the one that zscaler or whatever evil mitm is pointing us to because we have to in order to get any work done in this goddamn place. So, let's add that gargoyle to our personal/project java trusted certs. Use our function.
❯ getsslcert mirror.bazel.build
depth=2 C = US, ST = California, O = Zscaler Inc., OU = Zscaler Inc., CN = Zscaler Intermediate Root CA (zscalerthree.net), emailAddress = support@zscaler.com
verify error:num=20:unable to get local issuer certificate
verify return:0
DONEThis downloads it to our directory. It's fine to just keep working out of your project/repo dir. But, this next part's important. We'll import it into our custom java's cacerts:
❯ java_addcert ./mirror.bazel.build.crt zmirrorbazel
Warning: use -cacerts option to access cacerts keystore
Owner: CN=bcr.bazel.build
Issuer: CN="Zscaler Intermediate Root CA (zscalerthree.net) (t) ", OU=Zscaler Inc., O=Zscaler Inc., ST=California, C=US
Serial number: 6fb9f9bb58eea8e410e0f9dca2d0208e
--->8---
Trust this certificate? [no]: yes
Certificate was added to keystore
[Storing /Users/jwattenbarger/.asdf/installs/java/adoptopenjdk-11.0.0+28/lib/security/cacerts]Importantly, since you tried and failed running bazel build, bazel's daemon is likely still running. You need to stop it so that it can restart to pickup the certs (apparently).
$ bazel shutdownet voila:
❯ bazel build qa-lib/src/test:all
Starting local Bazel server and connecting to it...
INFO: Analyzed 3 targets (54 packages loaded, 1100 targets configured).
INFO: Found 3 targets...
INFO: Elapsed time: 16.030s, Critical Path: 4.06s
INFO: 24 processes: 11 internal, 6 darwin-sandbox, 7 worker.
INFO: Build completed successfully, 24 total actionsDisconnect from zscaler or your VPN for the bazel fetch operation. You can most likely re-connect and get into your write, build, run flow as long as you don't add new dependencies in between. However, for me this didn't always work! I struggled especially with the repos referenced in the bazel rules_go for some reason, even disconnected from VPN. It's a work laptop, so who knows? They might install their own system-wide certs that are out of date or something.
Another idea, and the first way I got it to work at all, was to download the packages individually from GitHub and then name them the way the WORKSPACE was expecting, but put them in a local dir and change all the stuff to reference that. Yep, this is as painful as it sounds. At one point, I think I was even building zips and changing hashes in WORKSPACE files. Not recommended.
Finally, probably the simplest alternative would be to use bazel's "offline" feature to create a seeded bazel base dir or whatever from a completely different computer (e.g., one on your home network w/o mitm re-writing proxies and where you know bazel fetch works). You don't have to run the full build, but you can do a bazel fetch. Then, copy over all those files to your work computer/the computer you want to bazel build on and tell bazel to use that base directory (look it up; it's in the manual). This might work fine or even great, depending on how well the transitive rules, etc. are. I could maybe see some issues with cross-compilation on different architectures or OSs. e.g., if you run it on Linux, it will likely fetch Linux binaries. So, in that case you should also look up how to force bazel to fetch for your target platform.
If you have a Java install with the right certs (e.g., if your company went the extra mile or you copied them over or imported from somewhere else), you could just make sure you always pass --server_javabase=$JAVA_HOME and that JAVA_HOME is right.
I'm sure there's lots of room for improvement here. Feel free to fork this gist or tell me I'm stupid in the comments.
One thing I just noticed is that I should add comments to the generated bazelrc files indicating they're generated. I could probably do some error-checking to make sure there's a valid $JAVA_HOME, etc.
I may add a nix version (vs. asdf) soon. If so, I'll put it in this gist. It would probably be simpler since it could all be in the shell.nix or default.nix--possibly even the cert setup, etc.