Sandbox slowness on OSX

[fenghaolw]

Description of the problem / feature request:

building has been extremely slow with the default darwin-sandbox

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

A mini repro could be found in https://github.com/alexeagle/rules_sass_repro
The repro contains 40 empty sass files
Running the sass compiler on them should be fast

bazel build :all takes ~60s on my mac

bazel build --strategy=SassCompiler=local :all takes ~4s

What operating system are you running Bazel on?

Mac OS 10.14.4

What's the output of bazel info release?

release 0.25.0

Have you found anything relevant by searching the web?

I found these issues: #902 and #1836 but they all seem obsolete.

JSON profile

According to https://docs.bazel.build/versions/master/skylark/performance.html#json-profile, I grabbed profiles for different strategies:

profiles.zip

[fenghaolw]

Note that on linux I also see slowness for the sandbox, but much better than mac.

It takes ~15s for linux-sandbox, and ~3s for strategy=SassCompiler=local on my Lenovo P920.

[meisterT]

cc @jmmv

[alexeagle]

discussed with @dslomov this morning, I think we should disable sandbox on mac by default. It has dubious benefits for keeping your build hermetic while causing lots of broken (non-hermetic) and slow builds

[dslomov]

ping @jmmv @philwo for opinions.

[philwo]

I think we should disable sandbox on mac by default

That's a bold strategy. Because without sandboxing, all actions will run in the same, shared execroot, which contains all files from your workspace and all output files, missing dependency declarations will no longer be detected. This can cause Bazel to assume that targets that should be rebuild are still up to date, which can result in wrong build results.

If you think that this doesn't matter for your particular use case or if you have other mitigations for this, then feel free to go ahead with the plan, but in general this is not recommended.

If your sass build steps are that slow, it can only mean that they have a huge number of input files. Is that the case?

while causing lots of broken (non-hermetic)

Please give concrete examples of broken builds due to sandboxing. If builds break due to sandboxing, it's almost always because the rules or the BUILD files don't declare all dependencies.

[Globegitter]

I have also noticed that sandboxing is generally slower on macOS compared to linux. Does anyone actually have any explanation of why that is? Is it due to some macOS reasons, or is the sandbox on macOS not as optimized?

Also @fenghaolw have you tested the same build with https://github.com/bazelbuild/sandboxfs? Apparently there is a chance it can make sandboxed builds on macOS quite a bit faster. Would be curios to see some comparisons with that.

[keith]

We tested sandboxfs with our iOS build without much luck bazelbuild/sandboxfs#76

[jmmv]

There are two problems regarding sandboxing slowness:

1. We have to create possibly thousands of symlinks per action, and that turns out to degrade badly on macOS. Microbenchmarks don't seem to expose this behavior easily though. sandboxfs is supposed to get rid of this problem but hasn't been successful for all builds yet (there is room for optimizations there that I haven't had a chance to tackle).

2. There is the theory that sandbox-exec is slow. Same as above, microbenchmarks don't seem to confirm this -- but I haven't looked into it with great detail. If this is inherently slow, then there is nothing much we can do (especially considering that this is deprecated).

There is this other idea floating around of changing the way sandboxing works: instead of preventing operations, we just log them all and then verify, after the action completes, that whatever the action accessed was allowed. This would avoid the symlinks completely and also bypass sandbox-exec. I think this is how Windows sandboxing will work. But... I don't think there is a way to do this kind of tracing on macOS out of the box that doesn't require either root or disabling SIP. It's worth investigating though.

[Globegitter]

@keith Yeah I saw that, very interesting. I have just done some quick tests on my 6 year old macbook Pro and for a tiny JS build at least (with already a fair amount of inputs due to npm modules) my build time halved from roughly 6s to 3s. Without sandbox it is 2s. That does look pretty good to me but I do wonder for what type of builds and what type of setup/configuration that differs.

Edit: Another slightly larger JS build of only 1 action went down from Elapsed time: 50.031s, Critical Path: 46.10s to 43.213s, Critical Path: 40.52s (I did multiple rounds each with similar results). So about 15% improvement for that also seems pretty good to me.