Directory artifact not present in sandbox using Bazel 6.0.0rc2

[gregmagolan]

Description of the bug:

Using Bazel 6.0.0rc1 or rc2 and with --remote_download_outputs=toplevel (or minimal), remote_cache (or disk_cache) & sandboxed execution, directory artifacts are not present in runfiles in the sandbox.

Conditions for issue

• Bazel 6.0.0rc1 or rc2
• directory artifact (created with ctx.actions.declare_directory) in runfiles of a test or binary target
• sandboxed execution
• --remote_cache (or --disk_cache enabled)
• --remote_download_outputs=toplevel (or minimal)

Issue

The output directory is not present in the sandbox when the test or binary target is run.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Clone https://github.com/gregmagolan/bazel_6_remote_download_outputs_repro

Run ./repro.sh to run a test that fails if the directory artifact is not present in runfiles. Removing the --remote_download_outputs flag -or- running with --spawn_strategy=local makes the test pass.

Which operating system are you running Bazel on?

MacOS (but the issue has also been observed on Linux CI)

What is the output of bazel info release?

release 6.0.0rc2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

git@github.com:gregmagolan/bazel_6_remote_download_outputs_repro.git
master
fatal: ambiguous argument 'master': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
ac7517ea0ef1dad51847b40899c25079b427fb6e

Have you found anything relevant by searching the web?

No

Any other information, logs, or outputs that you want to share?

First observed when updating rules_js to Bazel 6.0.0rc1. rules_js makes heavy use of directory artifacts for the node_modules virtual store

[gregmagolan]

cc @fmeum

[gregmagolan]

cc @alexeagle

[fmeum]

@bazel-io flag

[meteorcloudy]

@bazel-io fork 6.0.0

[meteorcloudy]

Does this issue also happen with Bazel@HEAD? I wonder what change we need to make/cherry-pick to fix this.

[gregmagolan]

I tried last_green in .bazelversion and that still fails:

$ bazel version
2022/11/18 08:03:14 Using unreleased version at commit 44918c540250a9e87eba8088e5676e3346695f29
Bazelisk version: development
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Jan 01 00:00:00 1970 (0)
Build timestamp: Thu Jan 01 00:00:00 1970 (0)
Build timestamp as int: 0

I'll try a local build @Head.

[gregmagolan]

@meteorcloudy I tested with a local built at HEAD:

commit 8349c95fc98beb4008085942a67a57f0c4da074b (HEAD -> master, origin/master, origin/HEAD)
Author: Fabian Meumertzheim <fabian@meumertzhe.im>
Date:   Fri Nov 18 08:00:19 2022 -0800

    Also collect clang resource directory with `-no-canonical-prefixes`
    
    `clang -print-resource-dir` without `-no-canonical-prefixes` returns a different path than is actually used to include `asan_blacklist.txt` on macOS with non-Apple clang.
    
    Closes #16730.
    
    PiperOrigin-RevId: 489475662
    Change-Id: If17f347d76f86e0ec5804f9e8789f44f46ab27b4

and the repro is still failing

[meteorcloudy]

/cc @coeuvre Can you look into this issue?

[coeuvre]

bisect points me to dfccbf9. Working on the fix.

[gregmagolan]

Confirmed that this change fixes the problem observed on rules_js with Bazel 6.0.0rc1 & rc2. aspect-build/rules_js#659 updates rules_js to last_green and CI is green with --remote_download_toplevel set.

Thanks for the quick fix @coeuvre !

[coeuvre]

Thanks for confirming!