Big picture of the Zig caching system.

zig_cache.md

This is outdated now. Compilation does not use the tmp directory anymore.

** Big picture of the Zig caching system. ** This is a big picture of the Zig caching system, mostly motivated for me to understand the necessary bits to solve ziglang/zig#11643. It reflects the status quo of commit fcfeafe99a3ecc694a3475735c81a0d75b6da6d0. The core logic is defined in src/Cache.zig.

zig-cache/h contains the manifest files with varying content. Manifest files hold relevant data from the compilation to decide, if the files at the used paths have changed (ie from editing or git). Typical values for libcs are build_options, zig_lib_dir, target.cpu.arch, target.abi, target_version.

zig-cache/tmp contains temporary files. For example, object, lib and executable files are first added to tmp before being moved to o and added to the caching system. zig-cache/o thus contains objects. zig-cache/z contains cached Zir code.

Generation of manifest file hashes in src/Cache.zig, ie in src/glibc.zig:

• obtain() gets empty hasher
• cache.hash.addBytes() and cache.hash.add() are used to combine hash things
• cache.hit() checks, if the hash value exists
• cache.final() gets the hash
• must validate with actual_hit to access via directory handle and ok file idea: ok file shows empty cache folder + correct cache dir

** How the bits are connected **

1. How is the global_cache_dir be picked up?

• in src/main.zig: global_cache_directory with resolveGlobalCacheDir()
• or an user-provided override

2. How is the local_cache_dir be picked up? in src/main.zig: local_cache_directory

• in run mode global_cache_directory
• if main_pkg then zig-cache from root_src_directory (This will change to the directory of the first found build.zig file during traversal to the root directory pending ziglang/zig#11743)
• or an user-provided override

3. Why are things are moved from tmp/ -> o/? Before having the accurate bytes, we cant compute the hash. Not doing so requires to keep all of them in memory (src/Compilation.zig). As one example, in src/main.zig: zig_cache_tmp_dir is used to compute source file that can import other files (C macros etc).
4. Where is the cached Air stored? Air is not cached on disk and stored in form of generated machine code in the binary (or in-memory in case of Wasm). Declaration updates (globals or functions) lead to regeneration of Air.

** Interesting bits for your usage. ** If you build Zig from source, you will see a directory zig-cache/options in the root directory of the compiler with hashes as filenames and content like

...
pub const version: [:0]const u8 = "0.10.0-dev.2460+cbb806da6";
pub const semver: @import("std").SemanticVersion = .{
    .major = 0,
    .minor = 10,
    .patch = 0,
    .pre = "dev.2460",
    .build = "cbb806da6",
};
...
pub const enable_logging: bool = true;

Those are generated as OptionsStep defined in lib/std/build/OptionsStep and used in build.zig with

    const exe_options = b.addOptions();
    exe.addOptions("build_options", exe_options);

and

    const test_cases_options = b.addOptions();
    test_cases.addOptions("build_options", test_cases_options);

TODO explain properly instead of this stuff and describe user-set and build.zig specific and cached compilation options.

tldr; You can use Zig code to provide the user with options to configure build options and everything is stored hashed in the used cache directory (local, global or user-overridden).