Building a substrate node template with a global cargo `target` directory

Question

I'm trying to build a substrate-node-template with the following command:

cargo build

I have my cargo target-dir set inside a global ~/.cargo/config.toml file:

[build]
target-dir = "/home/foo/.cargo/target"

The environment variable CARGO_TARGET_DIR is unset.

Normally this configuration builds everything inside the ~/.cargo/target directory - which saves a lot of space on disk (not having to duplicate build artifacts for every project).

Note that this also works for Wasm projects, which usually end up in the .cargo/target/wasm32-unknown-unknown directory.

That said, it seems that the build.rs script uses a wasm_builder crate, which does something unusual - it creates a new Wasm project inside of the target directory (under wbuild) and builds that project separately, causing the build to fail since it does not respect the global target-dir setting and also ignores the CARGO_TARGET_DIR variable so it can't find the correct build artifacts.

The cargo build hangs forever at the node-template-runtime step, most likely due to a deadlock, i.e. the parent cargo build is using the global ~/.cargo/target/ directory and the secondary runtime build is also using the global ~/.cargo/target/ directory instead of the nested ./target/debug/wbuild/node-template-runtime as it expects:

Building [=======================> ] 839/843: node-template-runtime(build)

If I export the CARGO_TARGET_DIR=target env var the build proceeds further but then fails at the build.rs step for node-template-runtime because the build.rs script ignores the CARGO_TARGET_DIR:

Compiling node-template-runtime-wasm v1.0.0 (/home/foo/substrate/substrate-node-template/target/debug/wbuild/node-template-runtime)
Finished release [optimized] target(s) in 52.13s
  thread 'main' panicked at 'Failed to compact generated WASM binary.: Error(HeapOther("Can't read from the file: Os { code: 2, kind: NotFound, message: \"No such file or directory\" }"))', /home/foo/substrate/substrate.mod/utils/wasm-builder/src/wasm_project.rs:618:14
  stack backtrace:
     0: rust_begin_unwind
               at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:498:5
     1: core::panicking::panic_fmt
               at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/panicking.rs:116:14
     2: core::result::unwrap_failed
               at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/result.rs:1690:5
     3: core::result::Result<T,E>::expect
               at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/result.rs:975:23
     4: substrate_wasm_builder::wasm_project::compact_wasm_file
               at /home/foo/substrate/substrate.mod/utils/wasm-builder/src/wasm_project.rs:617:3
     5: substrate_wasm_builder::wasm_project::create_and_compile
               at /home/foo/substrate/substrate.mod/utils/wasm-builder/src/wasm_project.rs:130:3
     6: substrate_wasm_builder::builder::build_project
               at /home/foo/substrate/substrate.mod/utils/wasm-builder/src/builder.rs:244:30
     7: substrate_wasm_builder::builder::WasmBuilder::build
               at /home/foo/substrate/substrate.mod/utils/wasm-builder/src/builder.rs:162:3
     8: build_script_build::main
               at ./build.rs:4:2
     9: core::ops::function::FnOnce::call_once
               at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/ops/function.rs:227:5
  note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
warning: build failed, waiting for other jobs to finish...
error: build failed

The only way I can successfully build it is if I unset the global target-dir option and leave cargo to use the default ./target directory for the entire build.

It seems that most of the build problems stem from the assumptions that the wasm_builder crate makes, hence the build.rs only works if it uses the default cargo target directory location.

Based on git history issue#7532, the reason the wasm_builder ignores the CARGO_TARGET_DIR is because it used to cause a cargo deadlock, since the build.rs script would try to build inside of the same target directory as the parent cargo process - but this seems to cause additional issues when not using the default target dir path.

For comparison, this is the successful build directory structure under ./target:

target
    debug
        wbuild
            node-template-runtime
                target
                    release
                    wasm32-unknown-unknown
                        release
                            node_template_runtime.wasm
                node_template_runtime.wasm

And this is the unsuccessful build directory structure:

target
    debug
        wbuild
            node-template-runtime
                <empty>
    release
    wasm32-unknown-unknown
        release
            target
            node_template_runtime.wasm

Answer 1

What seems to be happening is that the wasm_builder builds the runtime in a nested location in the target directory:

target
    debug
        wbuild
            node-template-runtime
                src
                Cargo.toml
                Cargo.lock

Which should create a nested target directory for the runtime build:

target
    debug
        wbuild
            node-template-runtime
                target
                src
                Cargo.toml
                Cargo.lock

But since the target-dir is set in .cargo/config.toml the build script puts both target directories in the global path instead:

.cargo/target
    debug (parent)
    release (child)
    wasm32-unknown-unknown (child)

Then when the build.rs script tries to compress the wasm binary it can't find it under the nested location and fails with the "No such file or directory" error as show above.

I'm supposing that a short-term solution would be to make the wasm_builder also ignore the target-dir setting the same way it ignores the CARGO_TARGET_DIR as they seem to be related problems.