Build extra components specified in manifest metadata

[N3xed]

The first bulk of this PR is refactoring the build configuration. This is done so that the configuration that before could only be specified with environment variables can now also be set in the package.metadata.esp-idf-sys table of the root crate's manifest.
Since we'll be using cargo metadata for specifying extra components anyway, it only makes sense that all configuration can be put into the Cargo.toml and not only configuration regarding extra esp-idf components.

The refactored build configuration uses serde, its derive macros, and a new dependency envy to deserialize environment variables and metadata values from a manifest. This has the advantage that all configuration options are now specified in a single struct and not littered throughout the build script. Additionally extending the configuration is much easier.

The second part is extending this configuration system to allow specifying extra, external esp-idf components (though solely in manifest metadata).
This is possible by adding an object to the package.metadata.esp-idf-sys.extra_components array of the manifest. For example:

[[package.metadata.esp-idf-sys.extra_components]]
component_dirs = ["esp-rainmaker/components/esp-insights/components", "esp-rainmaker/components"]
bindings_header = "src/bindings.h"

Fixes #32, fixes #72

[N3xed]

Okay, this should be pretty close to the finish line now. The only thing missing is more documentation (i.e. README, maybe book) and an embuild release (that's why CI is failing).

Generally, this PR does two things.

First, refactor the configuration to use serde and the envy crate, and allow all build configurations to be specified in the Cargo.toml. This is possible by setting the specific option (lowercased) in the package.metadata.esp-idf-sys object of the root crate's Cargo.toml. The build script gets the metadata by running cargo metadata --frozen --offline in the workspace directory.
If cargo metadata does not give a root crate, it can be specified with the ESP_IDF_SYS_ROOT_CRATE environment variable.

Second, it allows to specify extra esp-idf components (even multiple) in the package.metadata.esp-idf-sys.extra_components array of the root crate's and all direct dependencies' Cargo.toml. Such an extra component config can be specified like this:

[[package.metadata.esp-idf-sys.extra_components]]
# A single path or a list of paths to a component directory, or directory containing components.
#
# Each path can be absolute or relative. Relative paths will be relative to the
# folder containing the defining `Cargo.toml`.
component_dirs = ["dir1", "dirs2"] # or
component_dirs = "one_dir"

# The path to the C header to generate the bindings with. If this option is absent,
# **no** bindings will be generated.
#
# The path can be absolute or relative. A relative path will be relative to the
# folder containing the defining `Cargo.toml`.
#
# Optional
bindings_header = "bindings.h"

# If this option is present, the component bindings will be generated separately from
# the `esp-idf` bindings and put into their own module inside the `esp-idf-sys` crate.
# Otherwise, if absent, the component bindings will be added to the existing
# `esp-idf` bindings (which are available in the crate root).
#
# To put the bindings into its own module, a separate bindgen instance will generate
# the bindings. Note that this will result in duplicate `esp-idf` bindings if the
# same `esp-idf` headers that were already processed for the `esp-idf` bindings are
# included by the component(s).
#
# Optional
bindings_module = "name"

Note, the two [ ([[ and ]]). The above would be equivalent to:

[package.metadata.esp-idf-sys]
extra_components = [
    { component_dirs = [ "dir1", "dirs2" ], bindings_header = "bindings.h", bindings_module = "name" }
]

Reviews are welcome.

[MabezDev]

Thanks for tackling this @N3xed! Attempting to try it out now, running into a few build errors though:

error[E0432]: unresolved import `embuild::bindgen::BindgenExt`
  --> /home/mabez/.cargo/git/checkouts/esp-idf-sys-d4c2e811b42d99d6/db3254a/build/build.rs:11:5
   |
11 | use embuild::bindgen::BindgenExt;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ no `BindgenExt` in `bindgen`

error[E0425]: cannot find function `default_bindings_file` in module `bindgen_utils`
   --> /home/mabez/.cargo/git/checkouts/esp-idf-sys-d4c2e811b42d99d6/db3254a/build/build.rs:126:40
    |
126 |     let bindings_file = bindgen_utils::default_bindings_file()?;
    |                                        ^^^^^^^^^^^^^^^^^^^^^ not found in `bindgen_utils`

error[E0425]: cannot find function `cargo_fmt_file` in module `bindgen_utils`
   --> /home/mabez/.cargo/git/checkouts/esp-idf-sys-d4c2e811b42d99d6/db3254a/build/build.rs:178:20
    |
178 |     bindgen_utils::cargo_fmt_file(&bindings_file);
    |                    ^^^^^^^^^^^^^^ not found in `bindgen_utils`

error[E0425]: cannot find value `ENV_PATH_VAR` in module `embuild::build`
   --> /home/mabez/.cargo/git/checkouts/esp-idf-sys-d4c2e811b42d99d6/db3254a/build/build.rs:197:45
    |
197 |         cargo::set_metadata(embuild::build::ENV_PATH_VAR, env_path);
    |                                             ^^^^^^^^^^^^ not found in `embuild::build`

error[E0425]: cannot find value `ESP_IDF_PATH_VAR` in module `embuild::build`
   --> /home/mabez/.cargo/git/checkouts/esp-idf-sys-d4c2e811b42d99d6/db3254a/build/build.rs:202:25
    |
202 |         embuild::build::ESP_IDF_PATH_VAR,
    |                         ^^^^^^^^^^^^^^^^ not found in `embuild::build`

error[E0599]: no method named `headers` found for struct `bindgen::Builder` in the current scope
   --> /home/mabez/.cargo/git/checkouts/esp-idf-sys-d4c2e811b42d99d6/db3254a/build/build.rs:147:10
    |
147 |         .headers(headers)?
    |          ^^^^^^^ help: there is an associated function with a similar name: `header`

error[E0599]: no method named `headers` found for struct `bindgen::Builder` in the current scope
   --> /home/mabez/.cargo/git/checkouts/esp-idf-sys-d4c2e811b42d99d6/db3254a/build/build.rs:161:18
    |
161 |                 .headers(headers.into_iter().inspect(|h| cargo::track_file(h)))?
    |                  ^^^^^^^ help: there is an associated function with a similar name: `header`

Some errors have detailed explanations: E0425, E0432, E0599.
For more information about an error, try `rustc --explain E0425`.
error: could not compile `esp-idf-sys` due to 7 previous errors
warning: build failed, waiting for other jobs to finish...

[N3xed]

@MabezDev
Yup, you need embuild master right now to build (I should've made that clearer). That's also why CI is failing.

I didn't want to make a new embuild release, since I might be adding more stuff for this feature.

[MabezDev]

Thanks for the hint about embuild. So I've been playing with this and I can't seem to get it working, sorry for the hand holding I'll need here in advance, but can you see where I'm going wrong here: https://github.com/MabezDev/esp-rainmaker ?

[MabezDev]

Ugh ignore me, I missed an end quote on an include in binding.h 🤦. Seems to be working very well now! Will review a bit more thoroughly :)

[MabezDev]

This is working brilliantly, thanks for working on this! LGTM!

[czAdamV]

I got this warning when I played around with this branch:

warning: `cargo metadata` exited with an error: error: failed to download `atomic-polyfill v0.1.8`
warning: 
warning: Caused by:
warning:   attempting to make an HTTP request, but --frozen was specified

I tried to add atomic-polyfill to the list of dependencies and got the same warning about bare-metal and then some more crates after that when I added that one. The warnings stopped when I deleted the --offline and --frozen flags from the cargo metadata call, and weirdly enough, they didn't appear again even when I put the flags back. I don't know much about the cargo build process, and I have no idea how to debug this issue (or even reproduce it again), but I figured it might be helpful to at least report it here 🙂

[N3xed]

I got this warning when I played around with this branch:

warning: `cargo metadata` exited with an error: error: failed to download `atomic-polyfill v0.1.8`
warning: 
warning: Caused by:
warning:   attempting to make an HTTP request, but --frozen was specified

I've also seen that in the CI output, and it seems to me this is because of rust-lang/cargo#10801 (related rust-lang/cargo#10352). My assumption was that when running cargo metadata in a build script to get the metadata of the same build, it should never require changes to the Cargo.lock or downloading crate metadata (since at the start of the build this should've already been done).

So, I'll remove these flags for now.

[arashsm79]

Thanks for this!
After this was merged, I tried to use littlefs for my project and it worked flawlessly!

Here's the rundown for anyone who's interested:
I cloned the littlefs port for esp into a directory in my project (here it's called components).
Then added the following to my Cargo.toml

[[package.metadata.esp-idf-sys.extra_components]]
component_dirs = ["components"]
bindings_header = "src/bindings.h"

Contents of src/bindings.h:

#include "esp_littlefs.h"

Here's the partition table used when flashing:

# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you have increased the bootloader size, make sure to update the offsets to avoid overlap
nvs,      data, nvs,     ,        0x6000,
phy_init, data, phy,     ,        0x1000,
factory,  app,  factory, ,        2M,
littlefs, data, spiffs,  ,        0x1d0000,

After building the project and generating the bindings, I simply mounted the partition and registered the littlefs filesystem.

    let mut fs_conf = esp_idf_sys::esp_vfs_littlefs_conf_t {
        base_path: cstr!("/littlefs").as_ptr(),
        partition_label: cstr!("littlefs").as_ptr(),
        ..Default::default()
    };
    fs_conf.set_format_if_mount_failed(true as u8);
    fs_conf.set_dont_mount(false as u8);

    unsafe {esp!(esp_idf_sys::esp_vfs_littlefs_register(&fs_conf))?};
    let (mut fs_total_bytes, mut fs_used_bytes) = (0, 0);
    unsafe {esp!(esp_idf_sys::esp_littlefs_info(fs_conf.partition_label, &mut fs_total_bytes, &mut fs_used_bytes))?};
    info!("LittleFs Info: total bytes = {}, used bytes = {}.", fs_total_bytes, fs_used_bytes);

Finally, I could use the standard library file operations to read from and write to the filesystem.

    {
        let mut file = fs::OpenOptions::new().write(true).create(true).open("/littlefs/log")?;
        file.write_all(b"Hello from the Rustland!\n")?;
    }
    {
        let mut file = fs::OpenOptions::new().read(true).write(true).open("/littlefs/log")?; 
        let mut contents = String::new();
        file.read_to_string(&mut contents)?;
        println!("{}", contents);
    }