Set up `rustler_precompiled` #18

tmr08c · 2022-10-27T00:52:24Z

Closes #17

What

Adds rustler_precompiled as a dependency and creates a GitHub action
(release.yml) to attach precompiled versions of the fastrss crate as
artifacts to the release.

The value proposition of rustler_precompiled is that it can reduce the barrier
of entry for package users. By including a pre-compiled version of fastrss,
users no longer need to have the Rust toolchain installed locally to use this
package.

This change does add a bit more overhead to the release workflow. To reduce some
of this friction, this change also includes a checklist (RELEASE.md) to follow
when creating a release.

(Manually) Testing

I did not test the full Hex.pm release, but I locally tested pulling from a release on my fork.

You can see here that the action successfully built all targets.

To use the release, I made the following change to mix.exs:

-  @source_url "https://github.com/avencera/fast_rss"
-  @version "0.4.1"
+  @source_url "https://github.com/tmr08c/fast_rss"
+  @version "0.4.2-alpha-4"

and then tried a (clean) compile

mix clean
mix compile

On the first pass, they will warn you if you didn't generate the checksum file (this would be uploaded as a part of the flow documented in release.md).

› mix compile
Compiling 1 file (.ex)

19:56:00.316 [debug] Downloading NIF from https://github.com/tmr08c/fast_rss/releases/download/v0.4.2-alpha-4/libfastrss-v0.4.2-alpha-4-nif-2.16-x86_64-apple-darwin.so.tar.gz

== Compilation error in file lib/fast_rss.ex ==
** (RuntimeError) Error while downloading precompiled NIF: the precompiled NIF file does not exist in the checksum file. Please consider run: `mix rustler_precompiled.download FastRSS.Native --only-local` to generate the checksum file..

After following their suggestion, the compilation worked and includes a log, showing it fetched the precompiled asset from GitHub:

› mix compile                                                 
Compiling 1 file (.ex)

19:57:31.780 [debug] Copying NIF from cache and extracting to (...)/fast_rss/_build/dev/lib/fast_rss/priv/native/libfastrss-v0.4.2-alpha-4-nif-2.16-x86_64-apple-darwin.so
Generated fast_rss app

if I set @version to a fake version we don't have precopmiled assets for, it will fail:

21:37:28.524 [debug] Downloading NIF from https://github.com/tmr08c/fast_rss/releases/download/v0.4.2-alpha-fake/libfastrss-v0.4.2-alpha-fake-nif-2.16-x86_64-apple-darwin.so.tar.gz

== Compilation error in file lib/fast_rss.ex ==
** (RuntimeError) Error while downloading precompiled NIF: couldn't fetch NIF from https://github.com/tmr08c/fast_rss/releases/download/v0.4.2-alpha-fake/libfastrss-v0.4.2-alpha-fake-nif-2.16-x86_64-apple-darwin.so.tar.gz: {:ok, {{'HTTP/1.1', 404, 'Not Found'}, [{'cache-control', 'no-cache'}, {'date', 'Sun, 30 Oct 2022 01:37:28 GMT'}, {'server', 'GitHub.com'}, {'vary', 'X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Encoding, Accept, X-Requested-With'}, {'content-length', '9'}, {'content-type', 'text/plain; charset=utf-8'}, {'strict-transport-security', 'max-age=31536000; includeSubdomains; preload'}, {'x-frame-options', 'deny'}, {'x-content-type-options', 'nosniff'}, {'x-xss-protection', '0'}, {'referrer-policy', 'no-referrer-when-downgrade'}, {'expect-ct', 'max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"'}, {'content-security-policy', 'default-src \'none\'; base-uri \'self\'; connect-src \'self\'; form-action \'self\'; img-src \'self\' data:; script-src \'self\'; style-src \'unsafe-inline\''}, {'x-github-request-id', 'E024:8CCB:2A67147:3DB835D:635DD558'}], "Not Found"}}.

Create workflow for `release` which will handle using GitHub Actions to pre-compile `fastrss` for us. This was created using the following as examples: - https://github.com/kloeckner-i/mail_parser/blob/fe20dc953ea10f86b1ff3ad8532866a733a406ea/.github/workflows/release.yml - https://github.com/elixir-nx/explorer/blob/7a6c3bf64a31a6ff7cce9ea4f5ef220e8e70a9db/.github/workflows/release.yml

- Change macOS target condition - Add workaround for cross-compiling `x86_64-unknown-linux-musl` - Add config to make smaller builds when releasing

Follow the advice from rustler_precompiled and configure Cross to pass down `RUSTLER_NIF_VERSION`.

The repo is still using `master`, not `main`

tmr08c · 2022-10-29T23:22:40Z

.github/workflows/release.yml

@@ -0,0 +1,171 @@
+name: Build precompiled NIFs


This is mostly a copy/paste of the various examples that are already out there

https://github.com/philss/rustler_precompilation_example/blob/main/.github/workflows/release.yml

https://github.com/kloeckner-i/mail_parser/blob/main/.github/workflows/release.yml

https://github.com/elixir-nx/explorer/blob/main/.github/workflows/release.yml

The two main differences:

The project name (NIF_DIRECTORY, working-directory)

The use of master branch instead of main to match the projects default/primary branch

tmr08c · 2022-10-29T23:25:34Z

RELEASE.md

+
+1. Update [`CHANGELOG.md`](./CHANGELOG.md) to list the changes that will be included in the new version.
+2. Update the package version in [`README.md`](./README.md) and [`mix.exs`](./mix.exs).
+3. Commit the changes with a version bump commit, `git commit -m "Bump version"` .


This seems to be the commit phrase you use, so I suggested it to make copy/paste easier. We can drop or change it if there's a different workflow you'd prefer.

tmr08c · 2022-10-29T23:30:41Z

RELEASE.md

+5. Create a new release in GitHub, ensuring you are pointing to the version bump commit created above.
+6. Wait. Creating a new release will kick off the release action which uses [`rustler_precompiled`](https://hexdocs.pm/rustler_precompiled/RustlerPrecompiled.html) to precompile the `fast_rss` Rust dependency. You must wait until this process is complete before releasing to Hex.
+7. Checkout the recently create tag.
+8. Once the NIFs are built, use `mix rustler_precompiled.download FastRss.Native --all --print` to download generate the checksum file.


This project has a step in their checklist to include the checksums in their GitHub release notes:

Copy the output of the mix task and add it to the release notes

You can see the results here. I'm assuming some combination of rustler_precompiled and/or Hex will validate the checksum file automatically

The file generated will be named checksum-Elixir.RustlerPrecompilationExample.Native.exs and it's extremely important that you include this file in your Hex package (by updating the files: field in your mix.exs). Otherwise your package won't work.[^1]

I'm not sure if folks would feel better if they could see the checksums as well. explorer didn't seem to do this, so I did not suggest it. If this is a step you would like to take as a part of releasing the project, I can definitely add a note to this checklist.

tmr08c · 2022-10-29T23:31:40Z

RELEASE.md

+1. Update [`CHANGELOG.md`](./CHANGELOG.md) to list the changes that will be included in the new version.
+2. Update the package version in [`README.md`](./README.md) and [`mix.exs`](./mix.exs).
+3. Commit the changes with a version bump commit, `git commit -m "Bump version"` .
+4. Push the changes to GitHub.


Is this your preferred workflow? explorer suggests opening a PR, but I didn't see any "version bump" PRs, so I'm assuming this is not your current workflow.

tmr08c · 2022-10-29T23:33:04Z

native/fastrss/.cargo/config

+# Configuration for cross-compiling with `rustler_precompiled`
+# See https://hexdocs.pm/rustler_precompiled/precompilation_guide.html#additional-configuration-before-build
+
+[target.'cfg(target_os = "macos")']


[target.'cfg(target_os = "macos")'] seem to cover both [target.x86_64-apple-darwin] and [target.aarch64-apple-darwin]. If you are not comfortable with the change, I can use the existing two entries.

tmr08c · 2022-10-29T23:34:01Z

native/fastrss/.cargo/config

@@ -1,11 +1,23 @@
-[target.x86_64-apple-darwin]
+# Configuration for cross-compiling with `rustler_precompiled`
+# See https://hexdocs.pm/rustler_precompiled/precompilation_guide.html#additional-configuration-before-build


Essentially a copy/paste from https://hexdocs.pm/rustler_precompiled/precompilation_guide.html#additional-configuration-before-build

tmr08c · 2022-10-29T23:34:13Z

native/fastrss/Cross.toml

@@ -0,0 +1,6 @@
+# Pass relevant config when cross-compiling as a part of release


Copy/paste from https://hexdocs.pm/rustler_precompiled/precompilation_guide.html#additional-configuration-before-build

tmr08c · 2022-10-29T23:41:15Z

RELEASE.md

+7. Checkout the recently create tag.
+8. Once the NIFs are built, use `mix rustler_precompiled.download FastRss.Native --all --print` to download generate the checksum file.
+9. Run `mix hex.publish`.
+10. Bump the version in the `mix.exs` and add the `-dev` flag to it.


Adding the -dev suffix to the version will force the NIF to build locally¹. This may not be strictly necessary if you keep your local version matching the precompiled version or use the FORCE_BUILD flag, but I decided to add this suggestion because:

explorer has it

It looks like mail_parser may also find it useful to use

Footnotes

https://hexdocs.pm/rustler_precompiled/RustlerPrecompiled.html#module-options ↩

tmr08c · 2022-10-30T01:52:38Z

@praveenperera, no rush, but I wanted to ping you to let you know I have taken this out of draft.

If you still think this change could be useful, would you be open to adding a hacktoberfest-accepted label? This allows Hactoberfest to count the PR without you having to rush to review and approve¹. I am very behind, but hoping this can help towards getting a tree planted :)

https://hacktoberfest.com/participation/#pr-mr-details ↩

praveenperera · 2022-10-30T02:08:20Z

@tmr08c done! Thanks for this I’ll review this on Monday. Remind me if I don’t by the end of the week

tmr08c · 2022-11-14T14:16:54Z

@praveenperera – I am in no rush on this, but I wanted to ping you as you requested. Let me know if I can provide more annotations or explanations, or if something like a call would be helpful for you.

praveenperera · 2022-11-14T14:19:23Z

@tmr08c thanks for pining me. Looks good. Merged!

tmr08c · 2022-11-16T11:42:31Z

Thank you, @praveenperera!

I see you cut a new release, and it looks like precompiled assets were attached, so that's promising.

It doesn't look like the release has been pushed to Hex yet, so please let me know if you run into any issues with that.

praveenperera · 2022-11-17T14:14:51Z

@tmr08c thanks for the reminder, I was waiting for the release to complete before publishing, and I guess I forgot to come back and publish. Its published now. Thanks again for the PR.

tmr08c · 2022-11-22T11:57:29Z

thanks for the reminder,

My pleasure, @praveenperera; thank you for the project and for being open to trying out rustler_precompiled 💜 🦀

I was waiting for the release to complete before publishing, and I guess I forgot to come back and publish

This was definitely a negative side effect that I was worried about. Having to wait for all of the builds on release can definitely be a pain. If you find the cost is too high for the benefit, please let me know and I can revert the work. We could also reduce the number of build targets to reduce the amount of work that needs to be done.

fast_rss/.github/workflows/release.yml

Lines 32 to 40 in 367eeec

    
           - { target: arm-unknown-linux-gnueabihf , os: ubuntu-20.04 , use-cross: true } 
        
           - { target: aarch64-unknown-linux-gnu   , os: ubuntu-20.04 , use-cross: true } 
        
           - { target: aarch64-unknown-linux-musl  , os: ubuntu-20.04 , use-cross: true } 
        
           - { target: aarch64-apple-darwin        , os: macos-11      } 
        
           - { target: x86_64-apple-darwin         , os: macos-11      } 
        
           - { target: x86_64-unknown-linux-gnu    , os: ubuntu-20.04  } 
        
           - { target: x86_64-unknown-linux-musl   , os: ubuntu-20.04 , use-cross: true } 
        
           - { target: x86_64-pc-windows-gnu       , os: windows-2019  } 
        
           - { target: x86_64-pc-windows-msvc      , os: windows-2019  }

Its published now.

I upgraded the package version on a personal project and it used the precompiled binary.

==> fast_rss
Compiling 1 file (.ex)

06:48:37.388 [debug] Downloading NIF from https://github.com/avencera/fast_rss/releases/download/v0.4.2/libfastrss-v0.4.2-nif-2.16-x86_64-apple-darwin.so.tar.gz

06:48:38.243 [debug] NIF cached at (path to cache)
Generated fast_rss app

It looks like everything works!

thiagopromano · 2022-11-24T13:41:54Z

FastRSS user here!

It seems that the checksums file on hex.pm is called checksum-Elixir.FastRss.Native.exs instead of checksum-Elixir.FastRSS.Native.exs, so upgrading gives this error:

== Compilation error in file lib/fast_rss.ex ==
** (RuntimeError) Error while downloading precompiled NIF: the precompiled NIF file does not exist in the checksum file. Please consider run: `mix rustler_precompiled.download FastRSS.Native --only-local` to generate the checksum file..

You can force the project to build from scratch with:

    config :rustler_precompiled, :force_build, fast_rss: true

In order to force the build, you also need to add Rustler as a dependency in your `mix.exs`:

    {:rustler, ">= 0.0.0", optional: true}

    lib/fast_rss.ex:13: (module)
    lib/fast_rss.ex:6: (module)

It looks like everything works!

Maybe it worked on your system because your filesystem is case-insensitive. Anyway, this should be an easy fix.

BTW, congrats on the precompilation work!

praveenperera · 2022-11-24T17:35:16Z

@thiagopromano I published 0.4.3 after running the correct command. Can you try it now?

thiagopromano · 2022-11-24T19:59:01Z

Still no good, I've checked and the file is still with the wrong name:

https://preview.hex.pm/preview/fast_rss/0.4.3/show/checksum-Elixir.FastRss.Native.exs

thiagopromano · 2022-11-25T15:27:25Z

Hey @praveenperera, still no luck. I'm creating an issue just so other people find the conversation more easily.

praveenperera · 2022-11-25T15:28:53Z

I’m not sure how the precompiled assets work. I ran the command with the correct case this time. So im not sure why hex has the wrong files. Any ideas?

tmr08c added 5 commits October 26, 2022 19:33

Add rustler_precompiled

8f95e6f

Update rust build config

5e9cbca

- Change macOS target condition - Add workaround for cross-compiling `x86_64-unknown-linux-musl` - Add config to make smaller builds when releasing

Add configuration for Cross

22bedec

Follow the advice from rustler_precompiled and configure Cross to pass down `RUSTLER_NIF_VERSION`.

Add release instructions

673ad86

tmr08c force-pushed the tmr08c/setup-rustler-precompiled branch 2 times, most recently from c1e1d79 to 673ad86 Compare October 29, 2022 23:14

tmr08c added 3 commits October 29, 2022 19:20

Update branch in release

ba1e5b8

The repo is still using `master`, not `main`

Add -dev version suggestoin to RELEASE checklist

3a1b83b

Update deploy section of README

282b92c

tmr08c force-pushed the tmr08c/setup-rustler-precompiled branch from f3bde32 to 282b92c Compare October 30, 2022 01:44

tmr08c commented Oct 30, 2022

View reviewed changes

tmr08c marked this pull request as ready for review October 30, 2022 01:46

praveenperera added the hacktoberfest-accepted label Oct 30, 2022

praveenperera approved these changes Nov 14, 2022

View reviewed changes

praveenperera merged commit 5f889d5 into avencera:master Nov 14, 2022

tmr08c deleted the tmr08c/setup-rustler-precompiled branch November 16, 2022 11:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set up `rustler_precompiled` #18

Set up `rustler_precompiled` #18

tmr08c commented Oct 27, 2022 •

edited

Loading

tmr08c Oct 29, 2022

tmr08c Oct 29, 2022

tmr08c Oct 29, 2022

tmr08c Oct 29, 2022

tmr08c Oct 29, 2022

tmr08c Oct 29, 2022

tmr08c Oct 29, 2022

tmr08c Oct 29, 2022

tmr08c commented Oct 30, 2022

praveenperera commented Oct 30, 2022

tmr08c commented Nov 14, 2022

praveenperera commented Nov 14, 2022

tmr08c commented Nov 16, 2022

praveenperera commented Nov 17, 2022

tmr08c commented Nov 22, 2022

thiagopromano commented Nov 24, 2022 •

edited

Loading

praveenperera commented Nov 24, 2022

thiagopromano commented Nov 24, 2022

thiagopromano commented Nov 25, 2022

praveenperera commented Nov 25, 2022

		@@ -0,0 +1,6 @@
		# Pass relevant config when cross-compiling as a part of release

Set up rustler_precompiled #18

Set up rustler_precompiled #18

Conversation

tmr08c commented Oct 27, 2022 • edited Loading

What

(Manually) Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Footnotes

tmr08c commented Oct 30, 2022

Footnotes

praveenperera commented Oct 30, 2022

tmr08c commented Nov 14, 2022

praveenperera commented Nov 14, 2022

tmr08c commented Nov 16, 2022

praveenperera commented Nov 17, 2022

tmr08c commented Nov 22, 2022

thiagopromano commented Nov 24, 2022 • edited Loading

praveenperera commented Nov 24, 2022

thiagopromano commented Nov 24, 2022

thiagopromano commented Nov 25, 2022

praveenperera commented Nov 25, 2022

Set up `rustler_precompiled` #18

Set up `rustler_precompiled` #18

tmr08c commented Oct 27, 2022 •

edited

Loading

thiagopromano commented Nov 24, 2022 •

edited

Loading