alternate backends (brew, nix, npm, cargo, etc)

[jdx]

One of the most common feature requests for a long time is to support being able to directly install tools without having an asdf plugin for it. As an example: mise use -g cargo-eza or mise use -g npm-@antfu/ni maybe even homebrew?

I'm going to try to use this discussion as a central place to manage my solution which I've started spiking on.

Supported Backends

These are in rough order of priority:

• cargo via cargo install
• npm via npm i -g
• eget (might negate the need of others like cargo binstall)
• swift (someone tell me the best solution here, @pepicrft? I think you sent me something but I forget what it was called, also is eget perhaps an alternative?)
• pip via pipx
• go via go build OR https://gobinaries.com/
• cargo via cargo binstall
• node alternatives (pnpm, yarn, bun, etc)
• pkgx (tea successor)
• webinstall.dev (seems lacking very much so perhaps not—but does have nice aesthetics I would say)
• homebrew?
• nix?

Syntax

Right now I'm going with a dash to separate the backend from the tool name. I know this isn't great
because you could have tools with dashes in them, but it's really the only delimiter we can use in TOML
without needing to add quotes. I want to avoid something like this:

[tools]
node = "latest"
"cargo:ripgrep" = "latest"

It works fine, just looks ugly.

Alternatives considered:

• mise use -g cargo:eza
• mise use -g cargo/eza
• mise use -g cargo#eza
• mise use -g cargo!eza
• mise use -g cargo|eza

Runtime/compiler selection

For v1, I'm simply going to call whichever cargo or npm is on PATH. That may be something you've defined in your .mise.toml. It may be something in your system, just depends on how it is configured. This does mean you will need to manually setup cargo/npm/whatever ahead of time.

I think for v1 that's fine and possibly long-term as well. I think later if we detect the compiler/package manager isn't installed we can prompt to install it.

The runtime selection is a harder problem I think. If I install something with node-18, then run it with node-20, will it work ok? If memory serves, you actually need to run npm install again to get it to rebuild the right files (but I think there may have been a slightly easier command). Maybe that's better in more modern node? In any case, I'm jus talking about node here but I could see this being a problem for any scripting language. As you jump between projects the CLIs need to keep working.

I hope not but I may need to put a shim in place to detect if the runtime version changed and if so run whatever command I need to patch it for the current version.

Storage

Related to the above point is storage. I think these should just be like any asdf-plugin installed thing inside of ~/.local/share/mise/installs/cargo-eza—note the "cargo-" prefix to prevent the scenario where you have 2 things installed via different tools.

Scoped packages in node may be a problem. I'm not sure how mise will handle ~/.local/share/mise/installs/npm-@antfu/ni. Perhaps we should slugify that to ~/.local/share/mise/installs/npm-@antfu-ni. I actually think users will prefer this. I don't think they'll want to have npm CLIs grouped into individual directories.

An outstanding question (my spike will address) is how exactly I install something like this. npm install --into is the non-existent (at least I think non-existent) command I think I need.

My biggest question is

Questions

• is "backends" the right name? I was also thinking "registry" maybe but leaning towards "backends". I had actually started calling them "plugins" but that turned out to be a horrible idea—cuz they're very much not.

Existing issues/discussions

• A more generic shell binary manager #953
• why we need a plugin for every static binary tool？ #686
• Install via language package managers #580

[jdx]

nice! This actually wasn't too bad: #1452

[pepicrft]

swift (someone tell me the best solution here, @pepicrft? I think you sent me something but I forget what it was called, also is eget perhaps an alternative?)

Swift Package Manager doesn't have an API that users can leverage to add tasks to their projects. However, it supports adding plugin dependencies, for example this one to generate documentation, which extend swift package with additional commands. For example swift package generate-documentation. We should detect which packages are command plugins, and hook them up.

[pepicrft]

Oh, I see what you mean. Surprisingly, there's nothing official. Hence why, the community built its own tool.

[kattouf]

Mint would be great to see here :dreaming:

[kattouf]

@jdx I created a draft-pr with Mint backend and will be glad to receive comments or instructions about next steps.

[pepicrft]

🎉 this is amazing @kattouf 🎉 . Thanks a lot for going ahead with this.

[kattouf]

Hey guys, based on our last conversation in Mint backend pr, I'm back with a new "3rd party free" backend implementation.
#2241

[GeertJohan]

I would strongly prefer to use a more clear prefix than the dash. e.g. :. Even though the TOML becomes a bit ugglier, this indicates to the user that a different installation process is used. Using the prefix with a dash will feel like magic to the user and may lead to a lot of questions.
From a maintenance point of view, when scanning the .rtx.toml or rtx list output, it's also useful to quickly identify packages that use a plugin vs via package manager.

When considering Go, I'd prefer native go build/go install over gobinaries.com, although ofcourse both could be implemented (go:foo@latest vs gobinaries:foo@latest).

Lastly, very go-specific as I'm considering implementing that, Go uses import paths a lot. Where with cargo and npm developers have to 'reserve' their name, with Go it's possible to use any domain name to create an 'import path'. For example go install github.com/GeertJohan/gomatrix@latest downloads/compiles and installs a binary named gomatrix.

How would we translate this to rtx? A couple of options:

• go:github.com/GeertJohan/gomatrix@latest
• go:github-com-GeertJohan-gomatrix@latest
• go:github--com--GeertJohan--gomatrix@latest

I prefer the first one because it differs the least from original go usage. Need to look into implementation though. TOML will need quoting, but already needs it IF using the : as separator for the backend-prefix instead of -. The last one, where periods and slashes are replaced with a double dash (--) is cleaner when the original import path already has dashes in it. @jdx Any thoughts on this?

[jdx]

so I think colon is a bad choice. If I do that, I won't be able to use a colon in the directories so I'd have to use another separator.

[jdx]

that's not a dealbreaker, but I think it's going to make things a lot harder for both me and the users. I already have this problem with ref: which gets translated to ref-

[jdx]

/ makes for an interesting delimiter. It has a side effect that's actually maybe a good thing—it groups everything together, e.g.:

mise i npm/@antfu/ni
~/.local/share/mise/installs/npm/@antfu/ni/0.0.0/bin/ni -v

I don't know if I love it though, I will admit the CLI syntax of ":" does look better. I just don't like that there are different delimiters in different places.

I also wonder if I should even have the subdirectories. It seems to work fine, but maybe it would be better if I stripped all the subdirectories out and just had a flat directory structure.

[GeertJohan]

I think each backend should have it's own directory, and each backend could translate the package name to a directory name. This makes it the most flexible and the core shouldn't have to worry about it. I think it's fine if directory names and package names are not the same, this is 'under the hood', the user rarely sees it (shouldn't have to care) and the backend implementation can have freedom to do the best thing for it's own situation.

I would then also still stick with the colon, the name before the colon is the backend directory (created by the core rtx code) and then the part after the colon is passed to the backend implementation to translate into the directory.

The default 'asdf-plugin' way of working could also be one of these backends by the way, it just has a different install root directory, and it's default (no asdf-plugin: prefix required, without a backend prefix the asdf-plugin backend is assumed).

[GeertJohan]

e.g.:

~/.local/share/mise/installs/asdf-plugin/*       # All existing installs are moved here
~/.local/share/mise/installs/npm/*
~/.local/share/mise/installs/go/*

[RingOfStorms]

I am most looking forward to eget style behavior from mise with the ability to install any binary from releases in git.

Thus solving the pain point of writing asdf plugins which boil down to string manipulating the uname command into whatever random naming scheme some project uses for their binaries.

So maybe some process like
mise git-repo install https://github.com/jdx/mise mise (this could register the tool name mise to use the git-repo releases instead of an asdf plugin)
so that whenever you run mise install mise@[tag] it will look for that git Tag and get the binary and use that.

mise git-repo install https://github.com/zyedidia/eget eget
mise global eget@latest or mise local eget@v1.3.3 where v1.3.3 is a git Tag

Or maybe allow for
mise install repo_url@tag and it doesn't care the name of the tool itself.

Note that another challenge may be other files inside of release archives. For example, I want to use mise to install slack's Nebula https://github.com/slackhq/nebula/releases
They release an archive that contains two binaries (which I want both to be installed).

Other repos ship with other files that aren't just the binary files. Idk the best way to solve this some tools similar to eget just look for the largest file in the archive, etc.

So yea to two main issues are:

• Automatically detect current platform (this seems pretty doable based on eget/dra/tool-sync)
• Automatically detect what is actually installable (maybe this is not possible and can be specified for each repo somehow?)

[jdx]

I think for now it wouldn't be a good use of my time to try to map repos to binaries when so many alternatives are already in this space. I think the better solution is just to be a front-end to those other tools:

$ mise i egrep:user/repo
$ mise i dra:user/repo
$ mise i pkgx:user/repo

[RingOfStorms]

I feel like where mise could improve on all of those is being more declarative in the config file or command syntax, to get around those above issues. I was looking at the source code of some of the rust ones above and they fell back to really janky hacks like "Assume the largest file in an archive is the binary and use that one"

Those approaches don't work well and I don't feel like mise will be that great if it just proxies that bad behavior. You'd have to know the ins and out of every single one of these backends which kind of defeats the purpose in my mind. I want to use mise because I don't want to learn all these other backends or do what is required like write asdf-plugins (though maybe that is the sweet spot since that does solve these issues)

[RingOfStorms]

I guess something similar to the asdf plugin format but with tools built in and available that make this process easier would be great. The fact that everyone who writes an asdf plugin is grossly copying some uname -XYZ commands to determine some string of a binary is really annoying.

If there was just a simplified config for github releases native to mise that was easy to configure and declare that would be great. Where you specify generally what binaries exist in the release archives of a git repo. And some code in mise could do the platform specific stuff.

[jdx]

sorry I just don't have the appetite to take on this problem and I really think I've broadened the scope of mise plenty enough as it is. If you want a better tool for installing binaries from repos, consider creating one yourself and we could see if it might make sense to adopt it in mise either directly or via a plugin.

[zph]

I'm very interested to have a unified approach to installing binaries from Github Releases and I see that there's uncertain appetite for it being in mise directly. I currently use a combination of homebrew and hermit to handle this, but I'm not fully happy with my approach.

I took a few minutes tonight to extend mise to support it via your example @jdx of how npm backend was enabled... and I cheated for the prototype by shelling out to ubi (https://github.com/houseabsolute/ubi) which handles the release lookup and architecture matching but could be re-implemented or used as a library for a more durable solution.

My prototype implementation was simple to implement thanks to mise architecture.

I'll test drive my own implementation for a bit (https://github.com/zph/mise/tree/zh-add-ubi-forge) and if it holds up well I can polish it up to contribute back.

[jdx]

Not sure when I'll be able to get around to doing more of these, if someone wanted to help and had a particular backend they want, they're actually really easy to write though since I have refactored the codebase to allow for backends, here is npm: 8d678e0

I think the next problem might be how to deal with urls/repo names. I think it'll work out of the box but that might require some (probably minor) modifications.

Also, within the codebase, I've settled on calling these "forges"—the idea being they stamp out instances of "tools". A "forge" is something like asdf:node or npm:prettier and a "tool" is that w/ a version: asdf:node@20 and npm:prettier@20. This naming convention has actually been a problem since the beginning.

This was important in the codebase where "plugin" has been an overloaded term. These should not be thought of as plugins. In the future, a plugin may provide one or many forges or none at all in the case of a plugin for tasks and environments.

I have not made this change in anything user-facing (perhaps in some error messages) but that probably should be done at some point—though when that happens the change should be made everywhere at the same time for consistency. Part of the reason is I'm still a little undecided if "forge" should be the final name—and you all know how I can be about naming things :). It will make explaining mise a bit harder since right now there is a 1:1 mapping between plugins and forges but that will need to change.

[RingOfStorms]

Edit: Totally just assumed this is how it worked but jdx implemented it how I thought it should be already my bad for not trying it first.

Also for the npm i -g this bypasses one of my biggest issues with things installed with npm. Which is when I use mise in projects that define different node versions the global install is only for that specific version of node and I can lose a bunch of my installed tools just by moving around my directories in the command line unless I install them on every single node version managed by mise

It would be great for mise to be the tool that makes behavior consistent and pins things in the path when installed via mise. So in other words if I installed a npm based tool using mise I would expect it to ensure it is always on the path, regardless of which version of node is on my path at that time.

This is your tool so it is up to you, but I feel liek having mise just call commands of other tools direcly like this with no extra handling isn't super useful, and may even may mise more confusing to use. As a user who may only know 1-2 of the 20 possible backend tools I would have no idea what command I should run, esp when many may actually work but have large tradeoffs.

If I use mise install npm:cspell:latest and then ce into my typescript repo (with a different node version than where I ran the previous cspell install) and suddenly don't have cspell available on my path I'd be pretty confused. Then maybe I'd try to install it with something else mise install eget:cspell and now there are two installed in different places and one may be used instead of the other at weird times.

[jdx]

So in other words if I installed a npm based tool using mise I would expect it to ensure it is always on the path, regardless of which version of node is on my path at that time.

that is how it works though

[RingOfStorms]

oh sweet, my bad I haven't actually tried it yet 😅 I will refrain until I actually use the new features here

[jdx]

it might have problems though. I talked about this above in the description. How well a CLI installed with node-18 runs under node-20 I can't say.

[mbhutton]

For v1, I'm simply going to call whichever cargo or npm is on PATH. That may be something you've defined in your .mise.toml. It may be something in your system, just depends on how it is configured. This does mean you will need to manually setup cargo/npm/whatever ahead of time.

When bootstrapping a new system and running mise install, would the language level plugins (e.g. rust or python) always be installed and activated before doing operations with associated package manager (e.g. cargo or pipx)?

As a concrete example, if I have this in global or project settings and run mise install, would the python version be installed before the pip package?

[tools]
python = "3.11.0"
pipx=PyGitUp = "latest"

If homebrew/nix/apt are also in the mix, perhaps they should also precede backends like pipx/cargo etc, as that's how some users install pipx.

[jdx]

not currently, mise installs things in parallel so you'll need to explicitly run mise install python first. This could be fixed but it's not a trivial change.

[endigma]

Go is done as of #1470

[endigma]

not sure where $root is meant to be coming from

[endigma]

also, is there a way to run less than all of the e2e tests?

[jdx]

you should just be able to pass an argument to it

$root gets set here:

mise/src/cli/run.rs

Line 166 in 3412fa1

env.insert("root".into(), root.display().to_string());

project_root must be null for you, though I don't know why that would be if you're running from the mise root...

[jdx]

oh I bet your mise is just out of date

[endigma]

out of date was the issue

[endigma]

Installing in a special path with a special cache, pnpm, yarn, bun/etc are likely useless, the benefit of using those would be sharing global node modules between them, and in pnpm's case having the global symlink-store thing. They might be marginally faster on raw DL than npm though.

[jdx]

yeah I don't know who would need it, could maybe just be a setting for the npm backend

[endigma]

As far as I can tell, nix, pnpm and eget are DOA, no way to get them into the right paths or in eget's case no way to list versions, although listing GitHub releases through the API might solve that. #1583 fixes the go forge to work properly.

[andrewthauer]

I think ruby gems would be a good one to add to this list as well. I think gem should have all the requirements to work properly with forge.

[andrewthauer]

I was wondering how forge determines the runtime to use for installed packages. Looking at the npm backend, it seems the node_modules are vendored in the install location, but runs against whatever node version mise currently resolves to. I suspect this in general could result in unexpected and inconsistent behaviour. Unless I'm missing something in how it works?

[endigma]

It's probably possible to control which node version is used to install npm packages, but it's going to be very hard to set it up to use a specific one for a specific tool, it would require re-shimming as npm install-ed "executables" are just shims that call node anyway.

[jdx]

this is a known issue, I mentioned it in the initial description under "Runtime/compiler selection". Nobody has raised it yet but I'm sure it will be something that needs to be addressed and may require unique behavior in different languages.

[andrewthauer]

Looking at how homebrew handles this, it essentially creates a shim file with a special shebang pointing to the desired version of ruby, etc. to be executed. Not sure if that makes sense in mise or not though.

[andrewthauer]

I added a draft PR adding a gem backend here #1657.

[eproxus]

A backend for Elixir's Mix.Installmix escript.install. would be nice.

[pepicrft]

@eproxus is mix install able to install Elixir CLIs globally? From what I understood there, Mix.install is a runtime API to run inside iex.

[eproxus]

My bad, I was thinking of mix escript.install.

[BuildBackBuehler]

I'm glad to see rtx going in the right direction, hot damn there's a lot going on since I was last in the loop! Homebrew is going in the other direction with the python changes...sigh (and here I am as a result)

1. I like the move away from plugins for the big-hitters. Plug-ins should be reserved for auxiliary accoutrements to those fellas.

Backend is...ehhh.

Framework is easily #1 to me.

Some others that come to mind:
System, Base, Nexus, Design Entity (or Dgn. Sys.)

2. As for the meat & gravy, it would be nice to get Homebrew integrated into the rtx umbrella because the reason I'd gone back to Homebrew exclusively was because I could only use HB's basic dependencies if whatever "backend" was installed through HB. This was a problem because I couldn't use cmake/llvm/whatever to build some basic things I wanted to integrate into Ruby and/or Cargo.

That is, if you/team doesn't want to slide those kinda dependencies into the chaos 😂 (^ above was definitely a Mac-only problem...IIRC if I could hardlink, I could've worked around it. I'm also not proficient let alone an expert in code world.)

3. Hope that Mise focuses on...

Cargo, Web PM (anything besides npm, but pref. a flat-packer, whatever is next-gen pnpm, Deno? Bun? IDK), Python/Pip(x) should be #1 priorities for sure!

4. As for the great syntax debate -- I do believe that the dash / is too prevalent to be used for yet another delimiter. I love the whatever the duck | is called because it is probably the most convenient key out of all the delimiter characters at our disposal. But what about >? Makes sense with it being "This is Cargo's eza." I'm sure there's a reason they weren't mentioned I'm not privy to lol.

Tanx for the amazing work you do JDX!