wasi-parallel: implement CPU parallelism

[abrown]

This change implements wasi-parallel in Wasmtime. It addresses only the CPU parallelism (not GPU), since this would introduce additional complexity to an already difficult review (if you're interested about the GPU progress, talk to @egalli). Each commit implements a separate piece to the puzzle. Though most of the changed lines are not the core implementation (see tests, LICENSE, etc.), due to the size of the PR, it may be convenient to look at this commit by commit.

I see several issues highlighted by this change:

• lacking toolchain support: in the absence of any other feasible path, the WebAssembly bytes of the kernel code to be executed in parallel are embedded in the host WebAssembly module itself. This is problematic for many reasons, not least of which is the ability for the host WebAssembly module to tamper with the kernel (on the flip side: an intra-module JIT compiler feature!). But that is not the end of it: toolchain support is also lacking simply to produce modules that use atomics and shared memory. For C programs: How to distribute THREAD_MODEL=posix builds? WebAssembly/wasi-libc#326. For Rust programs: Fail to compile with WebAssembly atomics and shared memory rust-lang/rust#102157. Due to all this, most tests and examples in this crate are meticulously hand-crafted WAT files — once toolchain support improves these difficult-to-maintain files should be replaced.
• uncomfortable Wasmtime/Wiggle APIs: this implementation of wasi-parallel works by creating new instances of the kernel in each thread of a thread pool — the host module exports a shared memory and the kernel imports it. Getting access to the shared memory from within a wasi-parallel invocation is difficult with Wasmtime/Wiggle: the best way I could think to resolve this is to teach Wiggle to skip a WITX function and then manually implement add_to_linker for that function. This is tedious and error-prone, so I would be excited to discuss better solutions. Note that an almost identical version of this problem will arise when I try to implement wasi-threads.

Why now? The rationale behind merging this code behind the wasi-parallel feature flag is to enable users to try this out locally and to iterate on this in-tree (e.g., toolchain, GPU parts).

[penzn]

lacking toolchain support: in the absence of any other feasible path, the WebAssembly bytes of the kernel code to be executed in parallel are embedded in the host WebAssembly module itself. This is problematic for many reasons, not least of which is the ability for the host WebAssembly module to tamper with the kernel (on the flip side: an intra-module JIT compiler feature!). But that is not the end of it: toolchain support is also lacking simply to produce modules that use atomics and shared memory. For C programs: How to distribute THREAD_MODEL=posix builds? WebAssembly/wasi-libc#326. For Rust programs: Fail to compile with WebAssembly atomics and shared memory rust-lang/rust#102157. Due to all this, most tests and examples in this crate are meticulously hand-crafted WAT files — once toolchain support improves these difficult-to-maintain files should be replaced.

I think it is important to note that this PR does not depend on toolchain - implementing an API in consumer should not require a particular producer to be on board; multiple memories are a a different example of this. Also keep in mind that libc support is already implemented, WebAssembly/wasi-libc#326 is strictly about packaging and detecting it and we should be able to fix that before this PR gets merged.

[alexcrichton]

I have two minor comments below, but otherwise I am not certain that we'll want to land this in Wasmtime at this time. My main rationale for this is that wiggle is more-or-less "on its way out" with our transition to the component model and wit-bindgen. In the component model the hack you've implemented here with getting the shared memory to work will not work and has no way of working, there's no escape hatch to implement as you've done here.

At this time there's no concrete timeline of when wit-bindgen will become "official" or when we'll switch over to it. Additionally the existing wasi-common, wasi-nn, and wasi-crypto will likely all need to get removed or refactored or something with a transition away from wiggle. Despite all that though all these prior interfaces have what's at least a path forward to move into the component model rather than the wiggle-level of abstraction it has today. That's in contrast with this WASI proposal which I don't think has any path forward into the component model.

I'd be curious to see how others feel about this though. I am not sure how useful it's been to have wasi-nn and wasi-crypto in this repository, I am not aware of how much usage they're getting and feedback they're providing to the upstream proposals. In that sense I'm not sure how beneficial it will be to land this here vs have it externally.

Overall I personally feel that the multithreading story has a lot of baking to still do and while I'd like to see it get its baking time I'm not sure if it's this repository is the right place for that right now at this time.

[alexcrichton]

Mind bumping this to 2021 with edition.workspace = true?

[alexcrichton]

Personally I'm wary of continuing to take on large unvetted dependencies, even through the optional wasi proposals. The previous wasi proposals we have implemented were largely grandfathered in but I would personally prefer to take this opportunity to not continue to try to grandfather in new features.

Is it possible to slim down the proposal's dependency tree to not rqeuire so many exemptions? Or otherwise are these sorts of crates absolutely required? If so I would personally prefer to have an official vet on-the-record for the inclusion here.

[penzn]

That's in contrast with this WASI proposal which I don't think has any path forward into the component model.

Probably worth re-iterating the point @abrown made above, that this would block potential implementation of wasi-threads as well.

[alexcrichton]

That's true, yes, and I feel pretty similarly to the wasi-threads proposal as what's here as well. I also don't see a future for the wasi-threads proposal as-is in the component model right now. If these proposals want to integrate with the component model I think work needs to happen in the component model itself rather than assuming the functionality can be implemented with special host functions.

[penzn]

Just out of curiosity, as I haven't been following wasi meetings very closely, what is the background of the requirement that wasi interfaces only use component model? And does this only apply to wasmtime?

[alexcrichton]

I, too, don't follow wasi meetings closely so I won't pretend to speak authoritatively on this, but I believe that the intention is to define WASI APIs with *.wit files which is a component-model level abstraction. At that layer of abstraction there's no way to talk about a shared linear memory or something like that which must be imported/exported with a particular name or such.

I'm not sure what you mean though by if this only applies to wasmtime. The component model should look the same irrespective of where it's implemented, so it's not a wasmtime-specific limitation that wasi-{parallel,threads} can't be implemented in the component-model as-is, that's a component-model limitation. To be clear though the component model is in no way final by any stretch of the imagination so it's not like it can never support the underlying goals of wasi-{parallel,threads}, I'm instead commenting that as-is there's no way to implement this.

If you're instead asking if wasi-{parallel,threads} can be implemented at all in Wasmtime, this PR is certainly proof-enough that it's possible to do that. I don't doubt that this PR works as intended but my comments are taking a more long-term vision about how I believe as WASI, *.wit, and the component model all converge this implementation will cease to work, specifically around the workarounds here to get access to the shared memory through the host function implementation of parallel-exec. The parallel-exec function cannot be implemented in the component-model as-is.

[sunfishcode]

My understanding of how wasi-parallel is expected to fit into the component model is that it could be packaged as an import of a core-module, with the core-module importing the linear memory and exporting functions that operate on it directly. That isn't supported in wit or wit-bindgen today, so we'll need interim approaches, but eventually we should be able to pull these into the wit framework.

[penzn]

@alexcrichton thank you for the detailed answers.

I'm not sure what you mean though by if this only applies to wasmtime.

I meant to ask whether this is the approach adopted specifically by wasmtime or by component model, but you have answered that.

My understanding of how wasi-parallel is expected to fit into the component model is that it could be packaged as an import of a core-module, with the core-module importing the linear memory and exporting functions that operate on it directly. That isn't supported in wit or wit-bindgen today, so we'll need interim approaches, but eventually we should be able to pull these into the wit framework.

Does component model allow this yet? What can the interim approaches be? Maybe we should take it the component-model repo though.

[alexcrichton]

The component model does support importing a core wasm module, but that's not reflected in the *.wit textual description of APIs at this time. For interim things I suspect that'd be good to bring up on the component model repo yeah.

[abrown]

@alexcrichton, I think there are several points you should reconsider:

In the component model the hack you've implemented here with getting the shared memory to work will not work and has no way of working, there's no escape hatch to implement as you've done here.

Then there's a bug in the component model specification. If there is no way to support multiple WASI proposals (wasi-parallel, wasi-threads), no way to satisfy the recurring requests for access to the host (latest of which is bytecodealliance/wit-bindgen#313), and no way to then make use of the core WebAssembly threads proposal, then the WIT specification is not complete enough. If there isn't an escape hatch, then one should be built.

Additionally the existing wasi-common, wasi-nn, and wasi-crypto will likely all need to get removed or refactored or something with a transition away from wiggle

Sure, but that should not block progress. The issue you point out is not with those WASI contributions but rather with the unstable state of wit-bindgen. I tried to refactor wasi-nn to use WIT a month or two ago and the tools were simply not ready. If the spec is not ready and the tools are not ready, it seems unreasonable to block all other work on that. I think the more productive approach would be to allow WASI proposals to continue and ensure that both the WIT spec and tooling can do everything wiggle could.

I am not sure how useful it's been to have wasi-nn and wasi-crypto in this repository, I am not aware of how much usage they're getting and feedback they're providing to the upstream proposals.

You should talk to @geekbeast, also a Fastly employee. He was attempting to use wasi-nn in Viceroy and was able to do so based on the reference implementation here.

I also don't see a future for the wasi-threads proposal as-is in the component model right now. If these proposals want to integrate with the component model I think work needs to happen in the component model itself rather than assuming the functionality can be implemented with special host functions.

I am not involved in component model design... but you are 😁. This PR — along with WebAssembly/wasi-parallel#4, bytecodealliance/wit-bindgen#130, bytecodealliance/wit-bindgen#313 — should be great feedback for a component model issue that you would know how to craft but I would not. I don't understand all of the component model requirements, history, etc. so my ability to contribute there is limited. Why not just take this PR as feedback to the component model and propose a solution so that these WASI proposals can have a future?

[lukewagner]

Agreed with @sunfishcode above that the right way to support various kinds of shared-memory multi-threading in WASI in a component-model world is by expressing the imported threading functionality as core module imports, which are then instantiated by the component to explicitly share core memory with the other core modules inside the component. When core wasm eventually gets a fork instruction for creating threads purely from within core wasm, then these core module imports would even be virtualizable from core wasm, satisfying WASI's general goal to specify virtualizable interfaces.

This does raise the question of how to write the interface of a core module in Wit, since there currently isn't any syntactic support for this. But it's easy to imagine a Wit-like syntax for core:moduletype with a new top-level module keyword that can then be mentioned via import in a world. IIUC, this same Wit syntax would be independently useful for reflecting the types of components that import shared-everything dynamically-linked libraries, like a libc or libspidermonkey, using CanonicalABI.md saying how things get linked up by default, which is something @sunfishcode have also been talking about recently. So it's good to have 2 rather-symmetric motivating use cases for this new Wit functionality.

I don't have enough understanding (particularly around Wiggle) to say what we should do in the right-now term before the above Wit functionality gets implemented. But switching out Wiggle for Wit does seem like it should be our goal state.

[abrown]

Thanks @sunfishcode and @lukewagner for pointing out that the shared memory import/export will be expressible with WIT core modules. I still have questions on this (When and how can the core module syntax be implemented? Is help needed there and, if so, what? Should that block this PR?) but it seems like long term there should not be a problem with wasi-parallel moving to WIT related to shared memory import/export.

But, to take up what I think is @alexcrichton's point, the issue is actually about host access. Proposals like this one and wasi-threads and other use cases (like the ones linked above) want to touch the host itself, e.g., Wasmtime APIs like Caller, Module, Store, Engine. In wasi-threads, e.g., wasi_thread_spawn will need access to the Module of the parent thread in order to create new instances of it and access to the SharedMemory in the Store to import that in each new child thread. I think what @alexcrichton is saying is that the WIT syntax and tooling do not support this and never will (?), which I was asking him to reconsider.

I actually see several possibilities along a continuum:

a. dead in the water: the WIT syntax never has knowledge of "host access," the tools don't support it, and this class of proposal is dead in the water (from @alexcrichton's argument that engines like Wasmtime will not want to support non-WIT-able proposals)
b. hack: the WIT syntax still has no knowledge of "host access," the tools still don't support it, but this class of proposal can be implemented as a core module with some hacking on the linker (as done in this PR) to get access to the host (I don't understand how this would look in WIT!)
c. escape hatch: the WIT syntax still has no knowledge of "host access," the tools do support it as an escape hatch (e.g., an additional parameter to the WIT macros like host_access_functions: ["..."]), and this class of proposal can be more cleanly implemented
d. full support: the WIT syntax does gain knowledge of "host access" (perhaps with a new host keyword in the signature of functions, e.g., wasi_foo(host, x: i32, y: i32)), then the tools would support it, etc.

I've implemented "b" and I believe @alexcrichton was suggesting "a." I would prefer "c." @lukewagner, @sunfishcode: I can't tell from your comments but I think you are not saying "a"?

[abrown]

(Also, please prefix every sentence above with "IIUC" since I am not too familiar with the details of the component model and I'm reading into some of @alexcrichton's comments).

[alexcrichton]

To some degree I'm trying really hard here to be an impartial messenger, along the lines of "please don't shoot the messenger". I'm pretty intimately aware of all the constraints here ranging from the threads proposal to the desire for WASI threads to the component model itself, but I am trying to not take on the design myself and leave that to others. To that end I don't have great responses to comments such as:

If there isn't an escape hatch, then one should be built.

I'm pointing out the systems that are in play and their various designs, and all of these systems have intentional design behind them. I am not personally going to go to the component model repository and open a "please add a small escape hatch" issue because I'm aware of why one hasn't been added yet. Furthermore I'm not going to go to the wasi-parallel proposal and open an issue saying "you will never work with the component model" because I'm aware of the strong motivations for such a proposal. I'm trying to ideally state impartial facts here without trying to solve everything myself.

That being said I naturally have a lot of opinions about how this should all work. If a parallelism story for wasm requires an "escape hatch" that seems like a fundamental problem because parallelism is such a foundational part of any runtime environment that if it's not designed for from day one it seems to me like it has little chance of gaining any amount of larger traction. For example just because Wasmtime is capable of implementing escape hatches like this in no way means other runtimes can do so. The Caller context is not available on the web, for example, meaning that if the wasi-parallel proposal crucially relies on such an escape hatch then that would be unimplementable or otherwise problematic for the web.

Personally I even have hesitation about the whole kernel model in wasi-parallel. I don't understand how anyone could productively write a kernel that literally only imports shared memory and nothing else. Virtually no language can target a world where nothing is imported and if nothing is imported then a language can't even do something like print a line to standard out. While it's theoretically possible and technically possible to get something working here I feel that this is a far cry from a productive developer experience.

Again though I'm trying hard to not insert myself into the design here. These are some thoughts of mine but honestly I don't consider them relevant since I'm not the one championing or being involved in the proposal. At the same time though I feel like someone has to point out the actual limitations of the systems that are being built at some point as otherwise it feels like things are carried on as everyone's just blindly merging everyone else's work without thought to the consequences.

Why not just take this PR as feedback to the component model and propose a solution so that these WASI proposals can have a future?

My hope is that design issues like running wasi-parallel in the component model would have come up much earlier in the design process long before a PR came to Wasmtime here. This isn't really the right place to discuss deep design details of the proposal or modifications to the component model. Otherwise though I don't really think it's a productive developer process to create a proof of concept and then turn around to the component model and say "you figure out how to get this working it can't regress". The design here should be a collaborative process of all parties involved which doesn't just shove all the work onto one person to figure out everything.

---

Sorry for the long post, I have a lot of thoughts on this but now's probably not the time or the place. I'll reiterate that I personally would like to see threads-on-wasm via a standard like WASI. I don't think that the component model and WASI are in a state where things can be plucked off the shelf, mashed together, and a workable standard created. I feel there are deeper designs that need to be implemented and accounted for before moving forward with this iteration of wasi-parallel. For example the import-a-core-wasm-module idea seems plausible, but AFAIK the wasi-parallel proposal has not been designed with this in mind, runtime support in Wasmtime has not been implemented with this in mind, and support in developer tooling (a la *.wit) has not been designed with this in mind. That's a lot of hypothetical work to leave on the table while asking to merge this as an interim solution.

[mingqiusun]

I'd be curious to see how others feel about this though. I am not sure how useful it's been to have wasi-nn and wasi-crypto in this repository, I am not aware of how much usage they're getting and feedback they're providing to the upstream proposals. In that sense I'm not sure how beneficial it will be to land this here vs have it externally.

Is there a good mechanism to track usage of various features in Wasmtime? Usually I have no knowledge of certain features being used by a customer until they contact us with some issues, or they present/publish their usage somewhere. And these are probably a subset of total user base. For example, I know a big device manufacturer is in the process of embedding wasi-nn into their products. For wasi-crypto, it is an essential function for a Wasm runtime to be selected for a SASE project we are involved.

Here are a few links of wasi-nn usages from googling (not all on Wasmtime):
https://deislabs.io/posts/wasi-nn-onnx/
https://www.secondstate.io/articles/openvino-inferencing-using-wasmedge-wasi-nn/
https://wasmedge.org/book/en/write_wasm/rust/wasinn.html
https://cfp.cloud-native.rejekts.io/cloud-native-rejekts-eu-valencia-2022/talk/NXLTRQ/
https://www.youtube.com/watch?v=48ORmla7mak&list=PLj6h78yzYM2Ni0u-ONljTkv4uOutyjwq9&index=11

[lukewagner]

Just as an update: I realized my comment above was forgetting that we're talking about the core-instance-per-thread approach to multi-threading, for which the core module import approach I suggested above would need the imported core module to itself have a module import (the module to be instantiated for each new thread), which is not yet possible in core wasm. So that might not be the right approach here.

It does really seem like the best approach to exposing multi-threading to core wasm in a standard way is as proposed by Watt & Rossberg, as a core wasm feature. Although what we do to allow short-term experimentation and prototyping in Wasmtime is a different question.

[penzn]

For example just because Wasmtime is capable of implementing escape hatches like this in no way means other runtimes can do so. The Caller context is not available on the web, for example, meaning that if the wasi-parallel proposal crucially relies on such an escape hatch then that would be unimplementable or otherwise problematic for the web.

Web engines support this via a JavaScript callback which creates another copy of the instance. Web polyfill was part of the motivation for wasi-threads proposal, the other part being the fact that at least one more standalone engine supports it as well. Core spec supports multiple instance executing in parallel, this isn't an unknown either.

[alexcrichton]

@mingqiusun for sure yeah, I don't mean to say definitively whether or not those proposal should be removed from the repository, and it's great to hear that wasi-nn is getting such usage!

@penzn indeed you are correct. There are a huge number of caveats with the support on the web for threading, however, including:

• The threads proposal is not offically merged into the wasm spec yet
• I'm not aware of any general purpose "spawn a thread library" due to the instance-per-thread model which requires somehow getting a module's instance imports onto web workers. All solutions I know of are very runtime-specific or specific to the shape of module and what it imports. Everything I'm aware of at least is very far from "just call pthread_create and it all works"
• The web's abstraction layer is not components, which I believe wasi-{parallel,threads} want to target at this time. The component abstraction layer, at this time, does not allow for precisely the same capabilities of what core wasm allows.

One point that's probably worth clarifying is that it's not like I think Wasmtime shouldn't be able to implement a web-style threading scenario. In fact you can, already today, implement "threading" with Wasmtime since you as the host can spawn threads and figure out how to instantiate modules on new threads and share host state. There's nothing inherent to Wasmtime preventing this with Wasmtime's core wasm embedding API.

---

I talked with @abrown at great length yesterday about this and we're going to try to allocate time next week to discuss what it might look like for wasi-parallel to get integrated into the component model more completely. In the meantime my personal feelings on this PR is that I would like to make sure that this iteration of wasi-parallel and wasi-threads can be implemented with Wasmtime's embedding API, but otherwise this implementation I think might be best to reside in a different repository than the main Wasmtime repository.

[alexcrichton]

@abrown, @lukewagner, @sunfishcode, and I talked last week at great length about this PR and the various issues here, zooming out mostly to threads in the component model. We concluded a number of things that I hope to summarize here, and please correct me y'all if I miss anything.

Our general topic was what to do with threads, the component model, and having the two work in harmony with each other. To that end we broke down a possible plan of action in to three "stages" ordered from shorest-time-to-implement to longest-time-to-impelment.

Stage 1

The first thing for Wasmtime and threads is to get anything working all. The goal here would effectively, through some wasm-defined API using Wasmtime's embedding API, achieving the functionality proposed by wasi-{threads,parallel} today (I'm going to say only wasi-threads from now on but I mean both). There's a lot of things even here that need implementing/deciding such as:

• How are existing WASI hostcalls synchronized?
• How is a Store<T> created on the new thread?
• How does the new thread acquire a Module to instantiate?
• How does the new thread acquire a Linker<T> to create a new instance on this thread?

This is all just the bare bones of getting anything working, but should hopefully provide valuable implementation feedback to Wasmtime, the component model, and the wasi-threads proposal. All of this is likely design work that needs to be done anyway for any eventual formal wasi-threads proposal is the hope.

This local maximum, however, is not immediately compatible with the component model. Some of the fundamental abstractions here, such as importing a shared linear memory, do not work the same way in the component model (components can't import a linear memory). This means that this stage does not have access to host APIs defined by the component model, such as a hypothetical wasi-snapshot-preview2.

Stage 2

This stage is intended to be sequenced after the prior stage with the goal of hooking up component-model-level host definitions into a core wasm module that wants to use threads. As a base assumption the idea here is that the user-supplied "thing" is still a core wasm module. The high-level idea is that host-defined component-model definitions are "automatically lowered" into the corresponding core wasm functions to get imported into a core wasm module.

This sort of means that a wasmtime::Linker could be blended with a wasmtime::component::Linker to instantiate a wasmtime::Module. The thinking is that the work from stage 1, probably mostly present in a wasmtime::Linker, could pull in definitions from a wasmtime::component::Linker to get access to all host APIs while still retaining access to the abstraction layer of threads.

The main body of work in this stage would be to figure out how to robustly take a component-model level API and translate it into a core wasm API. This is roughly already done with the canonical ABI but I personally believe there's a lot of fiddly bits to work out such as how to reuse the lifting/lowering in Wasmtime without duplicating it. For example somehow a VMContext and a VMComponentContext need to be bridged between one another or ... something like that.

In discussing this though we wondered but weren't sure of how important such feature would be for migrating from wasi preview1 to preview2. There will, for a long time, be binaries that import wasi_snapshot_preview1 and there's a number of strategies to load that into a preview2-supporting runtime, but something it might be quite useful to allow merging of newly-defined and implemented APIs into older modules.

The downside of this stage is that there still isn't full integration with the component model. Everything is still working at the level of abstraction of a core wasm module and inherently continues to be incompatible with components. The main distinction from stage 1 is that component-model-level functionality, which is the presumed path that WASI is going, can still be imported into the module.

Stage 3

The next stage of evolution for support here would be to integrate threads into the component model proposal itself. This is a far-in-the-future vision though where there's a difference between a func and a shared func for example. there's a ton of design questions here that (at least to me) don't have obvious answers. There's quite a lot of design work here not only for core wasm but also the component model. This is the hypothetical end goal of components that support threads, but I'm mostly listing this here for completeness as the timescale for implementing this in in the many-numbers-of-years.

---

Given the possible staging here I concluded that my original stance of "we probably don't want to merge this" is no longer true. The staging here would explicitly continue to work at the core wasm layer and not the component layer. Furthermore for the time being there would be no path forward to implement this in the component layer but there would be a path to taking some of the major benefits of the component layer and bringing them to core wasm. I'd revise my previous stance, then, to it's ok to merge this into the main Wasmtime repository along the lines of all other WASI proposals.

That's not to discount the technical review here, however. This is an example of a fundamental issue in wiggle which will need to get ironed out before landing. Furthermore I personally at least feel that wasi-parallel may not be a best fit for the first proposal implemented here (the kernel feels a bit odd, it's JIT compiled so doesn't work with AOT compilation, the kernel itself can't import anything useful, unclear how toolchains will produce this kernel from "real" source code, etc). @abrown what would you think of implementing wasi-threads first? That feels like a more low-level proposal which would, for now at least, subsume the functionality implemented here for wasi-parallel I think?

[abrown]

Thanks @alexcrichton for the detailed write-up; I think it's a pretty accurate reflection of what we discussed and a decent high-level vision for how to move forward. I agree that wasi-threads is a more foundational proposal and, actually, I think it highlights some of the issues to overcome to get to stage 1 better than wasi-parallel does. wasi-parallel sidesteps a few things with its "bag of bytes" paradigm. I'll see if there's a way to get wasi-threads done first.

The current issues to get to stage 1 that @alexcrichton describes above are many and span multiple repositories (this one, wasi-libc, LLVM, Rust, etc.). For anyone else interested in progress on getting wasi-threads to the stage 1, this Zulip channel has the latest discussion. @haraldh has taken some of the pieces I and others have been working on and has build a working PoC! I will post my take on the remaining items to upstream, probably in an issue on the wasi-threads spec repository.