Add support for local Wasm plugins #3349

emcfarlane · 2024-09-27T22:18:29Z

Add support for running custom plugins compiled with WebAssembly. The plugins must conform to the WASI 0.1 specification. This system interface is supported by many languages. For bufplugin-go based plugins see the Go WASI docs. Supporting plugins with the WASI specification allows the running of plugins using the pluginrpc protocol without any code modifications required by the user.

Declare a plugin with the suffix .wasm to use the Wasm runtime to invoke the plugin. For example the plugin buf-plugin-field-option-safe-for-ml can be compiled with GOOS=wasip1 GOARCH=wasm and added to your buf.yaml config:

version: v2
plugins:
  - plugin: buf-plugin-field-option-safe-for-ml.wasm

github-actions · 2024-09-27T22:18:48Z

The latest Buf updates on your PR. Results from workflow Buf CI / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Oct 3, 2024, 2:43 PM

emcfarlane · 2024-10-01T14:10:53Z

private/usage/usage_unix.go

@@ -12,8 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

-//go:build aix || darwin || dragonfly || freebsd || (js && wasm) || linux || netbsd || openbsd || solaris
-// +build aix darwin dragonfly freebsd js,wasm linux netbsd openbsd solaris
+//go:build unix || wasip1 || js


Use the unix build constraint and drop support for +build as that is required for go versions 1.16 and earlier. Fixes builds for GOOS=wasip1

bufdev · 2024-10-01T19:45:41Z

private/buf/bufcli/cache.go

@@ -135,6 +139,17 @@ func NewCommitProvider(container appext.Container) (bufmodule.CommitProvider, er
 	)
 }

+// CreatePluginCacheDir creates the cache directory for plugins.
+//
+// This is used by the [bufwasm.WithLocalCacheDir] option.


Is the [] something that Godoc picks up? If not, we don't use it anywhere else. If so, good to know!

Removed [] use. It is supported in Godoc as a link but must be imported, so here it didn't link correctly.

bufdev · 2024-10-01T19:49:07Z

private/buf/bufcli/cache.go

@@ -103,6 +103,10 @@ var (
 	//
 	// Normalized.
 	v3CacheModuleLockRelDirPath = normalpath.Join("v3", "modulelocks")
+	// v3CachePluginsRelDirPath is the relative path to the plugins cache directory in its newest iteration.


This documentation isn't really accurate. If you dive down into the code, it appears you are passing this to the bufwasm.Runtime, which then happens to use this as a cache for some WASM compilation that is happening. Given the naming, and the documentation here, I would expect this to be a cache of downloaded plugins from the BSR, which it is not. This is effectively an implementation-specific cache.

This should be v3/wasmruntime or the like, whatever makes sense, and the documentation should reflect what this is specifically used for.

Renamed and updated the docs to the v3/wasmruntime dir.

bufdev · 2024-10-01T19:49:34Z

private/buf/bufcli/cache.go

@@ -135,6 +139,17 @@ func NewCommitProvider(container appext.Container) (bufmodule.CommitProvider, er
 	)
 }

+// CreatePluginCacheDir creates the cache directory for plugins.


This doesn't seem correct. See above.

Renamed to CreateWasmRuntimeCacheDir.

bufdev · 2024-10-01T19:51:17Z

private/bufpkg/bufcheck/multi_client.go

@@ -94,6 +95,7 @@ func (c *multiClient) Check(ctx context.Context, request check.Request) ([]*anno
 		jobs = append(
 			jobs,
 			func(ctx context.Context) error {
+				start := time.Now()


Such timings are the responsibilities of tracing.Tracers

Switched to tracing.

bufdev · 2024-10-01T19:51:57Z

private/bufpkg/bufpluginrunner/bufpluginrunner.go

+	"pluginrpc.com/pluginrpc"
+)
+
+// RunnerProvider provides pluginrpc.Runners for a given plugin config.


PluginConfig

bufdev · 2024-10-01T19:54:43Z

private/bufpkg/bufpluginrunner/provider.go

+}
+
+// wasmRunner is a runner that loads a Wasm plugin.
+type wasmRunner struct {


Why is this in this file?

Moved to pluginrpcutil/wasm_runner.go

bufdev · 2024-10-01T19:56:32Z

private/bufpkg/bufwasm/bufwasm.go

@@ -0,0 +1,81 @@
+// Copyright 2020-2024 Buf Technologies, Inc.


Why is this in bufpkg? It uses nothing buf-specific, and therefore should be in pkg/wasm.

Moved to pkg/wasm

bufdev · 2024-10-01T19:56:55Z

private/bufpkg/bufwasm/bufwasm.go

+	"pluginrpc.com/pluginrpc"
+)
+
+// Plugin is a Wasm module.


If Plugin is a WASM module, why is it named Plugin, especially given that plugins have a specific meaning in buf?

Originally thought that Wasm Plugin as its a Wasm module that implements the pluginrpc spec. Have renamed to CompiledModule.

Module also has specific meaning in Buf.

Nevermind on this - CompiledModule seems to be a thing for Wasm too

bufdev · 2024-10-01T19:57:22Z

private/bufpkg/bufwasm/bufwasm.go

+}
+
+// NewUnimplementedRuntime returns a new unimplemented Runtime.
+func NewUnimplementedRuntime() Runtime {


This can be var UnimplementedRuntime = unimplementedRuntime{}

bufdev · 2024-10-01T19:58:12Z

private/bufpkg/bufpluginrunner/provider.go

+		if r.wasmRuntime == nil {
+			return nil, syserror.Newf("wasm runtime is required for local wasm plugins")
+		}
+		return newWasmRunner(


It's weird that this is implemented in bufpluginrunner while the command.Runner-based Runner is implemented in pluginrpcutil. Why the split? I don't see any reason for one, both Runners should be implemented in pluginrpcutil.

They can't be implemented in pluginrpcutil as the pkg now depends on bufconfig for the bufconfig.PluginConfig. Could move pluginrpcutil under bufpkg?

the wasmRunner in no way depends on bufconfig.PluginConfig, I'm not sure what you are talking about.

Referring to the pluginrpcutil. All good on moving the wasm runner.

Moved the Wasm pkg to pkg/wasm. The pluginrpcutil pkg has a new function NewWasmRunner that takes a wasm.Runtime similar to the command.Runner. The wasm.Runtime is not implemented as a Run method to allow the caller to manage the caching of plugins.

bufdev · 2024-10-02T17:09:39Z

private/pkg/wasm/runtime.go

+		}
+		runtimeConfig = runtimeConfig.WithCompilationCache(cache)
+	}
+	r := wazero.NewRuntimeWithConfig(ctx, runtimeConfig)


We don't use single character variable names in this codebase beyond method receivers

bufdev · 2024-10-03T14:33:33Z

private/pkg/wasm/wasm.go

+}
+
+// UnimplementedRuntime is an unimplemented Runtime.
+var UnimplementedRuntime = unimplementedRuntime{}


global vars always go at the top, just below global consts

bufdev · 2024-10-03T14:42:37Z

I just merged a PR that will conflict with CHANGELOG.md, you'll need to merge

Init support for WASM plugins

6cd7178

emcfarlane requested a review from doriable September 27, 2024 22:18

emcfarlane added 5 commits September 30, 2024 14:12

Remove custom look path

a17f1c1

Fix test

58c9e6f

Fix test

ccdb801

Fix test

1fb09fe

Use Wasm abbr

4084811

emcfarlane commented Oct 1, 2024

View reviewed changes

emcfarlane added 2 commits October 1, 2024 10:14

Fix description

cc42495

Move RunnerProvider to bufpluginrunner pkg

5cd84ca

bufdev reviewed Oct 1, 2024

View reviewed changes

Rename Plugin to CompiledModule

0b4321b

emcfarlane marked this pull request as ready for review October 1, 2024 22:43

emcfarlane marked this pull request as draft October 2, 2024 00:25

emcfarlane added 15 commits October 1, 2024 20:40

Use tracer

ce2c42a

Unimplemented as var

8ef1070

Move wasm pkg

fa26987

move to bufpluginrpcutil

9429c87

Fixes

9634e9d

use wasmruntime cache dir

3498241

doc RunnerProvider config

767897e

fix wasm args

a69da63

cleanup wasm module docs

87676c0

fix bufconfig

e42f48d

fix naming

88ffcb7

fix banimports

85c7fbc

Merge branch 'main' into ed/pluginWASM

6a45022

fix pluginrpcutil

41f8127

fix doc

2be37aa

bufdev reviewed Oct 2, 2024

View reviewed changes

emcfarlane marked this pull request as draft October 2, 2024 17:19

emcfarlane added 19 commits October 2, 2024 13:25

Fix runtime options

f035402

fix naming to Close

4e161a6

fix var names

cec3d6d

fix unimplemented

64764cc

fix changelog

75fe87e

fix commandRunner naming

a5e1c43

fix delegate

799658a

clarify args

41705f9

use filepath.Ext

4bfc10c

expand doc on limits

d09ddbf

doc instantiation

2fc5bf6

fix see docs

1415eea

doc memory page limits

dc11fe5

note about size restriction

086a6ce

fix

6ab5dcb

doc limits

5e8e9d9

fix Runtime

1866ce1

fix closed

ef0d8a8

fix doc

6c4030a

emcfarlane marked this pull request as ready for review October 2, 2024 21:40

move unimplemented runtime

bf634a5

bufdev reviewed Oct 3, 2024

View reviewed changes

fix global var position

423e044

Merge branch 'main' into ed/pluginWASM

ef1a275

bufdev approved these changes Oct 3, 2024

View reviewed changes

bufdev merged commit e065fbb into main Oct 3, 2024
11 checks passed

bufdev deleted the ed/pluginWASM branch October 3, 2024 14:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for local Wasm plugins #3349

Add support for local Wasm plugins #3349

emcfarlane commented Sep 27, 2024 •

edited

Loading

github-actions bot commented Sep 27, 2024 •

edited

Loading

emcfarlane Oct 1, 2024 •

edited

Loading

bufdev Oct 1, 2024

emcfarlane Oct 2, 2024

bufdev Oct 1, 2024

emcfarlane Oct 2, 2024

bufdev Oct 1, 2024

emcfarlane Oct 2, 2024

bufdev Oct 1, 2024

emcfarlane Oct 2, 2024

bufdev Oct 1, 2024

emcfarlane Oct 2, 2024

bufdev Oct 1, 2024

emcfarlane Oct 2, 2024

bufdev Oct 1, 2024

emcfarlane Oct 2, 2024

bufdev Oct 1, 2024

emcfarlane Oct 2, 2024

bufdev Oct 2, 2024

bufdev Oct 2, 2024

bufdev Oct 1, 2024

emcfarlane Oct 2, 2024

bufdev Oct 1, 2024

emcfarlane Oct 1, 2024

bufdev Oct 2, 2024

emcfarlane Oct 2, 2024

emcfarlane Oct 2, 2024

bufdev Oct 2, 2024

bufdev Oct 3, 2024

bufdev commented Oct 3, 2024

		@@ -0,0 +1,81 @@
		// Copyright 2020-2024 Buf Technologies, Inc.

Add support for local Wasm plugins #3349

Add support for local Wasm plugins #3349

Conversation

emcfarlane commented Sep 27, 2024 • edited Loading

github-actions bot commented Sep 27, 2024 • edited Loading

emcfarlane Oct 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bufdev commented Oct 3, 2024

emcfarlane commented Sep 27, 2024 •

edited

Loading

github-actions bot commented Sep 27, 2024 •

edited

Loading

emcfarlane Oct 1, 2024 •

edited

Loading