Serverless Offline fails to resolve properly with Yarn PnP and the --useInProcess flag

[pomSense]

Bug Report

This is also related to #1577 and #1454.

Reproduction Repo - See README for quick reproduction steps

Current Behavior

When using YarnPnP, if we do not use the --useInProcess flag, we get the following error:

Error: Cannot find module '/Users/code/yarn-pnp-sls-offline/.yarn/__virtual__/serverless-offline-virtual-6c0c8d3381/0/cache/serverless-offline-npm-14.0.0-ceb31b15d2-3fc34ddd76.zip/node_modules/serverless-offline/src/lambda/handler-runner/worker-thread-runner/workerThreadHelper.js'
    at Module._resolveFilename (node:internal/modules/cjs/loader:1140:15)
    at Module._load (node:internal/modules/cjs/loader:981:27)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:128:12)
    at MessagePort.<anonymous> (node:internal/main/worker_thread:184:26)
    at [nodejs.internal.kHybridDispatch] (node:internal/event_target:786:20)
    at exports.emitMessage (node:internal/per_context/messageport:23:28)

Sample Code

Reproduction Repo - See README for quick reproduction steps

• file: serverless.yml

app: test-repo
service: my-service

custom:
  org: myOrg
  defaultStage: dev

  # ESBUILD
  esbuild:
    packager: 'yarn'

  serverless-offline:
    httpPort: 4400
    lambdaPort: 4402
    # When useChildProcesses is commented out, you will see the error
    # useInProcess: true

provider:
  name: aws
  runtime: nodejs18.x
  region: us-east-1

package:
  individually: true
plugins:
  - serverless-esbuild
  - serverless-offline

functions:
  my-function:
    handler: src/index.handler
    name: my-function
    events:
      - httpApi: '*'

• file: handler.js

export const handler = async (event) => {

    console.log('Hello World!');
    return event
}

Expected behavior/code

We should be able to use serverless offline start without the --useInProcess flag, and without having to use nodeLinker: node-modules in the .yarnrc.yml (as this defeats the purpose of yarn zero-install)

Environment

• serverless version: v3.38.0
• serverless-offline version: v14.0.0
• node.js version: v20.12.1
• OS: macOS 14.3.1

Possible Solution

Note a solution but some context in case it is helpful.

This seems to be in theWorkerThreadRunner.js when trying to resolve the workerThreadHelper.js:

import { MessageChannel, Worker } from 'node:worker_threads'
import { join } from 'desm'

export default class WorkerThreadRunner {
  #workerThread = null

  constructor(funOptions, env) {
    const { codeDir, functionKey, handler, servicePath, timeout } = funOptions

    this.#workerThread = new Worker(
      join(import.meta.url, 'workerThreadHelper.js'), // <--- here
  // ...r

My guess is that this happens because:

• Yarn PnP uses a virtual filesystem for package management, which changes how paths are resolved.
• Serverless Offline tries to resolve paths as if they are in a traditional file system
• Node.jsWorker threads expect an actual filesystem path to load their script.

As per Node.js docs for new Worker:

new Worker(filename[, options])
filename | The path to the Worker's main script or module. Must be either an absolute path or a relative path (i.e. relative to the current working directory) starting with ./ or ../, or a WHATWG URL object using file: or data: protocol. When using a data: URL, the data is interpreted based on MIME type using the ECMAScript module loader. If options.eval is true, this is a string containing JavaScript code rather than a path.

I'm not 100% sure if it is related to the Worker and the filename it expects, but just a rabbit hole that I went down and thought it might be relevant.

As per Yarn Docs, this error is emitted by the Node.js resolution pipeline which is supposed to forward resolution requests to Yarn but it didn't. I haven't been able to dig deeper into why it didn't forward.

[DorianMazur]

@pomSense
I created potential fix, can you try it?
#1819