deno cache failing with error sending request for url: dispatch task is gone: runtime dropped the dispatch task

[ishmam-mahmud]

This does not happen locally. It seems to only be affecting our Dockerfile when it runs on our render.com deployment.

Here's a section of the Dockerfile that's causing the issue

FROM denoland/deno:1.31.1

ARG SERVICE_NAME

EXPOSE 8000

WORKDIR /app

USER deno

# Copy dependencies files
ADD --chown=deno:deno ./import_map.json .
ADD --chown=deno:deno ./deps.ts .
# Cache dependencies
RUN deno cache --import-map=import_map.json deps.ts

# Copy shared dir
ADD --chown=deno:deno ./shared ./shared

# Copy service
ADD --chown=deno:deno ./services/${SERVICE_NAME} ./services/${SERVICE_NAME}

WORKDIR /app/services/${SERVICE_NAME}

RUN deno cache ./main.ts

CMD ["run", "--allow-net", "--allow-read", "--allow-env", "--allow-write", "--allow-sys", "main.ts"]

This is a monorepo structure, where each service is in a services folder, and the shared folder contains custom libraries used by multiple services.

The error message from the docker CLI itself:

29.63 Download https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.15.4.tgz
32.64 error: error sending request for url (https://deno.land/std@0.175.0/async/tee.ts): dispatch task is gone: runtime dropped the dispatch task
32.64     at https://deno.land/std@0.175.0/async/mod.ts:17:15
------
Dockerfile:15
--------------------
  13 |     ADD --chown=deno:deno ./deps.ts .
  14 |     # Cache dependencies
  15 | >>> RUN deno cache --import-map=import_map.json deps.ts
  16 |     
--------------------
error: failed to solve: process "/bin/sh -c deno cache --import-map=import_map.json deps.ts" did not complete successfully: exit code: 1
error: exit status 1

The url/specific dependency that causes the failure is different everytime, but it's usually a deno.land/std package or a deno.land/x package.

Here's the deps.ts file:

export * as server from "https://deno.land/std@0.175.0/http/server.ts";
import type * as ServerTypes from "https://deno.land/std@0.175.0/http/server.ts";
export { ServerTypes };

export * as dotenv from "https://deno.land/std@0.175.0/dotenv/mod.ts";
export * as datetime from "https://deno.land/std@0.175.0/datetime/mod.ts";
export * as oak from "https://deno.land/x/oak@v11.1.0/mod.ts";
export * as djwt from "https://deno.land/x/djwt@v2.0/mod.ts";
export * as io from "https://deno.land/std@0.176.0/io/mod.ts";
export * as crypto from "https://deno.land/std@0.170.0/node/crypto.ts";
export * as encodingBase64 from "https://deno.land/std@0.82.0/encoding/base64.ts";
export * as encodingBase32 from "https://deno.land/std@0.200.0/encoding/base32.ts";
export * as graphql_tag from "https://deno.land/x/graphql_tag@0.1.0/mod.ts";
export * as to_case from "https://deno.land/x/case@2.1.1/mod.ts";
export * as path from "https://deno.land/std@0.175.0/path/mod.ts";
export * as fs from "https://deno.land/std@0.175.0/fs/mod.ts";
export * as uuid from "https://deno.land/std@0.175.0/uuid/mod.ts";
export * as apollo_subgraph from "npm:@apollo/subgraph@2.3.2";
export * as graphql_scalars from "https://esm.sh/v131/graphql-scalars@1.20.1/es2022/graphql-scalars.mjs";
export * as bcrypt from "https://deno.land/x/bcrypt@v0.2.4/mod.ts";
export * as bcrypt_worker from "https://deno.land/x/bcrypt@v0.2.4/src/worker.ts";
export * as cookie from "https://deno.land/std@0.178.0/http/cookie.ts";
export * as jose from "https://deno.land/x/jose@v4.13.1/index.ts";
export * as encodingHex from "https://deno.land/std@0.204.0/encoding/hex.ts";
export * as Sentry from "npm:@sentry/node@7.49.0";
export * as Posthog from "npm:posthog-node@3.1.2";
import {
  ExtMapping,
  MimeMapping,
} from "https://deno.land/x/common_mime_types@0.1.1/mod.ts";
export { ExtMapping, MimeMapping };

import type * as GraphQLCodegenCliTypes from "npm:@graphql-codegen/cli";
export { GraphQLCodegenCliTypes };

export * as log from "https://deno.land/std@0.178.0/log/mod.ts";
export * as log_levels from "https://deno.land/std@0.178.0/log/levels.ts";
import type * as LogTypes from "https://deno.land/std@0.178.0/log/mod.ts";
export { LogTypes };

export * as graphql from "npm:graphql@16.6.0";
import type * as GraphqlTypes from "npm:graphql@16.6.0";
export { GraphqlTypes };

export * as zod from "https://esm.sh/v111/zod@3.21.4";
import type * as ZodTypes from "https://esm.sh/v111/zod@3.21.4";
export { ZodTypes };

export * as zodToJsonSchema from "https://esm.sh/v135/zod-to-json-schema@3.22.3";

export { default as postgres } from "https://deno.land/x/postgresjs@v3.3.4/mod.js";

export * as pgtyped_runtime from "npm:@pgtyped/runtime@2.0.0";
import type * as PgtypedRuntimeTypes from "npm:@pgtyped/runtime@2.0.0";
export { PgtypedRuntimeTypes };

export * as pgtyped_parser from "npm:@pgtyped/parser@2.0.0";
import type * as PgtypedParserTypes from "npm:@pgtyped/parser@2.0.0";
export { PgtypedParserTypes };

export * as graphql_request from "https://deno.land/x/graphql_request@v4.1.0/mod.ts";
import type * as GraphqlRequestTypes from "https://deno.land/x/graphql_request@v4.1.0/mod.ts";
import type * as GraphqlRequestDomTypes from "https://esm.sh/v111/*graphql-request@5.1.0/dist/types.dom";
export { GraphqlRequestDomTypes, GraphqlRequestTypes };

export * as graphql_middleware from "npm:graphql-middleware@6.1.33";
import type * as GraphqlMiddlewareTypes from "npm:graphql-middleware@6.1.33";
export { GraphqlMiddlewareTypes };

// "npm:graphql-yoga@3.1.2" is unstable live (graphql playground does not work)
// esm package v111 or v131 are both unstable during testing
// Use --no-check flag when testing locally until we find a stable version
export * as graphql_yoga from "https://esm.sh/v111/graphql-yoga@3.1.2?external=graphql";
export * as graphql_yoga_plugin_response_cache from "npm:@graphql-yoga/plugin-response-cache@1.9.0";
import type * as GraphqlYogaTypes from "https://esm.sh/v111/graphql-yoga@3.1.2?external=graphql";
export { GraphqlYogaTypes };

import * as envelop_core from "https://esm.sh/v124/@envelop/core@3.0.6";
export { envelop_core };
import type * as EnvelopCoreTypes from "https://esm.sh/v124/@envelop/core@3.0.6";
export { EnvelopCoreTypes };

export * as stripe from "https://esm.sh/v111/stripe@11.10.0?target=deno";
import type * as StripeTypes from "https://esm.sh/v111/stripe@11.10.0?target=deno";
export { StripeTypes };

export { default as sendgrid_mail } from "npm:@sendgrid/mail@7.7.0";
import type * as SendgridMailTypes from "npm:@sendgrid/mail@7.7.0";
export { SendgridMailTypes };

export * as sendgrid_helpers from "npm:@sendgrid/helpers@7.7.0";

export * as buffer from "https://deno.land/std@0.176.0/io/buffer.ts";
import type * as BufferTypes from "https://deno.land/std@0.176.0/io/buffer.ts";
export { BufferTypes };

// lodash
export { default as lodashChunk } from "https://deno.land/x/lodash@4.17.15-es/chunk.js";
export { default as lodashUniqBy } from "https://deno.land/x/lodash@4.17.15-es/uniqBy.js";
export { default as lodashIsEmpty } from "https://deno.land/x/lodash@4.17.15-es/isEmpty.js";
export { default as lodashOmit } from "https://deno.land/x/lodash@4.17.15-es/omit.js";
export { default as lodashGroupBy } from "https://deno.land/x/lodash@4.17.15-es/groupBy.js";
export { default as lodashSortBy } from "https://deno.land/x/lodash@4.17.15-es/sortBy.js";
export { default as lodashDifference } from "https://deno.land/x/lodash@4.17.15-es/difference.js";

// pothos
export { default as pothos_core } from "npm:@pothos/core@3.23.1";
export { default as pothos_plugin_relay } from "npm:@pothos/plugin-relay@3.30.0";
export { default as pothos_plugin_validation } from "npm:@pothos/plugin-validation@3.8.7";
export { default as pothos_plugin_simple_objects } from "npm:@pothos/plugin-simple-objects@3.6.7";
export { default as pothos_plugin_directives } from "npm:@pothos/plugin-directives@3.9.2";
export { default as pothos_plugin_federation } from "npm:@pothos/plugin-federation@3.9.1";

// octokit
export * as octokit_plugin_throttling from "https://esm.sh/v111/@octokit/plugin-throttling@5.0.1";
export * as octokit_plugin_paginate_rest from "https://esm.sh/v111/@octokit/plugin-paginate-rest@6.0.0";

export * as octokit_plugin_rest_endpoint_methods from "https://esm.sh/v111/@octokit/plugin-rest-endpoint-methods@7.0.1";
import type * as OctokitPluginRestEndpointMethodsTypes from "https://esm.sh/v111/@octokit/plugin-rest-endpoint-methods@7.0.1";
export { OctokitPluginRestEndpointMethodsTypes };

export * as octokit_rest from "https://esm.sh/v111/@octokit/rest@19.0.7";
import type * as OctokitRestTypes from "https://esm.sh/v111/@octokit/rest@19.0.7";
export { OctokitRestTypes };

export * as OctokitRequestError from "https://esm.sh/v111/@octokit/request-error@3.0.3";

import type * as OctokitWebhookEventTypes from "npm:@octokit/webhooks-types@6.11.0";
export { OctokitWebhookEventTypes };

export * as xstate from "https://esm.sh/v113/@xstate/fsm@2.0.0";

export * as preact_render_to_string from "https://esm.sh/preact-render-to-string@5.2.6";
import type * as PreactTypes from "preact";
export { PreactTypes };

export { default as Dataloader } from "https://esm.sh/v116/dataloader@2.2.2";

export * as norm from "https://cdn.jsdelivr.net/gh/Murcul/norm@1.3.0/src/mod.ts";

export { default as tld_extract } from "https://esm.sh/tld-extract@2.1.0";
export { default as emailProviders } from "https://esm.sh/email-providers@2.0.1/all.json" assert { type: "json" };

export { parse } from "https://deno.land/std@0.181.0/flags/mod.ts";

// S3
export * as client_s3 from "npm:@aws-sdk/client-s3";
export * as s3_request_presigner from "npm:@aws-sdk/s3-request-presigner";

export * as marked from "npm:marked@6";

export * as OTPAuth from "https://deno.land/x/otpauth@v9.1.4/dist/otpauth.esm.js";

export * as gitbeaker from "https://esm.sh/@gitbeaker/rest@39.25.0";
import type * as GitBeakerTypes from "https://esm.sh/@gitbeaker/rest@39.25.0";
export { GitBeakerTypes };

here's the import_map.json file, with the service names and shared library names redacted

{
  "imports": {
    "@root/deps.ts": "./deps.ts",
    "@root/dev_deps.ts": "./dev_deps.ts",
    "@svc.service_name_here/": "./services/service_name_here/",
    "@us/shared/shared_library_name_here": "./shared/shared_library_name_here/mod.ts",
    "graphql-request": "https://deno.land/x/graphql_request@v4.1.0/mod.ts",
    "graphql-tag": "https://deno.land/x/graphql_tag@0.1.0/mod.ts",
    "graphql-request/dist/types.dom": "https://esm.sh/v111/*graphql-request@5.1.0/dist/types.dom",
    "is-plain-obj": "https://esm.sh/v107/is-plain-obj@4.1.0",
    "graphql": "npm:graphql@16.6.0",
    "https://cdn.esm.sh/extract-files@12.0.0/": "https://unpkg.com/extract-files@12.0.0/",
    "preact": "https://esm.sh/preact@10.13.1",
    "preact/": "https://esm.sh/preact@10.13.1/",
    "preact/jsx-runtime": "https://cdn.skypack.dev/preact/jsx-runtime?dts",
    "preact/jsx-dev-runtime": "https://cdn.skypack.dev/preact/jsx-dev-runtime?dts",
    "preact-render-to-string": "https://esm.sh/*preact-render-to-string@5.2.6"
  }
}

[ishmam-mahmud]

The error is reproducible locally if I run docker build .
It just doesn't happen if I run deno cache myself

[magurotuna]

Hi @ishmam-mahmud. Have you seen this issue very recently, like in the past two hours or so?
We applied a small change to our backend a few hours ago. I wonder if it's related.

[ishmam-mahmud]

Hi @ishmam-mahmud. Have you seen this issue very recently, like in the past two hours or so? We applied a small change to our backend a few hours ago. I wonder if it's related.

We saw this issue the first time around 2024-01-09T17:04:27.000+08:00, or 2024-01-09T09:04:27.000Z in UTC

[magurotuna]

Thanks for the info. The time we applied a change was around 9:30 AM UTC, so I suppose this backend change has nothing to do with this issue. But we'll look into it a bit more.

[magurotuna]

Sorry I mistakenly closed it. Reopening now.

@ishmam-mahmud Could you try again your workflow? We've done a small change yet again to deno.land/x backend, although we're not sure if this fixes your issue.

[ishmam-mahmud]

We started getting a different error at first: 500 Internal Server Errors, but after a few retries we managed to deploy all our services successfully.
Everything's green now.

[lucab]

For reference, there is also a widespread GitHub outage going on today: https://www.githubstatus.com/incidents/pxg3dz4yg7lp

[magurotuna]

@ishmam-mahmud Thanks for confirmation. Glad that your issue has been solved.
I think all related issues have also been resolved by the change we made to the backend, so I'll close this issue now. Feel free to reopen if it still occurs.