Kaniko builds sometimes produces broken image (doesn't happen on first build, only building w/ cache?)

[timbrd]

I'm using kaniko 0.17.1 on my gitlab runner (Docker 19.03.5).

This is my dockerfile:

FROM docker.io/alpine:3.10
RUN apk add --no-cache openjdk9-jre-headless

This is the command to build the image:

/kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:ci --build-arg=SSH_PRIVATE_KEY="${SSH_PRIVATE_KEY_GIT}"

The image build succeeds, but when I pull the image from my registry, I get the following error (tested on multiple machines and operating systems):

docker pull gitlab.mydomain.net:5050/myname/toolbox:ci
ci: Pulling from myname/toolbox
4167d3e14976: Already exists
ae6394e466fb: Extracting [==================================================>]  71.28MB/71.28MB
failed to register layer: Error processing tar file(exit status 1): mkdir /usr/lib/jvm/default-jvm/bin: no such file or directory

When building the image with kaniko 0.16.0, the image works as expected.

[tarioch]

Similar issue with a slightly different error message on a custom image build

failed to register layer: ApplyLayer duplicates of file paths not supported stdout: {"layerSize":369586710} stderr:

[cigraphics]

I'm getting something like this:
The first build is fine. Then when i build a new one/new ones it will fail with this when i try to deploy to GKE i even tried on my local machine a docker pull [IMAGE-FROM-GCR] and i get the same error
failed to register layer: error processing tar file(exit status 1): failed to mknod no such file

[hberntsen]

This Dockerfile also fails on 0.17.1:

FROM ubuntu:19.10
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y libc6-dev

When I pull the generated image, it says:
failed to register layer: ApplyLayer exit status 1 stdout: stderr: link /lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu: no such file or directory

0.16.0 works fine

[drptbl]

Same issue with yarn here which is missing after container is built, also doesn't work on 0.16.0:

FROM node:10.19.0-buster
ENV YARN_VERSION 1.22.0
# install latest yarn
RUN curl -fSLO --compressed "https://yarnpkg.com/downloads/$YARN_VERSION/yarn-v$YARN_VERSION.tar.gz" \
    && tar -xzf yarn-v$YARN_VERSION.tar.gz -C /opt/ \
    && ln -snf /opt/yarn-v$YARN_VERSION/bin/yarn /usr/local/bin/yarn \
    && ln -snf /opt/yarn-v$YARN_VERSION/bin/yarnpkg /usr/local/bin/yarnpkg \
    && rm yarn-v$YARN_VERSION.tar.gz

[cvgw]

This is most likely the same issues as #1039, can you try tags a1af057f997316bfb1c4d2d82719d78481a02a79 or debug-a1af057f997316bfb1c4d2d82719d78481a02a79

[gfvirga]

@cvgw, I am having the same error with kaniko:debug
ERROR: Job failed: image pull failed: rpc error: code = Unknown desc = Error committing the finished image: error adding layer with blob "sha256:db37edea0ded56df3b03d6f76eedd9cd1303d1f6e4920588aa28b37c94d47b8f": Error processing tar file(exit status 1): open /bin/nc.traditional: no such file or directory

Tag debug-a1af057f997316bfb1c4d2d82719d78481a02a79 fixed the problem. What's the plan to push to the :debug tag? Is there a pull request I could follow?

[timbrd]

This is most likely the same issues as #1039, can you try tags a1af057f997316bfb1c4d2d82719d78481a02a79 or debug-a1af057f997316bfb1c4d2d82719d78481a02a79

Yes, building an image with debug-a1af057f997316bfb1c4d2d82719d78481a02a79, I can pull it afterwards.

[cvgw]

@cvgw, I am having the same error with kaniko:debug
ERROR: Job failed: image pull failed: rpc error: code = Unknown desc = Error committing the finished image: error adding layer with blob "sha256:db37edea0ded56df3b03d6f76eedd9cd1303d1f6e4920588aa28b37c94d47b8f": Error processing tar file(exit status 1): open /bin/nc.traditional: no such file or directory

Tag debug-a1af057f997316bfb1c4d2d82719d78481a02a79 fixed the problem. What's the plan to push to the :debug tag? Is there a pull request I could follow?

That code will get pushed with the debug tag when the next release of kaniko is cut. Should be end of this week or next cc @tejal29

[cvgw]

closing issue as it appears to be fixed, please re-open if needed

[nomatterz]

@cvgw
was using :debug image ( till yesterday) to build images from gitlab shared runner and push to AWS ECR. This week errors started to appear while pulling:

failed to register layer: Error processing tar file(exit status 1): no such file or directory

Referring to this issue and using debug-a1af057f997316bfb1c4d2d82719d78481a02a79 solved it.
So I'm afraid i the issue is still there on "latest"

[fabiotamagno]

I have the same behaviour when running on kubernetes as @nomatterz when using 0.17 and latest (0.18).

rpc error: code = Unknown desc = failed to register layer: Error processing tar file(exit status 1): no such file or directory

Using version debug-a1af057f997316bfb1c4d2d82719d78481a02a79 solved it for me as well.

Actually after a couple deployments it started breaking again. I'm back to 0.16 and after mixing some deployments it still works.

[kelvin-yue-scmp]

I still got

failed to register layer: Error processing tar file(exit status 1): failed to mknod("/usr/local/lib/node_modules/npm/appveyor.yml", S_IFCHR, 0): file exists

with using debug-v0.18.0

[swist]

Still having this issue on multi tenanted cluster running a mix of 0.17.1 and 0.19.0

[dave08]

Got this when I enabled cache use (works without it):

kaniko failed to register layer: Error processing tar file(exit status 1): file exists

Is this related to this issue? I'm using https://github.com/banzaicloud/drone-kaniko, which seems to be using 0.19.0

[bkempe]

Also seeing this on 0.19.0 and 0.20.0 with --cache=true:

failed to register layer: Error processing tar file(exit status 1): failed to mknod("/lib/x86_64-linux-gnu/libgpg-error.so.0.21.0", S_IFCHR, 0): file exists

It works with --cache=false.

[kumprj]

Experiencing this "only works with cache=false" with v0.19.0 as well. Going to try 0.23 when I get a chance.

edit: I did not experience it with 0.23 using cache=true.

edit2: not consistently working with 0.23.

[guyguy333]

We're also experiencing the same issue with Kaniko v1.3.0

Failed to pull image "XXXXXXX": rpc error: code = Unknown desc = failed to register layer: Error processing tar file(exit status 1): failed to mknod("/home/node", S_IFCHR, 0): file exists

[ghost]

any update about this?

Same problem using v1.3.0. and -snapshotMode=redo

[vichle]

I'm experiencing this issue with v1.6.0-debug as well.

Failed to pull image "XYZ": rpc error: code = Unknown desc = failed to register layer: Error processing tar file(exit status 1): failed to mknod("/app/.npmrc", S_IFCHR, 0): file exists

Dockerfile

FROM mirror.gcr.io/library/node:12-alpine AS builder
ARG NPM_TOKEN
RUN echo "XYZ" > ~/.npmrc
WORKDIR /src
COPY .npmrc package.json package-lock.json /src/
RUN npm install
COPY . /src/

FROM mirror.gcr.io/library/node:12-alpine AS app
RUN mkdir /app && chown 1000:1000 /app
USER 1000
WORKDIR /app
COPY --from=builder /src /app/
CMD ["node","app.js"]

Works like a charm the first time when the cache is empty, consistently produces broken images on subsequent runs.

[jflambert]

Got this issue using executor:v1.22.0-debug

at the end of extraction:
failed to register layer: failed to mknod("/usr/include/x86_64-linux-gnu", S_IFCHR, 0): file exists

As others have noted, it's a rare occurrence. Rebuilding solves the issue.