Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression on image built with kaniko between v0.16.0 and v0.17.1 #1039

Closed
tanguydelignieresaccenture opened this issue Feb 7, 2020 · 27 comments
Closed

Regression on image built with kaniko between v0.16.0 and v0.17.1 #1039

tanguydelignieresaccenture opened this issue Feb 7, 2020 · 27 comments
Assignees
@cvgw
Labels
area/filesystems For all bugs related to kaniko container filesystems (mounting issues etc) fixed-needs-verfication in progress kind/bug Something isn't working priority/p0 Highest priority. Break user flow. We are actively looking at delivering it. regression

Comments

@tanguydelignieresaccenture
Copy link

tanguydelignieresaccenture commented Feb 7, 2020 •
edited
Loading

Actual behavior
A clear and concise description of what the bug is.

  • An image, based on redhat Universal Based Image 7, installing gcc, gcc-c++ and make with yum package manager, built with kaniko v0.16.0 is usable as a FROM in another kaniko build and allows to successfully build a new sub image.

  • An image, based on redhat Universal Based Image 7, installing gcc, gcc-c++ and make with yum package manager, built with kaniko v0.17.1 is usable as a FROM in another kaniko build but does not allows to successfully build a new sub image and throws the following error:
    error building image: error building stage: failed to get filesystem from image: removing whiteout .wh.dev: unlinkat //dev/pts/ptmx: operation not permitted

Expected behavior
A clear and concise description of what you expected to happen.
An image, based on redhat Universal Based Image 7, installing gcc, gcc-c++ and make with yum package manager, built with kaniko v0.17.1 is usable as a FROM in another kaniko build and allows to successfully build a new sub image.

To Reproduce
Steps to reproduce the behavior:

  1. Build the image from the Dockerfile "build1" with gcr.io/kaniko-project/executor:debug-v0.16.0
    docker run --rm --entrypoint "" -v /host/path/to/kaniko/config.json:/kaniko/.docker/config.json -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build1:/workspace gcr.io/kaniko-project/executor:debug-v0.16.0 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile --destination index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.16.0
    output:
$ docker run --rm --entrypoint "" -v /host/path/to/kaniko/config.json:/kaniko/.docker/config.json -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build1:/workspace gcr.io/kaniko-project/executor:debug-v0.16.0 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile --destination index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.16.0
INFO[0001] Resolved base name registry.access.redhat.com/ubi7/ubi:7.7-214 to registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0001] Resolved base name registry.access.redhat.com/ubi7/ubi:7.7-214 to registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0001] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0004] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0006] Built cross stage deps: map[]
INFO[0006] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0007] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0009] Executing 0 build triggers
INFO[0009] Unpacking rootfs as cmd RUN yum --disableplugin=subscription-manager update -y     && yum --disableplugin=subscription-manager install -y         gcc-4.8.5-39.el7         gcc-c++-4.8.5-39.el7         make-3.82-24.el7     && yum --disableplugin=subscription-manager clean all requires it.
INFO[0024] Taking snapshot of full filesystem...
INFO[0025] RUN yum --disableplugin=subscription-manager update -y     && yum --disableplugin=subscription-manager install -y         gcc-4.8.5-39.el7         gcc-c++-4.8.5-39.el7         make-3.82-24.el7     && yum --disableplugin=subscription-manager clean all
INFO[0025] cmd: /bin/sh
INFO[0025] args: [-c yum --disableplugin=subscription-manager update -y     && yum --disableplugin=subscription-manager install -y         gcc-4.8.5-39.el7         gcc-c++-4.8.5-39.el7         make-3.82-24.el7     && yum --disableplugin=subscription-manager clean all]
Loaded plugins: ovl, product-id, search-disabled-repos
Resolving Dependencies
--> Running transaction check
---> Package binutils.x86_64 0:2.27-41.base.el7_7.1 will be updated
---> Package binutils.x86_64 0:2.27-41.base.el7_7.2 will be an update
---> Package curl.x86_64 0:7.29.0-54.el7_7.1 will be updated
---> Package curl.x86_64 0:7.29.0-54.el7_7.2 will be an update
---> Package libcurl.x86_64 0:7.29.0-54.el7_7.1 will be updated
---> Package libcurl.x86_64 0:7.29.0-54.el7_7.2 will be an update
---> Package nss.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-softokn.x86_64 0:3.44.0-5.el7 will be updated
---> Package nss-softokn.x86_64 0:3.44.0-8.el7_7 will be an update
---> Package nss-softokn-freebl.x86_64 0:3.44.0-5.el7 will be updated
---> Package nss-softokn-freebl.x86_64 0:3.44.0-8.el7_7 will be an update
---> Package nss-sysinit.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss-sysinit.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-tools.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss-tools.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-util.x86_64 0:3.44.0-3.el7 will be updated
---> Package nss-util.x86_64 0:3.44.0-4.el7_7 will be an update
---> Package sqlite.x86_64 0:3.7.17-8.el7 will be updated
---> Package sqlite.x86_64 0:3.7.17-8.el7_7.1 will be an update
---> Package systemd.x86_64 0:219-67.el7_7.2 will be updated
---> Package systemd.x86_64 0:219-67.el7_7.3 will be an update
---> Package systemd-libs.x86_64 0:219-67.el7_7.2 will be updated
---> Package systemd-libs.x86_64 0:219-67.el7_7.3 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                 Arch        Version                   Repository  Size
================================================================================
Updating:
 binutils                x86_64      2.27-41.base.el7_7.2      ubi-7      5.9 M
 curl                    x86_64      7.29.0-54.el7_7.2         ubi-7      270 k
 libcurl                 x86_64      7.29.0-54.el7_7.2         ubi-7      223 k
 nss                     x86_64      3.44.0-7.el7_7            ubi-7      854 k
 nss-softokn             x86_64      3.44.0-8.el7_7            ubi-7      330 k
 nss-softokn-freebl      x86_64      3.44.0-8.el7_7            ubi-7      224 k
 nss-sysinit             x86_64      3.44.0-7.el7_7            ubi-7       65 k
 nss-tools               x86_64      3.44.0-7.el7_7            ubi-7      528 k
 nss-util                x86_64      3.44.0-4.el7_7            ubi-7       79 k
 sqlite                  x86_64      3.7.17-8.el7_7.1          ubi-7      394 k
 systemd                 x86_64      219-67.el7_7.3            ubi-7      5.1 M
 systemd-libs            x86_64      219-67.el7_7.3            ubi-7      411 k

Transaction Summary
================================================================================
Upgrade  12 Packages

Total download size: 14 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /var/cache/yum/x86_64/7Server/ubi-7/packages/curl-7.29.0-54.el7_7.2.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Public key for curl-7.29.0-54.el7_7.2.x86_64.rpm is not installed
--------------------------------------------------------------------------------
Total                                              4.0 MB/s |  14 MB  00:03
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Importing GPG key 0xFD431D51:
 Userid     : "Red Hat, Inc. (release key 2) <security@redhat.com>"
 Fingerprint: 567e 347a d004 4ade 55ba 8a5f 199e 2f91 fd43 1d51
 Package    : redhat-release-server-7.7-10.el7.x86_64 (@anaconda/7.7)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Importing GPG key 0x2FA658E0:
 Userid     : "Red Hat, Inc. (auxiliary key) <security@redhat.com>"
 Fingerprint: 43a6 e49c 4a38 f4be 9abf 2a53 4568 9c88 2fa6 58e0
 Package    : redhat-release-server-7.7-10.el7.x86_64 (@anaconda/7.7)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : nss-util-3.44.0-4.el7_7.x86_64                              1/24
  Updating   : nss-softokn-freebl-3.44.0-8.el7_7.x86_64                    2/24
  Updating   : systemd-libs-219-67.el7_7.3.x86_64                          3/24
  Updating   : sqlite-3.7.17-8.el7_7.1.x86_64                              4/24
  Updating   : nss-softokn-3.44.0-8.el7_7.x86_64                           5/24
  Updating   : nss-sysinit-3.44.0-7.el7_7.x86_64                           6/24
  Updating   : nss-3.44.0-7.el7_7.x86_64                                   7/24
  Updating   : libcurl-7.29.0-54.el7_7.2.x86_64                            8/24
  Updating   : curl-7.29.0-54.el7_7.2.x86_64                               9/24
  Updating   : systemd-219-67.el7_7.3.x86_64                              10/24
Failed to get D-Bus connection: Operation not permitted
  Updating   : nss-tools-3.44.0-7.el7_7.x86_64                            11/24
  Updating   : binutils-2.27-41.base.el7_7.2.x86_64                       12/24
install-info: No such file or directory for /usr/share/info/as.info.gz
install-info: No such file or directory for /usr/share/info/binutils.info.gz
install-info: No such file or directory for /usr/share/info/gprof.info.gz
install-info: No such file or directory for /usr/share/info/ld.info.gz
install-info: No such file or directory for /usr/share/info/standards.info.gz
  Cleanup    : nss-tools-3.44.0-4.el7.x86_64                              13/24
  Cleanup    : curl-7.29.0-54.el7_7.1.x86_64                              14/24
  Cleanup    : systemd-219-67.el7_7.2.x86_64                              15/24
  Cleanup    : libcurl-7.29.0-54.el7_7.1.x86_64                           16/24
  Cleanup    : nss-sysinit-3.44.0-4.el7.x86_64                            17/24
  Cleanup    : nss-3.44.0-4.el7.x86_64                                    18/24
  Cleanup    : nss-softokn-3.44.0-5.el7.x86_64                            19/24
  Cleanup    : nss-softokn-freebl-3.44.0-5.el7.x86_64                     20/24
  Cleanup    : nss-util-3.44.0-3.el7.x86_64                               21/24
  Cleanup    : sqlite-3.7.17-8.el7.x86_64                                 22/24
  Cleanup    : systemd-libs-219-67.el7_7.2.x86_64                         23/24
  Cleanup    : binutils-2.27-41.base.el7_7.1.x86_64                       24/24
  Verifying  : binutils-2.27-41.base.el7_7.2.x86_64                        1/24
  Verifying  : nss-tools-3.44.0-7.el7_7.x86_64                             2/24
  Verifying  : nss-3.44.0-7.el7_7.x86_64                                   3/24
  Verifying  : sqlite-3.7.17-8.el7_7.1.x86_64                              4/24
  Verifying  : systemd-libs-219-67.el7_7.3.x86_64                          5/24
  Verifying  : nss-sysinit-3.44.0-7.el7_7.x86_64                           6/24
  Verifying  : curl-7.29.0-54.el7_7.2.x86_64                               7/24
  Verifying  : systemd-219-67.el7_7.3.x86_64                               8/24
  Verifying  : libcurl-7.29.0-54.el7_7.2.x86_64                            9/24
  Verifying  : nss-util-3.44.0-4.el7_7.x86_64                             10/24
  Verifying  : nss-softokn-freebl-3.44.0-8.el7_7.x86_64                   11/24
  Verifying  : nss-softokn-3.44.0-8.el7_7.x86_64                          12/24
  Verifying  : nss-tools-3.44.0-4.el7.x86_64                              13/24
  Verifying  : systemd-libs-219-67.el7_7.2.x86_64                         14/24
  Verifying  : systemd-219-67.el7_7.2.x86_64                              15/24
  Verifying  : nss-util-3.44.0-3.el7.x86_64                               16/24
  Verifying  : curl-7.29.0-54.el7_7.1.x86_64                              17/24
  Verifying  : libcurl-7.29.0-54.el7_7.1.x86_64                           18/24
  Verifying  : nss-sysinit-3.44.0-4.el7.x86_64                            19/24
  Verifying  : binutils-2.27-41.base.el7_7.1.x86_64                       20/24
  Verifying  : nss-softokn-freebl-3.44.0-5.el7.x86_64                     21/24
  Verifying  : nss-3.44.0-4.el7.x86_64                                    22/24
  Verifying  : nss-softokn-3.44.0-5.el7.x86_64                            23/24
  Verifying  : sqlite-3.7.17-8.el7.x86_64                                 24/24

Updated:
  binutils.x86_64 0:2.27-41.base.el7_7.2
  curl.x86_64 0:7.29.0-54.el7_7.2
  libcurl.x86_64 0:7.29.0-54.el7_7.2
  nss.x86_64 0:3.44.0-7.el7_7
  nss-softokn.x86_64 0:3.44.0-8.el7_7
  nss-softokn-freebl.x86_64 0:3.44.0-8.el7_7
  nss-sysinit.x86_64 0:3.44.0-7.el7_7
  nss-tools.x86_64 0:3.44.0-7.el7_7
  nss-util.x86_64 0:3.44.0-4.el7_7
  sqlite.x86_64 0:3.7.17-8.el7_7.1
  systemd.x86_64 0:219-67.el7_7.3
  systemd-libs.x86_64 0:219-67.el7_7.3

Complete!
Loaded plugins: ovl, product-id, search-disabled-repos
Resolving Dependencies
--> Running transaction check
---> Package gcc.x86_64 0:4.8.5-39.el7 will be installed
--> Processing Dependency: cpp = 4.8.5-39.el7 for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libgomp = 4.8.5-39.el7 for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libgomp.so.1()(64bit) for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-4.8.5-39.el7.x86_64
---> Package gcc-c++.x86_64 0:4.8.5-39.el7 will be installed
--> Processing Dependency: libstdc++-devel = 4.8.5-39.el7 for package: gcc-c++-4.8.5-39.el7.x86_64
---> Package make.x86_64 1:3.82-24.el7 will be installed
--> Running transaction check
---> Package cpp.x86_64 0:4.8.5-39.el7 will be installed
---> Package glibc-devel.x86_64 0:2.17-292.el7 will be installed
--> Processing Dependency: glibc-headers = 2.17-292.el7 for package: glibc-devel-2.17-292.el7.x86_64
--> Processing Dependency: glibc-headers for package: glibc-devel-2.17-292.el7.x86_64
---> Package libgomp.x86_64 0:4.8.5-39.el7 will be installed
---> Package libmpc.x86_64 0:1.0.1-3.el7 will be installed
---> Package libstdc++-devel.x86_64 0:4.8.5-39.el7 will be installed
---> Package mpfr.x86_64 0:3.1.1-4.el7 will be installed
--> Running transaction check
---> Package glibc-headers.x86_64 0:2.17-292.el7 will be installed
--> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.17-292.el7.x86_64
--> Processing Dependency: kernel-headers for package: glibc-headers-2.17-292.el7.x86_64
--> Running transaction check
---> Package kernel-headers.x86_64 0:3.10.0-1062.12.1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package              Arch        Version                      Repository  Size
================================================================================
Installing:
 gcc                  x86_64      4.8.5-39.el7                 ubi-7       16 M
 gcc-c++              x86_64      4.8.5-39.el7                 ubi-7      7.2 M
 make                 x86_64      1:3.82-24.el7                ubi-7      421 k
Installing for dependencies:
 cpp                  x86_64      4.8.5-39.el7                 ubi-7      6.0 M
 glibc-devel          x86_64      2.17-292.el7                 ubi-7      1.1 M
 glibc-headers        x86_64      2.17-292.el7                 ubi-7      687 k
 kernel-headers       x86_64      3.10.0-1062.12.1.el7         ubi-7      8.7 M
 libgomp              x86_64      4.8.5-39.el7                 ubi-7      158 k
 libmpc               x86_64      1.0.1-3.el7                  ubi-7       51 k
 libstdc++-devel      x86_64      4.8.5-39.el7                 ubi-7      1.5 M
 mpfr                 x86_64      3.1.1-4.el7                  ubi-7      203 k

Transaction Summary
================================================================================
Install  3 Packages (+8 Dependent packages)

Total download size: 42 M
Installed size: 86 M
Downloading packages:
--------------------------------------------------------------------------------
Total                                              5.8 MB/s |  42 MB  00:07
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : mpfr-3.1.1-4.el7.x86_64                                     1/11
  Installing : libmpc-1.0.1-3.el7.x86_64                                   2/11
  Installing : cpp-4.8.5-39.el7.x86_64                                     3/11
  Installing : kernel-headers-3.10.0-1062.12.1.el7.x86_64                  4/11
  Installing : glibc-headers-2.17-292.el7.x86_64                           5/11
  Installing : glibc-devel-2.17-292.el7.x86_64                             6/11
  Installing : libstdc++-devel-4.8.5-39.el7.x86_64                         7/11
  Installing : libgomp-4.8.5-39.el7.x86_64                                 8/11
  Installing : gcc-4.8.5-39.el7.x86_64                                     9/11
  Installing : gcc-c++-4.8.5-39.el7.x86_64                                10/11
  Installing : 1:make-3.82-24.el7.x86_64                                  11/11
  Verifying  : glibc-devel-2.17-292.el7.x86_64                             1/11
  Verifying  : gcc-c++-4.8.5-39.el7.x86_64                                 2/11
  Verifying  : 1:make-3.82-24.el7.x86_64                                   3/11
  Verifying  : mpfr-3.1.1-4.el7.x86_64                                     4/11
  Verifying  : libgomp-4.8.5-39.el7.x86_64                                 5/11
  Verifying  : libmpc-1.0.1-3.el7.x86_64                                   6/11
  Verifying  : cpp-4.8.5-39.el7.x86_64                                     7/11
  Verifying  : gcc-4.8.5-39.el7.x86_64                                     8/11
  Verifying  : libstdc++-devel-4.8.5-39.el7.x86_64                         9/11
  Verifying  : kernel-headers-3.10.0-1062.12.1.el7.x86_64                 10/11
  Verifying  : glibc-headers-2.17-292.el7.x86_64                          11/11

Installed:
  gcc.x86_64 0:4.8.5-39.el7            gcc-c++.x86_64 0:4.8.5-39.el7
  make.x86_64 1:3.82-24.el7

Dependency Installed:
  cpp.x86_64 0:4.8.5-39.el7
  glibc-devel.x86_64 0:2.17-292.el7
  glibc-headers.x86_64 0:2.17-292.el7
  kernel-headers.x86_64 0:3.10.0-1062.12.1.el7
  libgomp.x86_64 0:4.8.5-39.el7
  libmpc.x86_64 0:1.0.1-3.el7
  libstdc++-devel.x86_64 0:4.8.5-39.el7
  mpfr.x86_64 0:3.1.1-4.el7

Complete!
Loaded plugins: ovl, product-id, search-disabled-repos
Cleaning repos: ubi-7 ubi-7-rhah ubi-7-server-devtools-rpms
              : ubi-7-server-extras-rpms ubi-7-server-optional-rpms
              : ubi-server-rhscl-7-rpms
INFO[0065] Taking snapshot of full filesystem...
INFO[0065] Adding whiteout for /var/lib/yum/yumdb/n/939bd19e4f12fb2195f9571120092f8ae8d57dc1-nss-softokn-freebl-3.44.0-5.el7-x86_64
INFO[0065] Adding whiteout for /var/lib/yum/rpmdb-indexes/conflicts
INFO[0065] Adding whiteout for /usr/lib64/libbfd-2.27-41.base.el7_7.1.so
INFO[0065] Adding whiteout for /var/lib/yum/yumdb/b/4c369229881b300e3e564b9be2679d24a74368fb-binutils-2.27-41.base.el7_7.1-x86_64
INFO[0065] Adding whiteout for /var/lib/yum/yumdb/s/7488ccedb864f45573f8f76b7cf76141f97eb35f-sqlite-3.7.17-8.el7-x86_64
INFO[0065] Adding whiteout for /var/lib/yum/rpmdb-indexes/version
INFO[0065] Adding whiteout for /usr/lib64/libopcodes-2.27-41.base.el7_7.1.so
INFO[0065] Adding whiteout for /var/lib/yum/yumdb/l/317aba0360ab628f9efdd4f0b4b4a313039c89a2-libcurl-7.29.0-54.el7_7.1-x86_64
INFO[0065] Adding whiteout for /var/lib/yum/yumdb/n/1d886c11b8478068d9f8bdd7764b94501635306d-nss-sysinit-3.44.0-4.el7-x86_64
INFO[0065] Adding whiteout for /var/lib/yum/rpmdb-indexes/obsoletes
INFO[0065] Adding whiteout for /var/lib/yum/rpmdb-indexes/file-requires
INFO[0065] Adding whiteout for /var/lib/yum/yumdb/n/9c358fa289102ab9c0d28decb5de985391517ab5-nss-3.44.0-4.el7-x86_64
INFO[0065] Adding whiteout for /var/lib/yum/yumdb/s/8ea510704120f3378d00b516e3ffbfc7ee67f66d-systemd-219-67.el7_7.2-x86_64
INFO[0065] Adding whiteout for /var/lib/yum/yumdb/n/ce3e8fdc5b9723da97371f4e65ec44e8490aa84a-nss-softokn-3.44.0-5.el7-x86_64
INFO[0065] Adding whiteout for /var/lib/yum/yumdb/n/d700496ea41582ea9596d71c2c797a7258e98284-nss-util-3.44.0-3.el7-x86_64
INFO[0065] Adding whiteout for /var/lib/yum/yumdb/c/13e4cfa6c01c54f7bd005b45b2f86e023e0bf3d9-curl-7.29.0-54.el7_7.1-x86_64
INFO[0065] Adding whiteout for /var/lib/yum/yumdb/n/fd1dd9f95ea47fbb41a1b2c62b60edefa0e491c3-nss-tools-3.44.0-4.el7-x86_64
INFO[0065] Adding whiteout for /var/lib/yum/yumdb/s/3924e2a2ddea4327fa580f5430e3fc23d283c145-systemd-libs-219-67.el7_7.2-x86_64
INFO[0065] Adding whiteout for /var/lib/yum/rpmdb-indexes/pkgtups-checksums
  1. Build the image from the same Dockerfile "build1" with gcr.io/kaniko-project/executor:debug-v0.17.1
    docker run --rm --entrypoint "" -v /host/path/to/kaniko/config.json:/kaniko/.docker/config.json -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build1:/workspace gcr.io/kaniko-project/executor:debug-v0.17.1 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile --destination index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.17.1
    output:
$ docker run --rm --entrypoint "" -v /host/path/to/kaniko/config.json:/kaniko/.docker/config.json -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build1:/workspace gcr.io/kaniko-project/executor:debug-v0.17.1 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile --destination index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.17.1
INFO[0001] Resolved base name registry.access.redhat.com/ubi7/ubi:7.7-214 to registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0001] Resolved base name registry.access.redhat.com/ubi7/ubi:7.7-214 to registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0001] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0003] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0005] Built cross stage deps: map[]
INFO[0005] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0006] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0008] Unpacking rootfs as cmd RUN yum --disableplugin=subscription-manager update -y     && yum --disableplugin=subscription-manager install -y         gcc-4.8.5-39.el7         gcc-c++-4.8.5-39.el7         make-3.82-24.el7     && yum --disableplugin=subscription-manager clean all requires it.
INFO[0029] Taking snapshot of full filesystem...
INFO[0030] RUN yum --disableplugin=subscription-manager update -y     && yum --disableplugin=subscription-manager install -y         gcc-4.8.5-39.el7         gcc-c++-4.8.5-39.el7         make-3.82-24.el7     && yum --disableplugin=subscription-manager clean all
INFO[0030] cmd: /bin/sh
INFO[0030] args: [-c yum --disableplugin=subscription-manager update -y     && yum --disableplugin=subscription-manager install -y         gcc-4.8.5-39.el7         gcc-c++-4.8.5-39.el7         make-3.82-24.el7     && yum --disableplugin=subscription-manager clean all]
Loaded plugins: ovl, product-id, search-disabled-repos
Resolving Dependencies
--> Running transaction check
---> Package binutils.x86_64 0:2.27-41.base.el7_7.1 will be updated
---> Package binutils.x86_64 0:2.27-41.base.el7_7.2 will be an update
---> Package curl.x86_64 0:7.29.0-54.el7_7.1 will be updated
---> Package curl.x86_64 0:7.29.0-54.el7_7.2 will be an update
---> Package libcurl.x86_64 0:7.29.0-54.el7_7.1 will be updated
---> Package libcurl.x86_64 0:7.29.0-54.el7_7.2 will be an update
---> Package nss.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-softokn.x86_64 0:3.44.0-5.el7 will be updated
---> Package nss-softokn.x86_64 0:3.44.0-8.el7_7 will be an update
---> Package nss-softokn-freebl.x86_64 0:3.44.0-5.el7 will be updated
---> Package nss-softokn-freebl.x86_64 0:3.44.0-8.el7_7 will be an update
---> Package nss-sysinit.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss-sysinit.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-tools.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss-tools.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-util.x86_64 0:3.44.0-3.el7 will be updated
---> Package nss-util.x86_64 0:3.44.0-4.el7_7 will be an update
---> Package sqlite.x86_64 0:3.7.17-8.el7 will be updated
---> Package sqlite.x86_64 0:3.7.17-8.el7_7.1 will be an update
---> Package systemd.x86_64 0:219-67.el7_7.2 will be updated
---> Package systemd.x86_64 0:219-67.el7_7.3 will be an update
---> Package systemd-libs.x86_64 0:219-67.el7_7.2 will be updated
---> Package systemd-libs.x86_64 0:219-67.el7_7.3 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                 Arch        Version                   Repository  Size
================================================================================
Updating:
 binutils                x86_64      2.27-41.base.el7_7.2      ubi-7      5.9 M
 curl                    x86_64      7.29.0-54.el7_7.2         ubi-7      270 k
 libcurl                 x86_64      7.29.0-54.el7_7.2         ubi-7      223 k
 nss                     x86_64      3.44.0-7.el7_7            ubi-7      854 k
 nss-softokn             x86_64      3.44.0-8.el7_7            ubi-7      330 k
 nss-softokn-freebl      x86_64      3.44.0-8.el7_7            ubi-7      224 k
 nss-sysinit             x86_64      3.44.0-7.el7_7            ubi-7       65 k
 nss-tools               x86_64      3.44.0-7.el7_7            ubi-7      528 k
 nss-util                x86_64      3.44.0-4.el7_7            ubi-7       79 k
 sqlite                  x86_64      3.7.17-8.el7_7.1          ubi-7      394 k
 systemd                 x86_64      219-67.el7_7.3            ubi-7      5.1 M
 systemd-libs            x86_64      219-67.el7_7.3            ubi-7      411 k

Transaction Summary
================================================================================
Upgrade  12 Packages

Total download size: 14 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /var/cache/yum/x86_64/7Server/ubi-7/packages/curl-7.29.0-54.el7_7.2.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Public key for curl-7.29.0-54.el7_7.2.x86_64.rpm is not installed
--------------------------------------------------------------------------------
Total                                              3.7 MB/s |  14 MB  00:03
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Importing GPG key 0xFD431D51:
 Userid     : "Red Hat, Inc. (release key 2) <security@redhat.com>"
 Fingerprint: 567e 347a d004 4ade 55ba 8a5f 199e 2f91 fd43 1d51
 Package    : redhat-release-server-7.7-10.el7.x86_64 (@anaconda/7.7)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Importing GPG key 0x2FA658E0:
 Userid     : "Red Hat, Inc. (auxiliary key) <security@redhat.com>"
 Fingerprint: 43a6 e49c 4a38 f4be 9abf 2a53 4568 9c88 2fa6 58e0
 Package    : redhat-release-server-7.7-10.el7.x86_64 (@anaconda/7.7)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : nss-util-3.44.0-4.el7_7.x86_64                              1/24
  Updating   : nss-softokn-freebl-3.44.0-8.el7_7.x86_64                    2/24
  Updating   : systemd-libs-219-67.el7_7.3.x86_64                          3/24
  Updating   : sqlite-3.7.17-8.el7_7.1.x86_64                              4/24
  Updating   : nss-softokn-3.44.0-8.el7_7.x86_64                           5/24
  Updating   : nss-sysinit-3.44.0-7.el7_7.x86_64                           6/24
  Updating   : nss-3.44.0-7.el7_7.x86_64                                   7/24
  Updating   : libcurl-7.29.0-54.el7_7.2.x86_64                            8/24
  Updating   : curl-7.29.0-54.el7_7.2.x86_64                               9/24
  Updating   : systemd-219-67.el7_7.3.x86_64                              10/24
Failed to get D-Bus connection: Operation not permitted
  Updating   : nss-tools-3.44.0-7.el7_7.x86_64                            11/24
  Updating   : binutils-2.27-41.base.el7_7.2.x86_64                       12/24
install-info: No such file or directory for /usr/share/info/as.info.gz
install-info: No such file or directory for /usr/share/info/binutils.info.gz
install-info: No such file or directory for /usr/share/info/gprof.info.gz
install-info: No such file or directory for /usr/share/info/ld.info.gz
install-info: No such file or directory for /usr/share/info/standards.info.gz
  Cleanup    : nss-tools-3.44.0-4.el7.x86_64                              13/24
  Cleanup    : curl-7.29.0-54.el7_7.1.x86_64                              14/24
  Cleanup    : systemd-219-67.el7_7.2.x86_64                              15/24
  Cleanup    : libcurl-7.29.0-54.el7_7.1.x86_64                           16/24
  Cleanup    : nss-sysinit-3.44.0-4.el7.x86_64                            17/24
  Cleanup    : nss-3.44.0-4.el7.x86_64                                    18/24
  Cleanup    : nss-softokn-3.44.0-5.el7.x86_64                            19/24
  Cleanup    : nss-softokn-freebl-3.44.0-5.el7.x86_64                     20/24
  Cleanup    : nss-util-3.44.0-3.el7.x86_64                               21/24
  Cleanup    : sqlite-3.7.17-8.el7.x86_64                                 22/24
  Cleanup    : systemd-libs-219-67.el7_7.2.x86_64                         23/24
  Cleanup    : binutils-2.27-41.base.el7_7.1.x86_64                       24/24
  Verifying  : binutils-2.27-41.base.el7_7.2.x86_64                        1/24
  Verifying  : nss-tools-3.44.0-7.el7_7.x86_64                             2/24
  Verifying  : nss-3.44.0-7.el7_7.x86_64                                   3/24
  Verifying  : sqlite-3.7.17-8.el7_7.1.x86_64                              4/24
  Verifying  : systemd-libs-219-67.el7_7.3.x86_64                          5/24
  Verifying  : nss-sysinit-3.44.0-7.el7_7.x86_64                           6/24
  Verifying  : curl-7.29.0-54.el7_7.2.x86_64                               7/24
  Verifying  : systemd-219-67.el7_7.3.x86_64                               8/24
  Verifying  : libcurl-7.29.0-54.el7_7.2.x86_64                            9/24
  Verifying  : nss-util-3.44.0-4.el7_7.x86_64                             10/24
  Verifying  : nss-softokn-freebl-3.44.0-8.el7_7.x86_64                   11/24
  Verifying  : nss-softokn-3.44.0-8.el7_7.x86_64                          12/24
  Verifying  : nss-tools-3.44.0-4.el7.x86_64                              13/24
  Verifying  : systemd-libs-219-67.el7_7.2.x86_64                         14/24
  Verifying  : systemd-219-67.el7_7.2.x86_64                              15/24
  Verifying  : nss-util-3.44.0-3.el7.x86_64                               16/24
  Verifying  : curl-7.29.0-54.el7_7.1.x86_64                              17/24
  Verifying  : libcurl-7.29.0-54.el7_7.1.x86_64                           18/24
  Verifying  : nss-sysinit-3.44.0-4.el7.x86_64                            19/24
  Verifying  : binutils-2.27-41.base.el7_7.1.x86_64                       20/24
  Verifying  : nss-softokn-freebl-3.44.0-5.el7.x86_64                     21/24
  Verifying  : nss-3.44.0-4.el7.x86_64                                    22/24
  Verifying  : nss-softokn-3.44.0-5.el7.x86_64                            23/24
  Verifying  : sqlite-3.7.17-8.el7.x86_64                                 24/24

Updated:
  binutils.x86_64 0:2.27-41.base.el7_7.2
  curl.x86_64 0:7.29.0-54.el7_7.2
  libcurl.x86_64 0:7.29.0-54.el7_7.2
  nss.x86_64 0:3.44.0-7.el7_7
  nss-softokn.x86_64 0:3.44.0-8.el7_7
  nss-softokn-freebl.x86_64 0:3.44.0-8.el7_7
  nss-sysinit.x86_64 0:3.44.0-7.el7_7
  nss-tools.x86_64 0:3.44.0-7.el7_7
  nss-util.x86_64 0:3.44.0-4.el7_7
  sqlite.x86_64 0:3.7.17-8.el7_7.1
  systemd.x86_64 0:219-67.el7_7.3
  systemd-libs.x86_64 0:219-67.el7_7.3

Complete!
Loaded plugins: ovl, product-id, search-disabled-repos
Resolving Dependencies
--> Running transaction check
---> Package gcc.x86_64 0:4.8.5-39.el7 will be installed
--> Processing Dependency: cpp = 4.8.5-39.el7 for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libgomp = 4.8.5-39.el7 for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libgomp.so.1()(64bit) for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-4.8.5-39.el7.x86_64
---> Package gcc-c++.x86_64 0:4.8.5-39.el7 will be installed
--> Processing Dependency: libstdc++-devel = 4.8.5-39.el7 for package: gcc-c++-4.8.5-39.el7.x86_64
---> Package make.x86_64 1:3.82-24.el7 will be installed
--> Running transaction check
---> Package cpp.x86_64 0:4.8.5-39.el7 will be installed
---> Package glibc-devel.x86_64 0:2.17-292.el7 will be installed
--> Processing Dependency: glibc-headers = 2.17-292.el7 for package: glibc-devel-2.17-292.el7.x86_64
--> Processing Dependency: glibc-headers for package: glibc-devel-2.17-292.el7.x86_64
---> Package libgomp.x86_64 0:4.8.5-39.el7 will be installed
---> Package libmpc.x86_64 0:1.0.1-3.el7 will be installed
---> Package libstdc++-devel.x86_64 0:4.8.5-39.el7 will be installed
---> Package mpfr.x86_64 0:3.1.1-4.el7 will be installed
--> Running transaction check
---> Package glibc-headers.x86_64 0:2.17-292.el7 will be installed
--> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.17-292.el7.x86_64
--> Processing Dependency: kernel-headers for package: glibc-headers-2.17-292.el7.x86_64
--> Running transaction check
---> Package kernel-headers.x86_64 0:3.10.0-1062.12.1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package              Arch        Version                      Repository  Size
================================================================================
Installing:
 gcc                  x86_64      4.8.5-39.el7                 ubi-7       16 M
 gcc-c++              x86_64      4.8.5-39.el7                 ubi-7      7.2 M
 make                 x86_64      1:3.82-24.el7                ubi-7      421 k
Installing for dependencies:
 cpp                  x86_64      4.8.5-39.el7                 ubi-7      6.0 M
 glibc-devel          x86_64      2.17-292.el7                 ubi-7      1.1 M
 glibc-headers        x86_64      2.17-292.el7                 ubi-7      687 k
 kernel-headers       x86_64      3.10.0-1062.12.1.el7         ubi-7      8.7 M
 libgomp              x86_64      4.8.5-39.el7                 ubi-7      158 k
 libmpc               x86_64      1.0.1-3.el7                  ubi-7       51 k
 libstdc++-devel      x86_64      4.8.5-39.el7                 ubi-7      1.5 M
 mpfr                 x86_64      3.1.1-4.el7                  ubi-7      203 k

Transaction Summary
================================================================================
Install  3 Packages (+8 Dependent packages)

Total download size: 42 M
Installed size: 86 M
Downloading packages:
--------------------------------------------------------------------------------
Total                                              7.0 MB/s |  42 MB  00:06
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : mpfr-3.1.1-4.el7.x86_64                                     1/11
  Installing : libmpc-1.0.1-3.el7.x86_64                                   2/11
  Installing : cpp-4.8.5-39.el7.x86_64                                     3/11
  Installing : kernel-headers-3.10.0-1062.12.1.el7.x86_64                  4/11
  Installing : glibc-headers-2.17-292.el7.x86_64                           5/11
  Installing : glibc-devel-2.17-292.el7.x86_64                             6/11
  Installing : libstdc++-devel-4.8.5-39.el7.x86_64                         7/11
  Installing : libgomp-4.8.5-39.el7.x86_64                                 8/11
  Installing : gcc-4.8.5-39.el7.x86_64                                     9/11
  Installing : gcc-c++-4.8.5-39.el7.x86_64                                10/11
  Installing : 1:make-3.82-24.el7.x86_64                                  11/11
  Verifying  : glibc-devel-2.17-292.el7.x86_64                             1/11
  Verifying  : gcc-c++-4.8.5-39.el7.x86_64                                 2/11
  Verifying  : 1:make-3.82-24.el7.x86_64                                   3/11
  Verifying  : mpfr-3.1.1-4.el7.x86_64                                     4/11
  Verifying  : libgomp-4.8.5-39.el7.x86_64                                 5/11
  Verifying  : libmpc-1.0.1-3.el7.x86_64                                   6/11
  Verifying  : cpp-4.8.5-39.el7.x86_64                                     7/11
  Verifying  : gcc-4.8.5-39.el7.x86_64                                     8/11
  Verifying  : libstdc++-devel-4.8.5-39.el7.x86_64                         9/11
  Verifying  : kernel-headers-3.10.0-1062.12.1.el7.x86_64                 10/11
  Verifying  : glibc-headers-2.17-292.el7.x86_64                          11/11

Installed:
  gcc.x86_64 0:4.8.5-39.el7            gcc-c++.x86_64 0:4.8.5-39.el7
  make.x86_64 1:3.82-24.el7

Dependency Installed:
  cpp.x86_64 0:4.8.5-39.el7
  glibc-devel.x86_64 0:2.17-292.el7
  glibc-headers.x86_64 0:2.17-292.el7
  kernel-headers.x86_64 0:3.10.0-1062.12.1.el7
  libgomp.x86_64 0:4.8.5-39.el7
  libmpc.x86_64 0:1.0.1-3.el7
  libstdc++-devel.x86_64 0:4.8.5-39.el7
  mpfr.x86_64 0:3.1.1-4.el7

Complete!
Loaded plugins: ovl, product-id, search-disabled-repos
Cleaning repos: ubi-7 ubi-7-rhah ubi-7-server-devtools-rpms
              : ubi-7-server-extras-rpms ubi-7-server-optional-rpms
              : ubi-server-rhscl-7-rpms
INFO[0066] Taking snapshot of full filesystem...
  1. Try to build the image from the Dockerfile "build2" with gcr.io/kaniko-project/executor:debug-v0.17.1 (or gcr.io/kaniko-project/executor:debug-v0.16.0, same results), using the image "build1" with tag debug-v0.16.0 (see Dockerfile.debug-v0.16.0) > OK
    docker run --rm --entrypoint "" -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build2:/workspace gcr.io/kaniko-project/executor:debug-v0.17.1 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile.debug-v0.16.0 --no-push
    output:
$ docker run --rm --entrypoint "" -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build2:/workspace gcr.io/kaniko-project/executor:debug-v0.17.1 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile.debug-v0.16.0 --no-push
INFO[0000] Resolved base name index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.16.0 to index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.16.0
INFO[0000] Resolved base name index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.16.0 to index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.16.0
INFO[0000] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.16.0
INFO[0002] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.16.0
INFO[0007] Built cross stage deps: map[]
INFO[0007] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.16.0
INFO[0008] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.16.0
INFO[0009] Unpacking rootfs as cmd RUN echo test requires it.
INFO[0024] Taking snapshot of full filesystem...
INFO[0025] RUN echo test
INFO[0025] cmd: /bin/sh
INFO[0025] args: [-c echo test]
test
INFO[0025] Taking snapshot of full filesystem...
INFO[0026] Skipping push to container registry due to --no-push flag
  1. Try to build the image from the Dockerfile "build2" with gcr.io/kaniko-project/executor:debug-v0.17.1 (or gcr.io/kaniko-project/executor:debug-v0.16.0, same results), using the image "build1" with tag debug-v0.17.1 (see Dockerfile.debug-v0.17.1) > KO
    docker run --rm --entrypoint "" -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build2:/workspace gcr.io/kaniko-project/executor:debug-v0.17.1 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile.debug-v0.17.1 --no-push
    output:
$ docker run --rm --entrypoint "" -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build2:/workspace gcr.io/kaniko-project/executor:debug-v0.17.1 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile.debug-v0.17.1 --no-push
INFO[0000] Resolved base name index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.17.1 to index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.17.1
INFO[0000] Resolved base name index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.17.1 to index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.17.1
INFO[0000] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.17.1
INFO[0001] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.17.1
INFO[0003] Built cross stage deps: map[]
INFO[0003] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.17.1
INFO[0003] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.17.1
INFO[0004] Unpacking rootfs as cmd RUN echo test requires it.
error building image: error building stage: failed to get filesystem from image: removing whiteout .wh.dev: unlinkat //dev/pts/ptmx: operation not permitted

Additional Information

  • Dockerfile
    Please provide either the Dockerfile you're trying to build or one that can reproduce this error.

Dockerfile "build1"

FROM registry.access.redhat.com/ubi7/ubi:7.7-214

# Install GCC, GCC-C++ and make libraries for build environment
# Then clean caches
RUN yum --disableplugin=subscription-manager update -y \
    && yum --disableplugin=subscription-manager install -y \
        gcc-4.8.5-39.el7 \
        gcc-c++-4.8.5-39.el7 \
        make-3.82-24.el7 \
    && yum --disableplugin=subscription-manager clean all

Dockerfile.debug-v0.16.0 "build2"

FROM index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.16.0

RUN echo test

Dockerfile.debug-v0.17.1 "build2"

FROM index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.17.1

RUN echo test
  • Build Context
    Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)
    No specific files
  • Kaniko Image (fully qualified with digest)
    gcr.io/kaniko-project/executor:debug-v0.16.0
    gcr.io/kaniko-project/executor:debug-v0.17.1

Please note:

  • That "Build1" build with kaniko v0.16.0 has several Adding whiteout not present in build with kaniko v0.17.1

  • "Build1" images index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.16.0 and index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.17.1 have been pushed to my dockerhub if you need them.

  • I can't tell if this issue is related to error building image: error building stage GCC #763, you may have a check too.

Triage Notes for the Maintainers

Description Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile
@tejal29 tejal29 mentioned this issue Feb 7, 2020
Closed
@cvgw cvgw self-assigned this Feb 8, 2020
@cvgw
Copy link
Contributor

cvgw commented Feb 8, 2020

I'm able to repro (thank you @tanguydelignieresaccenture for the detailed report)

I'm seeing some strange output in a debugger at util/fs_util.go:158

print path => "/root/buildinfo/.wh..wh..opq"
print path => "/run/secrets/.wh..wh..opq"

It looks like files are being double whiteouted? I can't image a scenario where we would expect a file to have double .wh. prefixes like the examples above.

@cvgw cvgw added area/filesystems For all bugs related to kaniko container filesystems (mounting issues etc) kind/bug Something isn't working priority/p0 Highest priority. Break user flow. We are actively looking at delivering it. regression in progress labels Feb 8, 2020
@cvgw
Copy link
Contributor

cvgw commented Feb 8, 2020 •
edited
Loading

at util/fs_util.go

if err := os.RemoveAllI(filiepath.Join(dir, name)); err != nil {

dir => /
name => dev
path => /.wh.dev

os.RemoveAll("/dev") // OH NO!

ya trying to remove /dev is gonna be a bad time

@cvgw
Copy link
Contributor

cvgw commented Feb 8, 2020

/dev is listed in the whitelists (I'm not sure why, is it a volume?)

{Path: "/dev", PrefixMatchOnly: false}

If /dev is whitelisted then I would assume it doesn't get whiteout, but it seems that it does.

@cvgw
Copy link
Contributor

cvgw commented Feb 8, 2020

Looks like /dev gets whitelisted as part of this /proc logic (that I didn't know we had)

> github.com/GoogleContainerTools/kaniko/pkg/util.DetectFilesystemWhitelist() .go/src/github.com/GoogleContainerTools/kaniko/pkg/util/fs_util.go:407 (PC: 0x17128df)
   402:         }
   403:         defer f.Close()
   404:         reader := bufio.NewReader(f)
   405:         for {
   406:                 line, err := reader.ReadString('\n')
=> 407:                 logrus.Tracef("Read the following line from %s: %s", path, line)
   408:                 if err != nil && err != io.EOF {
   409:                         return err
   410:                 }
   411:                 lineArr := strings.Split(line, " ")
   412:                 if len(lineArr) < 5 {
(dlv) print line
"951 949 0:82 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=...+4 more"

@cvgw
Copy link
Contributor

cvgw commented Feb 8, 2020

I have a theory about this issue

In #971 this code was added

kaniko/pkg/snapshot/snapshot.go

Line 183 in 8d9e6b8

files, err := filesWithLinks(path)

In the issue example, this causes /etc/systemd/system/systemd-logind.service and its resolved link /dev/null to be appended to the filesToAdd list. The issue with this is that /dev is whitelisted and all of it's subdirectories should be skipped.

I think the solution here would be moving the filesWithLinks logic to the Walk block

kaniko/pkg/snapshot/snapshot.go

Line 133 in 8d9e6b8

godirwalk.Walk(s.directory, &godirwalk.Options{

or duplicating the whitelist checks in filesWithLinks

@tejal29 @samos123 WDYT?

This was referenced Feb 8, 2020
Closed
Closed
Open
@cvgw
Copy link
Contributor

cvgw commented Feb 8, 2020 •
edited
Loading

@tanguydelignieresaccenture can you try tag 1039-fix-test or debug-1039-fix-test?

I've just pushed these two tags and I'd like to test if they fix the issue you are seeing. Thanks!

@RemcodM
Copy link

RemcodM commented Feb 8, 2020 •
edited
Loading

@cvgw After your observations that the issues #1038 #1024 and #1028 might all be related, and my involvement with the issues #1024 and #1028, I figured I could do a test on my own.

I used the following Docker image:

# Eoan amd64 image
FROM ubuntu:eoan

ADD build /usr/local/bin/

ADD backports /backports

RUN apt-get update && \
  export DEBIAN_FRONTEND=noninteractive && \
  apt-get dist-upgrade -y && \
  apt-get install -qy sudo build-essential devscripts equivs lsb-release apt-transport-https && \
  cd /backports && \
  apt-get install -qy ./libdebhelper-perl_12.8_all.deb ./debhelper_12.8_all.deb ./dh-systemd_12.8_all.deb && \
  apt-get clean -y && \
  useradd -c "Build User" -d /tmp -M -s /bin/bash builduser

I have build this Docker image 4 times: One time using tag debug-v0.16.0, one time using debug-v0.17.0, one time using debug-v0.17.1 and one last time using debug-1039-fix-test. I have build this exact Dockerfile succesfully before on debug-v0.16.0 and I have seen builds of this file gone wrong on debug-v0.17.0 before.

These are the results:

# docker pull registry.tworem.nl/remcodm/kaniko-test/eoan/amd64:v0.16.0
v0.16.0: Pulling from remcodm/kaniko-test/eoan/amd64
93956c6f8d9e: Already exists 
46bddb84d1c5: Already exists 
15fa85048576: Already exists 
8aa40341c4fa: Already exists 
08d396ded471: Pull complete 
4ca40aa44069: Pull complete 
96ef31c5e575: Pull complete 
Digest: sha256:7305979dfc0692645ebc72bd7e81c31bd2c974a457f7698cf1630b6e2441d65a
Status: Downloaded newer image for registry.tworem.nl/remcodm/kaniko-test/eoan/amd64:v0.16.0
registry.tworem.nl/remcodm/kaniko-test/eoan/amd64:v0.16.0

# docker pull registry.tworem.nl/remcodm/kaniko-test/eoan/amd64:v0.17.0
v0.17.0: Pulling from remcodm/kaniko-test/eoan/amd64
93956c6f8d9e: Already exists 
46bddb84d1c5: Already exists 
15fa85048576: Already exists 
8aa40341c4fa: Already exists 
c72f51425616: Pull complete 
e7ae8616318f: Pull complete 
73ba141bedd1: Extracting [==================================================>]  177.3MB/177.3MB
failed to register layer: ApplyLayer exit status 1 stdout:  stderr: link /lib/systemd /usr/lib/systemd: no such file or directory

# docker pull registry.tworem.nl/remcodm/kaniko-test/eoan/amd64:v0.17.1
v0.17.1: Pulling from remcodm/kaniko-test/eoan/amd64
93956c6f8d9e: Already exists 
46bddb84d1c5: Already exists 
15fa85048576: Already exists 
8aa40341c4fa: Already exists 
97d612b5000c: Pull complete 
6eaaf8e851c4: Pull complete 
53c75cb1d7a9: Extracting [==================================================>]  177.3MB/177.3MB
failed to register layer: ApplyLayer exit status 1 stdout:  stderr: link /lib/systemd /usr/lib/systemd: no such file or directory

# docker pull registry.tworem.nl/remcodm/kaniko-test/eoan/amd64:1039-fix-test
1039-fix-test: Pulling from remcodm/kaniko-test/eoan/amd64
93956c6f8d9e: Already exists 
46bddb84d1c5: Already exists 
15fa85048576: Already exists 
8aa40341c4fa: Already exists 
cf5e7c37f679: Pull complete 
e4b591f8376c: Pull complete 
31603ad68aef: Extracting [==================================================>]  177.3MB/177.3MB
failed to register layer: ApplyLayer exit status 1 stdout:  stderr: link /lib/systemd /usr/lib/systemd: no such file or directory

So for me, debug-1039-fix-test did not resolve this issue I'm seeing. As you can see, debug-v0.16.0 does work fine.

@RemcodM
Copy link

RemcodM commented Feb 8, 2020

One more comment: I am not sure if it is related or useful at all, but I noticed when going over the build logs that debug-v0.16.0 seems to add some whiteouts for files at the end of the last layer, while version debug-v0.17.0 and later do not. Same for debug-1039-fix-test.

Build log debug-v0.16.0: https://git.tworem.nl/RemcodM/kaniko-test/-/jobs/17363/raw
Build log debug-v0.17.0: https://git.tworem.nl/RemcodM/kaniko-test/-/jobs/17364/raw
Build log debug-v0.17.1: https://git.tworem.nl/RemcodM/kaniko-test/-/jobs/17365/raw
Build log debug-1039-fix-test: https://git.tworem.nl/RemcodM/kaniko-test/-/jobs/17366/raw

@tanguydelignieresaccenture
Copy link
Author

@cvgw, I've made a test with debug-1039-fix-test tag, following the same steps I gave in this issue, and it's OK for me.
@RemcodM maybe you're issue is not the same, as I did not get pull error in debug-v0.17.1 nor debug-1039-fix-test.

Here are the results:

  1. Build the image from the Dockerfile "build1" with gcr.io/kaniko-project/executor:debug-1039-fix-test
$ docker run --rm --entrypoint "" -v /host/path/to/kaniko/config.json:/kaniko/.docker/config.json -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build1:/workspace gcr.io/kaniko-project/executor:debug-1039-fix-test /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile --destination index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test
INFO[0002] Resolved base name registry.access.redhat.com/ubi7/ubi:7.7-214 to registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0002] Resolved base name registry.access.redhat.com/ubi7/ubi:7.7-214 to registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0002] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0004] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0005] Built cross stage deps: map[]
INFO[0005] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0006] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0008] Unpacking rootfs as cmd RUN yum --disableplugin=subscription-manager update -y     && yum --disableplugin=subscription-manager install -y         gcc-4.8.5-39.el7         gcc-c++-4.8.5-39.el7         make-3.82-24.el7     && yum --disableplugin=subscription-manager clean all requires it.
INFO[0122] Taking snapshot of full filesystem...
INFO[0124] RUN yum --disableplugin=subscription-manager update -y     && yum --disableplugin=subscription-manager install -y         gcc-4.8.5-39.el7         gcc-c++-4.8.5-39.el7         make-3.82-24.el7     && yum --disableplugin=subscription-manager clean all
INFO[0124] cmd: /bin/sh
INFO[0124] args: [-c yum --disableplugin=subscription-manager update -y     && yum --disableplugin=subscription-manager install -y         gcc-4.8.5-39.el7         gcc-c++-4.8.5-39.el7         make-3.82-24.el7     && yum --disableplugin=subscription-manager clean all]
Loaded plugins: ovl, product-id, search-disabled-repos
https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi/server/7/7Server/x86_64/devtools/1/os/repodata/a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml: [Errno 12] Timeout on https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi/server/7/7Server/x86_64/devtools/1/os/repodata/a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml: (28, 'Operation timed out after 30001 milliseconds with 0 out of 0 bytes received')
Trying other mirror.
Resolving Dependencies
--> Running transaction check
---> Package binutils.x86_64 0:2.27-41.base.el7_7.1 will be updated
---> Package binutils.x86_64 0:2.27-41.base.el7_7.2 will be an update
---> Package curl.x86_64 0:7.29.0-54.el7_7.1 will be updated
---> Package curl.x86_64 0:7.29.0-54.el7_7.2 will be an update
---> Package libcurl.x86_64 0:7.29.0-54.el7_7.1 will be updated
---> Package libcurl.x86_64 0:7.29.0-54.el7_7.2 will be an update
---> Package nss.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-softokn.x86_64 0:3.44.0-5.el7 will be updated
---> Package nss-softokn.x86_64 0:3.44.0-8.el7_7 will be an update
---> Package nss-softokn-freebl.x86_64 0:3.44.0-5.el7 will be updated
---> Package nss-softokn-freebl.x86_64 0:3.44.0-8.el7_7 will be an update
---> Package nss-sysinit.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss-sysinit.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-tools.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss-tools.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-util.x86_64 0:3.44.0-3.el7 will be updated
---> Package nss-util.x86_64 0:3.44.0-4.el7_7 will be an update
---> Package sqlite.x86_64 0:3.7.17-8.el7 will be updated
---> Package sqlite.x86_64 0:3.7.17-8.el7_7.1 will be an update
---> Package systemd.x86_64 0:219-67.el7_7.2 will be updated
---> Package systemd.x86_64 0:219-67.el7_7.3 will be an update
---> Package systemd-libs.x86_64 0:219-67.el7_7.2 will be updated
---> Package systemd-libs.x86_64 0:219-67.el7_7.3 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                 Arch        Version                   Repository  Size
================================================================================
Updating:
 binutils                x86_64      2.27-41.base.el7_7.2      ubi-7      5.9 M
 curl                    x86_64      7.29.0-54.el7_7.2         ubi-7      270 k
 libcurl                 x86_64      7.29.0-54.el7_7.2         ubi-7      223 k
 nss                     x86_64      3.44.0-7.el7_7            ubi-7      854 k
 nss-softokn             x86_64      3.44.0-8.el7_7            ubi-7      330 k
 nss-softokn-freebl      x86_64      3.44.0-8.el7_7            ubi-7      224 k
 nss-sysinit             x86_64      3.44.0-7.el7_7            ubi-7       65 k
 nss-tools               x86_64      3.44.0-7.el7_7            ubi-7      528 k
 nss-util                x86_64      3.44.0-4.el7_7            ubi-7       79 k
 sqlite                  x86_64      3.7.17-8.el7_7.1          ubi-7      394 k
 systemd                 x86_64      219-67.el7_7.3            ubi-7      5.1 M
 systemd-libs            x86_64      219-67.el7_7.3            ubi-7      411 k

Transaction Summary
================================================================================
Upgrade  12 Packages

Total download size: 14 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /var/cache/yum/x86_64/7Server/ubi-7/packages/curl-7.29.0-54.el7_7.2.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Public key for curl-7.29.0-54.el7_7.2.x86_64.rpm is not installed
--------------------------------------------------------------------------------
Total                                              768 kB/s |  14 MB  00:19
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Importing GPG key 0xFD431D51:
 Userid     : "Red Hat, Inc. (release key 2) <security@redhat.com>"
 Fingerprint: 567e 347a d004 4ade 55ba 8a5f 199e 2f91 fd43 1d51
 Package    : redhat-release-server-7.7-10.el7.x86_64 (@anaconda/7.7)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Importing GPG key 0x2FA658E0:
 Userid     : "Red Hat, Inc. (auxiliary key) <security@redhat.com>"
 Fingerprint: 43a6 e49c 4a38 f4be 9abf 2a53 4568 9c88 2fa6 58e0
 Package    : redhat-release-server-7.7-10.el7.x86_64 (@anaconda/7.7)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : nss-util-3.44.0-4.el7_7.x86_64                              1/24
  Updating   : nss-softokn-freebl-3.44.0-8.el7_7.x86_64                    2/24
  Updating   : systemd-libs-219-67.el7_7.3.x86_64                          3/24
  Updating   : sqlite-3.7.17-8.el7_7.1.x86_64                              4/24
  Updating   : nss-softokn-3.44.0-8.el7_7.x86_64                           5/24
  Updating   : nss-sysinit-3.44.0-7.el7_7.x86_64                           6/24
  Updating   : nss-3.44.0-7.el7_7.x86_64                                   7/24
  Updating   : libcurl-7.29.0-54.el7_7.2.x86_64                            8/24
  Updating   : curl-7.29.0-54.el7_7.2.x86_64                               9/24
  Updating   : systemd-219-67.el7_7.3.x86_64                              10/24
Failed to get D-Bus connection: Operation not permitted
  Updating   : nss-tools-3.44.0-7.el7_7.x86_64                            11/24
  Updating   : binutils-2.27-41.base.el7_7.2.x86_64                       12/24
install-info: No such file or directory for /usr/share/info/as.info.gz
install-info: No such file or directory for /usr/share/info/binutils.info.gz
install-info: No such file or directory for /usr/share/info/gprof.info.gz
install-info: No such file or directory for /usr/share/info/ld.info.gz
install-info: No such file or directory for /usr/share/info/standards.info.gz
  Cleanup    : nss-tools-3.44.0-4.el7.x86_64                              13/24
  Cleanup    : curl-7.29.0-54.el7_7.1.x86_64                              14/24
  Cleanup    : systemd-219-67.el7_7.2.x86_64                              15/24
  Cleanup    : libcurl-7.29.0-54.el7_7.1.x86_64                           16/24
  Cleanup    : nss-sysinit-3.44.0-4.el7.x86_64                            17/24
  Cleanup    : nss-3.44.0-4.el7.x86_64                                    18/24
  Cleanup    : nss-softokn-3.44.0-5.el7.x86_64                            19/24
  Cleanup    : nss-softokn-freebl-3.44.0-5.el7.x86_64                     20/24
  Cleanup    : nss-util-3.44.0-3.el7.x86_64                               21/24
  Cleanup    : sqlite-3.7.17-8.el7.x86_64                                 22/24
  Cleanup    : systemd-libs-219-67.el7_7.2.x86_64                         23/24
  Cleanup    : binutils-2.27-41.base.el7_7.1.x86_64                       24/24
  Verifying  : binutils-2.27-41.base.el7_7.2.x86_64                        1/24
  Verifying  : nss-tools-3.44.0-7.el7_7.x86_64                             2/24
  Verifying  : nss-3.44.0-7.el7_7.x86_64                                   3/24
  Verifying  : sqlite-3.7.17-8.el7_7.1.x86_64                              4/24
  Verifying  : systemd-libs-219-67.el7_7.3.x86_64                          5/24
  Verifying  : nss-sysinit-3.44.0-7.el7_7.x86_64                           6/24
  Verifying  : curl-7.29.0-54.el7_7.2.x86_64                               7/24
  Verifying  : systemd-219-67.el7_7.3.x86_64                               8/24
  Verifying  : libcurl-7.29.0-54.el7_7.2.x86_64                            9/24
  Verifying  : nss-util-3.44.0-4.el7_7.x86_64                             10/24
  Verifying  : nss-softokn-freebl-3.44.0-8.el7_7.x86_64                   11/24
  Verifying  : nss-softokn-3.44.0-8.el7_7.x86_64                          12/24
  Verifying  : nss-tools-3.44.0-4.el7.x86_64                              13/24
  Verifying  : systemd-libs-219-67.el7_7.2.x86_64                         14/24
  Verifying  : systemd-219-67.el7_7.2.x86_64                              15/24
  Verifying  : nss-util-3.44.0-3.el7.x86_64                               16/24
  Verifying  : curl-7.29.0-54.el7_7.1.x86_64                              17/24
  Verifying  : libcurl-7.29.0-54.el7_7.1.x86_64                           18/24
  Verifying  : nss-sysinit-3.44.0-4.el7.x86_64                            19/24
  Verifying  : binutils-2.27-41.base.el7_7.1.x86_64                       20/24
  Verifying  : nss-softokn-freebl-3.44.0-5.el7.x86_64                     21/24
  Verifying  : nss-3.44.0-4.el7.x86_64                                    22/24
  Verifying  : nss-softokn-3.44.0-5.el7.x86_64                            23/24
  Verifying  : sqlite-3.7.17-8.el7.x86_64                                 24/24

Updated:
  binutils.x86_64 0:2.27-41.base.el7_7.2
  curl.x86_64 0:7.29.0-54.el7_7.2
  libcurl.x86_64 0:7.29.0-54.el7_7.2
  nss.x86_64 0:3.44.0-7.el7_7
  nss-softokn.x86_64 0:3.44.0-8.el7_7
  nss-softokn-freebl.x86_64 0:3.44.0-8.el7_7
  nss-sysinit.x86_64 0:3.44.0-7.el7_7
  nss-tools.x86_64 0:3.44.0-7.el7_7
  nss-util.x86_64 0:3.44.0-4.el7_7
  sqlite.x86_64 0:3.7.17-8.el7_7.1
  systemd.x86_64 0:219-67.el7_7.3
  systemd-libs.x86_64 0:219-67.el7_7.3

Complete!
Loaded plugins: ovl, product-id, search-disabled-repos
Resolving Dependencies
--> Running transaction check
---> Package gcc.x86_64 0:4.8.5-39.el7 will be installed
--> Processing Dependency: cpp = 4.8.5-39.el7 for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libgomp = 4.8.5-39.el7 for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libgomp.so.1()(64bit) for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-4.8.5-39.el7.x86_64
---> Package gcc-c++.x86_64 0:4.8.5-39.el7 will be installed
--> Processing Dependency: libstdc++-devel = 4.8.5-39.el7 for package: gcc-c++-4.8.5-39.el7.x86_64
---> Package make.x86_64 1:3.82-24.el7 will be installed
--> Running transaction check
---> Package cpp.x86_64 0:4.8.5-39.el7 will be installed
---> Package glibc-devel.x86_64 0:2.17-292.el7 will be installed
--> Processing Dependency: glibc-headers = 2.17-292.el7 for package: glibc-devel-2.17-292.el7.x86_64
--> Processing Dependency: glibc-headers for package: glibc-devel-2.17-292.el7.x86_64
---> Package libgomp.x86_64 0:4.8.5-39.el7 will be installed
---> Package libmpc.x86_64 0:1.0.1-3.el7 will be installed
---> Package libstdc++-devel.x86_64 0:4.8.5-39.el7 will be installed
---> Package mpfr.x86_64 0:3.1.1-4.el7 will be installed
--> Running transaction check
---> Package glibc-headers.x86_64 0:2.17-292.el7 will be installed
--> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.17-292.el7.x86_64
--> Processing Dependency: kernel-headers for package: glibc-headers-2.17-292.el7.x86_64
--> Running transaction check
---> Package kernel-headers.x86_64 0:3.10.0-1062.12.1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package              Arch        Version                      Repository  Size
================================================================================
Installing:
 gcc                  x86_64      4.8.5-39.el7                 ubi-7       16 M
 gcc-c++              x86_64      4.8.5-39.el7                 ubi-7      7.2 M
 make                 x86_64      1:3.82-24.el7                ubi-7      421 k
Installing for dependencies:
 cpp                  x86_64      4.8.5-39.el7                 ubi-7      6.0 M
 glibc-devel          x86_64      2.17-292.el7                 ubi-7      1.1 M
 glibc-headers        x86_64      2.17-292.el7                 ubi-7      687 k
 kernel-headers       x86_64      3.10.0-1062.12.1.el7         ubi-7      8.7 M
 libgomp              x86_64      4.8.5-39.el7                 ubi-7      158 k
 libmpc               x86_64      1.0.1-3.el7                  ubi-7       51 k
 libstdc++-devel      x86_64      4.8.5-39.el7                 ubi-7      1.5 M
 mpfr                 x86_64      3.1.1-4.el7                  ubi-7      203 k

Transaction Summary
================================================================================
Install  3 Packages (+8 Dependent packages)

Total download size: 42 M
Installed size: 86 M
Downloading packages:
--------------------------------------------------------------------------------
Total                                              778 kB/s |  42 MB  00:55
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : mpfr-3.1.1-4.el7.x86_64                                     1/11
  Installing : libmpc-1.0.1-3.el7.x86_64                                   2/11
  Installing : cpp-4.8.5-39.el7.x86_64                                     3/11
  Installing : kernel-headers-3.10.0-1062.12.1.el7.x86_64                  4/11
  Installing : glibc-headers-2.17-292.el7.x86_64                           5/11
  Installing : glibc-devel-2.17-292.el7.x86_64                             6/11
  Installing : libstdc++-devel-4.8.5-39.el7.x86_64                         7/11
  Installing : libgomp-4.8.5-39.el7.x86_64                                 8/11
  Installing : gcc-4.8.5-39.el7.x86_64                                     9/11
  Installing : gcc-c++-4.8.5-39.el7.x86_64                                10/11
  Installing : 1:make-3.82-24.el7.x86_64                                  11/11
  Verifying  : glibc-devel-2.17-292.el7.x86_64                             1/11
  Verifying  : gcc-c++-4.8.5-39.el7.x86_64                                 2/11
  Verifying  : 1:make-3.82-24.el7.x86_64                                   3/11
  Verifying  : mpfr-3.1.1-4.el7.x86_64                                     4/11
  Verifying  : libgomp-4.8.5-39.el7.x86_64                                 5/11
  Verifying  : libmpc-1.0.1-3.el7.x86_64                                   6/11
  Verifying  : cpp-4.8.5-39.el7.x86_64                                     7/11
  Verifying  : gcc-4.8.5-39.el7.x86_64                                     8/11
  Verifying  : libstdc++-devel-4.8.5-39.el7.x86_64                         9/11
  Verifying  : kernel-headers-3.10.0-1062.12.1.el7.x86_64                 10/11
  Verifying  : glibc-headers-2.17-292.el7.x86_64                          11/11

Installed:
  gcc.x86_64 0:4.8.5-39.el7            gcc-c++.x86_64 0:4.8.5-39.el7
  make.x86_64 1:3.82-24.el7

Dependency Installed:
  cpp.x86_64 0:4.8.5-39.el7
  glibc-devel.x86_64 0:2.17-292.el7
  glibc-headers.x86_64 0:2.17-292.el7
  kernel-headers.x86_64 0:3.10.0-1062.12.1.el7
  libgomp.x86_64 0:4.8.5-39.el7
  libmpc.x86_64 0:1.0.1-3.el7
  libstdc++-devel.x86_64 0:4.8.5-39.el7
  mpfr.x86_64 0:3.1.1-4.el7

Complete!
Loaded plugins: ovl, product-id, search-disabled-repos
Cleaning repos: ubi-7 ubi-7-rhah ubi-7-server-devtools-rpms
              : ubi-7-server-extras-rpms ubi-7-server-optional-rpms
              : ubi-server-rhscl-7-rpms
INFO[0255] Taking snapshot of full filesystem...
  1. Try to build the image from the Dockerfile "build2" with gcr.io/kaniko-project/executor:debug-1039-fix-test, using the image "build1" with tag debug-1039-fix-test (see Dockerfile.debug-1039-fix-test) > OK
$ docker run --rm --entrypoint "" -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build2:/workspace gcr.io/kaniko-project/executor:debug-1039-fix-test /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile.debug-1039-fix-test --no-push
INFO[0000] Resolved base name index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test to index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test
INFO[0000] Resolved base name index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test to index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test
INFO[0000] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test
INFO[0004] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test
INFO[0008] Built cross stage deps: map[]
INFO[0008] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test
INFO[0009] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test
INFO[0011] Unpacking rootfs as cmd RUN echo test requires it.
INFO[0178] Taking snapshot of full filesystem...
INFO[0179] RUN echo test
INFO[0179] cmd: /bin/sh
INFO[0179] args: [-c echo test]
test
INFO[0179] Taking snapshot of full filesystem...
INFO[0180] Skipping push to container registry due to --no-push flag

Dockerfile.debug-1039-fix-test

FROM index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test

RUN echo test

index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test available on my dockerhub.

@stephanf stephanf mentioned this issue Feb 12, 2020
Open
@olivierboudet
Copy link

@tanguydelignieresaccenture can you try tag 1039-fix-test or debug-1039-fix-test?

I've just pushed these two tags and I'd like to test if they fix the issue you are seeing. Thanks!

I have tested this image on my side too and it fix my similar issue.

@jlucktay
Copy link

I just started encountering this issue this morning.
My Dockerfile is here and the associated build script and Cloud Build YAML are alongside in the same directory.
It has failed with the same error 4 times in a row just now.

@whren whren mentioned this issue Feb 15, 2020
Closed
@dinvlad
Copy link

dinvlad commented Feb 17, 2020 •
edited
Loading

I'm getting the same error even with debug-v0.16.0, for the situation described in #1033. So atm there doesn't appear to be a workaround (other than using regular Docker builds)..

@kenanduman1988
Copy link

I'm getting the same error as well with gcr.io/kaniko-project/executor:debug

@diegosucaria
Copy link

same error here!

@tejal29
Copy link
Member

tejal29 commented Feb 19, 2020

@dinvlad Can you mention your error?
debug-v-.16.0 should work if it was due to a regression.

@dinvlad
Copy link

dinvlad commented Feb 19, 2020

@tejal29 the error is the same as in this issue, and the error reproduction and log are described in #1033 (however, the workaround no longer seems to work, even with 0.16.0).

@dinvlad
Copy link

dinvlad commented Feb 19, 2020 •
edited
Loading

This is breaking all of our multi-stage builds currently (i.e. not limited to the situation described in #1033). Moreover, switching to v0.16.0 doesn’t help. The only workaround is to switch to normal Docker builds (i.e. not using Kaniko) :(

@dinvlad
Copy link

dinvlad commented Feb 19, 2020

OK, apparently this issue has to do with caching. If I "break" the cache (either by removing all images from the cache directory on GCR, or by simply modifying a base layer) then the full rebuild succeeds. But when I rebuild it again (with the new cache), it fails..

@conblem
Copy link

conblem commented Feb 25, 2020

@dinvlad same issue here

@cvgw
Copy link
Contributor

cvgw commented Feb 25, 2020

We've committed a change which I hope will fix this. If anyone feels like testing tags a1af057f997316bfb1c4d2d82719d78481a02a79 and debug-a1af057f997316bfb1c4d2d82719d78481a02a79 have the new code

@cvgw cvgw mentioned this issue Feb 25, 2020
Merged
4 tasks
This was referenced Feb 25, 2020
Open
Closed
@tanguydelignieresaccenture
Copy link
Author

tanguydelignieresaccenture commented Feb 25, 2020 •
edited
Loading

I have re-run this issue context with the debug-a1af057f997316bfb1c4d2d82719d78481a02a79 tag and have confirm it fix it.
@cvgw, do you want details of test run?

Update: here it is

-- build 1

$ docker run --rm --entrypoint "" -v /host/path/to/kaniko/config.json:/kaniko/.docker/config.json -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build1:/workspace gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile --destination index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
Unable to find image 'gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79' locally
debug-a1af057f997316bfb1c4d2d82719d78481a02a79: Pulling from kaniko-project/executor
d746bf49e551: Pull complete                                                                                                  55bd16948852: Pull complete                                                                                                  e6a8536b92d3: Pull complete                                                                                                  ce4f88fff860: Pull complete                                                                                                  956cd6dc6b64: Pull complete                                                                                                  56587e5229ac: Pull complete                                                                                                  1ebb4d9355e1: Pull complete                                                                                                  Digest: sha256:dd28123388927a87f01c75dc6fddc222fe79273139fa040f8b4fa3045bacf538
Status: Downloaded newer image for gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
INFO[0001] Resolved base name registry.access.redhat.com/ubi7/ubi:7.7-214 to registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0001] Resolved base name registry.access.redhat.com/ubi7/ubi:7.7-214 to registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0001] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0004] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0006] Built cross stage deps: map[]
INFO[0006] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0008] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0009] Unpacking rootfs as cmd RUN yum --disableplugin=subscription-manager update -y     && yum --disableplugin=subscription-manager install -y         gcc-4.8.5-39.el7         gcc-c++-4.8.5-39.el7         make-3.82-24.el7     && yum --disableplugin=subscription-manager clean all requires it.
INFO[0025] Taking snapshot of full filesystem...
INFO[0025] Resolving paths
INFO[0027] RUN yum --disableplugin=subscription-manager update -y     && yum --disableplugin=subscription-manager install -y         gcc-4.8.5-39.el7         gcc-c++-4.8.5-39.el7         make-3.82-24.el7     && yum --disableplugin=subscription-manager clean all
INFO[0027] cmd: /bin/sh
INFO[0027] args: [-c yum --disableplugin=subscription-manager update -y     && yum --disableplugin=subscription-manager install -y         gcc-4.8.5-39.el7         gcc-c++-4.8.5-39.el7         make-3.82-24.el7     && yum --disableplugin=subscription-manager clean all]
Loaded plugins: ovl, product-id, search-disabled-repos
Resolving Dependencies
--> Running transaction check
---> Package binutils.x86_64 0:2.27-41.base.el7_7.1 will be updated
---> Package binutils.x86_64 0:2.27-41.base.el7_7.2 will be an update
---> Package curl.x86_64 0:7.29.0-54.el7_7.1 will be updated
---> Package curl.x86_64 0:7.29.0-54.el7_7.2 will be an update
---> Package libcurl.x86_64 0:7.29.0-54.el7_7.1 will be updated
---> Package libcurl.x86_64 0:7.29.0-54.el7_7.2 will be an update
---> Package nss.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-softokn.x86_64 0:3.44.0-5.el7 will be updated
---> Package nss-softokn.x86_64 0:3.44.0-8.el7_7 will be an update
---> Package nss-softokn-freebl.x86_64 0:3.44.0-5.el7 will be updated
---> Package nss-softokn-freebl.x86_64 0:3.44.0-8.el7_7 will be an update
---> Package nss-sysinit.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss-sysinit.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-tools.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss-tools.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-util.x86_64 0:3.44.0-3.el7 will be updated
---> Package nss-util.x86_64 0:3.44.0-4.el7_7 will be an update
---> Package sqlite.x86_64 0:3.7.17-8.el7 will be updated
---> Package sqlite.x86_64 0:3.7.17-8.el7_7.1 will be an update
---> Package systemd.x86_64 0:219-67.el7_7.2 will be updated
---> Package systemd.x86_64 0:219-67.el7_7.3 will be an update
---> Package systemd-libs.x86_64 0:219-67.el7_7.2 will be updated
---> Package systemd-libs.x86_64 0:219-67.el7_7.3 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                 Arch        Version                   Repository  Size
================================================================================
Updating:
 binutils                x86_64      2.27-41.base.el7_7.2      ubi-7      5.9 M
 curl                    x86_64      7.29.0-54.el7_7.2         ubi-7      270 k
 libcurl                 x86_64      7.29.0-54.el7_7.2         ubi-7      223 k
 nss                     x86_64      3.44.0-7.el7_7            ubi-7      854 k
 nss-softokn             x86_64      3.44.0-8.el7_7            ubi-7      330 k
 nss-softokn-freebl      x86_64      3.44.0-8.el7_7            ubi-7      224 k
 nss-sysinit             x86_64      3.44.0-7.el7_7            ubi-7       65 k
 nss-tools               x86_64      3.44.0-7.el7_7            ubi-7      528 k
 nss-util                x86_64      3.44.0-4.el7_7            ubi-7       79 k
 sqlite                  x86_64      3.7.17-8.el7_7.1          ubi-7      394 k
 systemd                 x86_64      219-67.el7_7.3            ubi-7      5.1 M
 systemd-libs            x86_64      219-67.el7_7.3            ubi-7      411 k

Transaction Summary
================================================================================
Upgrade  12 Packages

Total download size: 14 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /var/cache/yum/x86_64/7Server/ubi-7/packages/curl-7.29.0-54.el7_7.2.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Public key for curl-7.29.0-54.el7_7.2.x86_64.rpm is not installed
--------------------------------------------------------------------------------
Total                                              967 kB/s |  14 MB  00:15
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Importing GPG key 0xFD431D51:
 Userid     : "Red Hat, Inc. (release key 2) <security@redhat.com>"
 Fingerprint: 567e 347a d004 4ade 55ba 8a5f 199e 2f91 fd43 1d51
 Package    : redhat-release-server-7.7-10.el7.x86_64 (@anaconda/7.7)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Importing GPG key 0x2FA658E0:
 Userid     : "Red Hat, Inc. (auxiliary key) <security@redhat.com>"
 Fingerprint: 43a6 e49c 4a38 f4be 9abf 2a53 4568 9c88 2fa6 58e0
 Package    : redhat-release-server-7.7-10.el7.x86_64 (@anaconda/7.7)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : nss-util-3.44.0-4.el7_7.x86_64                              1/24
  Updating   : nss-softokn-freebl-3.44.0-8.el7_7.x86_64                    2/24
  Updating   : systemd-libs-219-67.el7_7.3.x86_64                          3/24
  Updating   : sqlite-3.7.17-8.el7_7.1.x86_64                              4/24
  Updating   : nss-softokn-3.44.0-8.el7_7.x86_64                           5/24
  Updating   : nss-sysinit-3.44.0-7.el7_7.x86_64                           6/24
  Updating   : nss-3.44.0-7.el7_7.x86_64                                   7/24
  Updating   : libcurl-7.29.0-54.el7_7.2.x86_64                            8/24
  Updating   : curl-7.29.0-54.el7_7.2.x86_64                               9/24
  Updating   : systemd-219-67.el7_7.3.x86_64                              10/24
Failed to get D-Bus connection: Operation not permitted
  Updating   : nss-tools-3.44.0-7.el7_7.x86_64                            11/24
  Updating   : binutils-2.27-41.base.el7_7.2.x86_64                       12/24
install-info: No such file or directory for /usr/share/info/as.info.gz
install-info: No such file or directory for /usr/share/info/binutils.info.gz
install-info: No such file or directory for /usr/share/info/gprof.info.gz
install-info: No such file or directory for /usr/share/info/ld.info.gz
install-info: No such file or directory for /usr/share/info/standards.info.gz
  Cleanup    : nss-tools-3.44.0-4.el7.x86_64                              13/24
  Cleanup    : curl-7.29.0-54.el7_7.1.x86_64                              14/24
  Cleanup    : systemd-219-67.el7_7.2.x86_64                              15/24
  Cleanup    : libcurl-7.29.0-54.el7_7.1.x86_64                           16/24
  Cleanup    : nss-sysinit-3.44.0-4.el7.x86_64                            17/24
  Cleanup    : nss-3.44.0-4.el7.x86_64                                    18/24
  Cleanup    : nss-softokn-3.44.0-5.el7.x86_64                            19/24
  Cleanup    : nss-softokn-freebl-3.44.0-5.el7.x86_64                     20/24
  Cleanup    : nss-util-3.44.0-3.el7.x86_64                               21/24
  Cleanup    : sqlite-3.7.17-8.el7.x86_64                                 22/24
  Cleanup    : systemd-libs-219-67.el7_7.2.x86_64                         23/24
  Cleanup    : binutils-2.27-41.base.el7_7.1.x86_64                       24/24
  Verifying  : binutils-2.27-41.base.el7_7.2.x86_64                        1/24
  Verifying  : nss-tools-3.44.0-7.el7_7.x86_64                             2/24
  Verifying  : nss-3.44.0-7.el7_7.x86_64                                   3/24
  Verifying  : sqlite-3.7.17-8.el7_7.1.x86_64                              4/24
  Verifying  : systemd-libs-219-67.el7_7.3.x86_64                          5/24
  Verifying  : nss-sysinit-3.44.0-7.el7_7.x86_64                           6/24
  Verifying  : curl-7.29.0-54.el7_7.2.x86_64                               7/24
  Verifying  : systemd-219-67.el7_7.3.x86_64                               8/24
  Verifying  : libcurl-7.29.0-54.el7_7.2.x86_64                            9/24
  Verifying  : nss-util-3.44.0-4.el7_7.x86_64                             10/24
  Verifying  : nss-softokn-freebl-3.44.0-8.el7_7.x86_64                   11/24
  Verifying  : nss-softokn-3.44.0-8.el7_7.x86_64                          12/24
  Verifying  : nss-tools-3.44.0-4.el7.x86_64                              13/24
  Verifying  : systemd-libs-219-67.el7_7.2.x86_64                         14/24
  Verifying  : systemd-219-67.el7_7.2.x86_64                              15/24
  Verifying  : nss-util-3.44.0-3.el7.x86_64                               16/24
  Verifying  : curl-7.29.0-54.el7_7.1.x86_64                              17/24
  Verifying  : libcurl-7.29.0-54.el7_7.1.x86_64                           18/24
  Verifying  : nss-sysinit-3.44.0-4.el7.x86_64                            19/24
  Verifying  : binutils-2.27-41.base.el7_7.1.x86_64                       20/24
  Verifying  : nss-softokn-freebl-3.44.0-5.el7.x86_64                     21/24
  Verifying  : nss-3.44.0-4.el7.x86_64                                    22/24
  Verifying  : nss-softokn-3.44.0-5.el7.x86_64                            23/24
  Verifying  : sqlite-3.7.17-8.el7.x86_64                                 24/24

Updated:
  binutils.x86_64 0:2.27-41.base.el7_7.2
  curl.x86_64 0:7.29.0-54.el7_7.2
  libcurl.x86_64 0:7.29.0-54.el7_7.2
  nss.x86_64 0:3.44.0-7.el7_7
  nss-softokn.x86_64 0:3.44.0-8.el7_7
  nss-softokn-freebl.x86_64 0:3.44.0-8.el7_7
  nss-sysinit.x86_64 0:3.44.0-7.el7_7
  nss-tools.x86_64 0:3.44.0-7.el7_7
  nss-util.x86_64 0:3.44.0-4.el7_7
  sqlite.x86_64 0:3.7.17-8.el7_7.1
  systemd.x86_64 0:219-67.el7_7.3
  systemd-libs.x86_64 0:219-67.el7_7.3

Complete!
Loaded plugins: ovl, product-id, search-disabled-repos
Resolving Dependencies
--> Running transaction check
---> Package gcc.x86_64 0:4.8.5-39.el7 will be installed
--> Processing Dependency: cpp = 4.8.5-39.el7 for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libgomp = 4.8.5-39.el7 for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libgomp.so.1()(64bit) for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-4.8.5-39.el7.x86_64
---> Package gcc-c++.x86_64 0:4.8.5-39.el7 will be installed
--> Processing Dependency: libstdc++-devel = 4.8.5-39.el7 for package: gcc-c++-4.8.5-39.el7.x86_64
---> Package make.x86_64 1:3.82-24.el7 will be installed
--> Running transaction check
---> Package cpp.x86_64 0:4.8.5-39.el7 will be installed
---> Package glibc-devel.x86_64 0:2.17-292.el7 will be installed
--> Processing Dependency: glibc-headers = 2.17-292.el7 for package: glibc-devel-2.17-292.el7.x86_64
--> Processing Dependency: glibc-headers for package: glibc-devel-2.17-292.el7.x86_64
---> Package libgomp.x86_64 0:4.8.5-39.el7 will be installed
---> Package libmpc.x86_64 0:1.0.1-3.el7 will be installed
---> Package libstdc++-devel.x86_64 0:4.8.5-39.el7 will be installed
---> Package mpfr.x86_64 0:3.1.1-4.el7 will be installed
--> Running transaction check
---> Package glibc-headers.x86_64 0:2.17-292.el7 will be installed
--> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.17-292.el7.x86_64
--> Processing Dependency: kernel-headers for package: glibc-headers-2.17-292.el7.x86_64
--> Running transaction check
---> Package kernel-headers.x86_64 0:3.10.0-1062.12.1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package              Arch        Version                      Repository  Size
================================================================================
Installing:
 gcc                  x86_64      4.8.5-39.el7                 ubi-7       16 M
 gcc-c++              x86_64      4.8.5-39.el7                 ubi-7      7.2 M
 make                 x86_64      1:3.82-24.el7                ubi-7      421 k
Installing for dependencies:
 cpp                  x86_64      4.8.5-39.el7                 ubi-7      6.0 M
 glibc-devel          x86_64      2.17-292.el7                 ubi-7      1.1 M
 glibc-headers        x86_64      2.17-292.el7                 ubi-7      687 k
 kernel-headers       x86_64      3.10.0-1062.12.1.el7         ubi-7      8.7 M
 libgomp              x86_64      4.8.5-39.el7                 ubi-7      158 k
 libmpc               x86_64      1.0.1-3.el7                  ubi-7       51 k
 libstdc++-devel      x86_64      4.8.5-39.el7                 ubi-7      1.5 M
 mpfr                 x86_64      3.1.1-4.el7                  ubi-7      203 k

Transaction Summary
================================================================================
Install  3 Packages (+8 Dependent packages)

Total download size: 42 M
Installed size: 86 M
Downloading packages:
--------------------------------------------------------------------------------
Total                                              6.8 MB/s |  42 MB  00:06
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : mpfr-3.1.1-4.el7.x86_64                                     1/11
  Installing : libmpc-1.0.1-3.el7.x86_64                                   2/11
  Installing : cpp-4.8.5-39.el7.x86_64                                     3/11
  Installing : kernel-headers-3.10.0-1062.12.1.el7.x86_64                  4/11
  Installing : glibc-headers-2.17-292.el7.x86_64                           5/11
  Installing : glibc-devel-2.17-292.el7.x86_64                             6/11
  Installing : libstdc++-devel-4.8.5-39.el7.x86_64                         7/11
  Installing : libgomp-4.8.5-39.el7.x86_64                                 8/11
  Installing : gcc-4.8.5-39.el7.x86_64                                     9/11
  Installing : gcc-c++-4.8.5-39.el7.x86_64                                10/11
  Installing : 1:make-3.82-24.el7.x86_64                                  11/11
  Verifying  : glibc-devel-2.17-292.el7.x86_64                             1/11
  Verifying  : gcc-c++-4.8.5-39.el7.x86_64                                 2/11
  Verifying  : 1:make-3.82-24.el7.x86_64                                   3/11
  Verifying  : mpfr-3.1.1-4.el7.x86_64                                     4/11
  Verifying  : libgomp-4.8.5-39.el7.x86_64                                 5/11
  Verifying  : libmpc-1.0.1-3.el7.x86_64                                   6/11
  Verifying  : cpp-4.8.5-39.el7.x86_64                                     7/11
  Verifying  : gcc-4.8.5-39.el7.x86_64                                     8/11
  Verifying  : libstdc++-devel-4.8.5-39.el7.x86_64                         9/11
  Verifying  : kernel-headers-3.10.0-1062.12.1.el7.x86_64                 10/11
  Verifying  : glibc-headers-2.17-292.el7.x86_64                          11/11

Installed:
  gcc.x86_64 0:4.8.5-39.el7            gcc-c++.x86_64 0:4.8.5-39.el7
  make.x86_64 1:3.82-24.el7

Dependency Installed:
  cpp.x86_64 0:4.8.5-39.el7
  glibc-devel.x86_64 0:2.17-292.el7
  glibc-headers.x86_64 0:2.17-292.el7
  kernel-headers.x86_64 0:3.10.0-1062.12.1.el7
  libgomp.x86_64 0:4.8.5-39.el7
  libmpc.x86_64 0:1.0.1-3.el7
  libstdc++-devel.x86_64 0:4.8.5-39.el7
  mpfr.x86_64 0:3.1.1-4.el7

Complete!
Loaded plugins: ovl, product-id, search-disabled-repos
Cleaning repos: ubi-7 ubi-7-rhah ubi-7-server-devtools-rpms
              : ubi-7-server-extras-rpms ubi-7-server-optional-rpms
              : ubi-server-rhscl-7-rpms
INFO[0080] Taking snapshot of full filesystem...
INFO[0081] Resolving paths

-- build 2

$ docker run --rm --entrypoint "" -v /host/path/to/dockerfile/directory/kaniko/20200207-001//build2:/workspace gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile.debug-a1af057f997316bfb1c4d2d82719d78481a02a79 --no-push
INFO[0000] Resolved base name index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79 to index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
INFO[0000] Resolved base name index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79 to index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
INFO[0000] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
INFO[0001] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
INFO[0003] Built cross stage deps: map[]
INFO[0003] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
INFO[0003] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
INFO[0004] Unpacking rootfs as cmd RUN echo test requires it.
INFO[0023] Taking snapshot of full filesystem...
INFO[0023] Resolving paths
INFO[0024] RUN echo test
INFO[0024] cmd: /bin/sh
INFO[0024] args: [-c echo test]
test
INFO[0024] Taking snapshot of full filesystem...
INFO[0024] Resolving paths
INFO[0025] No files were changed, appending empty layer to config. No layer added to image.
INFO[0025] Skipping push to container registry due to --no-push flag

-- Dockerfile.debug-a1af057f997316bfb1c4d2d82719d78481a02a79 used for build 2

FROM index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79

RUN echo test

@cvgw
Copy link
Contributor

cvgw commented Feb 25, 2020

@tanguydelignieresaccenture that info is great. Thanks for confirming the fix!

@cvgw cvgw closed this as completed Feb 25, 2020
@cvgw cvgw mentioned this issue Feb 26, 2020
Closed
@adamnovak adamnovak mentioned this issue Feb 28, 2020
Merged
@tejal29 tejal29 mentioned this issue Mar 6, 2020
Merged
@diegosucaria
Copy link

I see this is merged, but I have many Google Cloud Build pipelines using kaniko:latest which are still failing with the same problem..... any ideas why? should I use another tag?

Thanks

@dinvlad
Copy link

dinvlad commented Mar 6, 2020

That's because the :latest/:debug tags were not updated yet, to the build with the fix.

@yuchunyu97
Copy link 

error building image: error building stage: failed to get filesystem from image: removing whiteout etc/ca-certificates/.wh..wh..opq: fstatat /etc/ca-certificates/.wh..opq: operation not permitted

This issue also occurred in version v0.18.0, I don't know what happened?

@pbecotte
Copy link

Just to chime in, using Kaniko for the first time and am hitting a similar issue with invalid images on v0.19.0

@tejal29
Copy link
Member

tejal29 commented May 6, 2020

@pbecotte We added a fix in #1214 in v0.20.0 onwards. Can you please see if you still see the issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cvgw cvgw

Labels
area/filesystems For all bugs related to kaniko container filesystems (mounting issues etc) fixed-needs-verfication in progress kind/bug Something isn't working priority/p0 Highest priority. Break user flow. We are actively looking at delivering it. regression
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests