Task images not pulling from Kind local image registry

[oneillal]

Expected Behavior

When running Tekton on a local Kind cluster, Task definitions should see images existing in local Kind image registry (and not try to pull from remote) when imagePullPolicy: Never

Actual Behavior

Tekton appears to not find images in the local Kind registry (or find the registry itself) and always try's to connect to a remote registry for Tasks images, when imagePullPolicy: Never e.g. DockerHub

Steps to Reproduce the Problem

1. Create a Kind cluster
2. Install latest Tekton pipelines release
3. Build an image
4. Load the image into the Kind registry kind load docker-image my-image:1.01
5. Create a Task and TaskRun (manifests attached)
6. TaskRun failed as it tries to pull an image from a remote repository e.g. DockerHub

Additional Info

Kubernetes version:

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:53:42Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-15T00:36:28Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}

Tekton Pipeline version:

$ tkn version
Client version: 0.31.1
Pipeline version: v0.49.0

Kind version:

$ kind version
kind v0.20.0 go1.20.4 linux/amd64

Task and TaskRun definitions:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: my-task-1
spec:
  steps:
  - name: my-step-1
    image: my-image:1.01
    imagePullPolicy: Never

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: my-taskrun-1
spec:
  taskRef:
    name: my-task-1

Load an image into Kind registry:

$ kind load docker-image my-image:1.01         
Image: "my-image:1.01" with ID "sha256:eb4a57159180767450cb8426e6367f11b999653d8f185b5e3b78a9ca30c2c31d" not yet present on node "kind-control-plane", loading...

Ensure Kind manifest Can Access Image Registry

$ kubectl run my-image --image=my-image:1.01  --restart=Never
pod/my-image created

$ kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
my-image   1/1     Running   0          23s

TaskRun Logs

$ kubectl apply -f my-taskrun.yaml
taskrun.tekton.dev/my-taskrun-1 created

$ tkn taskrun logs        
task my-task-1 has failed: failed to create task run pod "my-taskrun-1": translating TaskSpec to Pod: GET https://index.docker.io/v2/library/my-image/manifests/1.01: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/my-image Type:repository]]. Maybe missing or invalid Task default/my-task-1
Error: pod for taskrun my-taskrun-1 not available yet

[vdemeester]

@oneillal Thanks for the issue. This is happening by design, due to the way tekton creates and executes the Tasks (also known as the entrypoint hack). In a gist, if you don't specify a command, the tekton controller need to know/find what command to execute (which is the "default command" of the image). In order to do this, it tries to query the image using the oci registry API (and not a "container runtime" API as.. we are in kubernetes, we have no way to do it). This is why it then tries to fetch the configuration from docker.io/library/my-image:1.01 as.. this is how the default behavior is with "resolving" oci image references.

The current workaround at your disposale today would be:

• specific the command in the Task definition
• use fully qualified image references (registry/image:tag) and thus push images to a registry (even a local one). It probably wouldn't work with what you described above, kind load directly loads the image to the container runtime, so there is no registry involved 😓

This is a relatively hard problem to "fix" even, as.. with Kubernetes, we have no way to "fetch" the oci image configuration from the local/node runtime (it's not part of the CRI API).

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[oneillal]

Thanks @vdemeester for the explanation. Apologies, GH notifications were turned off. Closing as resolved.