Add env option to build_docker_image build step

[jsopkin]

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar request and didn't find it.
• I searched the Prefect documentation for this feature.

Prefect Version

2.x

Describe the current behavior

The prefect deploy build_docker_image step does not support passing in an env variable. Exporting the variable prior to running prefect deploy or running a shell script to export the variable does not work either. For example, attempting to export the env var through the run_shell_script step results in this error FileNotFoundError: [Errno 2] No such file or directory: 'export'

Describe the proposed behavior

There are cases where it would be necessary to have an env variable set prior to building the image. Expected behavior would be to have an option to pass an env var(s) directly.

Example Use

  - prefect_docker.deployments.steps.build_docker_image:
      id: build_image
      requires: prefect-docker>=0.3.0
      image_name: jsopkin/my-test-image
      tag: latest
      dockerfile: Dockerfile
      push: true
      env: {MY_VARIABLE: "my-value"}

Additional context

This issue is referring to a use case where DOCKER_BUILDKIT needs to be set prior to building the Docker image

[zhen0]

@serinamarie @desertaxle - Can I get your help triaging this one? Is it something you've already considered? Thank you!

[desertaxle]

The build_docker_image step doesn't execute Docker commands via a shell and generally doesn't inspect any environment variables other than those needed for authentication. This is due to the behavior of the underlying docker-py library.

If I understand correctly, the requested feature is to support Docker's BuildKit in build_docker_image. That has been requested in #12922, and we've seen this request at least thrice. Unfortunately, docker-py doesn't support using BuildKit, and we would need to switch prefect-docker's docker-py dependency to a different Docker library.

I understand that BuildKit makes Docker image builds faster, which is a nice feature, but it shouldn't be blocking any users. If it blocks users or many users are requesting this, we can explore our options for adding BuildKit support.

[jsopkin]

@desertaxle Apologies about the duplicate. This was specifically requested by a client attempting to build images with private libraries using BuildKit. I can advise them that this isn't supported and to use another method

[desertaxle]

No worries @Bada-S! Feel free to add any details for their use case on the linked issue. If they absolutely need BuildKit, let us know and we can look into adding support.

[EmilRex]

Looks like perhaps it's settable at the daemon level. Might be a viable alternative.

[EmilRex]

Linking the relevant docker-py issue: docker/docker-py#2230

[desertaxle]

Thanks for the info @EmilRex! I've linked to your comment in #12922.

I'm going to close this issue as a duplicate of #12922, but feel free to add comments on that issue as needed.