Add support to CLI build #6865

ulyssessouza · 2019-08-22T14:47:37Z

This add support for building with the CLI by setting the env var COMPOSE_DOCKER_CLI_BUILD=1.
This feature can be used both by docker-compose build and docker-compose up --build.

Note that this feature is EXPERIMENTAL, therefore a warning message is printed on every execution

Signed-off-by: Nao YONASHIRO yonashiro@r.recruit.co.jp
Signed-off-by: Ulysses Souza ulysses.souza@docker.com

This PR is a squashed and resumed version of #6584

As a followup we need to check following points in other PRs

Add --progress flag in docker-compose build to control the format of the output
Add a warning if someone has this feature enabled, and uses the --compress or --parallel flag
Add integration tests for build and up --build

abhaychrungoo · 2019-08-25T11:31:00Z

I see a couple of issues. Both are likely related and may be summarized as

Not tagging the final image(s) correctly

Baseline: CentOS 7.5 x86_64. (Similar results seen on OSX Darwin 17.7.0)

abhay@ziraffe foo]$ git remote -v
origin  git@github.com:abhaychrungoo/compose-cli-build-test.git (fetch)
origin  git@github.com:abhaychrungoo/compose-cli-build-test.git (push)
[abhay@ziraffe foo]$ uname -s -a
Linux ziraffe.dev 4.18.7-1.el7.elrepo.x86_64 #1 SMP Sun Sep 9 09:02:34 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
[abhay@ziraffe foo]$ uname -s -r
Linux 4.18.7-1.el7.elrepo.x86_64
[abhay@ziraffe foo]$ ../dist/docker-compose-Linux-x86_64 -v
docker-compose version 1.25.0dev, build dc17a023

Case 1: Images not (re)tagged correctly after first build.

[abhay@ziraffe foo]$ ../dist/docker-compose-Linux-x86_64 -f bar/docker-compose.yml  -f baz/docker-compose.yml -p foo-02  build
.. snip ..
[abhay@ziraffe foo]$ ../dist/docker-compose-Linux-x86_64 -f bar/docker-compose.yml  -f baz/docker-compose.yml -p foo-02  up
WARNING: Native build is an experimental feature and could change at any time
Starting foo-02_bar_1 ... done
Starting foo-02_baz_1 ... done
Attaching to foo-02_baz_1, foo-02_bar_1
bar_1  | bar1
foo-02_baz_1 exited with code 0
foo-02_bar_1 exited with code 0


[abhay@ziraffe foo]$ echo baz1 > baz/hello
[abhay@ziraffe foo]$ ../dist/docker-compose-Linux-x86_64 -f bar/docker-compose.yml  -f baz/docker-compose.yml -p foo-02  build
WARNING: Native build is an experimental feature and could change at any time
Building baz
[+] Building 0.1s (8/8) FINISHED                                                                                             
..snip..
=> => writing image sha256:9b8b7b2d1c3260db6caab5d68b9c0670639d61136dc1b10ebc1cdb09e3c1cc5b                            0.0s
Successfully built 9b8b7b2d1c3260db6caab5d68b9c0670639d61136dc1b10ebc1cdb09e3c1cc5b
Building bar
[+] Building 0.1s (8/8) FINISHED                                                                                             
 .. snip..
 => => writing image sha256:2b4ca779f961e93c5a332010c62c4e4a110afb90f61b8562063859da2e64777f                            0.0s
Successfully built 2b4ca779f961e93c5a332010c62c4e4a110afb90f61b8562063859da2e64777f

[abhay@ziraffe foo]$ ../dist/docker-compose-Linux-x86_64 -f bar/docker-compose.yml  -f baz/docker-compose.yml -p foo-02  up
WARNING: Native build is an experimental feature and could change at any time
Starting foo-02_bar_1 ... done
Starting foo-02_baz_1 ... done
Attaching to foo-02_baz_1, foo-02_bar_1
bar_1  | bar1
foo-02_baz_1 exited with code 0
foo-02_bar_1 exited with code 0

Case 2:
On first run with build --up, the images are built twice. Once with buildkit, and once without

[abhay@ziraffe foo]$ ../dist/docker-compose-Linux-x86_64 -f bar/docker-compose.yml  -f baz/docker-compose.yml -p foo-03  up --build
WARNING: Native build is an experimental feature and could change at any time
Creating network "foo-03_default" with the default driver
Building bar
[+] Building 0.1s (8/8) FINISHED                                                                                             
 => [internal] load build definition from Dockerfile                                                                    0.0s
 => => transferring dockerfile: 37B                                                                                     0.0s
 => [internal] load .dockerignore                                                                                       0.0s
 => => transferring context: 2B                                                                                         0.0s
 => [internal] load metadata for docker.io/library/alpine:3.10.2                                                        0.0s
 => [internal] load build context                                                                                       0.0s
 => => transferring context: 53B                                                                                        0.0s
 => [1/3] FROM docker.io/library/alpine:3.10.2                                                                          0.0s
 => CACHED [2/3] WORKDIR /tmp                                                                                           0.0s
 => CACHED [3/3] ADD bar/hello /tmp/hello                                                                               0.0s
 => exporting to image                                                                                                  0.0s
 => => exporting layers                                                                                                 0.0s
 => => writing image sha256:2b4ca779f961e93c5a332010c62c4e4a110afb90f61b8562063859da2e64777f                            0.0s
Successfully built 2b4ca779f961e93c5a332010c62c4e4a110afb90f61b8562063859da2e64777f
Building baz
[+] Building 0.1s (8/8) FINISHED                                                                                             
 => [internal] load build definition from Dockerfile                                                                    0.0s
 => => transferring dockerfile: 133B                                                                                    0.0s
 => [internal] load .dockerignore                                                                                       0.0s
 => => transferring context: 2B                                                                                         0.0s
 => [internal] load metadata for docker.io/library/alpine:3.10.2                                                        0.0s
 => [internal] load build context                                                                                       0.0s
 => => transferring context: 55B                                                                                        0.0s
 => [1/3] FROM docker.io/library/alpine:3.10.2                                                                          0.0s
 => CACHED [2/3] WORKDIR /tmp                                                                                           0.0s
 => CACHED [3/3] ADD baz/hello /tmp/hello                                                                               0.0s
 => exporting to image                                                                                                  0.0s
 => => exporting layers                                                                                                 0.0s
 => => writing image sha256:221b24c992a4959b4c2c72a9031b35690208205125efd093aa61a521529052f9                            0.0s
Successfully built 221b24c992a4959b4c2c72a9031b35690208205125efd093aa61a521529052f9
Creating foo-03_bar_1 ... 
Creating foo-03_baz_1 ... 
Building bar
Building baz
Step 1/4 : FROM alpine:3.10.2 as alpine3
 ---> 961769676411
Step 2/4 : WORKDIR /tmp
 ---> Using cache
 ---> 4499fbb54a43
Step 3/4 : ADD baz/hello /tmp/hello
Step 1/4 : FROM alpine:3.10.2 as alpine3
 ---> 961769676411
Step 2/4 : WORKDIR /tmp
 ---> Using cache
 ---> 4499fbb54a43
Step 3/4 : ADD bar/hello /tmp/hello
 ---> Using cache
 ---> 35e885d72819
Step 4/4 : CMD cat /tmp/hello
 ---> Using cache
 ---> 9b02eb67a344
Successfully built 9b02eb67a344
Successfully tagged foo-03_bar:latest
WARNING: Image for service bar was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
 ---> 920204ba0094
Step 4/4 : CMD cat /tmp/hello
 ---> Running in 80b4b5de43b0
Removing intermediate container 80b4b5de43b0
 ---> 66e1fd303736
Successfully built 66e1fd303736
Successfully tagged foo-03_baz:latest
Creating foo-03_bar_1 ... done
Creating foo-03_baz_1 ... done
Attaching to foo-03_bar_1, foo-03_baz_1
bar_1  | bar1
foo-03_bar_1 exited with code 0
foo-03_baz_1 exited with code 0

orisano · 2019-08-25T11:49:32Z

build --up? Is it up --build?

abhaychrungoo · 2019-08-25T11:58:25Z

@orisano : up --build.
@ulyssessouza : What warning are you expecting to provide with the --parallel flag ?

orisano · 2019-08-25T12:08:30Z

Maybe resolve that
orisano@ee60803

abhaychrungoo · 2019-08-25T12:43:56Z

I will test this again and provide feedback. However I do feel that in the broader scheme of things, we should be adding something to the integration test suite in this PR.

Not sure of this is covered in any any testcases already.

the tagged image sha matches the sha of the final (or target stage) image.

I'm happy to dig into the code, if that helps to move this PR along.

abhaychrungoo · 2019-08-25T13:51:33Z

orisano/compose@ee60803

Yes, it fixes the issue for both build and up --build

orisano · 2019-08-27T20:39:54Z

orisano@4965627

I implemented a prototype of --progress flag on docker-compose build.

orisano · 2019-08-27T20:56:22Z

orisano@a633c9a
Added a warning if someone has this feature enabled, and uses the --compress or --parallel flag.

orisano · 2019-08-28T10:06:26Z

@ulyssessouza What could I do?

ulyssessouza · 2019-08-28T10:15:09Z

Thanks a lot @orisano !
Today I'll integrate this 3 commits, maybe rephrase the warnings and add integration tests to all of this.

Then I think the scope of this PR is closed.

orisano · 2019-08-28T15:14:47Z

Please add this commit.
orisano@8e1a715

Maybe, it is the mistake of conflict resolves.

This includes can be enabled by setting the env var `COMPOSE_NATIVE_BUILDER=1`. Signed-off-by: Nao YONASHIRO <yonashiro@r.recruit.co.jp> Signed-off-by: Ulysses Souza <ulysses.souza@docker.com>

Signed-off-by: Nao YONASHIRO <yonashiro@r.recruit.co.jp>

Signed-off-by: Ulysses Souza <ulysses.souza@docker.com>

ulyssessouza · 2019-08-28T18:12:31Z

@orisano Fixed

rumpl

LGTM

ulyssessouza · 2019-08-30T11:57:27Z

Thanks @orisano for the contribution! This feature will be available in the next RC.
Please feel free to reach us out in the #docker-compose channel in the docker community slack

thaJeztah · 2019-08-30T12:52:22Z

Thanks everyone! Looking forward to see people use this feature 👍 (be sure to post feedback!)

itaymelamed · 2019-09-04T07:56:01Z

When it would be possible to start using it in docker compose?

ulyssessouza · 2019-09-10T11:58:42Z

@itaymelamed In the next release candidate (1.25.0-rc3). It has no specific date but it's comming very soon (weeks).

abhaychrungoo · 2019-09-26T19:09:54Z

We've used this in our CI fairly extensively over the past few weeks.
Works like a charm for our usecases. No issues reported.

Thanks @ulyssessouza

thaJeztah · 2019-09-27T09:32:49Z

Thanks for the feedback, @abhaychrungoo ! That's good to hear 🤗

bkaid · 2019-10-08T18:16:25Z

First of all, it works great and getting a lot faster builds as a result. Thanks!

I installed latest via:

curl -L https://dl.bintray.com/docker-compose/master/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
docker-compose --version

Result: docker-compose version 1.25.0dev, build bc57a1bd

I noticed a difference now in behavior with restart: on-failure that isn't ideal. Previously if I did docker-compose up and the service restarted (in my case the database dependency hadn't finished starting), the restart would happen and the command would continue. Now if I do DOCKER_BUILDKIT=1 COMPOSE_DOCKER_CLI_BUILD=1 docker-compose up --build myservice the restart happens but docker-compose exits and doesn't reattach to the restarted instance.

thaJeztah · 2019-10-08T22:53:11Z

@bkaid thanks for reporting: would you be able to open a new issue and provide a minimum reproduction example?

bkaid · 2019-10-09T02:19:54Z

@thaJeztah Sorry - disregard. I tried on latest release version 1.24.1, build 4667896b and it works the same way so I don't think there is a change with this new version and I think I just misunderstood how it worked.

docker-compose.yml:

version: "3.7"

services:
  rabbitmq:
    image: rabbitmq:3.7.15-management-alpine

  myservice:
    image: alpine:3.10.2
    restart: on-failure
    depends_on:
      - rabbitmq
    entrypoint: sh -c "echo Starting up... && sleep 10 && echo Exiting on error && exit 1"

This will restart my service and keep showing updated logs and not exit:
docker-compose up rabbitmq myservice

This will restart my service in the background and exit:
docker-compose up myservice

lig · 2019-10-09T09:41:43Z

In the next release candidate (1.25.0-rc3). It has no specific date but it's comming very soon (weeks).

@ulyssessouza no pressure but it starting to become months :(

smamessier · 2019-10-25T09:23:55Z

I'd like to use --ssh default to forward the SSH agent at build time using compose. Is there a way we can forward all options to docker build CLI?

TortelliniLeap · 2019-10-25T20:00:14Z

@smamessier I'm interested in support for this as well.

chris-crone · 2019-10-28T12:45:24Z

This PR is included in the 1.25.0-rc3 release that you can find here. It will also be part of the next Docker Desktop edge release.

aleygues · 2019-10-30T12:53:08Z

@smamessier I'm also interested in this feature!

lig · 2019-11-04T17:10:02Z

@chris-crone it would be nice to see this mentioned in the release docs. It looks like a pretty significant new feature.

pcolaianni · 2020-04-02T11:57:09Z

How do I execute the equivalent of docker build --ssh default=/home/user/.ssh/keyfile . with docker-compose?
I have tried COMPOSE_DOCKER_CLI_BUILD=1 docker-compose build but it's not working.
I guess that only handles connections to an ssh-agent, instead of a file as in this case (/home/user/.ssh/keyfile).

thaJeztah · 2020-04-02T15:20:56Z

@pasqoo not supported yet, but there's a PR working on that to add it to the compose-file syntax; #7046. It won't be added as a command-line flag, because you want to control which builds have access to your ssh-agent (as the compose file may have multiple builds, and only some of them could need access)

shabeerak · 2020-05-07T02:49:59Z

@thaJeztah do you know when --ssh feature will be released?

ulyssessouza requested review from rumpl, thaJeztah and chris-crone August 22, 2019 14:47

ulyssessouza self-assigned this Aug 22, 2019

ulyssessouza mentioned this pull request Aug 22, 2019

feat: add --cli option #6584

Closed

chris-crone mentioned this pull request Aug 26, 2019

Support for docker build --secret for build-time docker secrets. #6358

Closed

graingert mentioned this pull request Aug 27, 2019

requirements: update jsonschema dependency #6631

Merged

GordonTheTurtle added the dco/no label Aug 28, 2019

ulyssessouza force-pushed the support-cli-build branch from bbaebd8 to c393540 Compare August 28, 2019 12:14

GordonTheTurtle removed the dco/no label Aug 28, 2019

docker deleted a comment from GordonTheTurtle Aug 28, 2019

ulyssessouza force-pushed the support-cli-build branch from c393540 to d752cc8 Compare August 28, 2019 12:15

orisano and others added 5 commits August 28, 2019 17:24

Add support to CLI build

cacbccc

This includes can be enabled by setting the env var `COMPOSE_NATIVE_BUILDER=1`. Signed-off-by: Nao YONASHIRO <yonashiro@r.recruit.co.jp> Signed-off-by: Ulysses Souza <ulysses.souza@docker.com>

fix: add build flags

862a13b

Signed-off-by: Nao YONASHIRO <yonashiro@r.recruit.co.jp>

feat: add --progress flag

81e223d

Signed-off-by: Nao YONASHIRO <yonashiro@r.recruit.co.jp>

feat: add a warning if someone uses the --compress or --parallel flag

15e8edc

Signed-off-by: Nao YONASHIRO <yonashiro@r.recruit.co.jp>

Rephrase warnings when building with the cli

ee8ca5d

Signed-off-by: Ulysses Souza <ulysses.souza@docker.com>

ulyssessouza force-pushed the support-cli-build branch from f9fb94a to a6de26d Compare August 28, 2019 15:24

rumpl approved these changes Aug 30, 2019

View reviewed changes

ulyssessouza merged commit 2199278 into docker:master Aug 30, 2019

thaJeztah mentioned this pull request Aug 30, 2019

build time only -v / --volume option moby/moby#14080

Open

itamarst mentioned this pull request Aug 31, 2019

/kind feature: podman-compose analogue of docker-compose containers/podman#3865

Closed

ghost mentioned this pull request Sep 17, 2019

Forward ssh key agent into container moby/moby#6396

Open

thaJeztah mentioned this pull request Oct 18, 2019

WORKDIR causes docker build to not use cache moby/moby#30081

Closed

jamiejackson mentioned this pull request Oct 23, 2019

Add support for specifying .dockerignore file with -i/--ignore moby/moby#12886

Closed

thaJeztah mentioned this pull request Oct 24, 2019

cache not used when using "cache_from" in docker compose but works with "docker build" moby/moby#40126

Closed

chris-crone added this to the 1.25.0 milestone Oct 28, 2019

tonistiigi mentioned this pull request Nov 7, 2019

Docker compose support moby/buildkit#685

Closed

ppg mentioned this pull request May 11, 2020

Introduce a new env var "COMPOSE_DOCKER_CLI_BUILD_EXTRA_ARGS" #7296

Closed

thaJeztah mentioned this pull request Jan 25, 2024

how to --build-context docker/docker-py#3220

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support to CLI build #6865

Add support to CLI build #6865

ulyssessouza commented Aug 22, 2019 •

edited

Loading

abhaychrungoo commented Aug 25, 2019 •

edited

Loading

orisano commented Aug 25, 2019

abhaychrungoo commented Aug 25, 2019

orisano commented Aug 25, 2019

abhaychrungoo commented Aug 25, 2019

abhaychrungoo commented Aug 25, 2019

orisano commented Aug 27, 2019

orisano commented Aug 27, 2019 •

edited

Loading

orisano commented Aug 28, 2019

ulyssessouza commented Aug 28, 2019

orisano commented Aug 28, 2019

ulyssessouza commented Aug 28, 2019

rumpl left a comment

ulyssessouza commented Aug 30, 2019

thaJeztah commented Aug 30, 2019

itaymelamed commented Sep 4, 2019

ulyssessouza commented Sep 10, 2019

abhaychrungoo commented Sep 26, 2019

thaJeztah commented Sep 27, 2019

bkaid commented Oct 8, 2019

thaJeztah commented Oct 8, 2019

bkaid commented Oct 9, 2019

lig commented Oct 9, 2019 •

edited

Loading

smamessier commented Oct 25, 2019

TortelliniLeap commented Oct 25, 2019

chris-crone commented Oct 28, 2019

aleygues commented Oct 30, 2019

lig commented Nov 4, 2019

pcolaianni commented Apr 2, 2020

thaJeztah commented Apr 2, 2020

shabeerak commented May 7, 2020

Add support to CLI build #6865

Add support to CLI build #6865

Conversation

ulyssessouza commented Aug 22, 2019 • edited Loading

abhaychrungoo commented Aug 25, 2019 • edited Loading

orisano commented Aug 25, 2019

abhaychrungoo commented Aug 25, 2019

orisano commented Aug 25, 2019

abhaychrungoo commented Aug 25, 2019

abhaychrungoo commented Aug 25, 2019

orisano commented Aug 27, 2019

orisano commented Aug 27, 2019 • edited Loading

orisano commented Aug 28, 2019

ulyssessouza commented Aug 28, 2019

orisano commented Aug 28, 2019

ulyssessouza commented Aug 28, 2019

rumpl left a comment

Choose a reason for hiding this comment

ulyssessouza commented Aug 30, 2019

thaJeztah commented Aug 30, 2019

itaymelamed commented Sep 4, 2019

ulyssessouza commented Sep 10, 2019

abhaychrungoo commented Sep 26, 2019

thaJeztah commented Sep 27, 2019

bkaid commented Oct 8, 2019

thaJeztah commented Oct 8, 2019

bkaid commented Oct 9, 2019

lig commented Oct 9, 2019 • edited Loading

smamessier commented Oct 25, 2019

TortelliniLeap commented Oct 25, 2019

chris-crone commented Oct 28, 2019

aleygues commented Oct 30, 2019

lig commented Nov 4, 2019

pcolaianni commented Apr 2, 2020

thaJeztah commented Apr 2, 2020

shabeerak commented May 7, 2020

ulyssessouza commented Aug 22, 2019 •

edited

Loading

abhaychrungoo commented Aug 25, 2019 •

edited

Loading

orisano commented Aug 27, 2019 •

edited

Loading

lig commented Oct 9, 2019 •

edited

Loading