Propagate the Buildkite pre-merge setup to GitHub Action #81077
Conversation
|
@llvm/pr-subscribers-github-workflow Author: Mehdi Amini (joker-eph) ChangesThis is a WIP, the Buildkite script is .ci/generate-buildkite-pipeline-premerge Full diff: https://github.com/llvm/llvm-project/pull/81077.diff 5 Files Affected:
diff --git a/.github/workflows/bolt-tests.yml b/.github/workflows/bolt-tests.yml
new file mode 100644
index 00000000000000..c10947b507b571
--- /dev/null
+++ b/.github/workflows/bolt-tests.yml
@@ -0,0 +1,37 @@
+name: Bolt Tests
+
+permissions:
+ contents: read
+
+on:
+ workflow_dispatch:
+ push:
+ branches:
+ - 'main'
+ paths:
+ - 'bolt/**'
+ - '.github/workflows/bolt-tests.yml'
+ - '.github/workflows/llvm-project-tests.yml'
+ pull_request:
+ branches:
+ - 'main'
+ paths:
+ - 'bolt/**'
+ - '.github/workflows/bolt-tests.yml'
+ - '.github/workflows/llvm-project-tests.yml'
+
+concurrency:
+ # Skip intermediate builds: always.
+ # Cancel intermediate builds: only if it is a pull request build.
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
+
+jobs:
+ check-all:
+ if: github.repository_owner == 'llvm'
+ name: Build and Test
+ uses: ./.github/workflows/llvm-project-tests.yml
+ with:
+ build_target: check-bolt
+ projects: bolt;lld;llvm
+ os_list: '["ubuntu-latest"]' # tests are not supported yet on Windows
\ No newline at end of file
diff --git a/.github/workflows/flang-tests.yml b/.github/workflows/flang-tests.yml
new file mode 100644
index 00000000000000..07f508fb674df3
--- /dev/null
+++ b/.github/workflows/flang-tests.yml
@@ -0,0 +1,36 @@
+name: Flang Tests
+
+permissions:
+ contents: read
+
+on:
+ workflow_dispatch:
+ push:
+ branches:
+ - 'main'
+ paths:
+ - 'flang/**'
+ - '.github/workflows/flang-tests.yml'
+ - '.github/workflows/llvm-project-tests.yml'
+ pull_request:
+ branches:
+ - 'main'
+ paths:
+ - 'flang/**'
+ - '.github/workflows/flang-tests.yml'
+ - '.github/workflows/llvm-project-tests.yml'
+
+concurrency:
+ # Skip intermediate builds: always.
+ # Cancel intermediate builds: only if it is a pull request build.
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
+
+jobs:
+ check-all:
+ if: github.repository_owner == 'llvm'
+ name: Build and Test
+ uses: ./.github/workflows/llvm-project-tests.yml
+ with:
+ build_target: check-flang
+ projects: flang;clang
\ No newline at end of file
diff --git a/.github/workflows/lld-tests.yml b/.github/workflows/lld-tests.yml
index 599c0975fa6858..7a22990e2ab6c8 100644
--- a/.github/workflows/lld-tests.yml
+++ b/.github/workflows/lld-tests.yml
@@ -8,6 +8,7 @@ on:
push:
branches:
- 'release/**'
+ - 'main'
paths:
- 'lld/**'
- '.github/workflows/lld-tests.yml'
@@ -16,6 +17,7 @@ on:
pull_request:
branches:
- 'release/**'
+ - 'main'
paths:
- 'lld/**'
- '.github/workflows/lld-tests.yml'
@@ -34,5 +36,5 @@ jobs:
name: Test lld
uses: ./.github/workflows/llvm-project-tests.yml
with:
- build_target: check-lld
- projects: lld
+ build_target: check-lld check-bolt check-cross-project-tests
+ projects: lld;
diff --git a/.github/workflows/llvm-tests.yml b/.github/workflows/llvm-tests.yml
index 127628d76f1913..7a13047882afde 100644
--- a/.github/workflows/llvm-tests.yml
+++ b/.github/workflows/llvm-tests.yml
@@ -8,6 +8,7 @@ on:
push:
branches:
- 'release/**'
+ - 'main'
paths:
- 'llvm/**'
- '.github/workflows/llvm-tests.yml'
@@ -15,6 +16,7 @@ on:
pull_request:
branches:
- 'release/**'
+ - 'main'
paths:
- 'llvm/**'
- '.github/workflows/llvm-tests.yml'
@@ -33,7 +35,7 @@ jobs:
uses: ./.github/workflows/llvm-project-tests.yml
with:
build_target: check-all
- projects: clang;lld;libclc;lldb
+ projects: clang-tools-extra;compiler-rt;flang;libc;lldb;openmp;cross-project-tests
abi-dump-setup:
if: github.repository_owner == 'llvm'
diff --git a/.github/workflows/mlir-tests.yml b/.github/workflows/mlir-tests.yml
new file mode 100644
index 00000000000000..83df617e0c3f9e
--- /dev/null
+++ b/.github/workflows/mlir-tests.yml
@@ -0,0 +1,36 @@
+name: MLIR Tests
+
+permissions:
+ contents: read
+
+on:
+ workflow_dispatch:
+ push:
+ branches:
+ - 'main'
+ paths:
+ - 'mlir/**'
+ - '.github/workflows/mlir-tests.yml'
+ - '.github/workflows/llvm-project-tests.yml'
+ pull_request:
+ branches:
+ - 'main'
+ paths:
+ - 'mlir/**'
+ - '.github/workflows/mlir-tests.yml'
+ - '.github/workflows/llvm-project-tests.yml'
+
+concurrency:
+ # Skip intermediate builds: always.
+ # Cancel intermediate builds: only if it is a pull request build.
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
+
+jobs:
+ check-all:
+ if: github.repository_owner == 'llvm'
+ name: Build and Test
+ uses: ./.github/workflows/llvm-project-tests.yml
+ with:
+ build_target: check-mlir check-flang
+ projects: mlir;flang;clang
|
joker-eph
force-pushed
the
ga-premerge
branch
from
6eca722
to
f7a04b6
Compare
This is a WIP, the Buildkite script is .ci/generate-buildkite-pipeline-premerge
joker-eph
force-pushed
the
ga-premerge
branch
from
f7a04b6
to
54b965f
Compare
There was a problem hiding this comment.
Thanks for working on this!
A couple points:
- Are we able to limit this to testing on Windows for now so we can focus on getting back coverage there without taking up too many runners?
- We believe that we might end up throughput constrained with the 60 job limit. I'm working on collecting statistics on this so we have more reliable numbers. In the mean time, maybe we introduce something like this gradually (one subproject at a time) to ensure we don't hit throughput limits?
- It would be nice if we could combine this into a single job and select the targets to build as necessary to avoid the issue you mentioned on the Discourse thread. Maybe even an adaptation of the current shell script in
.cithat selects stuff for Buildkite? This would probably be necessary in a separate job to not impact the release jobs, but that would be up to @tstellar.
As an aside, I'm working on setting up per-OS pipelines right now (with common stuff put into a separate action like llvm-project-tests.yml) and using optimized toolchains/containers to improve system throughput. I have put up #80951 for the time being as an initial version to start testing things out at scale. That's Linux/LLVM only currently though and the most basic configuration without most of the ideas implemented (other than using a well-optimized toolchain to pull buildtimes down significantly).
Yeah: we discussed it here: https://discourse.llvm.org/t/rfc-future-of-windows-pre-commit-ci/76840/34 I'll leave this PR as a reference, and send another one in this direction for consideration. |
@joker-eph Do you have a prototype for this? I was thinking about doing something similar for the Windows CI on GitHub Actions |
|
I don't: I was planning to setup a GitHub Action for the window CI that would reuse the buildkite script (just invoke it should be doable basically), but I haven't got to it yet. Do you want to give it a try? |
|
@joker-eph Yeah, I think I'll try it. |
This is a WIP, the Buildkite script is .ci/generate-buildkite-pipeline-premerge