Skip to content

Commit

Permalink
use Engage and GitHub Actions for CI
Browse files Browse the repository at this point in the history
Woodpecker has a severe deficiency in that caching artifacts across CI
runs is not supported. There has been an open issue for this feature for
over 1 year [here][0]. It is also not possible to run its CI definition
file locally; creating a pull request and pushing commits is the only
way to get the CI pipeline to check your code.

In Lemmy's case, the result of these facts is that the edit-compile-test
feedback loop is drawn out to 46 minutes just for the test portion. See
[here for an example][1]. Having to wait this long is a pretty serious
reduction in current productivity and potential productivity of developers
who lose interest/patience and give up before being able to contribute a
working patch.

While the maintainers understandably ["don't want to rely on
Microsoft"][0], the repository, code review, and issue tracker all live
on GitHub already. Lemmy can reduce its feedback loop latency
substantially by making use of GitHub Actions' ability to cache
artifacts between runs. Further, by defining the vast majority of the CI
pipeline via a third party tool, Lemmy can avoid relying heavily on
*any and all* CI vendors. I argue that the tradeoff between

* adding this 72 line GitHub Actions file
* large reduction in feedback loop latency and wasted CPU recompiling
  dependencies

is massively worth it.

Additionally, this particular third party tool is also capable of being
run locally without any middleware in the way like Docker. The result is
seamless caching of intermediate artifacts when running locally, which
amounts to a reduction in CI run time from 46 minutes in Woodpecker to 3
minutes on my computer with a warm cache.

[0]: woodpecker-ci/woodpecker#758
[1]: https://woodpecker.join-lemmy.org/LemmyNet/lemmy/pipeline/72/1
[2]: https://matrix.to/#/!tUnhsBRCsiePXfsIGe:matrix.org/$kV3XDo15eRPPb3xiq3gRrYr5Iix9fTGywTC53cMDRR0?via=matrix.org&via=chapo.chat&via=tchncs.de
  • Loading branch information
CobaltCause committed Jun 9, 2023
1 parent 5e602bd commit b41bacb
Show file tree
Hide file tree
Showing 3 changed files with 222 additions and 286 deletions.
72 changes: 72 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# This file just bootstraps caching and execution of the real CI system

name: CI

on:
pull_request:
push:
branches:
- main

env:
TERM: ansi

jobs:
ci:
runs-on: ubuntu-latest

steps:
- name: Sync repository
uses: actions/checkout@v3
with:
submodules: true

# GitHub Actions' `ubuntu-latest` ships with a bunch of random software
# pre-installed, including a distribution of Rust that will subtly break
# things if we don't delete it first
- name: Remove conflicting pre-installed tooling
run: rm -rf "$HOME/.cargo" "$HOME/.rustup"

- name: Install Nix with `flakes` and `nix-command` experimental features
uses: cachix/install-nix-action@v21
with:
# Use nixos-unstable; we only need direnv and nix-direnv from here,
# which are unlikely to cause nondeterministic breakages
nix_path: nixpkgs=channel:nixos-unstable

extraPullNames: nix-community

- name: Configure `nix-direnv` to use flakes
run: |
mkdir -p ""$HOME/.config/nixpkgs"
echo "[(self: super: { nix-direnv = super.nix-direnv.override { enableFlakes = true; }; })]" > ~/.config/nixpkgs/overlays.nix
echo 'source ~/.nix-profile/share/nix-direnv/direnvrc' > ~/.direnvrc
- name: Install `direnv` and `nix-direnv`
run: nix-env -f '<nixpkgs>' -i direnv -i nix-direnv

- name: Pop/push downloaded crate cache
uses: actions/cache@v3
with:
key: downloaded-crates
path: ~/.cargo

- name: Pop/push compiled crate cache
uses: actions/cache@v3
with:
key: compiled-crates
path: target

# Improves the signal-to-noise ratio of the logs of the next step
- name: Populate `/nix/store`
run: nix develop --command true

- name: Perform continuous integration
run: |
direnv allow
# Control the database lifecycle here since developers may want to do
# so manually while working locally rather than have Engage enforce it
docker compose up postgres -d
direnv exec . engage
docker compose stop
286 changes: 0 additions & 286 deletions .woodpecker.yml

This file was deleted.

Loading

0 comments on commit b41bacb

Please sign in to comment.