Setup caching for CI #2652
Labels
Comments
Nutomic
added
enhancement
Closed
CobaltCause
added a commit
to CobaltCause/lemmy
that referenced
this issue
Jun 9, 2023
Fixes <LemmyNet#2652>. Woodpecker has a severe deficiency in that caching artifacts across CI runs is not supported. There has been an open issue for this feature for over 1 year [here][0]. It is also not possible to run its CI definition file locally; creating a pull request and pushing commits is the only way to get the CI pipeline to check your code. In Lemmy's case, the result of these facts is that the edit-compile-test feedback loop is drawn out to 46 minutes just for the test portion. See [here for an example][1]. Having to wait this long is a pretty serious reduction in current productivity and potential productivity of developers who lose interest/patience and give up before being able to contribute a working patch. While the maintainers understandably ["don't want to rely on Microsoft"][0], the repository, code review, and issue tracker all live on GitHub already. Lemmy can reduce its feedback loop latency substantially by making use of GitHub Actions' ability to cache artifacts between runs. Further, by defining the vast majority of the CI pipeline via a third party tool, Lemmy can avoid relying heavily on *any and all* CI vendors. I argue that the tradeoff between * adding this 72 line GitHub Actions file * large reduction in feedback loop latency and wasted CPU recompiling dependencies[^1] is massively worth it. Additionally, this particular third party tool is also capable of being run locally without any middleware in the way like Docker. The result is seamless caching of intermediate artifacts when running locally, which amounts to a reduction in CI run time from 46 minutes in Woodpecker to 3 minutes on my computer with a warm cache. (It's likely you're reading this and wondering what Engage is, so I'll elaborate on that: Engage is a task scheduler and executor based on a directed acyclic graph of dependencies between those tasks, and you can read more about it [here][3]. I've seen a few similar tools but I'm partial to this one due to its simplicity and also the fact that I wrote it.) [^1]: With an empty cache, the CI run on GitHub Actions takes 30 minutes, which is already a 15 minute improvement over the previous Woodpecker setup. With a warm cache, the CI run on GitHub Actions takes 16 minutes, one third as much time as the previous Woodpecker setup. [0]: woodpecker-ci/woodpecker#758 [1]: https://woodpecker.join-lemmy.org/LemmyNet/lemmy/pipeline/72/1 [2]: https://matrix.to/#/!tUnhsBRCsiePXfsIGe:matrix.org/$kV3XDo15eRPPb3xiq3gRrYr5Iix9fTGywTC53cMDRR0?via=matrix.org&via=chapo.chat&via=tchncs.de [3]: https://or.computer.surgery/charles/engage
CobaltCause
added a commit
to CobaltCause/lemmy
that referenced
this issue
Jun 9, 2023
Fixes <LemmyNet#2652>. Woodpecker has a severe deficiency in that caching artifacts across CI runs is not supported. There has been an open issue for this feature for over 1 year [here][0]. It is also not possible to run its CI definition file locally; creating a pull request and pushing commits is the only way to get the CI pipeline to check your code. In Lemmy's case, the result of these facts is that the edit-compile-test feedback loop is drawn out to 46 minutes just for the test portion. See [here for an example][1]. Having to wait this long is a pretty serious reduction in current productivity and potential productivity of developers who lose interest/patience and give up before being able to contribute a working patch. While the maintainers understandably ["don't want to rely on Microsoft"][0], the repository, code review, and issue tracker all live on GitHub already. Lemmy can reduce its feedback loop latency substantially by making use of GitHub Actions' ability to cache artifacts between runs. Further, by defining the vast majority of the CI pipeline via a third party tool, Lemmy can avoid relying heavily on *any and all* CI vendors. I argue that the tradeoff between * adding this 72 line GitHub Actions file * large reduction in feedback loop latency and wasted CPU recompiling dependencies[^1] is massively worth it. Additionally, this particular third party tool is also capable of being run locally without any middleware in the way like Docker. The result is seamless caching of intermediate artifacts when running locally, which amounts to a reduction in CI run time from 46 minutes in Woodpecker to 3 minutes on my computer with a warm cache. (It's likely you're reading this and wondering what Engage is, so I'll elaborate on that: Engage is a task scheduler and executor based on a directed acyclic graph of dependencies between those tasks, and you can read more about it [here][3]. I've seen a few similar tools but I'm partial to this one due to its simplicity and also the fact that I wrote it.) [^1]: With an empty cache, the CI run on GitHub Actions takes 30 minutes, which is already a 15 minute improvement over the previous Woodpecker setup. With a warm cache, the CI run on GitHub Actions takes 16 minutes, one third as much time as the previous Woodpecker setup. [0]: woodpecker-ci/woodpecker#758 [1]: https://woodpecker.join-lemmy.org/LemmyNet/lemmy/pipeline/72/1 [2]: https://matrix.to/#/!tUnhsBRCsiePXfsIGe:matrix.org/$kV3XDo15eRPPb3xiq3gRrYr5Iix9fTGywTC53cMDRR0?via=matrix.org&via=chapo.chat&via=tchncs.de [3]: https://or.computer.surgery/charles/engage
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I would use this plugin, looks like it only needs a change to .drone.yml.
https://plugins.drone.io/plugins/volume-cache
The text was updated successfully, but these errors were encountered: