This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Allow caching artifacts without privileged containers and volumes #758
Labels
feature
add new functionality
Comments
|
Caching is a highly requested feature and we talked a lot about it already. I would prefer to have some official plugins which help the user to do caching. How the caching works really depends on the "backend" you want to use IMO. Most users will probably be fine with simply putting caches in a folder on the server, but as you can have multiple agents for your Woodpecker instance or could even use Kubernetes (#9) as backend, most larger deployments probably need some kind of clustered caching backend like s3 as well. Woodpecker should probably be the generic base for it and the caching plugins could do the magic in the background. If we have some nice plugins we should add a sample about them somewhere in the docs as well (#316). |
|
Although putting cache into S3 is a choice but it is still slow and wastes bandwidth. In some cases, we have predictable hosts to run agents. So on every host, we can cache the files separately. I do not care about if the cache is sync-ed. If there is cache, then I'd like to reuse it; if there is no cache, then I can let the job to init a new cache, instead of restoring it from somewhere else. ps: now I am using such local cache mechanism, and I use Docker-in-Docker to isolate the agents from hosts, then I can set |
|
"Allow agent to cache" would be the feature then I guess? |
3 tasks
3 tasks
CobaltCause
added a commit
to CobaltCause/lemmy
that referenced
this issue
Jun 9, 2023
Woodpecker has a severe deficiency in that caching artifacts across CI runs is not supported. There has been an open issue for this feature for over 1 year [here][0]. It is also not possible to run its CI definition file locally; creating a pull request and pushing commits is the only way to get the CI pipeline to check your code. In Lemmy's case, the result of these facts is that the edit-compile-test feedback loop is drawn out to 46 minutes just for the test portion. See [here for an example][1]. Having to wait this long is a pretty serious reduction in current productivity and potential productivity of developers who lose interest/patience and give up before being able to contribute a working patch. While the maintainers understandably ["don't want to rely on Microsoft"][0], the repository, code review, and issue tracker all live on GitHub already. Lemmy can reduce its feedback loop latency substantially by making use of GitHub Actions' ability to cache artifacts between runs. Further, by defining the vast majority of the CI pipeline via a third party tool, Lemmy can avoid relying heavily on *any and all* CI vendors. I argue that the tradeoff between * adding this 72 line GitHub Actions file * large reduction in feedback loop latency and wasted CPU recompiling dependencies is massively worth it. Additionally, this particular third party tool is also capable of being run locally without any middleware in the way like Docker. The result is seamless caching of intermediate artifacts when running locally, which amounts to a reduction in CI run time from 46 minutes in Woodpecker to 3 minutes on my computer with a warm cache. [0]: woodpecker-ci/woodpecker#758 [1]: https://woodpecker.join-lemmy.org/LemmyNet/lemmy/pipeline/72/1 [2]: https://matrix.to/#/!tUnhsBRCsiePXfsIGe:matrix.org/$kV3XDo15eRPPb3xiq3gRrYr5Iix9fTGywTC53cMDRR0?via=matrix.org&via=chapo.chat&via=tchncs.de
CobaltCause
added a commit
to CobaltCause/lemmy
that referenced
this issue
Jun 9, 2023
Woodpecker has a severe deficiency in that caching artifacts across CI runs is not supported. There has been an open issue for this feature for over 1 year [here][0]. It is also not possible to run its CI definition file locally; creating a pull request and pushing commits is the only way to get the CI pipeline to check your code. In Lemmy's case, the result of these facts is that the edit-compile-test feedback loop is drawn out to 46 minutes just for the test portion. See [here for an example][1]. Having to wait this long is a pretty serious reduction in current productivity and potential productivity of developers who lose interest/patience and give up before being able to contribute a working patch. While the maintainers understandably ["don't want to rely on Microsoft"][0], the repository, code review, and issue tracker all live on GitHub already. Lemmy can reduce its feedback loop latency substantially by making use of GitHub Actions' ability to cache artifacts between runs. Further, by defining the vast majority of the CI pipeline via a third party tool, Lemmy can avoid relying heavily on *any and all* CI vendors. I argue that the tradeoff between * adding this 72 line GitHub Actions file * large reduction in feedback loop latency and wasted CPU recompiling dependencies is massively worth it. Additionally, this particular third party tool is also capable of being run locally without any middleware in the way like Docker. The result is seamless caching of intermediate artifacts when running locally, which amounts to a reduction in CI run time from 46 minutes in Woodpecker to 3 minutes on my computer with a warm cache. [0]: woodpecker-ci/woodpecker#758 [1]: https://woodpecker.join-lemmy.org/LemmyNet/lemmy/pipeline/72/1 [2]: https://matrix.to/#/!tUnhsBRCsiePXfsIGe:matrix.org/$kV3XDo15eRPPb3xiq3gRrYr5Iix9fTGywTC53cMDRR0?via=matrix.org&via=chapo.chat&via=tchncs.de
CobaltCause
added a commit
to CobaltCause/lemmy
that referenced
this issue
Jun 9, 2023
Woodpecker has a severe deficiency in that caching artifacts across CI runs is not supported. There has been an open issue for this feature for over 1 year [here][0]. It is also not possible to run its CI definition file locally; creating a pull request and pushing commits is the only way to get the CI pipeline to check your code. In Lemmy's case, the result of these facts is that the edit-compile-test feedback loop is drawn out to 46 minutes just for the test portion. See [here for an example][1]. Having to wait this long is a pretty serious reduction in current productivity and potential productivity of developers who lose interest/patience and give up before being able to contribute a working patch. While the maintainers understandably ["don't want to rely on Microsoft"][0], the repository, code review, and issue tracker all live on GitHub already. Lemmy can reduce its feedback loop latency substantially by making use of GitHub Actions' ability to cache artifacts between runs. Further, by defining the vast majority of the CI pipeline via a third party tool, Lemmy can avoid relying heavily on *any and all* CI vendors. I argue that the tradeoff between * adding this 72 line GitHub Actions file * large reduction in feedback loop latency and wasted CPU recompiling dependencies is massively worth it. Additionally, this particular third party tool is also capable of being run locally without any middleware in the way like Docker. The result is seamless caching of intermediate artifacts when running locally, which amounts to a reduction in CI run time from 46 minutes in Woodpecker to 3 minutes on my computer with a warm cache. [0]: woodpecker-ci/woodpecker#758 [1]: https://woodpecker.join-lemmy.org/LemmyNet/lemmy/pipeline/72/1 [2]: https://matrix.to/#/!tUnhsBRCsiePXfsIGe:matrix.org/$kV3XDo15eRPPb3xiq3gRrYr5Iix9fTGywTC53cMDRR0?via=matrix.org&via=chapo.chat&via=tchncs.de
CobaltCause
added a commit
to CobaltCause/lemmy
that referenced
this issue
Jun 9, 2023
Woodpecker has a severe deficiency in that caching artifacts across CI runs is not supported. There has been an open issue for this feature for over 1 year [here][0]. It is also not possible to run its CI definition file locally; creating a pull request and pushing commits is the only way to get the CI pipeline to check your code. In Lemmy's case, the result of these facts is that the edit-compile-test feedback loop is drawn out to 46 minutes just for the test portion. See [here for an example][1]. Having to wait this long is a pretty serious reduction in current productivity and potential productivity of developers who lose interest/patience and give up before being able to contribute a working patch. While the maintainers understandably ["don't want to rely on Microsoft"][0], the repository, code review, and issue tracker all live on GitHub already. Lemmy can reduce its feedback loop latency substantially by making use of GitHub Actions' ability to cache artifacts between runs. Further, by defining the vast majority of the CI pipeline via a third party tool, Lemmy can avoid relying heavily on *any and all* CI vendors. I argue that the tradeoff between * adding this 72 line GitHub Actions file * large reduction in feedback loop latency and wasted CPU recompiling dependencies is massively worth it. Additionally, this particular third party tool is also capable of being run locally without any middleware in the way like Docker. The result is seamless caching of intermediate artifacts when running locally, which amounts to a reduction in CI run time from 46 minutes in Woodpecker to 3 minutes on my computer with a warm cache. [0]: woodpecker-ci/woodpecker#758 [1]: https://woodpecker.join-lemmy.org/LemmyNet/lemmy/pipeline/72/1 [2]: https://matrix.to/#/!tUnhsBRCsiePXfsIGe:matrix.org/$kV3XDo15eRPPb3xiq3gRrYr5Iix9fTGywTC53cMDRR0?via=matrix.org&via=chapo.chat&via=tchncs.de
CobaltCause
added a commit
to CobaltCause/lemmy
that referenced
this issue
Jun 9, 2023
Woodpecker has a severe deficiency in that caching artifacts across CI runs is not supported. There has been an open issue for this feature for over 1 year [here][0]. It is also not possible to run its CI definition file locally; creating a pull request and pushing commits is the only way to get the CI pipeline to check your code. In Lemmy's case, the result of these facts is that the edit-compile-test feedback loop is drawn out to 46 minutes just for the test portion. See [here for an example][1]. Having to wait this long is a pretty serious reduction in current productivity and potential productivity of developers who lose interest/patience and give up before being able to contribute a working patch. While the maintainers understandably ["don't want to rely on Microsoft"][0], the repository, code review, and issue tracker all live on GitHub already. Lemmy can reduce its feedback loop latency substantially by making use of GitHub Actions' ability to cache artifacts between runs. Further, by defining the vast majority of the CI pipeline via a third party tool, Lemmy can avoid relying heavily on *any and all* CI vendors. I argue that the tradeoff between * adding this 72 line GitHub Actions file * large reduction in feedback loop latency and wasted CPU recompiling dependencies[^1] is massively worth it. Additionally, this particular third party tool is also capable of being run locally without any middleware in the way like Docker. The result is seamless caching of intermediate artifacts when running locally, which amounts to a reduction in CI run time from 46 minutes in Woodpecker to 3 minutes on my computer with a warm cache. [^1]: With an empty cache, the CI run on GitHub Actions takes 30 minutes, which is already a 15 minute improvement over the previous Woodpecker setup. [0]: woodpecker-ci/woodpecker#758 [1]: https://woodpecker.join-lemmy.org/LemmyNet/lemmy/pipeline/72/1 [2]: https://matrix.to/#/!tUnhsBRCsiePXfsIGe:matrix.org/$kV3XDo15eRPPb3xiq3gRrYr5Iix9fTGywTC53cMDRR0?via=matrix.org&via=chapo.chat&via=tchncs.de
CobaltCause
added a commit
to CobaltCause/lemmy
that referenced
this issue
Jun 9, 2023
Fixes <LemmyNet#2652>. Woodpecker has a severe deficiency in that caching artifacts across CI runs is not supported. There has been an open issue for this feature for over 1 year [here][0]. It is also not possible to run its CI definition file locally; creating a pull request and pushing commits is the only way to get the CI pipeline to check your code. In Lemmy's case, the result of these facts is that the edit-compile-test feedback loop is drawn out to 46 minutes just for the test portion. See [here for an example][1]. Having to wait this long is a pretty serious reduction in current productivity and potential productivity of developers who lose interest/patience and give up before being able to contribute a working patch. While the maintainers understandably ["don't want to rely on Microsoft"][0], the repository, code review, and issue tracker all live on GitHub already. Lemmy can reduce its feedback loop latency substantially by making use of GitHub Actions' ability to cache artifacts between runs. Further, by defining the vast majority of the CI pipeline via a third party tool, Lemmy can avoid relying heavily on *any and all* CI vendors. I argue that the tradeoff between * adding this 72 line GitHub Actions file * large reduction in feedback loop latency and wasted CPU recompiling dependencies[^1] is massively worth it. Additionally, this particular third party tool is also capable of being run locally without any middleware in the way like Docker. The result is seamless caching of intermediate artifacts when running locally, which amounts to a reduction in CI run time from 46 minutes in Woodpecker to 3 minutes on my computer with a warm cache. (It's likely you're reading this and wondering what Engage is, so I'll elaborate on that: Engage is a task scheduler and executor based on a directed acyclic graph of dependencies between those tasks, and you can read more about it [here][3]. I've seen a few similar tools but I'm partial to this one due to its simplicity and also the fact that I wrote it.) [^1]: With an empty cache, the CI run on GitHub Actions takes 30 minutes, which is already a 15 minute improvement over the previous Woodpecker setup. With a warm cache, the CI run on GitHub Actions takes 16 minutes, one third as much time as the previous Woodpecker setup. [0]: woodpecker-ci/woodpecker#758 [1]: https://woodpecker.join-lemmy.org/LemmyNet/lemmy/pipeline/72/1 [2]: https://matrix.to/#/!tUnhsBRCsiePXfsIGe:matrix.org/$kV3XDo15eRPPb3xiq3gRrYr5Iix9fTGywTC53cMDRR0?via=matrix.org&via=chapo.chat&via=tchncs.de [3]: https://or.computer.surgery/charles/engage
CobaltCause
added a commit
to CobaltCause/lemmy
that referenced
this issue
Jun 9, 2023
Fixes <LemmyNet#2652>. Woodpecker has a severe deficiency in that caching artifacts across CI runs is not supported. There has been an open issue for this feature for over 1 year [here][0]. It is also not possible to run its CI definition file locally; creating a pull request and pushing commits is the only way to get the CI pipeline to check your code. In Lemmy's case, the result of these facts is that the edit-compile-test feedback loop is drawn out to 46 minutes just for the test portion. See [here for an example][1]. Having to wait this long is a pretty serious reduction in current productivity and potential productivity of developers who lose interest/patience and give up before being able to contribute a working patch. While the maintainers understandably ["don't want to rely on Microsoft"][0], the repository, code review, and issue tracker all live on GitHub already. Lemmy can reduce its feedback loop latency substantially by making use of GitHub Actions' ability to cache artifacts between runs. Further, by defining the vast majority of the CI pipeline via a third party tool, Lemmy can avoid relying heavily on *any and all* CI vendors. I argue that the tradeoff between * adding this 72 line GitHub Actions file * large reduction in feedback loop latency and wasted CPU recompiling dependencies[^1] is massively worth it. Additionally, this particular third party tool is also capable of being run locally without any middleware in the way like Docker. The result is seamless caching of intermediate artifacts when running locally, which amounts to a reduction in CI run time from 46 minutes in Woodpecker to 3 minutes on my computer with a warm cache. (It's likely you're reading this and wondering what Engage is, so I'll elaborate on that: Engage is a task scheduler and executor based on a directed acyclic graph of dependencies between those tasks, and you can read more about it [here][3]. I've seen a few similar tools but I'm partial to this one due to its simplicity and also the fact that I wrote it.) [^1]: With an empty cache, the CI run on GitHub Actions takes 30 minutes, which is already a 15 minute improvement over the previous Woodpecker setup. With a warm cache, the CI run on GitHub Actions takes 16 minutes, one third as much time as the previous Woodpecker setup. [0]: woodpecker-ci/woodpecker#758 [1]: https://woodpecker.join-lemmy.org/LemmyNet/lemmy/pipeline/72/1 [2]: https://matrix.to/#/!tUnhsBRCsiePXfsIGe:matrix.org/$kV3XDo15eRPPb3xiq3gRrYr5Iix9fTGywTC53cMDRR0?via=matrix.org&via=chapo.chat&via=tchncs.de [3]: https://or.computer.surgery/charles/engage
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Clear and concise description of the problem
Allow for the pipeline to cache artifacts, much like what Travis CI allows you to do: https://docs.travis-ci.com/user/caching/
This is currently possible in Woodpecker, but only if using volumes. It would be pretty neat if volumes weren't required and Woodpecker could cache items without exposing security issues.
Suggested solution
When the pipeline completes, save the indicated directories somewhere. Then, on subsequent runs of the pipeline, mount that directory automatically. Perhaps automatic volume creation?
Alternative
The current method is insecure and should not be available to untrusted users.
Additional context
No response
Validations
The text was updated successfully, but these errors were encountered: