-
-
Notifications
You must be signed in to change notification settings - Fork 360
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pipeline-scoped (temporary) volumes [solves caching, preparation steps, inter step communication, ...] #1452
Comments
Some thoughts:
|
Temporary volumes should be created when a pipeline is executed. To make the names unique we could use some random number and prefix it with For the other types of volumes, you can use some config to allow volumes with names / paths matching some pattern, but this is the same problem we have now with the current volume implementation. So other types of volumes will require trusted projects for now.
That is the idea of temporary volumes:
For other types of volume,s we have the same problem as today. maybe a cleanup job will help.
Probably woodpecker has to take the same route drone has taken: Have different config options per backend. I do not see how a local / ssh backend could support it, but for those you can use a directory with some naming convention. They can write on any place the user has permissions for. BTW: What are people using local / ssh backends for?
Yes please. Maybe we once will externl need volume provider for these scenarios.
My use case is to put some files in the $HOME directory (ssh keys, .npmrc, settings.xml ,...) and to share a docker socket with a serivce container. Having this in the workspace does not help much, as you have to do a lot of config besides just using a third party program off the shelf. |
This would ease the sharing of a Docker socket with a service container. It would be a nice feature! |
Need to keep in mind that docker volumes won't help much if multiple agents are deployed on different hosts or in swarm where agents on restart can change swarm node. Also if you are allowed to specify host path for volume that would definitely create security issues |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Clear and concise description of the problem
In some pipelines/workflows it is necessary to store data created by one step outside the source directory for use in other steps. E.g.
At the moment the only way to share this between the steps is to either
Suggested solution
Main idea
Take the idea drone has already implemented and add a pipeline scoped configuration for volumes and change the semantic for volume configuration in a step to reference these volumes.
On pipeline/workflow level:
In services / steps:
Temporary volumes should be created before the pipeline is run and deleted after the last step of a pipeline has finished.
The use of temporary volumes should not require privileged pipelines
Possible enhancements
Volumes on different kind
This could be enhanced on the pipeline level to provide different types of volumes
To solve the caching issue auto-magically, there could be volumes of type
cache
, which could be handled by some woodpecker magic, if still required. I consider a general caching solution a hard problem, but it could work something like this:Volumes on different scope
One could provide volumes for different scopes (pipeline vs. workflow). This would require the possibility to provide configurations on a pipeline level (multiple workflows)
Quota for volumes
The only bad thing a step could do is to fill up the complete disk where the volumes are stored. There could be some limits on that (defined on the agent level) or in the project settings. Needs further ideas on how / what to limit
To be discussed
To be discussed: What would be the requirement for matrix builds?
E.g.
or something else?
Alternative
At the moment the following is possible (don't know if it is by intention):
Executing the pipeline will create a docker volume on the host (or reuse an existing one) and bind it into the steps container.
But this
Additional context
There are already quite some issues for that or similair topic, but no common sulution
And a PoC on caching:
But i would like to see a common and consistent solution for most (all?) of these issues.
Validations
The text was updated successfully, but these errors were encountered: