feat: added central caching #257
Conversation
Every vendir directory is processed individually and placed into a central cache.
|
Hey 🖖🏽,
I haven't taken a closer look at the code yet, but I have to say the way it works looks really promising. |
|
Oh, I've found a disadvantage, but maybe you have an idea: |
Thanks for the feedback @kbudde! Good point about the .myks folder. I will move it as well. |
I was thinking about that. We could add a slug to the cached folder names that contains versions or other characteristics, e.g. paths for local directories, branch names for git repos. |
|
Tested again, today with bad internet connection ;) There is still one unexpected difference. #@ load("@ytt:data", "data")
#@ app = data.values.application
---
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: #@ "charts/" + app.name
contents:
- path: .
helmChart:
name: #@ app.name
version: #@ app.version
repository:
url: #@ app.url
# just copy paste from above and rename as a very simple example of a second chart:
- path: #@ "charts/" + app.name + "-backup"
contents:
- path: .
helmChart:
name: #@ app.name
version: #@ app.version
repository:
url: #@ app.urlOnly the last chart is in the rendered folder with the new myks |
| #! Configuration options for caching during vendir sync step. | ||
| cache: | ||
| #! If true, the cache is enabled for all apps per default and can be selectively disabled with the vendir lazy flag. | ||
| #! If false, the cache is disabled for all apps per default and can be selectively enabled with the vendir lazy flag. | ||
| enabled: true |
There was a problem hiding this comment.
What do you think about moving this under the sync "namespace"?
| #! Configuration options for caching during vendir sync step. | |
| cache: | |
| #! If true, the cache is enabled for all apps per default and can be selectively disabled with the vendir lazy flag. | |
| #! If false, the cache is disabled for all apps per default and can be selectively enabled with the vendir lazy flag. | |
| enabled: true | |
| #! Configuration options for the sync step. | |
| sync: | |
| cache: | |
| #! If true, the cache is enabled for all apps per default and can be selectively disabled with the vendir lazy flag. | |
| #! If false, the cache is disabled for all apps per default and can be selectively enabled with the vendir lazy flag. | |
| enabled: true |
|
@fritzduchardt I made some changes to the caching logic in 0bded65. Now, instead of deciding whether to run I also restricted vendir configs to a single content item (0bded65#diff-057be4d85a28f33bb6feddad323f77f05bd9484f8b01ea17fabe6491358ddb60R179-R186). This might be not needed, but it's too late and I can't think about this thoroughly anymore 🙃 A non-logic change is refactoring: the cache- and patch-related code is moved out of the |
|
I fear it got worse... 😿 I still had my example repo with two prototypes, where I copied the httpbingo prototype to have two helm charts in the Prototypes:
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: #@ "charts/" + app.name
contents:
- path: .
helmChart:
name: #@ app.name
version: #@ app.version
repository:
url: #@ app.url
- path: #@ "charts/" + app.name + "-backup"
contents:
- path: .
helmChart:
name: #@ app.name
version: #@ app.version
repository:
url: #@ app.url
The vendir cache looks like this: The "normal" httpbingo prototype now suddenly includes the -backup files This might be due to my strange example repo. I think I have to start using multi source prototypes in other projects :) |
|
Yes, it doesn't seem right 😁 |
|
I've found an easier example: The described behavior always happens, if you have to prototypes with a helm chart with the same version but different path. directories:
- path: #@ "charts/foo"
...
---
directories:
- path: #@ "charts/httpbingo"
...will lead to |
|
Just thinking after looking at the code. ---
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: #@ "charts/" + app.name #! or "charts/foo"
contents:
#! <<-- hashed
- path: .
helmChart:
name: #@ app.name
version: #@ app.version
repository:
url: #@ app.url
#! hashed -->>Another option would be to move the path inside contents and enforce "." path on higher level. This would as well ensure a clean hash. ---
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: . #! should this be enforced?
contents:
#! <<-- hashed
- path: #@ "charts/" + app.name #! or "charts/foo"
helmChart:
name: #@ app.name
version: #@ app.version
repository:
url: #@ app.url
#! hashed -->> |
|
Superseded by #274 |
myks render ALL