Introduce cache layer to improve performance

[wy65701436]

In the currely design, we found that in large scale scenario usage, Harbor in some particular case will reache the DB connection limit and high CPU usage.

However, it doesn't have to let all get/list requests go directly to the database. If it introduces an cache layer before DB, it will significantly improve concurrency capability.

[Vad1mo]

We should also evaluate the option to use a third party DB layer cache present in Connection Pooler ( PgBouncer vs. Pgpool-II)

Also, if you plan to use a cache, we need to have an in Code Distributed Cache (ICDC) such as https://github.com/buraksezer/olric in order to scale horizontally.

Here I propose to switch to pgx the golang PostgreSQL driver which has some caching options as well.
related: #15209

[buraksezer]

Hi all,

Thank you for mentioning Olric. I'm the author of that library. If you have any questions, it will be my pleasure to help you.

[xaleeks]

let's aim for a concise problem statement and a design doc completed by v2.5, including estimation of performance improvement etc. Assigning to @wy65701436

[chlins]

Move to 2.6 after discussions. cc @xaleeks

[chlins]

Action Items

• Replace harbor-core redis library. #16455
• Implement beego session provider. #16456
• Cache layer for harbor resource. #16457
• Cache layer for manifest. #16459

[schrej]

Have you considered implementing connection pooling and re-using existing connections instead of opening one per request?
With our deployment we mainly have issues with the amount of connections getting opened by harbor, and it seems like it's one for each individual request. With authentication enabled that leads to pretty significant CPU overhead for authentication.

Imo it would be a better approach to try optimising the usage of the database before adding an additional layer.

Edt: After digging through the code, it seems like connection pooling is already enabled. Why is it that it needs that many connections then? Do they get locked up by transactions?
How does the database interaction work when uploading images for example, is it creating a transaction that takes as long as the upload?

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[chlins]

@schrej The cache layer is an abstract concept which based on the harbor codebase, there is no additional component will be introduced, we just cache the mostly used resource to the redis for quick search and reduce the database connection, the question you mentioned need to analyze case by case, so feel free to file issue when you met the db connection issue and describe your scenario, thanks.

[chlins]

Close this epic as engineer story has been completed.