Deployment, monitoring, and performance optimization of a microservices technology stack on Kafka, Kubernetes, GCP.
Déploiement, monitoring et optimisation de performances d'une pile technologique de Micro services, sur Kafka, Kubernetes, GCP
- Docker
- Kubernetes (MiniKube, K3s)
- Kafka
- Ansible
- Prometheus
- Grafana
- Python
- Helm
- ansible folder which contains the
.yaml
files to automatically deploy the application and its environment - kubernetes folder which contains the
.yaml
files used by the Ansible scripts to deploy all the Kubernetes ressources - scripts folder which contains the following
.sh
files:chmodAll.sh
: give the execution rights to all the scriptsdeploy-k3s-cluster.sh
: deploy a K3s cluster on a list of hostsdeployEnv.sh
: deploy all the environment needed for the applicationlaunchExperience.sh
: launch the execution of the application, retrieve the data and generate the graphs in thepython/output
foldermnk-requirements1.sh
: install Docker for Minikubemnk-requirements2.sh
: install Kubectl, Minikube, Helm, Python packages and Ansible
NB: You must be in the root folder to use the scripts.
- 18 CPU cores
- 18 GB of RAM
- Execute the following command in order to be able to execute all the scripts
chmod +x scripts/chmodAll.sh && scripts/chmodAll.sh
- Install Docker
scripts/mnk-requirements1.sh
-
Once the previous script is done, exit the machine and reconnect to it in order to update the Docker users group, thus avoiding using the
sudo su
command everytime -
Install Kubectl, Minikube, Helm, Python packages and Ansible
scripts/mnk-requirements2.sh
- Deploy all the ressources
scripts/deployEnv.sh
- Launch the experience
scripts/launchExperience.sh
- An account on Grid5000
- All information about Grid5000 can be found on getting started
- Clone the project on your home directory on a site of Grid5000 (for exemple sophia)
- Get the number of hosts that you want for the K8s cluster
# Exemple of 2 nodes for 2 hours
oarsub -I -l host=2,walltime=2 -t deploy
kadeploy3 debian11-min
- Clone this project in your Grid5000 home directory
git clone https://github.com/antoinebqt/PER2023-045.git
- Modify the IP addresses in K3s/hosts.ini (only one master is allowed)
- Deploy the K3s cluster
NB: You must be in the root folder of the project to use the scripts.
cd PER2023-045
chmod +x scripts/chmodAll.sh && scripts/chmodAll.sh
vim k3s/hosts.ini
pip install ansible # not necessary if already installed
scripts/deploy-k3s-cluster.sh
- Connect to your master node
ssh root@<ip_address>
ssh root@<grid_node_name>
- Deploy the stack
cd ~/PER2023-045
scripts/deployEnv.sh
- Wait 10 minutes then launch the experience
scripts/launchExperience.sh
- Wait until the end of the experience
- Retrieve the experience data
# from your home Grid5000, not the master node
scp -r root@<ip_address>:~/PER2023-045/python/output ~
The data will be in your home directory