Deployment

This repository contains all deployment and server configuration details for Lavinia.

Requirements

• A linux machine/VM (This document will use Ubuntu as reference)
• Terraform ( >= 0.13.0, download page)
• Ansible ( >= 3.8.2, apt-get install ansible)
• Openstack client (should be installed automatically by terraform, >= 5.3.1, pip3 install python-openstackclient)
• An NREC API password
• A Lavinia SSH key (If contributing to the current setup)
• A GPG public encryption key (If contributing to the current setup)
• A registered domain name (If setting up new instance)
• An account for Nginx Amplify (If setting up new instance)

First time set-up

1. git pull https://github.com/Project-Lavinia/Deployment.git
2. Create the file Deployment/keystone_rc.sh as explained in this guide.
3. Create the file: Deployment/ansible/private.yaml with the content:

       letsencrypt_email: <email of project admin>
       amplify_key: <API key from Amplify>
       key_fingerprint: <Public key fingerprint>
   

   (Public key fingerprint will be generated later, if setting up a new instance)
4. In Deployment/terraform, setup the ansible inventory as per this guide (The inventory directory should have the path Deployment/terraform/inventory)
5. In Deployment/terraform/terraform.tfvars modify which IPs should have http/ssh access to the servers.
6. In Deployment/terraform/variables.tf modify the flavour and number of each server type, as well as the domain that should be used.
7. Go through the rest of the terraform files and ensure that you are happy with the settings.
8. In Deployment do source keystone_rc.sh
9. In the Deployment/terraform directory, use the command terraform init

Set up a new instance

1. Complete the First time set-up.
2. Generate an SSH key pair by following the first code-block of this guide.
3. Generate a second key in the directory ~/.ssh/jenkins/
4. Generate a GPG key with gpg --full-generate-key, use the same email as used in Deployment/ansible/private.yaml
5. Export the public key to Deployment/ansible/gpg/lavinia.key with gpg --output <path to key file> --armor --export <email>
6. Use gpg --list-key to get the fingerprint of your key, and add it to the Deployment/ansible/private.yaml file.
7. Log into your domain registrar and change the name servers of your domain as according to this guide. (If you encounter problems, try using ns1.uh-iaas.no and ns2.uh-iaas.no, instead of the nrec domain)
8. In Deployment/terraform do terraform apply
9. Then do .\update_all.sh
10. Open https://jenkins.<your domain> in your browser
11. In Jenkins, log in with username: admin, password: admin, and immediately change the password
12. In Jenkins, install the following jenkins plugins:
    • Blue Ocean
    • Ansible plugin
    • SSH Credentials Plugin
    • Basic Branch Build Strategies Plugin
    • MSTest plugin
13. In Github -> Personal access tokens: Create a new access token called Jenkins_Hooks with the permission: admin:org_hook
14. In Jenkins -> Manage Jenkins -> Configure System -> GitHub:
    • Name: GitHub
    • API URL: leave the default value
    • Credentials -> Add -> Jenkins:
      • Domain: Global credentials
      • Kind: Secret text
      • Scope: System
      • Secret: the Jenkins_Hooks access token
      • ID: github_hooks
      • Description: Github Hooks
    • Manage hooks: checked
15. In Github -> Personal access tokens: Create a new access token called Jenkins_Release with the permission: repo
16. In Jenkins -> Manage Jenkins -> Manage Credentials -> Stores: Jenkins -> System: Global credentials -> Add Credentials:
    • Kind: Secret text
    • Scope: Global
    • Secret: the Jenkins_Release access token
    • ID: jenkins_release_token
    • Description: Github token for uploading releases
17. In Jenkins -> Manage Jenkins -> Manage Credentials -> Stores scoped to Jenkins: Jenkins -> System: Global credentials -> Add Credentials:
    • Scope: Global
    • ID: ansible_key
    • Description: SSH key for Ansible
    • Username: centos
    • Private Key: Enter directly -> Paste the contents of the file ~/.ssh/jenkins/id_rsa
    • Passphrase: Leave empty
18. In Jenkins -> Blue Ocean: create two new pipelines (for Lavinia-API and Lavinia-Client), it should assist you with GitHub configuration
19. For each pipeline, In Jenkins -> Pipeline -> Configure -> Behaviors: Edit the Behaviors so they contain exactly:
    • Discover branches (Exclude branches that are also filed as PRs)
    • Discover pull requests from origin (Merging the pull request with the current target branch revision)
    • Discover tags
    • Clean before checkout (Delete untracked nested repositories: checked)
    • Clean after checkout (Delete untracked nested repositories: checked)
20. For each pipeline, In Jenkins -> Pipeline -> Configure -> Build strategied -> Add:
    • Tags (Ignore tags newer than: (leave empty), Ignore tags older than: 7)

Add/remove web/api instances

It is safe to modify both the number of web and api instanced at the same time, just perform the actions for both web and api at the points where you should do one or the other.

1. Modify the number of instances in Deployment/terraform/variables.tf
2. In the Deployment/terraform directory, use the command terraform apply
3. Then do:
   1. ansible-playbook -i inventory ../ansible/api.yaml or ansible-playbook -i inventory ../ansible/web.yaml
   2. ansible-playbook -i inventory ../ansible/load_balancer.yaml
   3. ansible-playbook -i inventory ../ansible/jenkins.yaml

Handling issues

If you get an error stating that the snap seeding progress is not complete

1. Restart the playbook, it should work the second time around

If nginx is not able to start, the issue is most likely that the ports are already bound

1. SSH into the server
2. Run sudo netstat -tulpn, making a note of the PID of the service using the 80/TCP and 443/TCP ports (probably the nginx service)
3. Run sudo kill -2 <PID>
4. Rerun the playbook

If you get an error while running the playbooks stating that Snap "core" has "install-snap" change in progress.

1. SSH into the related server
2. Run snap changes to identify the task that has status Doing, making a note of the related task ID.
3. Run sudo snap abort <task ID>
4. Rerun the playbook