CI/CD: Deploying our demo

Too long;didn't read

Visit corner.builders for the live previous demo. Use your finger/mouse to look around. On mobile the view is better in landscape orientation, but you have to refresh the page after switching for now.

Our goal:

Defining our deployment strategy: how do we go from code in editor, to modifying our website.

The whole process depends on the complexity of the project and the size of the collaboration, or maybe even regulations for some industries. For us it will be quite straightforward, as we are a team of 1. But even developing solo requires some structure. What if I work for 1 week in my code editor and I realize it's not the right path? How to revert? Should I save each time in a different file? Software development has answered these questions with version control tools. For this project I use Gitlab for this purpose, as well as building, and deploying the build artifacts to our AWS s3 bucket.

A Simple Branching model: Trunk Based development

How do I integrate changes in the main project? Here are some key requirements:

1. Each chunk of work should be focused on a small/atomic and well defined functionality, that is called a feature.
2. These features should be relatively short lived: I should not start 5 features in parallel for weeks, until a point where each one diverges significantly from the others, and merging them becomes a feature of its own
3. As I am alone and cannot rely on other peer reviewer (and also because I cannot trust myself to be right first try), I should set some automatic strict tooling to help me catch errors that is what we call the CI part: Continuous Integration of change. This is done by various tests and analyzer, and blocking the merge of the feature into main, until problems are addressed.
4. We a feature is done, deployment to our website should be automatic. This is the CD part: Continuous Deployment.

.gitlab-ci.yml

All those steps can be described as code in gitlab in a .gitlab-ci.yml file. This file principal components are jobs that can be executed by workers to fulfill so checks/tasks. Let's have a practical look of my .gitlab-ci.yml for my Rust project.

I will separate by jobs for clarity, and also share the file at the end for those interested. The gitlab documentation is quite extensive with plenty of examples if you need a reference.

image + before script

image: rust:latest

before_script:
  - echo "gitlab.com $SSH_GITLAB_DOT_COM_PUBLIC_KEY" >> /etc/ssh/ssh_known_hosts
  - eval $(ssh-agent -s)
  - ssh-add <(echo "$SSH_PRIVATE_KEY")         

image: rust:latest means we will work in a docker image that has already some tools and program useful for rust development, so we won't have to install them ourselves

before_script this script will execute before each script. Cargo which is the build tool of rust will have to fetch some of my own forks (not yet on crates.io) and needs to communicate with gitlab.com. The pair ($SSH_GITLAB_DOT_COM_PUBLIC_KEY/$SSH_PRIVATE_KEY) is used to access my repos. Note that for bigger collaborative projects, other methods should be used.

stages:
  - check
  - build
  - deploy        

stages: For organization, our jobs are separated in 3 stages, some checks, some build, and the final deploy.

variables:
  CARGO_HOME: $CI_PROJECT_DIR/cargo

cache:
  key: "$CI_JOB_NAME"
  paths:
    - ./target/
    - $CARGO_HOME        

cache: We will try to reuse the output of previous jobs and not always start from scratch, the cache can allow that.

check:
  stage: check
  script:
    - cargo check        

cargo check: In rust this is quite the most important, as your compiler is your biggest check. Because the rust compiler comes with so much guarantees about memory safety, thread safety, lifetime correctness, if your code compiles, you are already 95% there, and if it does not compile, the compiler will usually tell you very precisely where to look and even give suggestions on how to fix.

format:
  stage: check
  script:
    - rustup component add rustfmt
    - cargo fmt -- --check        

cargo format: you usually want to keep the formatting consistent (because why having it non-consistent if you can avoid?) this ensures it. Those tools can (should) be integrated in your code editor, so committing and pushing from it should already autoformat, but still nothing prevents change that is not compliant, so it's a nice safe-guard to have in the CI.

clippy:
  stage: check
  script:
    - rustup component add clippy
    - cargo clippy -- -D warnings        

cargo clippy: this is a tool that will yell at you because even if your code is correct, it thinks it could be better written and will tell you how... and you should probably listen. There are several level of strictness available.

test:
  stage: check
  script:
    - cargo test        

cargo test: This will execute your tests. Always running your tests and adding new ones ensures what you did at least didn't break the previous tests. Otherwise it will also fail the CI pipeline, and you have to fix them (and thank your previous self to have written those)

coverage:
  stage: check
  image: xd009642/tarpaulin
  script: cargo tarpaulin --ignore-tests --out Xml --out Html --output-dir cov
  coverage: /^\d+.\d+% coverage/
  artifacts:
    paths:
      - cov/
    reports:
      coverage_report:
        coverage_format: cobertura
        path: cov/cobertura.xml        

coverage: this job will do some stats on how many lines of your code are actually tested. The regex is for gitlab itself to pick up the number from the job log, that can show you in the merge request presentation and in the general view the percentage. The higher the better, usually.

cargo-audit:
  stage: check
  script:
    - cargo install cargo-audit
    - cargo audit        

cargo-audit: This one will warn if there are security vulnerabilities in the dependencies.

pages:
  stage: deploy
  script:
    - mkdir public
    - cp cov/tarpaulin-report.html public/index.html
  artifacts:
    paths:
      - public
  only:
    - main        

Coverage report: this job will publish the coverage report, so it's browsable as a website, allowing to go to each module/function/lines and see what is tested and not.

build:
  stage: build
  image: rust:latest
  script:
    - rustup target add wasm32-unknown-unknown
    - cargo install --locked trunk --version 0.17.5
    - cargo install --locked wasm-bindgen-cli
    - export PATH=/builds/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/cargo/bin:$PATH
    - trunk build --release
  artifacts:
    paths:
      - dist/
  only:
    - main        

Building the release bundle: this job use trunk to generate a static website in a dist/ deployable folder.

deploy:
  stage: deploy
  image: amazonlinux:latest
  before_script:
    - yum install -y openssh-clients
    - echo "gitlab.com $SSH_GITLAB_DOT_COM_PUBLIC_KEY" >> /etc/ssh/ssh_known_hosts
    - eval $(ssh-agent -s)
    - ssh-add <(echo "$SSH_PRIVATE_KEY")
  script:
    - yum install -y awscli
    - aws s3 sync dist/ s3://www.corner.builders/ --delete
    - aws cloudfront create-invalidation --distribution-id $CLOUDFRONT_DISTRIB_ID --paths "/*"
  only:
    - main        

Deploy to s3: the final job, will copy to s3 (and delete the previous files) and force a resynchronisation of the CloudFront edge locations (not necessary, but make the changes available faster, otherwise you get the previous cached version for a few minutes still).

Also of course, deploying only happen on main, so after merging (you should protect and disable pushing change directly to main now). You can also note that the before_script is again here, because we switch our image to amazonlinux.

To conclude:

we have a pipeline that propagate our code change directly to corner.builders.

I triggered the pipeline, so now the previous demo is available in the website!

You should be able to look around with a finger on the screen. It works better in landscape orientation, but resizing in bugged for now, so you should reload the page after switching to landscape. Let's go have a look!

What's next

Integrating a GUI (Graphical User Interface) to add some ways to interact.

See you soon!