HTTPS on your GitLab hosted blog with Let's Encrypt

December 27, 2017    hugo gitlab https letsencrypt

Setting up this blog learnt me a couple of things. It’s a Hugo blog, hosted for free on GitLab Pages, with a custom domain name harenslak.nl. One thing I was particularly fond of, was getting encrypted traffic with HTTPS with auto-renewing certificates signed by Let’s Encrypt to work. In this post I explain how the setup works.

GitLab Pages

I choose GitLab for my hosting since it gives me free private repositories. Besides this, GitLab comes with a ton of free and integrated features such as CI/CD and Pages, a free website hosting service perfect for static site generator-blogs. For my blog I choose Hugo. GitLab will automatically publish your website on https://[namespace].gitlab.io. So lastly, I bought my domain name harenslak.nl and configured it to point to my GitLab Pages.

Free certificates with Let’s Encrypt

In order to get SSL/TLS encrypted traffic for your website, a.k.a. HTTPS, you need a certificate signed by a certificate authority (CA), a trusted authority that issues certificates. Before the start of Let’s Encrypt in September 2015, you needed to buy a certificate from such a CA. Let’s Encrypt is the first free CA and allows to generate your certificates within seconds. The underlying certificate generation protocol is Automatic Certificate Management Environment (ACME) and requires one or more “challenges” to verify if you are indeed the owner of the website. For more information on how Let’s Encrypt works, see https://letsencrypt.org/how-it-works.

Generating certificates manually

To understand the process, let’s first generate our certificates manually. You will need certbot for this. Install certbot and run sudo certbot certonly --manual -d harenslak.nl -d www.harenslak.nl. You will be asked several questions:

Enter email address (used for urgent renewal and security notices) (Enter 'c' to
cancel):

Enter your email here.

Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must
agree in order to register with the ACME server at
https://acme-v01.api.letsencrypt.org/directory
-------------------------------------------------------------------------------
(A)gree/(C)ancel:

Agree.

Would you be willing to share your email address with the Electronic Frontier
Foundation, a founding partner of the Let's Encrypt project and the non-profit
organization that develops Certbot? We'd like to send you email about EFF and
our work to encrypt the web, protect its users and defend digital rights.
-------------------------------------------------------------------------------
(Y)es/(N)o:

Optional Y/N.

Obtaining a new certificate
Performing the following challenges:
http-01 challenge for myblog.com
http-01 challenge for www.myblog.com

-------------------------------------------------------------------------------
NOTE: The IP of this machine will be publicly logged as having requested this
certificate. If you're running certbot in manual mode on a machine that is not
your server, please ensure you're okay with that.

Are you OK with your IP being logged?
-------------------------------------------------------------------------------
(Y)es/(N)o:

Yes is required here.

At this point the ACME protocol performs the HTTP challenge which requires you to place a file1 on your webserver for each domain:

Create a file containing just this data:

aWVhKnPtNwv5QB6sFmgVrCVyyvWCFoJkaEjjDnYLHqY.yhwVOtHwhcz8hsMI6GPY0ou8ZxZiD0zKpXAeF2MChF8

And make it available on your web server at this URL:

http://harenslak.nl/.well-known/acme-challenge/aWVhKnPtNwv5QB6sFmgVrCVyyvWCFoJkaEjjDnYLHqY

-------------------------------------------------------------------------------
Press Enter to Continue

-------------------------------------------------------------------------------
Create a file containing just this data:

RazqZqhy5oVRg3tvyLgX9Po6Ac6d97kM5zYt-e2ZVu4.yhwVOtHwhcz8hsMI6GPY0ou8ZxZiD0zKpXAeF2MChF8

And make it available on your web server at this URL:

http://www.harenslak.nl/.well-known/acme-challenge/RazqZqhy5oVRg3tvyLgX9Po6Ac6d97kM5zYt-e2ZVu4

-------------------------------------------------------------------------------
Press Enter to Continue

After the last domain, do not continue yet, place the files on your webserver first! For a Hugo-blog, place the files in /static/.well-known/acme-challenge.

Only after the challenge files are available on your webserver, continue and hopefully you see a congratulations message:

Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/harenslak.nl/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/harenslak.nl/privkey.pem
   Your cert will expire on 2018-03-28. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Copy-paste the certificate and key in your domains in GitLab Pages and you should now have a valid https:// website (the update could take a couple of seconds).

CAA DNS record

Received an error message like this?

DNS problem: SERVFAIL looking up CAA
Failed authorization procedure. www.harenslak.nl (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: DNS problem: SERVFAIL looking up CAA for www.harenslak.nl

As of September 8 2017, every Certificate Authority (CA) is obliged to check for CAA (Certificate Authority Authorisation) DNS records. CAA records control which CAs are allowed to issue certificates for your domain. The goal of CAA records is to reduce the risk of unauthorised issuance of SSL certificates.

Let’s Encrypt requires a CAA record for the verification process so you’ll need to create one if your domain doesn’t have it yet. To inspect the CAA records of any website use dig, for example2:

dig +short google.com CAA
0 issue "pki.goog"

The CAA syntax is: CAA <flags> <tag> <value>, e.g. CAA 0 issue "letsencrypt.org". For this blog, I set CAA records to this value, allowing only letsencrypt.org to issue certificates for my domain.

CAA flag value

While googling, you read lots of discussions on the correct value for the flag. This is what the RFC6844 document says about it:

  • Bit 0: issuer critical flag: if the value is set to ‘1’, the critical flag is asserted and the property MUST be understood if the CAA record is to be correctly processed by a certificate issuer.
  • According to the conventions set out in RFC1035, bit 0 is the Most Significant Bit and bit 7 is the Least Significant Bit. Thus, the Flags value 1 means that bit 7 is set while a value of 128 means that bit 0 is set according to this convention.

In several discussions online, you’ll see flag values 0, 1 or 128. So 0 meaning the CA may continue to issue the certificate if it does not understand the record. 128 meaning the CA may not issue the certificate if it does not understand the record. In practice, Let’s Encrypt interprets both flags 1 and 128 equally. Other values are reserved for possible future usage.

Automatic renewal

At this point, I assume you have HTTPS up and running on your blog. Let’s Encrypt certificates are valid for 90 days and you probably don’t want to manually renew your certificates every 90 days, so I automated the renewal with the help of GitLab Pipeline Schedules.

Certbot supports several plugins for obtaining certificates, we used the manual plugin above. To renew certificates generated with the manual plugin, one can run certbot renew or re-run the same command to renew. However this assumes you renew from the same machine (there’s an /etc/letsencrypt directory with some config files). GitLab Pages are hosted by GitLab and the CI/CD pipeline runs inside a Docker container which clones the blog repository at build time. This makes the renewal a bit tricky. I initially generated certificates on my local machine and pushed the ACME challenge files to my master branch in order to deploy, we need to do something similar but automated.

To do this, I use 3 shell scripts:

letsencrypt_generate.sh

#!/bin/bash

end_epoch=$(date -d "$(echo | openssl s_client -connect harenslak.nl:443 -servername harenslak.nl 2>/dev/null | openssl x509 -enddate -noout | cut -d'=' -f2)" "+%s")
current_epoch=$(date "+%s")
renew_days_threshold=30
days_diff=$((($end_epoch - $current_epoch) / 60 / 60 / 24))

if [ $days_diff -lt $renew_days_threshold ]; then
  echo "Certificate is $days_diff days old, renewing now."
  certbot certonly --manual --debug --preferred-challenges=http -m $GITLAB_USER_EMAIL --agree-tos --manual-auth-hook letsencrypt_authenticator.sh --manual-cleanup-hook letsencrypt_cleanup.sh --manual-public-ip-logging-ok -d harenslak.nl -d www.harenslak.nl
  echo "Certbot finished. Updating GitLab Pages domains."
  curl --request PUT --header "PRIVATE-TOKEN: $CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN" --form "certificate=@/etc/letsencrypt/live/harenslak.nl/fullchain.pem" --form "key=@/etc/letsencrypt/live/harenslak.nl/privkey.pem" https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pages/domains/harenslak.nl
  curl --request PUT --header "PRIVATE-TOKEN: $CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN" --form "certificate=@/etc/letsencrypt/live/harenslak.nl/fullchain.pem" --form "key=@/etc/letsencrypt/live/harenslak.nl/privkey.pem" https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pages/domains/www.harenslak.nl
else
  echo "Certificate still valid for $days_diff days, no renewal required."
fi

letsencrypt_authenticator.sh

#!/bin/bash

mkdir -p $CI_PROJECT_DIR/static/.well-known/acme-challenge
echo $CERTBOT_VALIDATION > $CI_PROJECT_DIR/static/.well-known/acme-challenge/$CERTBOT_TOKEN
git add $CI_PROJECT_DIR/static/.well-known/acme-challenge/$CERTBOT_TOKEN
git commit -m "GitLab runner - Added certbot challenge file for certificate renewal"
git push https://$GITLAB_USER_LOGIN:$CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN@gitlab.com/BasPH/blog.git HEAD:master

interval_sec=15
max_tries=80 # 20 minutes
n_tries=0
while [ $n_tries -le $max_tries ]
do
  status_code=$(curl -L --write-out "%{http_code}\n" --silent --output /dev/null https://harenslak.nl/.well-known/acme-challenge/$CERTBOT_TOKEN)
  if [[ $status_code -eq 200 ]]; then
    exit 0
  fi

  n_tries=$((n_tries+1))
  sleep $interval_sec
done

exit 1

letsencrypt_cleanup.sh

#!/bin/bash

git rm $CI_PROJECT_DIR/static/.well-known/acme-challenge/$CERTBOT_TOKEN
git commit -m "GitLab runner - Removed certbot challenge file"
git push https://$GITLAB_USER_LOGIN:$CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN@gitlab.com/BasPH/blog.git HEAD:master

It is scheduled to run with GitLab Pipeline Schedules with this .gitlab-ci.yml job:

letsencrypt-renewal:
  only:
    - schedules
  variables:
    CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN: $CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN
  script:
    - echo "deb http://ftp.debian.org/debian jessie-backports main" >> /etc/apt/sources.list
    - apt-get update
    - apt-get install certbot -t jessie-backports -y
    - export PATH=$PATH:$CI_PROJECT_DIR
    - git config --global user.name $GITLAB_USER_LOGIN
    - git config --global user.email $GITLAB_USER_EMAIL
    - ./letsencrypt_generate.sh

Let’s break it down. The .gitlab-ci.yml contains a job called letsencrypt-renewal.

GitLab Pipeline Schedule

I created a Pipeline Schedule with the same name to run every day at 04:003. The schedule and CI job names are both “letsencrypt-renewal”, however not related. The CI job simply starts when the schedule starts because of the only - schedules property (and not during normal builds, after a push to master). AFAIK, there is no way to target specific CI jobs from the Pipeline Schedule at this point in time.

I set a variable CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN in the Pipeline Schedule, whose value is a personal access token granting access to the GitLab API, since we need this to push the challenge files and update pages domain certificates. The variable becomes available in the .gitlab-ci.yml file. In the .gitlab-ci.yml, we need to pass it again, this time as an environment variable in the build environment.

Next up, the letsencrypt_generate.sh script. This is the starting point for generating certificates. Certbot can renew certificates super easy with certbot renew, however our build environment is thrown away after building so there’s no letsencrypt history and thus no way to renew previous certificates. The renew command checks if certificates expire within 30 days, so I wrote my own code to achieve the same by checking the certificate enddate from harenslak.nl:

end_epoch=$(date -d "$(echo | openssl s_client -connect harenslak.nl:443 -servername harenslak.nl 2>/dev/null | openssl x509 -enddate -noout | cut -d'=' -f2)" "+%s")
current_epoch=$(date "+%s")
renew_days_threshold=30
days_diff=$((($end_epoch - $current_epoch) / 60 / 60 / 24))

if [ $days_diff -lt $renew_days_threshold ]; then

If this passes, I run the same certbot command as in the manual process, but with some extra flags:

certbot certonly
--manual # Manual plugin
--preferred-challenges=http # Set preferred challenge to http (= check if file exists on webserver)
-m $GITLAB_USER_EMAIL # Set my email
--agree-tos # Agree to ToS
--manual-public-ip-logging-ok # Allow public IP logging
--manual-auth-hook letsencrypt_authenticator.sh # Validation script
--manual-cleanup-hook letsencrypt_cleanup.sh # Cleanup script
-d harenslak.nl -d www.harenslak.nl # Domains

During a CI build, GitLab provides several environment variables, such as my GitLab email address which I use for renewal. Certbot allows custom validation hooks, which I used with --manual-auth-hook and --manual-cleanup-hook. Within these scripts, Certbot provides several environment variables such as the validation string (CERTBOT_VALIDATION) and challenge file name (CERTBOT_TOKEN), for you to use in whatever way you want. Since deploying a file on GitLab Pages requires a push to master and CI deployment, I scripted this part. For each domain I create and commit a challenge file, causing the CI to build and deploy, and for about max. 20 minutes the script continuously checks for HTTP status 200:

interval_sec=15
max_tries=80 # 20 minutes
n_tries=0
while [ $n_tries -le $max_tries ]
do
  status_code=$(curl -L --write-out "%{http_code}\n" --silent --output /dev/null https://harenslak.nl/.well-known/acme-challenge/$CERTBOT_TOKEN)
  if [[ $status_code -eq 200 ]]; then
    exit 0
  fi

  n_tries=$((n_tries+1))
  sleep $interval_sec
done

For my blog, the CI usually takes about a minute and a half to complete and thus for the Certbot authentication script to complete. After both authentications are completed successfully the letsencrypt_cleanup.sh script is called which removes and commits both challenge files again. After this, Certbot should have successfully verified my domains and generated the certificates in /etc/letsencrypt/live/harenslak.nl/. Last piece of the puzzle is to update the GitLab Pages domain using the GitLab API:

curl --request PUT --header "PRIVATE-TOKEN: $CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN" --form "certificate=@/etc/letsencrypt/live/harenslak.nl/fullchain.pem" --form "key=@/etc/letsencrypt/live/harenslak.nl/privkey.pem" https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pages/domains/harenslak.nl
curl --request PUT --header "PRIVATE-TOKEN: $CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN" --form "certificate=@/etc/letsencrypt/live/harenslak.nl/fullchain.pem" --form "key=@/etc/letsencrypt/live/harenslak.nl/privkey.pem" https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pages/domains/www.harenslak.nl

Final words

Quite a read, but I was happy to get the auto-renewing certificates to work. The expiration check is run daily, and only after 60 days are the certificates renewed. This results in 4 commits on my master branch, but I can live with that.

Four commits after certificate renewal

Other ACME challenges are also available and possible, e.g. validation via DNS TXT records. I didn’t investigate that since my provider gives a 2-4 hour timeframe for updating of DNS records.

Thoughts? Comments? Better ways to do this? Always happy to discuss, let me know!

References


  1. For details of the ACME HTTP challenge, read https://tools.ietf.org/html/draft-ietf-acme-acme-09#section-8.3 and https://tools.ietf.org/html/draft-ietf-acme-acme-09#section-8.1. [return]
  2. Although the CAA record is widely supported by now, there was a period in which the new CAA record was unknown for several tools. In case your tool didn’t support the CAA record type yet, you could query with type257, the type id assigned to the CAA resource record by the IANA in RFC6844. [return]
  3. During testing, you might want to renew more often than once a day. Let’s Encrypt has a staging environment for this purpose with much higher rate limits. Run certbot with --staging to use. [return]


comments powered by Disqus