Context
As I may have mentioned in previous articles I run my services using Ansible to deal with the deployment and upgrade of different components. If we generalize the setup, it mostly consists of a reverse proxy (Caddy) listening on 443 and app stacks running under docker on their own network that connects to Caddy’s one. Something that looks like that:
Issue
My Ansible playbook is structured in a way where the reverse proxy is spawned first and then each app is deployed one by one, registering its own configuration against Caddy’s API in order to be served. This way of structuring it has lead me to a nice place where each service is described as a var of my role which then iterates over each service. The var describes among other things the docker-compose definition, networking, volumes to be backed up, DNS, and the part that gets our attention today: the Caddy API payload. While I love this organization, I have recently grown weary of the time it takes to execute the complete playbook and as such I started looking for culprits. You guessed it, a big bottleneck is indeed the declaration to Caddy’s API.
For reference, here is the kind of vars describing the services we are talking about:
---
dns_domain: mydomain.com
applications:
- dns_entry:
record: app_a
value: xx.xx.xx.xx
state: present
payload:
match:
- host:
- "app_a.mydomain.com"
handle:
- handler: reverse_proxy
upstreams:
- dial: "app_a:8080"
terminal: true
- dns_entry:
record: app_b
value: xx.xx.xx.xx
state: present
payload:
match:
- host:
- "app_b.mydomain.com"
handle:
- handler: reverse_proxy
upstreams:
- dial: "app_b:80"
terminal: true
vpn: true
The issue has nothing to do with Caddy itself which is a very nice piece of software, but everything to do with the architecture of the Playbook which falls on me. By pursuing this idea of declaring each service independently I found myself forced to do two things:
- Having an ID for each part of the configuration so that I can query it back later when it eventually changes.
- Querying the API multiple times for each service.
The need of an ID is no big deal in itself, but I kinda dislike them when they are not really needed, you never know how it is going to evolve and if you’re going to end up with conflicts or whatnot. But I’m forced to query the API multiple times during declaration which leads to a lot of unnecessary Ansible’s tasks and HTTP calls. In fact, I need to call the API exactly 4 times in the worst case, worst case being that the service already has a configuration registered by Caddy:
- Insert the hostname in the TLS automation part of Caddy’s configuration.
- Get current configuration for service if any.
- Delete current configuration if it is different from the new one.
- Register the new configuration for the service.
For reference, here is an example of what Caddy’s payload looks like in full:
{
"apps":{
"http":{
"servers":{
"myserver":{
"listen":[
":443"
],
"routes":[
{
"handle":[
{
"handler":"reverse_proxy",
"upstreams":[
{
"dial":"app_a:8080"
}
]
}
],
"match":[
{
"host":[
"app_a.mydomain.com"
]
}
],
"terminal":true
}
]
},
"myserver-vpn":{
"listen":[
":5556"
],
"routes":[
{
"handle":[
{
"handler":"reverse_proxy",
"upstreams":[
{
"dial":"app_b:80"
}
]
}
],
"match":[
{
"host":[
"app_b.mydomain.com"
]
}
],
"terminal":true
}
]
}
}
},
"tls":{
"automation":{
"policies":[
{
"issuers":[
{
"challenges":{
"dns":{
"provider":{
"api_token":"myapitoken",
"name":"myprovider"
}
}
},
"module":"acme"
},
{
"challenges":{
"dns":{
"provider":{
"api_token":"myapitoken",
"name":"myprovider"
}
}
},
"module":"zerossl"
}
],
"subjects":[
"app_a.mydomain.com",
"app_b.mydomain.com"
]
}
]
}
}
}
}
And here is what the Ansible playbook looks like:
## main.yml
---
- name: Deploy application
include_tasks: deploy-application.yml
loop: "{{ applications }}"
loop_control:
loop_var: application
## deploy-application.yml
---
- include_tasks: register-dns.yml
- include_tasks: inject-reverse-proxy-payload.yml
- include_tasks: deploy-compose-stack.yml
- include_tasks: connect-reverse-proxy-to-containers.yml
## inject-reverse-proxy-payload.yml
---
- name: Register hostname in TLS automation
# Whatever needs to be done here
- name: Get currently loaded configuration for route
# Whatever needs to be done here
- name: Load configuration for route if needed
# Whatever needs to be done here, DELETE and POST
Solution
After thinking it through, I realized that it does not make sense with our use case, we could potentially compute the whole payload at once and push it once. This would solve both of my issues: only one call to Caddy’s API instead of 4xServices (worst case scenario), and no need to use IDs anymore as I won’t be modifying parts of it anymore, if I need to update the configuration I re-send the whole one to Caddy.
In order to implement this, we can leverage on Jinja2 which is the templating engine bundled with Ansible. The payload being “rather big”, I prefer isolating it inside a template file instead of directly building it inside the playbook. I can then render the template inside a variable using lookup('template', 'path/to/template.j2').
The templates looks like this:
{% set public_payloads_list = [] %}
{% set private_payloads_list = [] %}
{% for application in applications %}
{% if 'vpn' in application and application.vpn == true %}
{{ private_payloads_list.append(application.payload) }}
{% else %}
{{ public_payloads_list.append(application.payload) }}
{% endif %}
{% endfor %}
{% set domain_names = [] %}
{% for application in applications %}
{{ domain_names.append(application.dns_entry.record + "." + dns_domain) }}
{% endfor %}
{
"apps": {
"http": {
"servers": {
"{{ inventory_hostname }}": {
"listen": [
":443"
],
"routes": {{ public_payloads_list | to_json }}
},
"{{ inventory_hostname }}-vpn": {
"listen": [
":5556"
],
"routes": {{ private_payloads_list | to_json }}
}
}
},
"tls": {
"automation": {
"policies": [
{
"subjects": {{ domain_names | to_json }},
"issuers": [
{
"challenges": {
"dns": {
"provider": {
"api_token": "{{ provider_dns_secret }}",
"name": "myprovider"
}
}
},
"module": "acme"
},
{
"challenges": {
"dns": {
"provider": {
"api_token": "{{ provider_dns_secret }}",
"name": "myprovider"
}
}
},
"module": "zerossl"
}
]
}
]
}
}
}
}
The template starts with two loops, one to render to json and dispatch the service’s payload between private_payloads_list and public_payloads_list based on whether or not it is supposed to be access via the VPN, and the other one to collect and rebuild all the hostnames of the services. All thoses lists are then rendered inside the template after going through the to_json filter in order to format them correctly for our payload.
Now that the template is built, we can render it directly inside a variable but not before sending it through a from_json filter so that it fails if anything is broken at this point. Overall, the Ansible playbook now looks like this:
## main.yml
---
- import_tasks: register-dns.yml
- import_tasks: inject-reverse-proxy-payload.yml
- name: Deploy application
include_tasks: deploy-application.yml
loop: "{{ applications }}"
loop_control:
loop_var: application
## inject-reverse-proxy-payload.yml
---
- name: Inject Caddy payload
vars:
caddy_payload: "{{ lookup('template', '../path/to/caddy-payload.json.j2') | from_json }}"
community.docker.docker_container_exec:
container: caddy
command: |
curl -X POST http://localhost:2019/load
-H "Content-Type: application/json"
-d '{{ caddy_payload | to_json }}'
Conclusion
The deployment of my stack is now much faster than it was. Of course there are other areas of my playbook that could still be optimized but this removed the most obvious time consuming part of it. I guess the conlusion is that although I philosophically preferred the previous way of dealing with services registration in Caddy in the sense that it felt more “encapsulated”, this new version is much more practical and elegant (no more IDs yay), much so that I tried to apply the same principles to the dns registration part but sadly could not find a way as it seems that the Ansible module for my provider only allows the definition of one domain at a time.