This post should encapsulate all you need to understand how Burgershot forums and associated services are deployed.
The system is (almost) fully automated and makes use of the following technologies:
- Docker (ઉꦢ)
- Wadsworth
- Watchtower
- Traefik
- Vault from Hashicorp
Wadsworth watches https://github.com/openmultiplayer/deployment and will run `docker-compose up -d` every time it's changed. Unfortunately Wadsworth will not periodically run these commands or compare the current Docker state to the desired state (like Kubernetes) so if something goes down and can't come back up for whatever reason, you must do it manually.
Doing it manually simply involves restarting Wadsworth. Currently, Wadsworth does not run inside a container for one annoying reason that I want to fix soon. It runs inside a tmux instance under the `southclaws` user on the machine. It reads environment variables from .env (temporary solution... sucks, I know).
So, to restart you just `tmux attach` and then re-run the command:
wadsworth run [email protected]:Southclaws/infra_config 2>&1 | logger --tag=wadsworth
Note the pipe into `logger`. If you need to inspect output of failed starts, remove this. This logger goes to syslog and loggly.
Wadsworth makes use of Vault to store credentials securely. The token it has is refreshed every 24 hours and has a TTL of 768 hours. On the offchance Wadsworth is offline for longer than 768 hours, the token will expire and you will need to request a new token with the necessary policy.
To do this, make sure your local instance of Vault is authenticated with vault.southcla.ws and run:
vault token create -policy=wadsworth-velox -ttl=768h
This will output a list of key:value pairs, `token` is what you need. Drop that into the .env file and re-run Wadsworth.
---
Docker issue from 2019-12-17
Two containers got stuck in a broken state: CJ's mongodb and samp-servers-api. These could not be killed with any of the standard commands, remained alive over docker daemon restarts and even a full system upgrade.
The resolution was `aa-remove-unknown`.