This repository contains code to run Deuxfleurs' infrastructure on NixOS.
Our abstraction stack
We try to build a generic abstraction stack between our different resources (CPU, RAM, disk, etc.) and our services (Chat, Storage, etc.), we develop our own tools when needed.
Our first abstraction level is the NixOS level, which installs a bunch of standard components:
- Wireguard: provides encrypted communication between remote nodes
- Nomad: schedule containers and handle their lifecycle
- Consul: distributed key value store + lock + service discovery
- Docker: package, distribute and isolate applications
Fabio is an HTTP and TCP reverse proxy that configures itself with data from Consul.
Traditional load balancers and reverse proxies need to be configured with a config file. The configuration contains the hostnames and paths the proxy is forwarding to upstream services. This process can be automated with tools like consul-template that generate config files and trigger a reload.
Fabio works differently since it updates its routing table directly from the data stored in Consul as soon as there is a change and without restart or reloading.
When you register a service in Consul all you need to add is a tag that announces the paths the upstream service accepts, e.g. urlprefix-/user or urlprefix-/order and fabio will do the rest.
The agent has various configuration options that can be specified via the command-line or via configuration files. All of the configuration options are completely optional. Defaults are specified with their descriptions.
We've started our first agent and registered and queried a service on that agent. This showed how easy it is to use Consul but didn't show how this could be extended to a scalable, production-grade service discovery infrastructure. In this step, we'll create our first real cluster with multiple members.
When a Consul agent is started, it begins without knowledge of any other node: it is an isolated cluster of one. To learn about other cluster members, the agent must join an existing cluster. To join an existing cluster, it only needs to know about a single existing member. After it joins, the agent will gossip with this member and quickly discover the other members in the cluster. A Consul agent can join any other agent, not just agents in server mode.