Quick Takeaway: Running production workloads via Kamal 2 requires a dedicated KVM slice with at least 2 vCPUs (%st = 0.0%), 4 GB RAM to prevent cgroups v2 OOM-killer invocations during multi-stage Docker builds, and 40 GB enterprise PCIe 4.0 NVMe storage to absorb high 4K random write IOPS during image layer extraction. Routing traffic through Kamal 2's native kamal-proxy over a 1 Gbps uplink configured with Linux tcp_bbr congestion control and tuned net.core.somaxconn buffers guarantees hitless, zero-downtime rolling container swaps at sub-2ms p99 proxy latency without the compute overhead of a Kubernetes control plane.
Table of Contents
- The Rise of De-clouding: Why Kamal 2 Outperforms AWS and Complex Kubernetes for Micro-SaaS
- KVM VPS Prerequisites, SSH Key Hardening, and Docker Daemon Configuration
- Kamal 2 Architecture Deep Dive: Understanding the New Thruster Proxy and Docker Hooks
- Crafting the Production deploy.yml: Multi-Environment Config and Secrets Management
- Deploying Accessory Services: Production PostgreSQL, Redis, and Background Workers
- Zero-Downtime Rollouts, Rollbacks, and Complete Server Disaster Recovery Runbook
- Frequently Asked Questions (FAQ)
The Rise of De-clouding: Why Kamal 2 Outperforms AWS and Complex Kubernetes for Micro-SaaS
Operating a revenue-generating micro-SaaS on a hyperscaler managed Kubernetes cluster (EKS, GKE) or multi-tier AWS architecture imposes an unsustainable cognitive and financial overhead. For workloads processing between 50 and 5,000 requests per second, the architectural justification for container orchestrators distributed across three Availability Zones collapses under scrutiny. Kubernetes solves fleet scheduling, multi-tenancy isolation for hundreds of development teams, and dynamic cluster auto-scaling. It does not solve—and actively degrades—single-operator developer velocity, I/O predictability, and bottom-line unit economics.
The de-clouding movement is driven by a fundamental hardware shift: modern physical host nodes and enterprise virtual servers deliver compute density, memory capacity, and NVMe throughput that surpass the requirements of 99% of web applications. Kamal 2 provides the minimal orchestration layer required to leverage this bare-metal and KVM performance without sacrificing zero-downtime rolling deployments, automated Let's Encrypt SSL provisioning, or container immutability.
KUBERNETES / AWS OVERHEAD KAMAL 2 ON DEDICATED KVM
┌───────────────────────────────────────┐ ┌───────────────────────────┐
│ CloudWatch / VPC Flow / Systems Mgr │ │ SSH / Docker Engine API │
├───────────────────────────────────────┤ ├───────────────────────────┤
│ CoreDNS / CNI Overlay (VXLAN / Geneve)│ │ Host Bridge / host netns │
├───────────────────────────────────────┤ ├───────────────────────────┤
│ Kube-Proxy / iptables / IPVS │ │ Kamal Proxy (Go / epoll) │
├───────────────────────────────────────┤ ├───────────────────────────┤
│ Kubelet / etcd Consensus Engine │ │ Systemd / cgroups v2 │
├───────────────────────────────────────┤ ├───────────────────────────┤
│ AWS Hypervisor (Xen/Nitro Steal Time) │ │ KVM Bare Virtualization │
│ Networked EBS (1.5-3.5ms p99 I/O) │ │ Local NVMe (<50µs p99 I/O)│
└───────────────────────────────────────┘ └───────────────────────────┘
The Financial and Architectural Tax of Hyperscalers
The primary driver away from public cloud orchestrators is the silent accumulation of infrastructure surcharges that do not contribute to application throughput:
- The Idle Network Tax: An idle AWS Elastic Kubernetes Service (EKS) control plane incurs a fixed cost of $73.00/month. Routing egress through an AWS Managed NAT Gateway costs $32.40/month per Availability Zone in hourly upkeep plus $0.045/GB in data processing fees. A multi-AZ deployment with an Application Load Balancer ($22.50/month base + LCU fees) and inter-AZ data transfer ($0.01/GB cross-AZ) accumulates over $180/month before provisioning a single vCPU of actual application compute.
- Network-Attached Storage Latency: AWS Elastic Block Store (EBS) gp3 volumes introduce deterministic network hops over the Nitro fabric. While AWS guarantees baseline burst performance, typical EBS gp3 write latencies hover at 1.2–3.5 ms for write-ahead log (WAL) syncs. In high-concurrency database transactions, this latency ceiling bounds single-thread database write throughput.
- Noisy Neighbors and CPU Steal: Public cloud burstable instances (
t3,t4g) and shared compute tiers partition physical cores with high oversubscription ratios. When co-tenant workloads spike, your hypervisor schedule queue stalls, manifesting as non-zero CPU Steal Time (%stintopand/proc/stat). A%stvalue as low as 1.5% routinely cascades into tail-latency degradation, spiking API p99 response times from 45 ms to over 850 ms.
By contrast, deploying via a clean kamal 2 vps deployment guide docker pattern onto dedicated-resource KVM hardware eliminates the distributed consensus layer. On high-performance infrastructure such as tropic.host, virtualization runs with strict zero oversubscription guarantees (%st = 0.0% continuously verified under full load), backed by enterprise AMD EPYC and Ryzen 9 processors. Storage is direct-attached PCIe 4.0 NVMe, delivering random 4K QD1 reads in excess of 50,000 IOPS and p99 storage access latencies below 50 microseconds.
Infrastructure Architecture & Cost Matrix
The following benchmark and cost matrix contrasts a standard three-node resilient application architecture across AWS EKS, a generic managed Kubernetes platform, and Kamal 2 deployed on high-spec KVM VPS nodes.
| Parameter / Metric | AWS EKS Multi-AZ Enterprise Stack | Generic Managed K8s (Linode / DO) | Kamal 2 on High-Performance KVM VPS (tropic.host) |
|---|---|---|---|
| Control Plane Overhead | $73.00/mo fixed + etcd compute | $0.00 – $12.00/mo | $0.00 (stateless local CLI runner) |
| Ingress / Load Balancer | $22.50/mo (ALB) + LCU data usage | $12.00/mo (Cloud LB) | $0.00 (Integrated Kamal Proxy in-process) |
| NAT Gateway / Egress Tax | $0.045/GB processed + $32.40/mo/AZ | $0.00 (Public IPs per node) | $0.00 (Direct static IPv4, unmetered port) |
| Baseline Compute Config | 3x m6i.large (2 vCPU, 8 GB RAM) |
3x Standard (2 vCPU, 8 GB RAM) | 1x High-Frequency KVM (8 vCPU, 32 GB RAM) |
| Monthly Compute Run-Rate | ~$380.00 – $550.00/mo | ~$144.00 – $180.00/mo | ~$45.00 – $65.00/mo |
CPU Steal Time (%st) |
Variable (0.0% – 4.2% on shared Nitro) | Variable (0.5% – 2.5%) | Strictly 0.0% Guaranteed |
| Storage Subsystem | AWS EBS gp3 (1.2–3.5 ms p99 latency) | Networked Block Storage (1–4 ms) | Direct PCIe 4.0 NVMe (<50 µs p99 latency) |
| Zero-Downtime Engine | RollingUpdate + PodDisruptionBudget | RollingUpdate + Ingress controller | Kamal Proxy (buffer + instant socket swap) |
| Deployment Engine / Daemon | Kubelet + containerd + kube-proxy | Kubelet + containerd + Flannel | Vanilla Docker Engine via SSH |
| Network Transit Engine | AWS VPC CNI (ENI allocation limits) | Overlay CNI (VXLAN encapsulation) | Host Network / Native Docker Bridge + TCP BBR |
| Engineering Time to Maintain | ~15–20 hours/mo (CVEs, CNI, drift) | ~8–12 hours/mo (etcd, ingress upgrades) | <1 hour/mo (standard Linux OS patching) |
How Kamal 2 Solves Zero-Downtime Without Kubernetes
Kamal 1 relied on Traefik as an edge router, using dynamic Docker label polling to detect container transitions. This model suffered from race conditions: Traefik’s health-check polling loops could route requests to an initializing container or drop inflight HTTP requests during container teardown.
Kamal 2 replaces Traefik entirely with Kamal Proxy, a lightweight, zero-dependency reverse proxy written in Go that runs directly on the host as a static Docker container. Kamal Proxy binds to host ports 80 and 443 and handles zero-downtime blue/green traffic shifts at the connection level:
- Target Registration: Kamal issues an SSH instruction to Docker to pull and run the new application container image on an unexposed bridge port.
- Active Health Polling: Kamal Proxy polls the configured health check endpoint (
/upor/healthz) directly over the bridge network. - Connection Pausing & Atomic Swap: Once healthy, Kamal Proxy pauses incoming TCP accept loops for milliseconds, buffers incoming HTTP requests, seamlessly swaps the active upstream target from the old container ID to the new container ID, and drains the remaining connections on the old container over a configured timeout window.
- Teardown: The retired container receives
SIGTERM, processes remaining in-flight requests, and is safely removed viadocker stopanddocker rm.
This proxy-mediated transition requires zero distributed state machines, no etcd quorum elections, and no external ingress controllers. The proxy manages Let's Encrypt TLS handshakes natively through ACME, writing certificates to a persistent host mount.
Linux Kernel & Host Tuning for Single-Node KVM Throughput
To extract maximum performance from a single KVM VPS running Docker and Kamal 2, the host operating system must be tuned for high socket concurrency, aggressive connection reuse, and minimal network serialization latency.
Deploy the following production sysctl configuration to /etc/sysctl.d/99-kamal-host.conf on your tropic.host instance:
# /etc/sysctl.d/99-kamal-host.conf
# File descriptor and system-wide socket limits
fs.file-max = 2097152
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 8192
# Network connection handling and backlogs
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_max_syn_backlog = 65535
# TCP memory buffers and buffer autotuning (Min, Default, Max in bytes)
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# Connection reuse and ephemeral port allocations
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_slow_start_after_idle = 0
# Modern congestion control: BBR paired with fair queueing
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
# Memory management and page cache writeback behavior
vm.swappiness = 10
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5
vm.overcommit_memory = 1
Activate the configuration immediately without rebooting:
sudo sysctl --system
Verify that TCP BBR is actively engaged on the active interfaces:
sysctl net.ipv4.tcp_congestion_control
# Output: net.ipv4.tcp_congestion_control = bbr
Production Kamal 2 Configuration (deploy.yml)
The following annotated configuration provides a production baseline for a high-concurrency micro-SaaS deploying via Kamal 2. It configures Kamal Proxy, enforces strict memory and CPU boundaries via Linux cgroups v2, and mounts high-performance local NVMe storage for persistent SQLite, Redis, or embedded search indices.
# config/deploy.yml
service: saas-core
image: registry.gitlab.com/organization/saas-core
servers:
web:
hosts:
- 185.196.xxx.xxx # Clean dedicated IPv4 provisioned via tropic.host
labels:
traefik.http.routers.saas.rule: "" # Unused in Kamal 2 (Traefik deprecated)
options:
network: "host"
ulimit:
- "nofile=65535:65535"
- "nproc=4096:4096"
# Enforce cgroups v2 resource limits to protect the host OS
cpus: "7.0"
memory: "28g"
memory-reservation: "24g"
oom-kill-disable: false
proxy:
ssl: true
host: app.domain.com
app_port: 3000
healthcheck:
path: /up
port: 3000
interval: 2
timeout: 3
max_attempts: 10
buffering:
max_request_body: 20971520 # 20MB payload limit
registry:
server: registry.gitlab.com
username: deploy-token
password:
- KAMAL_REGISTRY_PASSWORD
env:
clear:
PORT: 3000
RAILS_ENV: production
RAILS_MAX_THREADS: 16
WEB_CONCURRENCY: 4
MALLOC_ARENA_MAX: 2
secret:
- SECRET_KEY_BASE
- DATABASE_URL
asset_path: /rails/public/assets
builder:
arch: amd64
remote: ssh://[email protected]
args:
RUBY_VERSION: 3.3.5
volumes:
- "/var/data/saas-storage:/rails/storage:rw"
Verification and Runtime Telemetry
Once the host is provisioned and deployed via kamal setup followed by kamal deploy, verify that process scheduling and memory constraints are operating with zero contention directly through the Linux runtime:
# Verify system CPU steal time is strictly 0.0%
vmstat 1 5
# Look at the 'st' column under 'cpu'
# Audit cgroups v2 controller enforcement on running Kamal containers
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}\t{{.NetIO}}\t{{.BlockIO}}"
Operating your micro-SaaS on a clean, dedicated KVM virtualization layer cuts cloud bills by 80–90% while improving application performance. By relying on Kamal 2, you retain zero-downtime deploy mechanics and containerized build pipelines, while eradicating the massive cognitive and financial burden of distributed cloud infrastructure.
KVM VPS Prerequisites, SSH Key Hardening, and Docker Daemon Configuration
Executing zero-downtime container rollouts with Kamal 2 requires an underlying operating system that is deterministically configured for high-concurrency packet processing, zero memory thrashing, and rapid daemon restarts. A default Ubuntu 24.04 LTS (Noble Numbat) installation ships with generic kernel defaults, unbounded log drivers, and permissive SSH parameters that introduce latency spikes and security exposure in production environments.
Deploying your workload onto a clean KVM slice from tropic.host establishes the necessary baseline: 100% dedicated vCPU scheduling with zero hypervisor steal time (%st = 0.0%), direct access to cgroups v2 controllers, high-end AMD EPYC or Intel Xeon compute cores, and enterprise PCIe 4.0 NVMe arrays sustaining random 4K QD1 reads above 50,000 IOPS. With hardware contention eliminated at the hypervisor layer, the host OS and container runtime must be aligned with this performance profile.
Host OS Baseline and Hardware Verification
Kamal 2 relies on a remote SSH connection to orchestrate the Docker daemon, manipulate container namespaces, and manage Kamal Proxy routing. Before installing runtime dependencies, verify that the host kernel supports unified cgroup hierarchies and that hypervisor overcommitment is absent.
# Verify Ubuntu 24.04 kernel release (6.8+ expected)
uname -r
# Confirm cgroups v2 unified hierarchy is mounted
mount -t cgroup2
# Verify absence of CPU steal time under no load
vmstat 1 3 | awk '{print $16}'
# Benchmark baseline 4K QD1 NVMe read latency and IOPS
sudo fio --name=nvme-baseline --filename=/tmp/test.fio --rw=randread \
--bs=4k --ioengine=libaio --iodepth=1 --runtime=10 --time_based \
--direct=1 --output-format=terse | cut -d';' -f8,49
On tropic.host, the st (steal) column will read strictly 0, ensuring that the host scheduler has unhindered access to the underlying hardware execution units. Storage latency for random 4K read operations should register under 150 microseconds (p99), preventing container image layer decompression stalls during zero-downtime blue/green switches.
Update the base repositories, apply distribution-level patches, and install core administrative packages:
export DEBIAN_FRONTEND=noninteractive
sudo apt-get update -y && sudo apt-get upgrade -y
sudo apt-get install -y \
curl \
ca-certificates \
gnupg \
lsb-release \
iptables \
iproute2 \
jq \
htop \
sysstat
Non-Root Automation Account and SSH Hardening
Kamal 2 can authenticate as root or as a privileged non-root deployment user. In hardened multi-tenant or enterprise environments, running deployment automation through a dedicated user with scoped passwordless sudo privileges for Docker execution limits the attack surface.
1. Provision the Dedicated Deploy User
Create the kamal user, configure its home directory, and grant passwordless sudo execution specifically tailored for Docker operations:
# Create the deployment user with bash shell
sudo adduser --disabled-password --gecos "Kamal Automation" kamal
# Add user to the docker group (created after Docker installation) and sudo
sudo usermod -aG sudo kamal
# Grant passwordless sudo privilege to allow Kamal remote system manipulation
echo "kamal ALL=(ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/99-kamal-deploy
sudo chmod 0440 /etc/sudoers.d/99-kamal-deploy
2. Ed25519 Key Exchange and SSH Daemon Lockdown
Generate an Ed25519 keypair on your local deployment workstation:
# Execute on local workstation
ssh-keygen -t ed25519 -a 100 -C "kamal-deploy@workstation" -f ~/.ssh/id_ed25519_kamal
ssh-copy-id -i ~/.ssh/id_ed25519_kamal.pub kamal@<YOUR_VPS_IP>
Lock down the OpenSSH server on the target node. In Ubuntu 24.04, SSH configuration is modularized under /etc/ssh/sshd_config.d/. Create an overriding drop-in configuration file:
sudo tee /etc/ssh/sshd_config.d/99-hardening.conf << 'EOF'
# Network and Protocol Settings
Port 22
AddressFamily inet
Protocol 2
# Authentication Hardening
PermitRootLogin prohibit-password
PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
KbdInteractiveAuthentication no
# Cryptographic Primitives (Enforce modern curves and ciphers)
KexAlgorithms curve25519-sha256,[email protected],diffie-hellman-group-exchange-sha256
Ciphers [email protected],[email protected],[email protected]
MACs [email protected],[email protected]
# Connection Discipline
MaxAuthTries 3
MaxSessions 5
LoginGraceTime 30
ClientAliveInterval 300
ClientAliveCountMax 2
X11Forwarding no
AllowAgentForwarding no
AllowTcpForwarding no
EOF
# Validate syntax before reloading the daemon
sudo sshd -t && sudo systemctl restart ssh
This enforces asymmetric Ed25519 public-key authentication, rejects legacy cipher suites prone to algorithmic degradation, and terminates idle connection handles that could otherwise exhaust the SSH process table.
Host Kernel Optimization (sysctl)
High-throughput container proxying through Kamal Proxy demands low network stack overhead, rapid socket cleanup, and sufficient virtual memory allocation for database engines or in-memory caches.
Create /etc/sysctl.d/99-kamal-host.conf to optimize the Linux network and memory subsystem:
sudo tee /etc/sysctl.d/99-kamal-host.conf << 'EOF'
# ====================================================================
# File Descriptors and System Inotify Limits
# ====================================================================
fs.file-max = 2097152
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 1024
# ====================================================================
# Virtual Memory and Swappiness
# ====================================================================
# Reduce swap aggression; reserve swap for emergency memory relief only
vm.swappiness = 10
# Allow memory overcommit for standard Redis/NodeJS/Ruby runtime forks
vm.overcommit_memory = 1
# Required by embedded search, vector stores, and elastic search engines
vm.max_map_count = 262144
# Flush dirty memory aggressively to protect NVMe queues during spikes
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
# ====================================================================
# Network Core and Socket Buffer Tuning
# ====================================================================
# Enable TCP BBR Congestion Control (standard on tropic.host uplinks)
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
# Increase socket backlog buffer for burst traffic
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 16384
# Maximize socket read/write buffers (16MB max)
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# ====================================================================
# TCP Connection Lifecycle and Security
# ====================================================================
# Enable SYN Cookies against SYN flood attacks
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 8192
# Enable rapid reuse of TIME_WAIT sockets for outgoing proxy connections
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
# Expand ephemeral port range to prevent local port exhaustion
net.ipv4.ip_local_port_range = 10240 65535
# Fast fail dead TCP connections
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_keepalive_probes = 5
# ====================================================================
# IP Forwarding and Bridge Routing (Required for Docker and Kamal Proxy)
# ====================================================================
net.ipv4.ip_forward = 1
net.ipv4.conf.all.forwarding = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
# Load and enforce parameters immediately
sudo sysctl --system
With tcp_tw_reuse = 1 and an expanded local port range (10240 65535), Kamal Proxy can sustain tens of thousands of upstream connections to local application container ports without suffering from ephemeral socket starvation. The activation of TCP BBR paired with fq queuing ensures low-latency p99 network roundtrips across the 1–10 Gbps unmetered uplinks routed by tropic.host.
Docker Engine Installation and Daemon Architecture
Kamal 2 requires standard Docker CE. Avoid distribution wrapper packages (docker.io or Ubuntu Snap packages), as they frequently lag behind upstream engine updates and break cgroups v2 resource delegation.
1. Official Docker CE Repository Setup
# Remove outdated or conflicting engine binaries
sudo apt-get remove -y docker docker-engine docker.io containerd runc
# Add Docker's official GPG signing key
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Register the official upstream repository
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update -y
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin
Add your deployment user to the docker socket group so Kamal can control the daemon without root escalation:
sudo usermod -aG docker kamal
Production Docker Engine Optimization (daemon.json)
By default, the Docker daemon routes traffic to exposed ports through an internal Go userspace proxy binary (docker-proxy). This introduces additional context switching overhead, increases per-connection RAM consumption, and raises p99 request latency under load. Furthermore, default Docker installations write unbounded JSON logs directly to the host filesystem, eventually filling the root NVMe partition and crashing production stateful nodes.
Configure /etc/docker/daemon.json to eliminate these architectural bottlenecks:
sudo tee /etc/docker/daemon.json << 'EOF'
{
"log-driver": "local",
"log-opts": {
"max-size": "50m",
"max-file": "3",
"compress": "true"
},
"storage-driver": "overlay2",
"exec-opts": [
"native.cgroupdriver=systemd"
],
"live-restore": true,
"userland-proxy": false,
"icc": true,
"no-new-privileges": false,
"default-ulimits": {
"nofile": {
"Name": "nofile",
"Hard": 65535,
"Soft": 65535
},
"nproc": {
"Name": "nproc",
"Hard": 65535,
"Soft": 65535
}
},
"dns": [
"1.1.1.1",
"8.8.8.8"
],
"features": {
"buildkit": true
}
}
EOF
Architectural Rationale for Key Directives
"userland-proxy": false: Disables the userspacedocker-proxyprocess. Port forwarding is handed over directly to Linux kernel Netfilter/iptables hairpin NAT rules. Packets arriving on ports 80/443 for Kamal Proxy bypass userspace context switches, reducing CPU overhead by up to 15% during connection surges and shaving 1–3 ms off p99 latency."live-restore": true: Allows application and proxy containers to remain running when thedockerdprocess terminates or updates during unattended security patching. This prevents unexpected downtime when maintaining the container host."log-driver": "local": Unlike the legacyjson-filedriver, thelocaldriver writes binary, size-capped logs with automated compression and memory-mapped reads. It prevents write lock contention on the host storage controller during verbose log outputs."native.cgroupdriver=systemd": Aligns Docker's cgroup management withsystemd, eliminating dual-controller race conditions when Kamal 2 applies dynamic CPU and memory constraints."default-ulimits": Sets the soft and hard file descriptor ceiling to65535, preventing high-throughput web processes or reverse proxies from hittingEMFILE: too many open fileserrors.
Restart and activate the daemon:
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl enable docker containerd
Verification and Runtime Telemetry
Verify that the runtime optimizations, socket configurations, and cgroup alignments are fully operational:
# Verify Docker operational state and cgroup driver
docker info --format 'Cgroup Driver: {{.CgroupDriver}}, Cgroup Version: {{.CgroupVersion}}, Live Restore: {{.LiveRestoreEnabled}}'
# Confirm userland-proxy is disabled (should output zero running docker-proxy processes)
pgrep docker-proxy || echo "userland-proxy successfully disabled (direct kernel NAT active)"
# Test local logging subsystem
docker run --rm --name test-logger alpine:latest sh -c "echo 'Host verification passed'"
docker inspect --format '{{.HostConfig.LogConfig.Type}}' $(docker run -d --rm alpine:latest sleep 1)
The output confirms that Cgroup Version: 2, Live Restore: true, and the local log driver are enforced. The host is now primed for Kamal 2 configuration orchestration, maintaining the low-latency baseline provided by tropic.host's dedicated KVM NVMe infrastructure.
Kamal 2 Architecture Deep Dive: Understanding the New Thruster Proxy and Docker Hooks
Kamal 1 relied on Traefik as its ingress reverse proxy, utilizing dynamic Docker label inspection to route traffic to running application containers. While functional, that architecture introduced operational friction: Traefik required mounting the host's /var/run/docker.sock inside the container—creating an elevated privilege vector—consumed between 80 MB and 160 MB of resident memory (RSS), and introduced unpredictable reconciliation latency when polling the Docker daemon for container lifecycle events. Rapid zero-downtime rollouts occasionally triggered transient 502 Bad Gateway errors due to timing skews between container health checks and Traefik’s internal routing table convergence.
Kamal 2 fundamentally shifts this paradigm by decoupling ingress edge routing from the internal application runtime. The monolithic Traefik instance is replaced by two purpose-built components:
kamal-proxy: A lightweight (under 15 MB RSS) Go binary operating directly at the host edge (binding ports80and443). It manages zero-downtime container swaps via an explicit HTTP management API rather than passive Docker socket polling.Thruster: A high-performance sidecar/init proxy written in Go, bundled directly inside the application container to wrap the application process (e.g., Puma, Unicorn, Node.js, or Go). Thruster handles TLS termination, HTTP/2 multiplexing, static asset caching, and compression before traffic ever reaches the application runtime.
Public Traffic (Ports 80 / 443)
│
▼
┌────────────────────────────────────────────────────────┐
│ Host Layer: kamal-proxy (Port 80/443 Ingress) │
│ - Zero-downtime cutover & instant socket pausing │
│ - Controlled via localhost:2999 (No docker.sock mount) │
└───────────┬────────────────────────────────┬───────────┘
│ (Target A: Active) │ (Target B: Deploying)
▼ ▼
┌──────────────────────────────┐ ┌──────────────────────────────┐
│ App Container: web-v1 │ │ App Container: web-v2 │
│ ┌──────────────────────────┐ │ │ ┌──────────────────────────┐ │
│ │ Thruster (Port 3000) │ │ │ │ Thruster (Port 3000) │ │
│ │ - HTTP/2 & Asset Caching │ │ │ │ - HTTP/2 & Asset Caching │ │
│ │ - Gzip / Brotli / X-Send │ │ │ │ - Gzip / Brotli / X-Send │ │
│ └────────────┬─────────────┘ │ │ └────────────┬─────────────┘ │
│ ▼ │ │ ▼ │
│ ┌──────────────────────────┐ │ │ ┌──────────────────────────┐ │
│ │ Puma / App (Port 3001) │ │ │ │ Puma / App (Port 3001) │ │
│ └──────────────────────────┘ │ │ └──────────────────────────┘ │
└──────────────────────────────┘ └──────────────────────────────┘
On tropic.host KVM compute instances—where zero CPU oversubscription (%st = 0.0%) and AMD EPYC / Ryzen 9 cores eliminate scheduling latency—this separation allows kamal-proxy and Thruster to maintain flat p99 response times (< 8 ms) even during heavy deployment churn.
Inside Thruster: Eliminating Web Server Overhead and Managing Assets
Web application runtimes like Ruby's Puma or Python's Gunicorn are inefficient at serving static assets, terminating TLS, and managing high-concurrency keep-alive pools. Serving a 400 KB gzipped JavaScript bundle through a Puma worker ties up an entire interpreter thread that should otherwise execute database queries or business logic.
Thruster executes inside the application container as the primary process (ENTRYPOINT), wrapping the underlying server command:
# Production Dockerfile snippet integrating Thruster
FROM ruby:3.3.5-slim AS base
# Install Thruster binary
COPY --from=ghcr.io/basecamp/thruster:latest /usr/bin/thrust /usr/bin/thrust
WORKDIR /rails
EXPOSE 3000
# Thruster listens on port 3000, forwards dynamic requests to Puma on port 3001
CMD ["thrust", "./bin/rails", "server", "-p", "3001", "-b", "127.0.0.1"]
1. In-Memory Static Asset Acceleration
When an incoming request matches a static asset path (e.g., /assets/*, /packs/*, or fingerprinted build artifacts), Thruster intercepts the request in Go user-space: * Zero Application Traversal: The request never touches the Puma worker thread. * Direct File System Cache: Thruster streams assets straight from the local container overlay filesystem. On tropic.host PCIe 4.0 NVMe storage arrays (exceeding 50,000 random 4K QD1 IOPS), cold disk reads are serviced in under 40 microseconds, after which files remain mapped in the Linux page cache (vm.vfs_cache_pressure = 50). * Built-in Compression: Thruster evaluates the client's Accept-Encoding header and natively serves pre-compressed .gz or .br files, or compresses in-flight assets with minimal CPU penalty using optimized Go routines. * Aggressive Caching Headers: Injected headers automatically enforce Cache-Control: public, max-age=31536000, immutable for asset digests, eliminating duplicate round trips.
2. Automatic TLS Termination (ACME Integration)
When Thruster is deployed directly facing the internet (without an external load balancer or when running in standalone mode), it executes an integrated ACME client: * Automates Let's Encrypt certificates using standard HTTP-01 and TLS-ALPN-01 challenges. * Stores and rotates certificates automatically in persistent volumes without requiring external cert-manager daemons. * Supports modern cipher suites exclusively: TLS 1.3 and TLS 1.2 with ECDHE key exchanges (X25519, P-256) and ChaCha20-Poly1305 / AES-128-GCM authenticated encryption.
3. Request Buffering and Slow Client Protection
Application servers written in interpreted languages are vulnerable to slow-client resource exhaustion (Slowloris attacks or high-latency mobile networks uploading payloads slowly). Thruster buffers the entire incoming HTTP request body into memory or temporary storage before forwarding the complete payload to the backend process over a fast loopback socket (127.0.0.1). Puma receives the request instantly, processes it at wire speed, and returns the response immediately to Thruster, freeing up runtime threads instantly.
kamal-proxy: Deterministic Edge Orchestration
Kamal 2 places kamal-proxy on the host to route public traffic to the appropriate container instances. Unlike Traefik, which dynamically tracks the Docker socket API, kamal-proxy operates as a stateful, deterministic reverse proxy commanded exclusively via Kamal's deploy tasks.
The Zero-Downtime Swap Protocol
When kamal deploy runs, the coordination sequence proceeds through precise kernel and socket states:
Step 1: Deploy New Container ──> web-v2 starts on dynamic high-port (e.g., :32768)
Step 2: Health Validation ──> kamal-proxy probes http://127.0.0.1:32768/up
Step 3: Target Registration ──> POST /targets (Register web-v2 into routing pool)
Step 4: Atomic Traffic Shift ──> Ingress SYNs instantly directed to web-v2
Step 5: Drain Old Container ──> Allow web-v1 in-flight requests to complete (30s)
Step 6: Target Eviction ──> DELETE /targets/web-v1 & docker stop web-v1
- New Container Rollout: Kamal deploys the new image tag (
web-v2) and maps its Thruster port to an ephemeral host port (e.g.,127.0.0.1:32768). - Local Health Probing:
kamal-proxyor Kamal executes health validation against the target endpoint (configured viadeploy.ymlaspath: /up). - Target Registration: Kamal issues an authenticated command to
kamal-proxy's local control API (http://localhost:2999):bash curl -X POST http://localhost:2999/targets \ -H "Content-Type: application/json" \ -d '{"service": "app-web", "target": "127.0.0.1:32768", "tls": true}' - Instant Switchover:
kamal-proxyupdates its internal routing pointer atomically. New incoming TCP handshakes are immediately dispatched toweb-v2. - Connection Draining: Existing connections to
web-v1are not severed;kamal-proxymaintains established TCP sockets until transactions finish or thedrain_timeout(default: 30 seconds) expires. - Eviction and Teardown: Once connections reach zero, Kamal issues a stop command to
web-v1and removes the container.
Because kamal-proxy interacts with Kamal via its local administrative socket rather than querying the Docker engine, host security is significantly hardened. The Docker daemon socket (/var/run/docker.sock) remains strictly owned by root:docker with 0660 permissions, entirely unexposed to external containers.
Kamal 2 Docker Hooks: Lifecycle Automation
Kamal 2 introduces a structured hook architecture located in .kamal/hooks/. Hooks are executable scripts (POSIX shell, Ruby, Python, or Go binaries) triggered on the local management machine or remotely across target servers at specific stages of the rollout lifecycle.
Hook Execution Hierarchy
| Hook File | Execution Target | Trigger Point | Common Use Case |
|---|---|---|---|
docker-setup |
Remote Host | Before Docker runtime initialization | Enforcing sysctl limits, cgroups v2 mount checks, storage driver audits |
pre-build |
Local Machine | Before executing docker buildx |
Validating git commit cleanliness, asset pre-compilation, linting |
post-build |
Local Machine | Immediately after image push | Scanning images for vulnerabilities (e.g., Trivy, Grype) |
pre-deploy |
Remote Host | Before starting new app containers | Running database migrations, checking available disk space and memory |
post-deploy |
Remote Host | Immediately after traffic cutover | Purging external CDNs, notifying Datadog/Sentry, warming cache |
pre-connect |
Remote Host | Before container attaches to kamal-proxy |
Seeding local tenant databases, warming worker processes |
post-disconnect |
Remote Host | After old container is dropped | Rotating logs, cleaning stale Docker layers, vacuuming SQLite databases |
Kamal automatically passes deployment state to these scripts through environment variables: * KAMAL_RECORDED_AT: ISO 8601 UTC timestamp of execution. * KAMAL_PERFORMER: User identity initiating the deployment. * KAMAL_VERSION: The Git SHA or image tag being applied. * KAMAL_HOSTS: Comma-delimited list of target hosts. * KAMAL_ROLE: The targeted role (e.g., web, worker). * KAMAL_DESTINATION: Multi-environment target name (e.g., production, staging).
Production Implementation: pre-deploy Safety Gate
Create .kamal/hooks/pre-deploy on your local repository and mark it executable (chmod +x .kamal/hooks/pre-deploy). This hook runs on the remote host to verify that the environment has adequate hardware headroom before a deployment starts:
#!/usr/bin/env bash
set -euo pipefail
# .kamal/hooks/pre-deploy
# Halts deployment if host hardware thresholds are breached.
echo "--> [Hook: pre-deploy] Validating host resources for version ${KAMAL_VERSION} on ${HOSTNAME}"
# 1. Enforce minimum disk headroom on root partition (threshold: 15% free)
DISK_FREE_PCT=$(df -k / | awk 'NR==2 {gsub("%","",$5); print 100 - $5}')
MIN_DISK_THRESHOLD=15
if [ "${DISK_FREE_PCT}" -lt "${MIN_DISK_THRESHOLD}" ]; then
echo "CRITICAL: Insufficient disk space on ${HOSTNAME}. Available: ${DISK_FREE_PCT}%, required: ${MIN_DISK_THRESHOLD}%." >&2
exit 1
fi
# 2. Check cgroups v2 memory pressure
if [ -f /sys/fs/cgroup/memory.pressure ]; then
SOME_AVG10=$(awk '/some/ {for(i=1;i<=NF;i++) if($i ~ /^avg10=/) {sub(/avg10=/,"",$i); print $i}}' /sys/fs/cgroup/memory.pressure)
# If average memory pressure in the last 10 seconds exceeds 20%, abort to prevent OOM
PRESSURE_EXCEEDED=$(awk -v val="${SOME_AVG10}" 'BEGIN {print (val > 20.00)}')
if [ "${PRESSURE_EXCEEDED}" -eq 1 ]; then
echo "CRITICAL: Host memory pressure too high (some avg10=${SOME_AVG10}%). Aborting deploy." >&2
exit 1
fi
fi
# 3. Ensure kamal-proxy is responding locally
if ! curl -s --max-time 2 http://localhost:2999/healthz > /dev/null; then
echo "CRITICAL: kamal-proxy local API on port 2999 is unresponsive." >&2
exit 1
fi
echo "--> [Hook: pre-deploy] Host telemetry nominal. Proceeding with deployment."
exit 0
Production Implementation: post-deploy Cache Warming and Telemetry
Create .kamal/hooks/post-deploy to execute asset cache priming and notify external telemetry endpoints:
#!/usr/bin/env bash
set -euo pipefail
# .kamal/hooks/post-deploy
# Executes cache priming and emits audit logs after successful cutover.
echo "--> [Hook: post-deploy] Finalizing release ${KAMAL_VERSION} on ${HOSTNAME}"
# 1. Warm Thruster internal asset cache via local loopback
echo "--> Warming local Thruster cache..."
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -H "Host: yourdomain.com" http://127.0.0.1/up || true)
if [ "${HTTP_CODE}" != "200" ]; then
echo "WARNING: Local health endpoint returned HTTP ${HTTP_CODE} post-deploy." >&2
fi
# 2. Prune obsolete Docker image layers to keep NVMe storage clean
docker image prune -af --filter "until=72h" > /dev/null 2>&1 || true
# 3. Emit structured syslog event for SIEM auditing
logger -t kamal-deploy "SUCCESS: Service ${KAMAL_ROLE} updated to ${KAMAL_VERSION} by ${KAMAL_PERFORMER}"
echo "--> [Hook: post-deploy] Post-flight tasks completed."
exit 0
Verification and Edge Diagnostics
Inspect the operational status of kamal-proxy and Thruster to confirm that connection draining, routing states, and kernel sockets operate correctly:
# 1. Inspect running kamal-proxy daemon state and listening sockets
docker ps --filter "name=kamal-proxy" --format "table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Ports}}"
# 2. Query kamal-proxy's live target table
curl -s http://localhost:2999/targets | jq .
# Expected Output:
# [
# {
# "service": "production-web",
# "target": "127.0.0.1:32768",
# "state": "active",
# "active_connections": 142
# }
# ]
# 3. Validate socket state and network queue depth
ss -tulpn | grep -E ':(80|443|2999)'
# 4. Confirm TCP BBR and zero drop queues on host ingress interfaces
ss -ti '( sport = :443 )' | head -n 10
The output confirms that kamal-proxy holds ingress ports 80 and 443, routes directly to the application container’s Thruster instance, and communicates with zero process friction.
Running this architecture on tropic.host ensures that high network throughput is maintained by native TCP BBR congestion control and 1–10 Gbps unmetered uplinks with direct peering across Frankfurt, Amsterdam, and London Internet Exchanges. Because tropic.host enforces clean hardware virtualization with dedicated resources, TLS handshakes processed by Thruster experience consistent, jitter-free execution across all deployment phases.
Crafting the Production deploy.yml: Multi-Environment Config and Secrets Management
The primary deployment orchestrator in Kamal 2 is driven by config/deploy.yml. When migrating workloads from basic container runners to bare-metal or KVM-backed instances, the deployment configuration must enforce three operational boundaries: deterministic multi-stage inheritance across deployment tiers, leak-proof runtime secrets injection, and strict kernel-enforced cgroups v2 resource quotas.
Kamal 2 eliminates the Traefik dependency present in early iterations, standardizing routing and zero-downtime cutovers on kamal-proxy. Consequently, the configuration syntax directly bridges Docker daemon runtime directives, registry authentication protocols, and edge healthchecks into a consolidated infrastructure-as-code schema.
┌────────────────────────────────────────┐
│ config/deploy.yml │
│ (Global Base: Image, Service, Proxy) │
└───────────────────┬────────────────────┘
│
Inherits & Merges │ kamal deploy -d production
▼
┌────────────────────────────────────────┐
│ config/deploy.production.yml │
│ (Dedicated Hosts, Sizing, Overrides) │
└───────────────────┬────────────────────┘
│
Evaluates Envs & Secrets │ Evaluates local / vault commands
▼
┌──────────────────────────────────────────────────────────────────┐
│ Target Host Instance │
│ ┌───────────────────────────┐ ┌───────────────────────────┐ │
│ │ kamal-proxy (Port 80/443)│───▶│ Application Container │ │
│ │ Healthcheck: /up │ │ cgroups v2: cpu.max, mem │ │
│ └───────────────────────────┘ └───────────────────────────┘ │
└──────────────────────────────────────────────────────────────────┘
Hierarchical Architecture: Base Deploy and Destination Overrides
Do not maintain monolithic, duplicate configuration files for staging and production. Kamal 2 establishes a hierarchical inheritance model via destinations (-d or --destination).
The root manifest config/deploy.yml contains universal application declarations: * Image coordinates and container registry endpoints. * Universal build architecture directives (Docker Buildx multi-arch platforms). * Core kamal-proxy edge parameters, SSL bindings, and unified healthcheck paths.
Destination manifests—placed inside config/deploy.staging.yml and config/deploy.production.yml—override target inventory, per-node host IP allocations, scaling replicas, and dedicated hardware sizing parameters. When executing kamal deploy -d production, Kamal evaluates config/deploy.yml, deeply merges config/deploy.production.yml on top, and interpolates target-specific environment variables.
Registry Authentication and Ephemeral Token Evaluation
Pushing and pulling deployment artifacts requires isolated registry credentials. Baking access keys or raw personal access tokens (PAT) into deploy.yml is an architectural anti-pattern. Instead, Kamal evaluates dynamic shell outputs and local environment variables on the deployment orchestrator machine before connecting to the target fleet over SSH.
For high-throughput pipelines targeting GitHub Packages (GHCR), AWS Elastic Container Registry (ECR), or self-hosted Harbor registries, configure registry authentication with short-lived tokens:
registry:
server: ghcr.io
username: infra-deployer
password:
- KAMAL_REGISTRY_PASSWORD
When evaluated, Kamal pulls KAMAL_REGISTRY_PASSWORD from the local deployer shell or executes a runtime lookup through external password stores. Target servers run docker login through an isolated, non-interactive SSH session, storing the authentication hash in /root/.docker/config.json with 0600 POSIX file permissions.
Multi-Tier Secret Injection with Zero Host-Side Disk Leaks
A critical failure point in multi-tenant or bare-cloud VPS configurations is leaving plaintext .env files persistent on disk across /var/lib/docker/ directories. Kamal 2 enforces strict separation between clear configuration parameters and encrypted secrets through native secrets managers or external CLI adapters.
1. Clear vs. Secret Variables
Clear environment variables (non-sensitive runtime indicators such as log formats, port bindings, or application performance monitoring service tags) belong in the env.clear stanza. Sensitive credentials (database connection strings, master encryption keys, object storage private secrets) reside exclusively under env.secret:
env:
clear:
NODE_ENV: production
LOG_LEVEL: info
PORT: 80
THRUSTER_RAM_CACHE_SIZE: 64
secret:
- DATABASE_URL
- REDIS_URL
- RAILS_MASTER_KEY
- S3_SECRET_ACCESS_KEY
2. Resolving Secrets via .kamal/secrets
Kamal 2 utilizes an executable secrets evaluation pattern in .kamal/secrets (or target-specific variants such as .kamal/secrets.production). This file is evaluated locally on the deployer's workstation or CI runner, never shipped directly to the remote server.
To fetch credentials from 1Password CLI, Bitwarden, or HashiCorp Vault on demand without touching local disk files, configure .kamal/secrets:
#!/usr/bin/env bash
# .kamal/secrets - Evaluates production credentials into memory
set -euo pipefail
# 1. Fetch registry credential from 1Password
KAMAL_REGISTRY_PASSWORD=$(op read "op://Infrastructure/GHCR_Token/credential")
# 2. Extract database URI and state keys from HashiCorp Vault or Cloud Secret Engine
DATABASE_URL=$(vault kv get -field=DATABASE_URL secret/production/db)
REDIS_URL=$(vault kv get -field=REDIS_URL secret/production/redis)
RAILS_MASTER_KEY=$(op read "op://Infrastructure/Rails_Prod/master_key")
S3_SECRET_ACCESS_KEY=$(op read "op://Infrastructure/S3_Storage/secret_key")
# 3. Export to Kamal evaluation context
export KAMAL_REGISTRY_PASSWORD DATABASE_URL REDIS_URL RAILS_MASTER_KEY S3_SECRET_ACCESS_KEY
During execution, Kamal streams these variables over the established SSH tunnel directly into the docker run execution array as temporary environment definitions (-e VAR=VALUE), preventing disk-persisted .env artifacts from accumulating in the remote root directory.
Healthchecks and Zero-Downtime Transition under kamal-proxy
Kamal 2 replaces Traefik with kamal-proxy, drastically simplifying ingress routing and health checks. kamal-proxy operates as an internal reverse proxy running directly on the host interface, routing requests to dynamic local ports exposed by new and old container generations.
The proxy block in deploy.yml controls the cutover threshold. Rather than sending traffic to a newly provisioned container immediately, kamal-proxy polls the internal health endpoint until a 200 OK HTTP status is received consistently within defined timeout and retry bounds:
proxy:
ssl: true
host: api.production.tropic-cluster.net
app_port: 80
healthcheck:
path: /up
interval: 1s
timeout: 2s
cord: /cord
[Inbound Traffic]
│
▼
┌──────────────┐ Polling /up: 200 OK
│ kamal-proxy │────────────────────────┐
└──────┬───────┘ ▼
│ Drain Old Connections ┌──────────────┐
▼ │ Container v2 │
┌──────────────┐ │ (Healthy) │
│ Container v1 │ └──────────────┘
│ (Terminating)│
└──────────────┘
path: Direct application healthcheck endpoint (e.g.,/upor/healthz). It must validate internal dependencies (database read socket, local disk availability) within under 200 milliseconds.interval: Frequency of internal polls dispatched bykamal-proxy. Setting this to1sminimizes deployment wait overhead.timeout: Drop-off window before marking the boot phase as unviable and triggering an automatic rollback.cord: An optional zero-downtime drain hook. When scaling down or cycling instances, calling this path alerts application workers to stop accepting new requests, allowing existing client connections to flush cleanly.
Enforcing cgroups v2 Resource Sizing and CFS Latency Bounds
In bare containerized architectures, an unconstrained background worker or unexpected memory leak will trigger the Linux kernel's Out-Of-Memory (OOM) killer, jeopardizing the host operating system or dropping concurrent network sockets. Kamal 2 exposes raw Docker runtime flags through the options key, translating user parameters into Linux cgroups v2 control primitives.
options:
cpus: "3.5"
memory: "4096m"
memory-reservation: "3072m"
memory-swap: "4096m"
pids-limit: 1024
ulimit:
- nofile=65535:65535
- nproc=4096:4096
Understanding the Kernel Enforcement Mechanics
- CFS Bandwidth Control (
cpus: "3.5"):
The Linux Completely Fair Scheduler (CFS) enforces quotas over a period window (typically100000usor 100ms). Settingcpus: "3.5"configures:/sys/fs/cgroup/system.slice/docker-<id>.scope/cpu.max -> 350000 100000If worker threads exceed 350ms of CPU execution time within any 100ms period, the kernel CFS immediately throttles container threads. This prevents CPU starvation on neighboring services, stabilizing p99 latency across inbound requests. - Memory Hierarchy (
memory.maxvsmemory.high): memory: "4096m"sets the hard cgroups v2 ceiling (memory.max). Breaching this limit triggers an immediate kernel OOM kill on the process consuming the largest anonymous memory footprint.memory-reservation: "3072m"configures the soft reclamation ceiling (memory.high). When total system memory becomes constrained, the Linux page cache manager actively reclaims page buffers from the container without killing the primary daemon.memory-swap: "4096m"sets swap equal to physical memory. This disables swap expansion, stopping disk thrashing on NVMe storage when memory consumption spikes.- File Descriptor Saturation (
ulimit: nofile=65535:65535):
High-concurrency servers using HTTP/2, Thruster, or SSE run out of default POSIX file descriptors (1024) within minutes under traffic spikes. Raising container limits to65535guarantees that TCP socket handshakes are never dropped at the kernel boundary due toEMFILE (Too many open files)errors.
To guarantee that CPU throttling and I/O waits remain at absolute zero under heavy cgroups confinement, deployments must run on uncompromised virtualization. Deploying these manifests on tropic.host gives containers access to clean, non-oversubscribed KVM virtualization with zero CPU Steal Time (%st = 0.0%) powered by high-frequency AMD EPYC and Ryzen 9 processors. Because tropic.host provisions dedicated PCIe 4.0 NVMe drives yielding over 50,000 random 4K QD1 IOPS, kernel memory paging operations and Docker image layer caching execute without storage controller latency.
Production Manifest: config/deploy.yml and config/deploy.production.yml
Below is the hardened, production-tested base configuration and its corresponding production override, configured for multi-node deployments with dedicated SSL, automated healthchecks, and strict resource isolation.
Base Manifest (config/deploy.yml)
# config/deploy.yml - Universal Application Topology
service: api-core
image: ghcr.io/infra-team/api-core
# Deploy to AMD64 architecture instances
builder:
arch: amd64
remote: false
cache:
type: registry
options: mode=max
# Docker Registry Integration
registry:
server: ghcr.io
username: infra-deployer
password:
- KAMAL_REGISTRY_PASSWORD
# Zero-Downtime Edge Ingress via kamal-proxy
proxy:
ssl: true
host: api.production.tropic-cluster.net
app_port: 80
healthcheck:
path: /up
interval: 1s
timeout: 2s
# Base Environment Variables
env:
clear:
PORT: 80
LOG_LEVEL: info
RAILS_SERVE_STATIC_FILES: "true"
MALLOC_ARENA_MAX: "2"
secret:
- DATABASE_URL
- REDIS_URL
- SECRET_KEY_BASE
# SSH Deployment User Configuration
ssh:
user: root
keys: ["~/.ssh/id_ed25519_deploy"]
port: 22
# Asset and Docker Retention
asset_path: /rails/public/assets
retain_containers: 5
Production Destination Override (config/deploy.production.yml)
# config/deploy.production.yml - Production Environment Overrides
# Invocation: kamal deploy -d production
servers:
web:
hosts:
- 198.51.100.10
- 198.51.100.11
labels:
tier: edge-web
options:
cpus: "3.5"
memory: "4096m"
memory-reservation: "3072m"
memory-swap: "4096m"
pids-limit: 1024
ulimit:
- nofile=65535:65535
- nproc=4096:4096
sysctl:
net.core.somaxconn: 4096
worker:
hosts:
- 198.51.100.12
cmd: bundle exec sidekiq -C config/sidekiq.yml
options:
cpus: "2.0"
memory: "2048m"
memory-swap: "2048m"
pids-limit: 512
# Environment additions specific to production
env:
clear:
NODE_ENV: production
RACK_ENV: production
SIDEKIQ_CONCURRENCY: "10"
Kernel-Level Verification and cgroups Inspection
Once the configuration is applied via kamal deploy -d production, run verification commands directly on the host nodes to inspect the cgroups v2 hierarchy and ensure the Docker daemon accurately enforced kernel ceilings:
# 1. Identify active application container ID
CONTAINER_ID=$(docker ps --filter "name=api-core-web" --format "{{.ID}}" | head -n 1)
# 2. Extract active cgroups v2 scope assigned to the container
CGROUP_PATH=$(docker inspect --format '{{.Id}}' "$CONTAINER_ID")
CGROUP_DIR="/sys/fs/cgroup/system.slice/docker-${CGROUP_PATH}.scope"
# Fallback check for systems using cgroupfs driver directly
if [ ! -d "$CGROUP_DIR" ]; then
CGROUP_DIR="/sys/fs/cgroup/docker/${CGROUP_PATH}"
fi
# 3. Read active CFS CPU quotas and period windows
echo "--- CPU Limits (Quota / Period) ---"
cat "${CGROUP_DIR}/cpu.max"
# Expected Output: 350000 100000 (Matches cpus: "3.5")
# 4. Read strict memory limit ceiling
echo "--- Memory Ceilings (Bytes) ---"
cat "${CGROUP_DIR}/memory.max"
# Expected Output: 4294967296 (Matches 4096m)
# 5. Verify process thread boundaries
echo "--- Process / Thread Limits ---"
cat "${CGROUP_DIR}/pids.max"
# Expected Output: 1024
# 6. Audit host-level open file descriptors for the container process
APP_PID=$(docker inspect --format '{{.State.Pid}}' "$CONTAINER_ID")
prlimit --pid "$APP_PID" --nofile
# Expected Output:
# RESOURCE DESCRIPTION SOFT HARD UNITS
# NOFILE max number of open files 65535 65535 files
Confirming these kernel boundaries ensures that the application operates strictly within its designated hardware envelope. The host remains protected against resource exhaustions, memory allocation cascades, and connection-limit timeouts across all production deployment cycles.
Deploying Accessory Services: Production PostgreSQL, Redis, and Background Workers
Stateless web containers scheduled through Kamal’s rolling deployments are disposable by design, but stateful persistence layers demand an isolated operational lifecycle. In Kamal 2, stateful backends (such as PostgreSQL, Redis, ClickHouse for analytical telemetry, or Qdrant for semantic search embeddings) are managed via the accessories stanza. Unlike application roles, accessories do not cycle during code pushes; they remain running across application upgrades, retain static private network bindings, and mount persistent host storage volumes directly from the underlying NVMe storage subsystem.
Running stateful services alongside background workers on a single host or across dedicated nodes requires strict resource segmentation, host-level kernel tuning, and network isolation to prevent unauthenticated access and I/O starvation.
Host Kernel Tuning for Stateful Workloads
Before configuring Docker accessories, the host OS must be tuned for stateful throughput. Default Linux kernel parameters are biased toward generic desktop or bursty network workloads, which can trigger aggressive page swapping, Redis memory allocation failures during snapshotting, and PostgreSQL disk commit bottlenecks.
Apply the following production parameters to /etc/sysctl.d/99-stateful.conf:
# /etc/sysctl.d/99-stateful.conf
# Permit Redis background BGSAVE/fork operations without out-of-memory errors
vm.overcommit_memory = 1
# Reduce swappiness to keep database working sets in physical RAM
vm.swappiness = 10
# Flush dirty pages to NVMe progressively to avoid synchronous I/O freezes
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
# Widen system socket listen backlogs for connection bursts to Redis and Postgres
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
# Increase ephemeral port range for container proxying
net.ipv4.ip_local_port_range = 10240 65535
# Enable TCP BBR congestion control algorithm
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
Apply the changes immediately without rebooting:
sudo sysctl --system
Redis latency p99 is particularly sensitive to Linux Transparent Huge Pages (THP). When THP is active, memory page allocations during fork() operations balloon from 4 KB to 2 MB, inducing latency spikes during background persistence (BGSAVE and AOF compaction). Disable THP via a systemd unit:
sudo tee /etc/systemd/system/disable-thp.service << 'EOF'
[Unit]
Description=Disable Linux Transparent Huge Pages (THP) for Redis Performance
DefaultDependencies=no
After=sysinit.target local-fs.target
Before=docker.service
[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/enabled && echo never > /sys/kernel/mm/transparent_hugepage/defrag'
[Install]
WantedBy=basic.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now disable-thp.service
Verify that THP is permanently disabled:
cat /sys/kernel/mm/transparent_hugepage/enabled
# Expected Output: always madvise [never]
Volume Architecture and NVMe Filesystem Layout
Stateful containers must map to absolute paths on the host's high-speed NVMe partition. Avoid generic Docker managed volumes (/var/lib/docker/volumes/...), as direct path binds simplify file-level backups, disaster recovery, and storage monitoring.
Prepare dedicated persistence paths on the host:
# Create dedicated directories for accessory persistence
sudo mkdir -p /var/data/postgres /var/data/redis /etc/postgres/custom
# Restrict permissions: PostgreSQL (UID 999), Redis (UID 999)
sudo chown -R 999:999 /var/data/postgres
sudo chown -R 999:999 /var/data/redis
sudo chmod 700 /var/data/postgres
sudo chmod 700 /var/data/redis
On tropic.host KVM VPS instances, the underlying storage architecture runs on enterprise-grade PCIe 4.0 NVMe drives delivering over 50,000 IOPS on random 4K QD1 reads and sub-millisecond write latency. To capitalize on this hardware baseline, ensure the underlying mount (typically / or /var) uses modern ext4 or XFS mount options with noatime set in /etc/fstab to eliminate metadata write amplification on read operations:
UUID=xxxx-xxxx-xxxx /var ext4 defaults,noatime,barrier=1,errors=remount-ro 0 2
Kamal 2 Configuration: Production config/deploy.yml
In Kamal 2, background workers are defined under the roles stanza, whereas databases and caches are declared under accessories. Stateful services must never expose ports to 0.0.0.0. Instead, bind them strictly to 127.0.0.1 for local debugging or leave ports unmapped to force all communication through the shared internal Docker bridge network (kamal).
Here is a production-hardened config/deploy.yml implementation:
service: enterprise-core
image: registry.example.com/org/enterprise-core
servers:
web:
hosts:
- 203.0.113.10
labels:
kamal-proxy-port: 3000
options:
cpus: "3.5"
memory: "4096m"
# Asynchronous job workers sharing the same application image
workers:
hosts:
- 203.0.113.10
cmd: "bundle exec sidekiq -C config/sidekiq.yml -e production"
options:
cpus: "2.0"
memory: "2048m"
registry:
server: registry.example.com
username: deploy-bot
password:
- KAMAL_REGISTRY_PASSWORD
builder:
arch: amd64
# Internal Docker network shared across web, workers, and accessories
network: "kamal"
accessories:
# -------------------------------------------------------------
# Production PostgreSQL 16 on NVMe
# -------------------------------------------------------------
db:
image: postgres:16-alpine
host: 203.0.113.10
# Do NOT map 5432:5432 to 0.0.0.0. Bind only to loopback for direct tunnel access.
port: "127.0.0.1:5432:5432"
env:
clear:
POSTGRES_DB: "core_production"
POSTGRES_USER: "core_app"
secret:
- POSTGRES_PASSWORD
directories:
- "/var/data/postgres:/var/lib/postgresql/data"
files:
- "/etc/postgres/custom/postgresql.conf:/etc/postgresql/postgresql.conf:ro"
options:
cpus: "3.0"
memory: "4096m"
shm-size: "1024m"
restart: "always"
ulimit:
- "nofile=65535:65535"
# -------------------------------------------------------------
# Production Redis 7 Cache & Job Broker
# -------------------------------------------------------------
redis:
image: redis:7-alpine
host: 203.0.113.10
port: "127.0.0.1:6379:6379"
cmd: >
redis-server
--requirepass ${REDIS_PASSWORD}
--maxmemory 1536mb
--maxmemory-policy allkeys-lru
--save 300 1
--save 60 100
--appendonly yes
--appendfsync everysec
--tcp-backlog 65535
directories:
- "/var/data/redis:/data"
options:
cpus: "1.0"
memory: "2048m"
restart: "always"
Hardware-Aligned PostgreSQL Tuning
The default postgresql.conf ships with conservative settings designed for 128 MB RAM footprints. Write a customized configuration directly to /etc/postgres/custom/postgresql.conf on the host, tuned for a 4-core, 8 GB to 16 GB instance on tropic.host:
# /etc/postgres/custom/postgresql.conf
# Connectivity and Concurrency
listen_addresses = '*'
max_connections = 150
# Memory Allocation (Sized for a 4GB Container Limit)
shared_buffers = 1024MB
effective_cache_size = 3072MB
maintenance_work_mem = 256MB
work_mem = 16MB
temp_buffers = 16MB
# Write-Ahead Log (WAL) & Checkpoint Throttling
wal_buffers = 16MB
min_wal_size = 1GB
max_wal_size = 8GB
checkpoint_completion_target = 0.9
checkpoint_timeout = 15min
# Storage Engine (Calibrated for High-IOPS PCIe 4.0 NVMe)
random_page_cost = 1.1
seq_page_cost = 1.0
effective_io_concurrency = 200
# Parallel Query Execution
max_worker_processes = 4
max_parallel_workers_per_gather = 2
max_parallel_workers = 4
max_parallel_maintenance_workers = 2
# Logging and Diagnostics
logging_collector = on
log_min_duration_statement = 250 # Capture slow queries exceeding 250ms
log_line_prefix = '%m [%p] %q%u@%d '
log_lock_waits = on
The parameter random_page_cost = 1.1 is critical: conventional HDDs set this to 4.0 due to seek penalties. On high-throughput NVMe arrays without virtualization overcommit (%st = 0.0%), random page reads execute at near-zero latency penalty relative to sequential scans, allowing the PostgreSQL cost planner to favor index lookups over inefficient sequential sweeps.
Booting and Verifying Accessories
Deploy the database and Redis instances using Kamal accessory commands:
# 1. Initialize and start PostgreSQL
kamal accessory boot db
# 2. Initialize and start Redis
kamal accessory boot redis
# 3. Verify container states and restart policies
docker ps --filter "name=enterprise-core-" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
Expected output:
NAMES STATUS PORTS
enterprise-core-db Up 2 minutes 127.0.0.1:5432->5432/tcp
enterprise-core-redis Up 2 minutes 127.0.0.1:6379->6379/tcp
Test internal DNS resolution and container-to-container connectivity within the Docker bridge network. Launch an ephemeral verification container attached to the kamal network:
docker run --rm --network kamal alpine:latest sh -c \
"apk add --no-cache bind-tools netcat-openbsd && \
nslookup enterprise-core-db && \
nc -zv enterprise-core-db 5432 && \
nc -zv enterprise-core-redis 6379"
Verify persistent NVMe write synchronization directly on the database data path using PostgreSQL’s pg_test_fsync utility:
docker exec -it enterprise-core-db pg_test_fsync
Review the fdatasync metric in the benchmark output:
Compare file sync methods using one 8kB write:
open_datasync sync 3412.1 ops/sec 293.1 usec/op
fdatasync 3398.4 ops/sec 294.3 usec/op
fsync 3124.6 ops/sec 320.0 usec/op
A sub-350 µs fdatasync commit latency confirms that the underlying disk subsystem handles write-heavy transactions without blocking kernel write queues. Combined with isolated cgroups quotas, this deployment topology protects stateful accessories from container cross-talk, memory starvation, and unplanned deployment restarts.
Zero-Downtime Rollouts, Rollbacks, and Complete Server Disaster Recovery Runbook
Kamal 2 eliminates deployment downtime by replacing Traefik with kamal-proxy, an in-house HTTP/TLS reverse proxy written in Go that runs as a lightweight host container bound directly to TCP ports 80 and 443. While Traefik relied on dynamic configuration polling over the Docker daemon socket—often introducing a 50 ms to 200 ms propagation delay where incoming connections faced race conditions—kamal-proxy exposes a local Unix domain socket control plane (http://localhost:2999).
During a deployment, kamal-proxy directs client requests to the running target while Kamal orchestrates the new container alongside it, validates application readiness, atomically shifts router targets, and drains active HTTP keep-alive connections without dropping packets or terminating TCP streams with ECONNRESET.
Executing Zero-Downtime Rollouts
A production release executes through a single entrypoint that compiles assets, synchronizes secrets, builds cross-platform container layers via Docker Buildx, transfers images to the remote registry, and instructs the host nodes to perform atomic cutovers.
# Execute a standard production release
kamal deploy
Under the hood, the deployment engine executes eight distinct operational phases:
- Lock Acquisition: Kamal contacts the target node and places a coordination lock in
/tmp/kamal.lock(or within the configured accessory) to block concurrent administrative operations. - Local Multi-Arch Build: Docker Buildx builds the image layer cache against the local build engine, tags it with the short Git commit SHA (e.g.,
git rev-parse --short HEAD), and pushes it to the registry. - Pre-Deployment Hook Execution: If configured in
.kamal/hooks/pre-deploy, local linting and migration pre-flight checks run before modifying server state. - Target Image Prefetch: The remote VPS downloads the target image over its external uplink:
bash docker pull ghcr.io/org/enterprise-core:a81f3d9 - Parallel Container Initialization: Kamal spawns the incoming container using a distinct generated name (e.g.,
enterprise-core-web-a81f3d9) while the legacy container (enterprise-core-web-7c2e1b4) continues handling production ingress. - Health Buffer Verification: The engine polls the path specified in
deploy.yml(healthcheck.path: /up) every 1000 ms. The container must return HTTP 200 withinhealthcheck.timeout(default: 30s). - Atomic Socket Switchover: Once healthy, Kamal calls
kamal-proxyto register the new container’s IP and port on the Docker bridge network.kamal-proxybegins routing all new SYN packets to the new container. - Drain and Prune: Active TCP connections on the old container enter a graceful shutdown phase governed by
drain_timeout: 30s. After the drain period expires, Kamal sendsSIGTERM, waits up to 10 seconds for clean worker termination, sendsSIGKILLif unclosed file descriptors linger, and purges the old container.
To inspect the routing table of kamal-proxy in real time during a rollout, query the control socket directly on the host:
docker exec kamal-proxy curl -s http://localhost:2999/targets | jq .
Expected routing output indicating dual-container presence during cutover:
[
{
"service": "enterprise-core",
"target": "172.17.0.4:3000",
"state": "active",
"drain": false
},
{
"service": "enterprise-core",
"target": "172.17.0.3:3000",
"state": "draining",
"drain": true
}
]
For zero-overhead updates where Dockerfile definitions, system libraries, and base OS packages remain unchanged (e.g., pure application code patches), bypass the multi-stage builder and execute a fast restart:
# Skips registry push if images are pre-staged; initiates instant target rotation
kamal redeploy
Instant Rollback Runbook
When an unhandled exception path, memory leak, or latency regression bypasses automated integration tests and pollutes production, immediate rollback to the preceding immutable container image is mandatory. Kamal avoids full redeploy cycles by preserving recent application tags on the host disk.
Step 1: Audit Deployed Releases
Identify currently cached images and active container versions on the target node:
kamal app containers
Output:
App enterprise-core on 194.61.28.45:
CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES
9a4b2c8e1f0a ghcr.io/org/enterprise-core:a81f3d9 "bin/server" 2 minutes ago Up 2 minutes enterprise-core-web-a81f3d9
Check the cryptographic Git deploy audit trail stored on the server:
kamal audit
Step 2: Trigger Instant Rollback
Pass the target image tag directly to the rollback command. Because the image layers already reside in the host's /var/lib/docker/overlay2 engine storage, Kamal skips network transfer entirely:
kamal rollback 7c2e1b4
Execution flow: - Spawns enterprise-core-web-7c2e1b4. - Validates the /up healthcheck endpoint. - Updates kamal-proxy target configuration within 15 ms. - Sets drain: true on enterprise-core-web-a81f3d9. - Eliminates bad traffic within one healthcheck cycle.
Step 3: Clear Stale Concurrency Locks
If an SSH transport connection drops abruptly mid-deployment due to an upstream BGP routing failure, subsequent deployment commands will fail with:
Acquiring deploy lock...
Error: Deploy lock already taken by 'deployer@workstation' on 2026-10-11 12:44:10 UTC
Verify that no active docker build or container extraction process is running on the host, then forcefully release the lock:
kamal lock release
Stateful Accessory Backup Pipeline
Zero-downtime application rollouts protect stateless compute, but node disaster recovery depends strictly on regular, uncorrupted, off-host backups of stateful accessories.
PostgreSQL running inside an accessory container (enterprise-core-db) requires deterministic streaming backups that write to encrypted, off-site S3 storage without acquiring exclusive table locks or impacting disk I/O latency p99.
Create a hardened backup automation script at /usr/local/bin/pg-backup-s3.sh on the host:
#!/usr/bin/env bash
set -Eeuo pipefail
BACKUP_DATE=$(date -u +"%Y%m%d_%H%M%SZ")
BACKUP_DIR="/var/backups/postgres"
CONTAINER_NAME="enterprise-core-db"
DB_NAME="enterprise_production"
DB_USER="postgres"
S3_BUCKET="s3://infra-backups-secure/kamal/enterprise-core"
mkdir -p "${BACKUP_DIR}"
chmod 700 "${BACKUP_DIR}"
BACKUP_FILE="${BACKUP_DIR}/${DB_NAME}_${BACKUP_DATE}.sql.zst"
# Pipe pg_dump directly through modern zstandard compression (level 3, multi-threaded)
# --clean --if-exists avoids restore failures on dirty target schemas
# --no-owner --no-privileges ensures portable restores across unprivileged service users
docker exec "${CONTAINER_NAME}" pg_dump \
-U "${DB_USER}" \
-d "${DB_NAME}" \
-F p \
--clean \
--if-exists \
--no-owner \
--no-privileges | zstd -T0 -3 > "${BACKUP_FILE}"
# Validate archive integrity before transfer
zstd -t "${BACKUP_FILE}"
# Stream encrypted snapshot to off-site object storage via AWS CLI / S3-compatible API
aws s3 cp "${BACKUP_FILE}" "${S3_BUCKET}/${DB_NAME}_${BACKUP_DATE}.sql.zst" \
--storage-class STANDARD_IA
# Maintain a rolling 7-day retention policy locally
find "${BACKUP_DIR}" -type f -name "*.sql.zst" -mtime +7 -delete
echo "[$(date -u)] Backup completed and verified: ${BACKUP_FILE}"
Make the script executable and schedule it using a native systemd timer to guarantee execution isolation independent of cron daemon failures:
chmod 700 /usr/local/bin/pg-backup-s3.sh
Create /etc/systemd/system/pg-backup.service:
[Unit]
Description=PostgreSQL Kamal Accessory S3 Backup Pipeline
After=docker.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/pg-backup-s3.sh
StandardOutput=journal
StandardError=journal
Create /etc/systemd/system/pg-backup.timer:
[Unit]
Description=Trigger PostgreSQL Backup Every 6 Hours
[Timer]
OnCalendar=*-*-* 00,06,12,18:00:00
Persistent=true
[Install]
WantedBy=timers.target
Enable and start the timer:
systemctl daemon-reload
systemctl enable --now pg-backup.timer
systemctl list-timers pg-backup.timer
Complete Server Disaster Recovery Runbook
When a host suffers irreversible node loss, silent kernel panics, or unrecoverable filesystem corruption, follow this deterministic recovery sequence to restore the complete infrastructure from scratch.
Disaster Recovery Runbook: 0% Downtime Rebuild
┌────────────────────────────────────────────────────────┐
│ 1. Provision Fresh KVM Node on tropic.host │
│ (Dedicated IPv4, AMD EPYC/Ryzen 9, PCIe 4.0 NVMe) │
└───────────────────────────┬────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ 2. Apply Linux Kernel & sysctl Hardening │
│ (net.ipv4.tcp_congestion_control=bbr, somaxconn) │
└───────────────────────────┬────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ 3. Bootstrap Host via Kamal Control Plane │
│ kamal setup (Installs Docker, kamal-proxy, network) │
└───────────────────────────┬────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ 4. Initialize Database Accessory & Ingest S3 Snapshot │
│ kamal accessory boot db && pg_restore │
└───────────────────────────┬────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ 5. Execute Zero-Downtime Application Rollout │
│ kamal deploy │
└───────────────────────────┬────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ 6. Verify Production SLA (p99 Latency & %st = 0.0%) │
└────────────────────────────────────────────────────────┘
Step 1: Provision Clean Hardware Target
Deploy a new compute instance on tropic.host to ensure dedicated KVM virtualization guarantees. High-write workloads running transactional PostgreSQL and real-time Kamal ingress require unshared compute:
- CPU Core Guarantees: 100% dedicated hardware slices powered by AMD EPYC, Ryzen 9, or high-clock Intel Xeon processors. CPU Steal Time must remain strictly verified at
%st = 0.0%under peak loads to prevent worker execution stalls. - NVMe Subsystem: Enterprise PCIe 4.0 NVMe storage providing random 4K QD1 read throughput exceeding 50,000 IOPS and zero thermal write throttling.
- Networking: Dedicated, static, clean public IPv4 address without abuse reputation, tied to 1–10 Gbps uplinks with native TCP BBR congestion control and hardware-level L3/L4/L7 DDoS filtering.
Step 2: Apply Base System Hardening and Kernel Parameters
Log in via root SSH and apply performance-critical sysctl parameters to /etc/sysctl.d/99-kamal-performance.conf:
# Buffer limits for high-throughput ingress
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_max_syn_backlog = 65535
# Fast connection recycling and socket tracking
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65535
# Enable BBR congestion control over high-bandwidth links
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
# Memory allocation headroom for containerized PostgreSQL
vm.overcommit_memory = 1
vm.swappiness = 10
Load the sysctl configuration immediately:
sysctl --system
Install Docker CE and configure /etc/docker/daemon.json with hardened cgroups v2 quotas and log rotation policies to prevent uncontrolled container stdout logging from exhausting disk space:
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "50m",
"max-file": "3"
},
"storage-driver": "overlay2",
"live-restore": true
}
Restart Docker to apply the configuration:
systemctl restart docker
Step 3: Run Remote Kamal Initialization
On your local deployment workstation, update the host IP address in config/deploy.yml with the new tropic.host static IPv4 address. Run the bootstrap routine:
kamal setup
This command connects via SSH, verifies Docker installation, creates the bridge network (kamal), deploys kamal-proxy, provisions the proxy's TLS directory, and binds host ports 80 and 443.
Step 4: Restore Stateful Data from Off-Site Snapshot
Before deploying stateless application workers, initialize the database accessory and ingest the backup snapshot:
# 1. Spin up the clean PostgreSQL container accessory
kamal accessory boot db
# 2. Fetch the latest verified snapshot from offsite S3
aws s3 cp s3://infra-backups-secure/kamal/enterprise-core/enterprise_production_latest.sql.zst /tmp/restore.sql.zst
# 3. Stream the uncompressed SQL stream into the running container
zstd -d -c /tmp/restore.sql.zst | docker exec -i enterprise-core-db psql \
-U postgres \
-d enterprise_production
# 4. Clean up temporary host artifacts
rm -f /tmp/restore.sql.zst
Verify table counts, index validity, and database connectivity:
docker exec -it enterprise-core-db psql -U postgres -d enterprise_production -c "\dt+"
Step 5: Execute Application Launch
Boot the secondary accessories (e.g., Redis) and launch the stateless app containers:
# Boot remaining accessories
kamal accessory boot redis
# Deploy production application fleet
kamal deploy
Step 6: Post-Recovery Validation and Latency Audit
Confirm that zero-downtime routing functions correctly by executing an edge latency benchmark directly against the production ingress. Monitor CPU steal time and I/O wait on the host:
# Verify 0.0% CPU Steal Time (%st) and zero wait overhead (%wa)
vmstat 1 5
Expected output on dedicated tropic.host KVM hardware:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 12891244 32400 3120400 0 0 0 12 1204 954 4 1 95 0 0
0 0 0 12891244 32400 3120400 0 0 0 0 1198 942 2 1 97 0 0
Run a concurrency load verification using wrk across 100 persistent connections:
wrk -t4 -c100 -d30s --latency https://enterprise.example.com/up
Target latency benchmarks for production SLA sign-off:
Latency Distribution:
50% 1.12ms
75% 1.34ms
90% 1.89ms
99% 2.41ms
Requests/sec: 8412.33
Transfer/sec: 1.22MB
A latency p99 below 3 ms coupled with %st = 0.0% confirms that the server rebuild is complete, kamal-proxy is routing without socket overhead, and stateful NVMe write pipelines are operating at peak efficiency.
Stateful collaboration tools such as Mattermost illustrate how Kamal 2 handles persistent NVMe volume attachments and database accessory containers.
Frequently Asked Questions (FAQ)
How does Kamal 2 achieve zero-downtime deployments without Kubernetes?
Kamal 2 boots the new container version, performs health checks via Thruster proxy, seamlessly swaps traffic to the new container port, and gracefully stops the old container without dropping active HTTP requests.
Can I host multiple different web applications on a single KVM VPS with Kamal 2?
Yes. Kamal 2 supports multi-destination and multi-service deployments on a single host, allowing you to run multiple apps and their databases on one powerful KVM NVMe VPS.