Three Small Nodes or One Big Server? Cluster Math for Labs

Every lab builder hits this fork: one well-fed 2U, or three modest 1U nodes for the same money? The answer depends on what you are trying to learn, because the two paths teach different things — and cost different amounts to run every month thereafter.

What only a real cluster teaches

Quorum behavior, live migration, rolling upgrades, what actually happens when a node dies mid-write — these are cluster properties. Kubernetes on three physical nodes with etcd spread across them behaves like production; Kubernetes in three VMs on one host behaves like a simulation, because the failure domain everyone worries about (the machine) is shared. If the goal is operational fluency — the confidence that comes from having drained, failed and rejoined real nodes — hardware count is the feature. Three cheap R440s or R630s deliver it.

What the big box does better

Everything else, frankly. One R740 or DL380 Gen10 with 2×16 cores and 384GB gives more usable capacity than three small nodes at the same spend, because you buy one chassis, one set of PSUs, one boot mirror — and lose no RAM to two extra hypervisor/OS overheads. It is one thing to power (150–200W vs 250–350W for three), one thing to update, one thing making noise. VMs simulate any topology you like, including a nine-node Kubernetes practice cluster, as long as you accept the shared physical fate.

The numbers side by side

Three-node path: 3× R440 (1× Silver 4110, 64GB each) ≈ 24 cores / 192GB across three failure domains, plus a small switch with enough 10GbE for storage replication if you run Ceph or Longhorn properly. Single-box path: 1× R740 (2× Gold 6130, 384GB) ≈ 32 cores / 384GB in one domain, no east-west traffic to engineer at all. Capacity per dollar favors the single box by a comfortable margin; realism per dollar favors the cluster absolutely.

Storage decides more than people expect

Distributed storage is the hidden cost of the multi-node path: Ceph and friends want three replicas, fast networking and patience, and they will consume a chunk of each node's RAM and disks just to exist. The single box sidesteps it with local ZFS/LVM and replicates nothing. If your interest is apps and platforms, that simplicity is a gift; if your interest is infrastructure itself, the storage pain is the curriculum.

A sensible middle

Plenty of labs land on two tiers: one substantial box for the permanent estate (services, storage, the VMs you rely on), plus two or three disposable small nodes that exist purely to be clustered, broken and reinstalled. Refurbished pricing makes the pattern affordable — the practice nodes cost less than a mid-range GPU — and it keeps experiments from taking down the things your household quietly depends on.

Buy for the lesson you want. Capacity: one big box. Scars and fluency: three small ones.