Everyone said "just use managed Kubernetes." We said no, and saved millions. This is the story of how a counterintuitive infrastructure bet paid off bigger than anyone expected, and the specific, unglamorous work that made it possible.
Why Bare Metal? The Hyperscaler Math
When you run Kubernetes on AWS or GCP, you're paying for three things: compute, the managed control plane markup, and data transfer (the silent killer). For an e-commerce platform with a million storefronts, data transfer fees alone were running $15K a month. You're also paying the hyperscaler's margin on top of hardware that costs them a fraction of what they charge you.
Bare metal eliminates the markup. You pay for the server, the colo space, and the bandwidth. That's it. But you give up something real: the cloud provider does a lot of invisible work that you now have to do yourself. For us, the most important piece of that invisible work was global routing.
The BGP Design: 9 Data Centres, One IP Prefix
The architecture we built uses BGP Anycast: the same /24 IP prefix announced from 9 different Points of Presence: Mumbai, Singapore, Frankfurt, Dubai, São Paulo, Tokyo, New York, London, Sydney. The internet's BGP routing protocol naturally directs each user to the topologically nearest announcement. No GeoDNS, no latency-based routing policies, no application layer tricks. The routing happens at the network layer, handled by 30-year-old infrastructure that powers the global internet.
The routing daemon was BIRD. It's been on production networks since 1998, has meticulous documentation, and a parser that will refuse to start if you misplace a semicolon. Here's the core config that ran at every PoP:
# BIRD config — identical file, 9 locations
# The IP prefix is the same everywhere. BGP does the rest.
protocol bgp upstream_isp {
local as 65001;
neighbor 198.51.100.1 as 65000; # this PoP's upstream ISP router
ipv4 {
import none;
export filter {
if net ~ [203.0.113.0/24] then accept; # our Anycast /24
reject;
};
};
hold time 90;
keepalive time 30;
}MetalLB in BGP Mode: Kubernetes LoadBalancers Without a Cloud
On AWS, when you create a Kubernetes LoadBalancer service, the cloud-controller-manager calls the AWS API and provisions an ALB for you. On bare metal, there's no magic. You need something to assign IPs and make them routable. That's what MetalLB does in BGP mode.
MetalLB peers with BIRD on each node. When you kubectl apply a LoadBalancer service,
MetalLB picks an IP from your Anycast pool and tells BIRD to start announcing it. The service becomes
globally reachable via BGP in seconds. No cloud API. No managed load balancer. Just routing.
apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
name: bird-router
namespace: metallb-system
spec:
myASN: 65001
peerASN: 65001
peerAddress: 127.0.0.1 # BIRD on the same node
peerPort: 179
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: anycast-pool
namespace: metallb-system
spec:
addresses:
- 203.0.113.0/24 # our Anycast VIP range
---
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
name: anycast-advert
namespace: metallb-system
spec:
ipAddressPools:
- anycast-pool
communities:
- 65000:100 # no-export community — don't leak to peersThe DC7 Incident: What Happens When You Get BGP Community Wrong
Six weeks into the rollout, we pushed a BIRD config change to DC7 (our Dubai PoP) to add a new BGP community tag that we wanted upstream ISPs to respect as a traffic-engineering signal. The community value was valid syntax. The logic was correct. The problem: our Dubai ISP used that specific community value internally to mean "redistribute this route to our entire upstream peering mesh."
Our /24 started appearing in BGP tables it had no business being in. Traffic from unexpected regions started arriving at DC7, a PoP sized for Dubai traffic, suddenly absorbing Pakistani, Iranian, and Central Asian traffic because the ISP was redistributing our announcement. DC7's CPU spiked. NGINX started queueing. Latency climbed. We had no idea why, for about 8 minutes.
We caught it in 8 minutes via anomaly detection on per-PoP traffic volume. Fix: withdraw the community tag, reload BIRD. The ISP withdrew the spurious redistribution. Total duration: 11 minutes. We spent the next hour documenting every ISP's community string reservations before touching BIRD config on any other PoP.
BGP Timer Tuning: The Detail That Matters
BGP sessions have a "hold timer": if keepalives stop arriving for this long, the session dies and routes are withdrawn. The default in many implementations is 90 seconds, with keepalives every 30. That means three missed keepalives before a session dies.
One of our ISPs had a non-standard default: 30-second hold timer, 10-second keepalives. Under normal conditions, fine. Under a brief burst of network jitter, our keepalive missed their window and the session dropped, pulling all our routes from that PoP. Traffic failed over to the next nearest PoP (fine), but we'd created an unnecessary blip. After that we explicitly set hold times in every BIRD config to match what we'd negotiated with each ISP. No more surprises from that.
Production Results
Who Should Do This (and Who Shouldn't)
Do this if: you're paying significant cloud bills for workloads with predictable capacity, you have engineers who understand BGP or can learn it, and your users are spread across geographies where latency to a single cloud region is measurably bad.
Don't do this if: you need elastic capacity that spikes unpredictably, your team has no network engineering experience, or your traffic is concentrated in one region where a single cloud PoP is already close enough. The operational overhead is real. BIRD configs don't manage themselves. Hardware fails and someone has to deal with it at 2am.
The cost reduction only makes sense if your team can run BGP safely. If not, Cloudflare or AWS Global Accelerator do this work for you at a markup. Once you can do it yourself, that markup becomes optional.