In January 2022, Dukaan was paying $80,000 a month to AWS. By March, that number was $5,000. Here's exactly how we pulled it off — and nearly broke everything in the process.
The bill had been growing for months. Over a million seller storefronts, EKS clusters across nine regions, ALBs in every availability zone, RDS read replicas, and — the silent killer — data transfer fees. AWS charges for every byte that leaves its data centers, and when you're serving storefronts globally, bytes leave constantly. Every product image. Every API call. Every byte. Money.
Exec pressure to cut costs was at an all-time high. Someone finally asked the question out loud in a planning meeting:
A silence fell over the room. Not the uncomfortable kind — the kind where everyone is doing math.
The Idea: BGP Anycast Across 9 Data Centres
Here's the thing about BGP Anycast: Cloudflare uses it. DNS root servers use it. The entire backbone of the internet runs on it. It is not magic. It's a 30-year-old protocol that routes users to the topologically nearest announcement of a given IP prefix. We could run it ourselves.
The plan: nine bare-metal Points of Presence, one in each key region (India, Singapore, Frankfurt, Dubai, São Paulo, Tokyo, New York, London, Sydney). Each PoP would announce the same /24 prefix via BGP. Users anywhere in the world would automatically hit their nearest PoP — not because we told them to, but because the internet's routing tables would make it happen.
How BGP Dual-Announcement Works (The Scary Part)
We couldn't just flip a switch and move a million storefronts off AWS overnight. The strategy was BGP dual-announcement: run both AWS Global Accelerator and our bare-metal nodes announcing the same /24 simultaneously for six weeks. Traffic would naturally prefer the shorter AS path (for most regions, that meant bare metal). AWS became the automatic fallback.
The routing daemon we chose was BIRD, an open-source BGP implementation that's been running on production networks since 1998. Powerful, battle-tested, and completely unforgiving about semicolons.
# BIRD config at each PoP — same file, same prefix, 9 locations
# The internet figures out the rest
protocol bgp upstream_isp {
local as 65001;
neighbor 198.51.100.1 as 65000; # upstream ISP router at this PoP
ipv4 {
import none; # we don't accept routes from ISP
export filter {
if net ~ [203.0.113.0/24] then accept; # announce our Anycast /24
reject;
};
};
hold time 90;
keepalive time 30;
}If this worked, a million storefronts would silently migrate to bare metal. If it failed, if we leaked a route, if BIRD crashed, if a BGP peer went down at the wrong moment, stores would go dark and we'd have no AWS to fall back on mid-cutover.
k3s + MetalLB: Kubernetes Without a Cloud
We chose k3s for Kubernetes. The lightweight distribution with no cloud-controller-manager dependency, single binary, sub-5-minute node bootstrap. For bare metal where there's no magical cloud provider to hand you a LoadBalancer IP, k3s was the right call.
MetalLB in BGP mode fills the gap: it peers with BIRD on each node and
announces service IPs into the BGP routing table. When you kubectl apply a
LoadBalancer service, MetalLB assigns it an IP from our Anycast pool and tells BIRD to
announce it. The service becomes globally reachable in seconds.
The first time we watched kubectl get svc show an EXTERNAL-IP that was
actually external — announced to a real upstream router over real BGP — three of us
stood around a laptop in silence for a moment. Then someone said "that actually works" and
we immediately worried we'd jinxed it.
The 72-Hour Cutover Weekend
We picked a Friday night in late February for the final AWS decommission. All nine PoPs were live and had been handling 30% of real traffic for two weeks without incident. The dual-announcement window was closing. It was time.
Everything was going smoothly until Saturday morning, around 3am.
DC3 — our Frankfurt PoP — had a BIRD peer flap. The upstream ISP router sent a BGP notification and BIRD withdrew all routes. For 47 seconds, European traffic had no nearby announcement to follow. It didn't drop (Mumbai picked it up), but latency spiked to 280ms for EU users and the dashboards turned amber.
The root cause: we had misconfigured the BGP hold timer on the Frankfurt ISP's router. It was set to 30 seconds; we were sending keepalives every 30 seconds, and network jitter caused a missed keepalive, which the router interpreted as a dead session. One config change, one reload, one very quiet Slack message at 3:17am: "DC3 back. Hold timer fixed to 90s. Going to sleep."
By Sunday evening, all nine PoPs were running clean. We withdrew the AWS Global Accelerator announcements one region at a time, watching the traffic meters in Grafana. Each withdrawal showed traffic shifting cleanly to the nearest bare-metal PoP. No store went dark. No support tickets. Nothing. I almost expected something to break.
The Result
The finance team's monthly email stopped having the subject line "Can we talk about infrastructure?" and started having the subject line "Is this correct?"
It was correct.
What We Learned
Bare metal isn't scary if you treat it like a distributed systems problem, not a hardware problem. The networking is learnable. BIRD has good documentation. MetalLB has sensible defaults. What's actually scary is doing the cutover without a fallback, so we didn't. Dual-announce. Validate in production. Cut over one region at a time.
We gave up managed upgrades, elastic capacity, and the comfortable feeling that if something breaks it's probably AWS's fault. We got 94% cost reduction, sub-50ms global latency, and full control over our own routing table. For a million-storefront platform that needed to load fast everywhere, that trade was worth every sleepless Saturday.