The Connection Limit Nightmare
People assume that because it is in the cloud it just works. It doesnt.
Take Postgres for example. We moved our massive on-prem Postgres instance into AWS RDS. On day one we ran into connection limits. Our app was written in Node and every time a container spun up it grabbed a handful of connections. RDS has a hard limit on connections based on how much RAM you pay for. In our on-prem setup we just threw more memory at the VM. In RDS you cant. You just hit the max_connections limit and the database starts rejecting queries. The app goes down. We had to scramble to put PgBouncer in front of everything to multiplex the connections.
Disk Bloat and Dead Tuples
We also didn't watch our disk metrics. We had a session tracking table getting hit 50 times a second. I logged in one morning and a 5GB database was suddenly taking up 60GB of disk space. The default autovacuum settings were completely failing to clear out the dead tuples. Postgres was just writing new rows and ignoring the garbage. I ended up spending two days messing with the AWS parameter group to force autovacuum to run almost constantly so the disk wouldn't fill up and crash the instance.
The NoSQL Trap
Someone decided our new microservice needed to use DynamoDB because it was web scale. DynamoDB is fast if you know what you are doing but we didn't understand the partition logic.
The developer used a timestamp as the partition key. Every single write hit the exact same physical partition on Amazon's backend. AWS heavily throttles you when you do that. We were paying for thousands of write capacity units but the app was crawling because of that one hot partition.
Egress Fees and Replication Lag
Storing data in S3 or RDS is cheap but moving it is where they get you. We had a team running BI tools in Google Cloud while our databases were in AWS us-east-1. They were pulling hundreds of gigabytes across the public internet every night. AWS charges around 9 cents a gig for outbound data. We were paying thousands of dollars a month literally just to move our own data out of their data center.
We spun up a Postgres read replica to offload heavy reporting queries. It works great until your primary database gets hit with a massive batch update. Because the replication is asynchronous the read replica falls behind trying to apply the same changes. I watched our replication lag spike to 40 minutes once during a Black Friday sale.
If a user bought something on the primary database and then clicked on their profile which read from the replica their order was just missing. People panicked. They thought they got charged for nothing. We ended up having to write terrible application logic to force users to read from the primary database for 5 minutes after they made a purchase just to hide the lag from them.
Backup Costs and Azure Throttling
Even the backups cost a fortune. You check the box that says retain automated backups for 30 days because it sounds like the responsible thing to do. But snapshot storage costs money. Since our database had such a high churn rate those incremental snapshots were massive. We were paying for terabytes of snapshot storage on a database that was only a fraction of that size.
Azure SQL uses DTUs which is basically a made up metric combining CPU memory and IO. You buy a 100 DTU database and everything is fine until month end reporting starts. Then you hit the DTU ceiling and Azure just chokes the database. Everything hangs. Since DTU is a blended metric you can't even tell if you are out of RAM or if the disk is too slow. You literally just slide the billing bar to the right and hope the problem goes away.
Serverless is a Lie
To save money on development environments we tried using Serverless databases. Aurora Serverless scales down to zero when nobody is using it. It seemed brilliant for staging environments that only get used during business hours.
Except when QA logs in at 9 AM and clicks a button the database has to physically wake up. That takes a few seconds. If your API gateway has a 3 second timeout QA gets a 504 Gateway Timeout error before the database even turns on. They report a bug. You spend an hour debugging only to realize the database was just asleep. We had to write a dummy cron job just to ping the database every 15 minutes to keep it awake which completely defeated the purpose of it being serverless.
The Mongo Sharding Disaster
Before the Postgres migration we had a massive Mongo cluster running in Atlas. We hit about 3TB and had to turn on sharding. Whoever set it up picked tenant ID as the shard key which you cant change later without literally wiping the database. It was a disaster. We had a couple of giant enterprise customers and thousands of small ones. The shard holding the giant customers was constantly pegging its CPU at 100% while the rest of the cluster did absolutely nothing. And any cross-tenant queries would just scatter-gather across the whole network and time out. We lived with that garbage performance for two years because management wouldn't approve the downtime required to dump and restore a 4TB cluster.
The cloud doesnt actually fix bad architecture it just hides it behind a massive monthly invoice. You trade swapping dead hard drives at 3 AM for digging through AWS Cost Explorer trying to figure out who spun up an unattached terabyte disk. You dont patch operating systems anymore you just wake up to an email saying AWS is rebooting your primary database in ten minutes for mandatory maintenance whether you are ready or not.
