A three-tier web app — load balancer, private application server, private database — built inside a properly segmented VPC. The app itself is a minimal guestbook; the actual subject of this project is what happens after you deploy something: what breaks, how you find out why, and how you stop it from happening again.
Architecture
| Tier | What's there |
|---|---|
| Web | Application Load Balancer, public subnets, two AZs |
| App | EC2 running Flask behind gunicorn, managed by systemd, private subnet |
| Data | RDS MySQL, private subnet, reachable only from the app tier |
Three tiered security groups, each accepting traffic only from the tier in front of it — referenced by security group ID, not by IP range. Public subnets route to an Internet Gateway; the private subnets reach the internet only outbound, through a NAT Gateway. There's no SSH key and no bastion host: the EC2 instance has no public IP and is reached exclusively through SSM Session Manager, with an IAM role scoped to SSM access and permission to read exactly one secret.
The /health endpoint the ALB polls returns 200 only if the app can also reach the database — so a broken DB connection shows up as an unhealthy target, not a silently-failing app that looks fine to the load balancer.
Monitoring, tested — not assumed
A CloudWatch dashboard tracks EC2 CPU, ALB request count and 5XX errors, and RDS connections. Alarms notify an SNS email on an unhealthy target, on application-level 5XX errors, and on high CPU. Every alarm was tested by actually inducing the failure it's supposed to catch, rather than configured once and trusted — that practice is what caught the bug in incident 04 below, an alarm that looked correctly configured and would never have fired.
The NAT Gateway and the ALB are the only two components here billed hourly, so the whole environment is built, tested, and torn down in a single session rather than left running.
Five real incidents
These weren't scripted. They're the actual problems hit while building and operating this environment, each written up as symptom → hypotheses → diagnosis → root cause → fix → prevention — the same structure a support engineer uses on a real ticket.
| # | Incident | Root cause |
|---|---|---|
| 1 | SSM Session Manager wouldn't connect | The EC2 role's trust policy didn't allow the EC2 service to assume it — no trust, no credentials, no registration |
| 2 | SSM connection timed out after recreating the NAT Gateway | The new NAT Gateway was built in a private subnet instead of a public one, so it had no route to the internet |
| 3 | The app couldn't reach the database after a reboot | Environment variables set with export don't survive a reboot; the app fell back to a default of localhost |
| 4 | A CloudWatch alarm never fired with the app down | The threshold was > 1 on a single-instance target group — a count that can only ever reach 1, so it could never breach |
| 5 | The systemd service failed to start | The service ran as ec2-user, but the app code and its Python packages lived under a different user's home directory |
Incident 02, in full: the NAT Gateway in the wrong subnet
After stopping the environment overnight to save on NAT and ALB costs, I rebuilt the NAT Gateway the next day and tried to reconnect over SSM. It timed out — but the error was different from incident 01's credential failure: this time the agent had credentials and was failing to reach ssm.us-east-1.amazonaws.com on port 443 with an i/o timeout.
A timeout to a public AWS endpoint from a private instance points at routing, not identity — the instance had no working outbound path. I traced the private subnet's route table (correct, pointing 0.0.0.0/0 at the NAT), then checked the NAT Gateway's own subnet, and that's where the fault was: the NAT Gateway had been created in a private subnet. A NAT Gateway has to live in a public subnet — one with its own route to the Internet Gateway — or it has no way out itself, and everything routed through it times out.
NAT Gateways can't be moved between subnets, so the fix was to delete it, release the Elastic IP, recreate the NAT in a subnet I first confirmed had an Internet Gateway route, and repoint the private subnet's route table at the new NAT. Two minutes later the SSM agent reconnected on its own, confirmed with curl -I https://aws.amazon.com from the instance.
The lesson that generalizes: "NAT in a private subnet" creates successfully and only fails at traffic time, which makes it a classic silent misconfiguration. Now the habit is to check a subnet's route table for the Internet Gateway entry before deciding it's public.
The other four incidents — the IAM trust policy, the lost environment variables, the alarm threshold, and the systemd user/permission cascade — are written up in full in the troubleshooting log.
What I'd do differently next
- Terraform. This environment was built by hand once, following a checklist, specifically so I'd feel every step that's tedious and error-prone enough to automate. The next iteration is
terraform applyto build it andterraform destroyto tear it down. - HTTPS. The ALB listener is plain HTTP right now. An ACM certificate and an 80→443 redirect are the obvious next addition.
- Real high availability. There's one EC2 instance behind the ALB today. A second instance in the second AZ would make this actually survive an AZ failure, not just look like it's set up to.
The repository, including the full troubleshooting log, is public: karan-sohi/aws-three-tier-webapp.