OnCallReady

Lesson 31.18 · AWS III: CloudWatch, CloudTrail, Cost & Incidents · 17 min read

The cost model: what you pay for, and the bills that surprise people

In plain words

AWS is a utility like electricity: you pay for what you use. Machines are paid by the hour, storage by how much you keep each month, and moving data by the gigabyte. Some things cost money just by existing, even when nobody uses them.

Cost Explorer is the itemised bill. It can split the total by service, by kind of usage and by day, so when the bill jumps you can see exactly which line grew and when.

The cost model: what you pay for, and the bills that surprise people

On AWS a bad deploy can take the shop down, and a bad architecture decision can quietly cost more than the team's salaries. Cost is an operational signal like latency: it has a baseline, it has spikes, and a spike has a cause you can find with the same discipline as an outage. This lesson is the model - what AWS bills for - and Cost Explorer from the CLI, plus the handful of surprises behind most "why is the bill three times higher?" threads.

Need to know: you pay per unit of use: instance-hours, GB-months of storage, GB moved, requests. aws ce get-cost-and-usage breaks the bill down by SERVICE, USAGE_TYPE, REGION or a tag, DAILY or MONTHLY - and its End date is exclusive. NAT gateway charges, EBS volumes and data transfer between AZs are billed under "EC2 - Other". Data into AWS is free; out to the internet, between AZs, and anything through a NAT gateway is not. Each Cost Explorer API request costs $0.01.

The units

whathow it is billedeu-central-1 list price
an m7g.large instanceper second while running, quoted per hour$0.0957/hour
a gp3 EBS volumeper GB-month provisioned (used or not, attached or not)$0.0952/GB-month
a NAT gatewayper hour and per GB processed$0.052/hour + $0.052/GB
data between two AZsper GB, each direction$0.01/GB
data out to the internetper GB (first 100 GB/month free)$0.09/GB
a public IPv4 addressper hour, attached or not (since February 2024)$0.005/hour
S3 Standardper GB-month stored, plus per request$0.0245/GB-month
CloudWatch Logsper GB ingested, then per GB-month stored$0.63/GB + $0.0324/GB-month
an Application Load Balancerper hour plus LCUs (load)$0.027/hour + $0.008/LCU-hour

Prices change and differ per Region; the AWS Pricing Calculator and each service's pricing page are the reference. The shape matters more than the cents: anything provisioned costs per hour or per GB-month whether you use it or not, and anything that moves data costs per GB.

Cost Explorer from the CLI

Month to date, per service:

$ M=$(date +%Y-%m-01); T=$(date -d tomorrow +%F); echo "$M -> $T"
2026-09-01 -> 2026-09-23
$ aws ce get-cost-and-usage --time-period Start=$M,End=$T --granularity MONTHLY --metrics UnblendedCost --group-by Type=DIMENSION,Key=SERVICE --query 'ResultsByTime[0].Groups[].[Keys[0],Metrics.UnblendedCost.Amount]' --output text | sort -t$'\t' -k2 -rn | awk -F'\t' '{printf "%-45s %10.2f\n", $1, $2}'
EC2 - Other                                       239.64
Amazon Elastic Compute Cloud - Compute            200.59
Amazon Simple Storage Service                     143.90
AmazonCloudWatch                                  124.98
Amazon Elastic Container Service for Kubernetes      52.40
Amazon Elastic Load Balancing                      19.68
Amazon Virtual Private Cloud                       13.10
AWS Key Management Service                          4.31
Amazon Route 53                                     0.36
AWS Cost Explorer                                   0.01

End is exclusive: End=$(date +%F) stops at yesterday midnight, so "month to date" needs tomorrow's date. Today's costs are partial and every period of the current month is Estimated: Cost Explorer refreshes a few times a day and can lag by up to 24 hours. Amounts are strings with ten decimals; --metrics picks the number (UnblendedCost is what you were charged; AmortizedCost spreads upfront reservations; UsageQuantity is the hours, GB or requests).

The second-biggest line is EC2 - Other, and it is not instances. Drill into it by usage type:

$ aws ce get-cost-and-usage --time-period Start=$M,End=$T --granularity MONTHLY --metrics UnblendedCost UsageQuantity --filter '{"Dimensions": {"Key": "SERVICE", "Values": ["EC2 - Other"]}}' --group-by Type=DIMENSION,Key=USAGE_TYPE --query 'ResultsByTime[0].Groups[].[Keys[0],Metrics.UnblendedCost.Amount,Metrics.UsageQuantity.Amount,Metrics.UsageQuantity.Unit]' --output text
EUC1-DataTransfer-Out-Bytes	52.2173217187	580.1924635413	GB
EUC1-DataTransfer-Regional-Bytes	12.3475627903	1234.7562790322	GB
EUC1-EBS:SnapshotUsage	34.8932496392	646.1712896155	GB-Mo
EUC1-EBS:VolumeUsage.gp3	64.9186487376	681.918579176	GB-Mo
EUC1-NatGateway-Bytes	48.011105746	923.2904951163	GB
EUC1-NatGateway-Hours	27.2480606667	524.0011666667	Hrs

A usage type reads like a sentence: EUC1 (the Region: Frankfurt; EU is Ireland, us-east-1 has no prefix) - NatGateway-Bytes (what) - and the unit in UsageQuantity. "EC2 - Other" collects EBS volumes and snapshots, NAT gateway hours and bytes, Elastic IPs and data transfer - the lines people do not think of as "EC2".

Data transfer, the expensive arrows

Draw the arrows of your architecture and price each one:

The classic surprise: a batch job in a private subnet copies a few TB a day from S3. Without an S3 gateway endpoint every byte goes through the NAT gateway: 3 TB a day is about $160 a day in NAT processing, for traffic that never left the Region. If the job runs in another AZ than the NAT gateway it pays inter-AZ transfer on top. You will find exactly that in the incident later in this chapter.

$ aws ce get-cost-and-usage --time-period Start=$(date -d '-7 days' +%F),End=$(date +%F) --granularity DAILY --metrics UnblendedCost --filter '{"Dimensions": {"Key": "USAGE_TYPE", "Values": ["EUC1-NatGateway-Bytes", "EUC1-DataTransfer-Regional-Bytes"]}}' --query 'ResultsByTime[].[TimePeriod.Start,Total.UnblendedCost.Amount]' --output text
2026-09-15	2.6181735552
2026-09-16	2.8324480688
2026-09-17	2.6312348128
2026-09-18	2.719798112
2026-09-19	2.7786983984
2026-09-20	2.6632366
2026-09-21	2.8965727808

The usual surprises

Most bill spikes are one of these:

  1. Idle or forgotten resources: an instance from a test, an unattached EBS volume (billed per GB-month even when nothing uses it), old snapshots, a load balancer with no targets (hourly charge).
  2. Data transfer: NAT gateway processing, cross-AZ chatter, egress after a launch or a misconfigured CDN.
  3. Logs and metrics: debug logging in production (ingestion per GB), log groups that never expire, high-cardinality custom metrics.
  4. Public IPv4 addresses: $3.65 a month each, every one of them, since February 2024.
  5. Scaling that did not scale back: an Auto Scaling group whose minimum was raised in an incident and never lowered.

The unattached volume and the unused address from this lesson's setup:

$ aws ec2 describe-volumes --filters Name=status,Values=available --query 'Volumes[].[VolumeId,Size,VolumeType,CreateTime]' --output text
vol-0aa11bb22cc33dd44	500	gp3	2026-07-24T20:00:03.000+00:00
$ aws ce get-cost-and-usage --time-period Start=$M,End=$T --granularity MONTHLY --metrics UnblendedCost UsageQuantity --filter '{"Dimensions": {"Key": "USAGE_TYPE", "Values": ["EUC1-PublicIPv4:InUseAddress"]}}' --query 'ResultsByTime[0].Total'
{
    "UnblendedCost": {
        "Amount": "13.10003125",
        "Unit": "USD"
    },
    "UsageQuantity": {
        "Amount": "2620.00625",
        "Unit": "Hrs"
    }
}

Forecasts, the estimated-charges metric, and the cost of asking

$ aws ce get-cost-forecast --time-period Start=$(date -d tomorrow +%F),End=$(date -d "$(date +%Y-%m-01) +1 month" +%F) --metric UNBLENDED_COST --granularity MONTHLY --query 'Total'
{
    "Amount": "292.6920481027",
    "Unit": "USD"
}
$ aws cloudwatch get-metric-statistics --region us-east-1 --namespace AWS/Billing --metric-name EstimatedCharges --dimensions Name=Currency,Value=USD --start-time $(date -u -d '-1 day' +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 21600 --statistics Maximum --query 'sort_by(Datapoints, &Timestamp)[-1].Maximum'
799

The forecast API spells metrics in capitals (UNBLENDED_COST) where get-cost-and-usage uses UnblendedCost - a real inconsistency of the API. AWS/Billing EstimatedCharges is the month-to-date total, published every few hours only in us-east-1 (after billing alerts are enabled in the billing preferences); the oldest billing alarm pattern is an alarm on it. And every Cost Explorer API request is billed at $0.01 - a dashboard that polls it every minute costs $430 a month. It shows up in the bill as "AWS Cost Explorer":

$ aws ce get-cost-and-usage --time-period Start=$(date +%F),End=$T --granularity DAILY --metrics UsageQuantity UnblendedCost --filter '{"Dimensions": {"Key": "SERVICE", "Values": ["AWS Cost Explorer"]}}' --query 'ResultsByTime[0].Total'
{
    "UsageQuantity": {
        "Amount": "6",
        "Unit": "Requests"
    },
    "UnblendedCost": {
        "Amount": "0.06",
        "Unit": "USD"
    }
}

In an interview: "The AWS bill doubled this month. How do you find out why?" - "Cost Explorer grouped by service, daily, to find the line and the day it jumped; then that service grouped by usage type - often it is EC2 - Other, which is NAT gateway bytes, EBS or data transfer; then by resource or tag to find the owner. The day it started usually matches a deploy or a new job. Typical causes: data through a NAT gateway, cross-AZ traffic, forgotten resources, log ingestion."

You can now: explain what AWS bills for, read month-to-date and daily costs from the CLI (End is exclusive), decode usage types, look inside "EC2 - Other", price the data-transfer arrows of an architecture including the NAT gateway trap, list the usual surprises, and forecast the month.

Why it helps

A bill that triples without anyone noticing is an incident that nobody paged for. Knowing what AWS charges for - and especially the charges people do not think of, such as NAT gateway processing, traffic between zones and logs - turns a scary invoice into a few queries and a fix.

It is also what makes architecture reviews concrete: every arrow in a diagram either costs nothing or costs per gigabyte, and you should know which before the arrow goes to production.

Commands in this lesson

aws

FAQ

Why does my Cost Explorer query miss today?

The End date is exclusive: End=2026-10-08 stops at the end of October 7. Use tomorrow's date to include today, and remember today's costs are partial and the whole current month is marked Estimated until the invoice is final.

What is "EC2 - Other" on my bill?

It is not instances. It collects EBS volumes and snapshots, NAT gateway hours and bytes, Elastic IPs and data transfer. Filter Cost Explorer on that service and group by USAGE_TYPE to see which one grew.

Why is traffic to S3 from a private subnet expensive?

Without a VPC endpoint it goes through the NAT gateway, which charges per gigabyte processed, plus transfer between zones if the gateway is in another one. An S3 gateway endpoint is free and takes that traffic off the NAT gateway without changing the application.

Does using Cost Explorer cost money?

The console is free, but each Cost Explorer API request is billed at one cent. A dashboard that polls the API every minute costs hundreds of dollars a month; cache the results or query a few times a day.

Where is the total cost metric in CloudWatch?

AWS/Billing EstimatedCharges, with the dimension Currency=USD, published every few hours and only in us-east-1, after billing alerts are enabled in the billing preferences. An alarm on it is the oldest and simplest form of billing alert.

In an interview Mid

The AWS bill doubled this month. How do you find out why?

Cost Explorer get-cost-and-usage grouped by SERVICE, DAILY, to find the line and the day it jumped; then that service grouped by USAGE_TYPE - often it is EC2 - Other, which is NAT gateway bytes, EBS or data transfer; then by tag or resource to find the owner. The day usually matches a deploy or a new job. Typical causes: data through a NAT gateway, cross-AZ traffic, forgotten resources, log ingestion.

Also asked: Which data transfer on AWS is free, and which is not? · How would you cut a NAT gateway bill? · What costs money on AWS even when nothing uses it?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.