OnCallReadyStart learning

AWS Lesson 39.23 · AWS VI: Architecting on AWS & the exams · 15 min read

Streaming and data ingestion: Kinesis, Firehose and the analytics map

In plain words

A queue is like a to-do list: someone takes a task and crosses it off. A stream is more like a security camera recording: everything is kept in order for a while, and several people can watch the same recording, rewind it and watch again. Kinesis Data Streams is the recorder, split into channels called shards. Firehose is a courier who collects the footage every minute and drops it in the archive, without anyone having to watch it.

Streaming and data ingestion: Kinesis, Firehose and the analytics map

Queues hand one task to one worker and forget it. Clickstreams, application logs, IoT readings and change events are different: a stream of records, in order per key, that several consumers read independently - the fraud model, the dashboard, the archive - and can re-read from a point in time. On AWS that is Kinesis Data Streams (or Amazon MSK for Kafka). When all you need is "get these records into S3 (or Redshift, OpenSearch, Splunk) in batches", Amazon Data Firehose does it without any consumer code. Behind both sits the analytics side - Glue, Athena, EMR, Lake Formation - which this lesson maps without simulating it.

Need to know: a Kinesis stream is made of shards. Each record has a partition key; its hash picks the shard, so records with the same key stay in order on one shard. A shard takes 1 MB/s or 1,000 records/s in and serves 2 MB/s out, shared by its consumers (or 2 MB/s per consumer with enhanced fan-out). Records are kept 24 hours by default, up to 365 days. Provisioned mode = you choose the shard count; on-demand mode scales shards for you. Firehose buffers records (by size and time) and writes them as objects; you never read from it.

A stream, keys and shards

$ aws kinesis create-stream --stream-name try-clicks --shard-count 2
$ aws kinesis describe-stream-summary --stream-name try-clicks --query 'StreamDescriptionSummary.[StreamStatus,OpenShardCount,RetentionPeriodHours,StreamModeDetails.StreamMode]' --output text
CREATING	2	24	PROVISIONED
$ sleep 30
$ aws kinesis describe-stream-summary --stream-name try-clicks --query 'StreamDescriptionSummary.[StreamStatus,OpenShardCount]' --output text
ACTIVE	2
$ aws kinesis list-shards --stream-name try-clicks --query 'Shards[].[ShardId,HashKeyRange.StartingHashKey]' --output text
shardId-000000000000	0
shardId-000000000001	170141183460469231731687303715884105728

Each shard owns a range of the 128-bit hash space. Write a few clicks for two users:

$ for p in /home /cart /pay; do aws kinesis put-record --stream-name try-clicks --partition-key user-42 --data "{\"user\":42,\"page\":\"$p\"}" --query '[ShardId,SequenceNumber]' --output text; done
shardId-000000000000	49630000000000000000000000000000000000000000000000000001
shardId-000000000000	49630000000000000000000000000000000000000000000000000002
shardId-000000000000	49630000000000000000000000000000000000000000000000000003
$ aws kinesis put-record --stream-name try-clicks --partition-key user-8 --data '{"user":8,"page":"/home"}' --query ShardId --output text
shardId-000000000001

All of user 42's records went to the same shard with increasing sequence numbers. Reading is pull-based: get an iterator for a shard and a starting position (TRIM_HORIZON = the oldest kept record, LATEST, AT_SEQUENCE_NUMBER, AT_TIMESTAMP), then get-records with it; Data is base64:

$ S=$(aws kinesis put-record --stream-name try-clicks --partition-key user-42 --data '{"user":42,"page":"/done"}' --query ShardId --output text)
$ IT=$(aws kinesis get-shard-iterator --stream-name try-clicks --shard-id $S --shard-iterator-type TRIM_HORIZON --query ShardIterator --output text)
$ aws kinesis get-records --shard-iterator $IT --query 'Records[].Data' --output text | tr '\t' '\n' | while read d; do echo $d | base64 -d; echo; done
{"user":42,"page":"/home"}
{"user":42,"page":"/cart"}
{"user":42,"page":"/pay"}
{"user":42,"page":"/done"}

Nobody writes consumers like this in production: the Kinesis Client Library (or a Lambda event source mapping, or Managed Service for Apache Flink) tracks a checkpoint per shard, balances shards across workers and follows resharding. An iterator expires after 5 minutes, and a consumer that falls behind shows it in GetRecords.IteratorAgeMilliseconds - the stream's equivalent of a queue's oldest-message age.

A shard that gets more than its limits answers ProvisionedThroughputExceededException; the fix is more shards (update-shard-count, or on-demand mode), better spread keys (a hot key cannot be fixed by resharding), or batching with put-records.

Firehose: records to S3 without code

A delivery stream with direct PUT, an S3 destination, a prefix and buffering hints - write an object when 1 MiB has arrived or 60 seconds have passed, whichever comes first. The role try-firehose-role trusts firehose.amazonaws.com and may write to the bucket:

$ aws firehose create-delivery-stream --delivery-stream-name try-clicks-archive --delivery-stream-type DirectPut --extended-s3-destination-configuration 'RoleARN=arn:aws:iam::111122223333:role/try-firehose-role,BucketARN=arn:aws:s3:::oncall-try-clicks-111122223333,Prefix=clicks/,BufferingHints={SizeInMBs=1,IntervalInSeconds=60}' --query DeliveryStreamARN --output text
arn:aws:firehose:eu-central-1:111122223333:deliverystream/try-clicks-archive
$ sleep 10
$ aws firehose describe-delivery-stream --delivery-stream-name try-clicks-archive --query 'DeliveryStreamDescription.[DeliveryStreamStatus,Destinations[0].ExtendedS3DestinationDescription.BufferingHints]' --output json
[
    "ACTIVE",
    {
        "SizeInMBs": 1,
        "IntervalInSeconds": 60
    }
]
$ cd ~/oncall-lab/labs/aws/try6
$ cat records.json
[
  {
    "Data": "{\"page\":\"/home\"}\n"
  },
  {
    "Data": "{\"page\":\"/cart\"}\n"
  },
  {
    "Data": "{\"page\":\"/pay\"}\n"
  }
]
$ aws firehose put-record-batch --delivery-stream-name try-clicks-archive --records file://records.json --query FailedPutCount
0
$ aws s3 ls s3://oncall-try-clicks-111122223333/ --recursive | wc -l
0
$ sleep 61
$ aws firehose put-record --delivery-stream-name try-clicks-archive --record file://late.json --query Encrypted --output text
False
$ aws s3 ls s3://oncall-try-clicks-111122223333/ --recursive
2026-09-22 20:01:46         50 clicks/2026/09/22/20/try-clicks-archive-1-2026-09-22-20-01-46-94659338-9d31-4277-9345-c5fa68888bd7
$ aws s3 cp s3://oncall-try-clicks-111122223333/$(aws s3 ls s3://oncall-try-clicks-111122223333/ --recursive | awk 'NR==1{print $4}') -
{"page":"/home"}
{"page":"/cart"}
{"page":"/pay"}

The first three records waited in the buffer; the interval elapsed, and the next record flushed them as one object under clicks/YYYY/MM/DD/HH/ (UTC). Firehose adds no newline between records

  • send them with one, or every downstream reader sees one long line. It can also convert JSON to

Parquet / ORC with a schema from the Glue Data Catalog, compress, partition by fields of the record (dynamic partitioning), transform with a Lambda function, and take a Kinesis stream as its source. Its latency is the buffer: seconds to minutes, never milliseconds - near real time.

The ingestion and analytics map (not simulated)

NeedServiceNotes
real-time stream, many consumers, replayKinesis Data Streams / Amazon MSKshards / Kafka partitions; MSK for Kafka APIs
land records in S3, Redshift, OpenSearchAmazon Data Firehosebuffered, no consumer code, format conversion
process streams with SQL or JavaManaged Service for Apache Flinkthe old Kinesis Data Analytics
catalog and transform data (ETL)AWS Gluecrawlers fill the Data Catalog; Spark jobs turn .csv into .parquet
query files in S3 with SQLAmazon Athenaserverless, pay per TB scanned: Parquet + partitions cut cost
big data frameworks (Spark, Hive, Presto)Amazon EMRclusters on EC2, EKS, or EMR Serverless
permissions on a data lake (tables, columns, rows)AWS Lake Formationon top of the Glue catalog
dashboards and BIAmazon Quick (formerly QuickSight)reads Athena, Redshift, S3
a data warehouseAmazon Redshiftcolumnar, SQL; Redshift Spectrum queries S3
move files online from on-premisesAWS DataSync / Transfer FamilyDataSync: NFS/SMB/HDFS to S3/EFS/FSx, scheduled; Transfer Family: SFTP/FTPS/FTP endpoints into S3
on-premises apps that need cloud storageAWS Storage Gatewayfile, volume and tape gateways with a local cache
terabytes to petabytes offlineAWS Snowballa device shipped to you

A typical data lake: producers -> Kinesis or Firehose -> raw S3 (JSON) -> a Glue job (or Firehose conversion) -> curated S3 (Parquet, partitioned by date) -> the Glue Data Catalog -> Athena / Redshift Spectrum / EMR -> Quick dashboards, with Lake Formation deciding who sees which table. Secure the ingestion points: IAM for producers (no long-lived keys in devices - Cognito or IoT credentials), VPC endpoints for Kinesis and Firehose, encryption with KMS on the stream and the bucket.

On the exam (SAA-C03 3.5 "high-performing data ingestion and transformation"; CLF-C02 3.8): Kinesis Data Streams for real-time with replay and multiple consumers, Firehose for near-real-time delivery to S3 / Redshift / OpenSearch with no code, Glue for ETL and the catalog, Athena for ad-hoc SQL on S3 (Parquet + partitions = less scanned = cheaper), EMR for Spark / Hadoop, Lake Formation for data-lake permissions, DataSync / Storage Gateway / Snowball for transfers.

In an interview: "Kinesis Data Streams or Firehose?" - "Data Streams when consumers need the records in real time, in order per key, several of them, with replay - I own the consumers and the shard capacity. Firehose when the goal is to land the data somewhere, like S3 as Parquet, with a minute of buffering: no consumer code, it scales itself. They combine: a stream for the real-time consumers, with Firehose as one more consumer that archives it."

You can now: create a stream, explain shards, partition keys and their limits, write and read records with iterators, spot a hot partition and iterator age, archive records to S3 with Firehose buffering and know its newline and latency traps, and place Glue, Athena, EMR, Lake Formation, Quick, Redshift, DataSync, Storage Gateway, Transfer Family and Snowball on the map.

Why it helps

Clickstreams, logs, metrics and change events arrive continuously and are needed by several consumers at once - fraud detection, dashboards, the data lake. Choosing between Kinesis Data Streams, Firehose and queues, sizing shards and partition keys, and knowing the analytics services that read the data afterwards (Glue, Athena, EMR, Lake Formation) is exactly the data ingestion and transformation part of the Solutions Architect exam.

Commands in this lesson

aws sleep cd cat

FAQ

What does a partition key do?

Kinesis hashes it to choose the shard, so all records with the same key go to the same shard and stay in order there. A good key spreads load evenly across shards while keeping related records together, such as a user or device id; a key with few values creates hot shards that resharding cannot fix.

How do I scale a Kinesis stream?

Each shard takes 1 MB or 1,000 records per second in and serves 2 MB per second out. Add shards with update-shard-count, or use on-demand mode and let Kinesis manage capacity. ProvisionedThroughputExceededException means a shard, often one hot key, is over its limit.

Why does Firehose not deliver my records immediately?

Firehose buffers records and writes an object when the buffer reaches the configured size or interval, whichever comes first. That makes delivery near real time - seconds to minutes - and produces fewer, larger objects. If you need records within milliseconds, consumers should read Kinesis Data Streams directly.

What are Glue, Athena and EMR for?

Glue catalogs data in S3 with crawlers and runs ETL jobs, for example converting CSV to Parquet. Athena runs SQL directly on files in S3 and bills by data scanned. EMR runs big data frameworks such as Spark on clusters or serverless. They are concepts in this course, not simulated.

Why convert data to Parquet?

Parquet is a columnar, compressed format: queries read only the columns they need, and much less data is scanned. With Athena, which bills per terabyte scanned, Parquet plus partitioning by date can cut both cost and query time by an order of magnitude compared with raw JSON or CSV.

In an interview Mid

Kinesis Data Streams or Firehose - how do you choose?

Kinesis Data Streams when consumers need records in real time, in order per partition key, several independent consumers, and replay within the retention period - I own the consumers and the shard capacity. Firehose when the goal is to land data somewhere such as S3, Redshift or OpenSearch with about a minute of buffering, optional Parquet conversion and no consumer code. They often combine: a stream for real-time consumers and Firehose as one consumer that archives it.

Also asked: How would you design a data lake on AWS? · What is a hot shard and how do you avoid it? · When would you use Athena instead of Redshift?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.