Streaming and data ingestion: Kinesis, Firehose and the analytics map
Queues hand one task to one worker and forget it. Clickstreams, application logs, IoT readings and change events are different: a stream of records, in order per key, that several consumers read independently - the fraud model, the dashboard, the archive - and can re-read from a point in time. On AWS that is Kinesis Data Streams (or Amazon MSK for Kafka). When all you need is "get these records into S3 (or Redshift, OpenSearch, Splunk) in batches", Amazon Data Firehose does it without any consumer code. Behind both sits the analytics side - Glue, Athena, EMR, Lake Formation - which this lesson maps without simulating it.
Need to know: a Kinesis stream is made of shards. Each record has a partition key; its hash picks the shard, so records with the same key stay in order on one shard. A shard takes 1 MB/s or 1,000 records/s in and serves 2 MB/s out, shared by its consumers (or 2 MB/s per consumer with enhanced fan-out). Records are kept 24 hours by default, up to 365 days. Provisioned mode = you choose the shard count; on-demand mode scales shards for you. Firehose buffers records (by size and time) and writes them as objects; you never read from it.
A stream, keys and shards
$ aws kinesis create-stream --stream-name try-clicks --shard-count 2
$ aws kinesis describe-stream-summary --stream-name try-clicks --query 'StreamDescriptionSummary.[StreamStatus,OpenShardCount,RetentionPeriodHours,StreamModeDetails.StreamMode]' --output text
CREATING 2 24 PROVISIONED
$ sleep 30
$ aws kinesis describe-stream-summary --stream-name try-clicks --query 'StreamDescriptionSummary.[StreamStatus,OpenShardCount]' --output text
ACTIVE 2
$ aws kinesis list-shards --stream-name try-clicks --query 'Shards[].[ShardId,HashKeyRange.StartingHashKey]' --output text
shardId-000000000000 0
shardId-000000000001 170141183460469231731687303715884105728
Each shard owns a range of the 128-bit hash space. Write a few clicks for two users:
$ for p in /home /cart /pay; do aws kinesis put-record --stream-name try-clicks --partition-key user-42 --data "{\"user\":42,\"page\":\"$p\"}" --query '[ShardId,SequenceNumber]' --output text; done
shardId-000000000000 49630000000000000000000000000000000000000000000000000001
shardId-000000000000 49630000000000000000000000000000000000000000000000000002
shardId-000000000000 49630000000000000000000000000000000000000000000000000003
$ aws kinesis put-record --stream-name try-clicks --partition-key user-8 --data '{"user":8,"page":"/home"}' --query ShardId --output text
shardId-000000000001
All of user 42's records went to the same shard with increasing sequence numbers. Reading is pull-based: get an iterator for a shard and a starting position (TRIM_HORIZON = the oldest kept record, LATEST, AT_SEQUENCE_NUMBER, AT_TIMESTAMP), then get-records with it; Data is base64:
$ S=$(aws kinesis put-record --stream-name try-clicks --partition-key user-42 --data '{"user":42,"page":"/done"}' --query ShardId --output text)
$ IT=$(aws kinesis get-shard-iterator --stream-name try-clicks --shard-id $S --shard-iterator-type TRIM_HORIZON --query ShardIterator --output text)
$ aws kinesis get-records --shard-iterator $IT --query 'Records[].Data' --output text | tr '\t' '\n' | while read d; do echo $d | base64 -d; echo; done
{"user":42,"page":"/home"}
{"user":42,"page":"/cart"}
{"user":42,"page":"/pay"}
{"user":42,"page":"/done"}
Nobody writes consumers like this in production: the Kinesis Client Library (or a Lambda event source mapping, or Managed Service for Apache Flink) tracks a checkpoint per shard, balances shards across workers and follows resharding. An iterator expires after 5 minutes, and a consumer that falls behind shows it in GetRecords.IteratorAgeMilliseconds - the stream's equivalent of a queue's oldest-message age.
A shard that gets more than its limits answers ProvisionedThroughputExceededException; the fix is more shards (update-shard-count, or on-demand mode), better spread keys (a hot key cannot be fixed by resharding), or batching with put-records.
Firehose: records to S3 without code
A delivery stream with direct PUT, an S3 destination, a prefix and buffering hints - write an object when 1 MiB has arrived or 60 seconds have passed, whichever comes first. The role try-firehose-role trusts firehose.amazonaws.com and may write to the bucket:
$ aws firehose create-delivery-stream --delivery-stream-name try-clicks-archive --delivery-stream-type DirectPut --extended-s3-destination-configuration 'RoleARN=arn:aws:iam::111122223333:role/try-firehose-role,BucketARN=arn:aws:s3:::oncall-try-clicks-111122223333,Prefix=clicks/,BufferingHints={SizeInMBs=1,IntervalInSeconds=60}' --query DeliveryStreamARN --output text
arn:aws:firehose:eu-central-1:111122223333:deliverystream/try-clicks-archive
$ sleep 10
$ aws firehose describe-delivery-stream --delivery-stream-name try-clicks-archive --query 'DeliveryStreamDescription.[DeliveryStreamStatus,Destinations[0].ExtendedS3DestinationDescription.BufferingHints]' --output json
[
"ACTIVE",
{
"SizeInMBs": 1,
"IntervalInSeconds": 60
}
]
$ cd ~/oncall-lab/labs/aws/try6
$ cat records.json
[
{
"Data": "{\"page\":\"/home\"}\n"
},
{
"Data": "{\"page\":\"/cart\"}\n"
},
{
"Data": "{\"page\":\"/pay\"}\n"
}
]
$ aws firehose put-record-batch --delivery-stream-name try-clicks-archive --records file://records.json --query FailedPutCount
0
$ aws s3 ls s3://oncall-try-clicks-111122223333/ --recursive | wc -l
0
$ sleep 61
$ aws firehose put-record --delivery-stream-name try-clicks-archive --record file://late.json --query Encrypted --output text
False
$ aws s3 ls s3://oncall-try-clicks-111122223333/ --recursive
2026-09-22 20:01:46 50 clicks/2026/09/22/20/try-clicks-archive-1-2026-09-22-20-01-46-94659338-9d31-4277-9345-c5fa68888bd7
$ aws s3 cp s3://oncall-try-clicks-111122223333/$(aws s3 ls s3://oncall-try-clicks-111122223333/ --recursive | awk 'NR==1{print $4}') -
{"page":"/home"}
{"page":"/cart"}
{"page":"/pay"}
The first three records waited in the buffer; the interval elapsed, and the next record flushed them as one object under clicks/YYYY/MM/DD/HH/ (UTC). Firehose adds no newline between records
- send them with one, or every downstream reader sees one long line. It can also convert JSON to
Parquet / ORC with a schema from the Glue Data Catalog, compress, partition by fields of the record (dynamic partitioning), transform with a Lambda function, and take a Kinesis stream as its source. Its latency is the buffer: seconds to minutes, never milliseconds - near real time.
The ingestion and analytics map (not simulated)
| Need | Service | Notes |
|---|---|---|
| real-time stream, many consumers, replay | Kinesis Data Streams / Amazon MSK | shards / Kafka partitions; MSK for Kafka APIs |
| land records in S3, Redshift, OpenSearch | Amazon Data Firehose | buffered, no consumer code, format conversion |
| process streams with SQL or Java | Managed Service for Apache Flink | the old Kinesis Data Analytics |
| catalog and transform data (ETL) | AWS Glue | crawlers fill the Data Catalog; Spark jobs turn .csv into .parquet |
| query files in S3 with SQL | Amazon Athena | serverless, pay per TB scanned: Parquet + partitions cut cost |
| big data frameworks (Spark, Hive, Presto) | Amazon EMR | clusters on EC2, EKS, or EMR Serverless |
| permissions on a data lake (tables, columns, rows) | AWS Lake Formation | on top of the Glue catalog |
| dashboards and BI | Amazon Quick (formerly QuickSight) | reads Athena, Redshift, S3 |
| a data warehouse | Amazon Redshift | columnar, SQL; Redshift Spectrum queries S3 |
| move files online from on-premises | AWS DataSync / Transfer Family | DataSync: NFS/SMB/HDFS to S3/EFS/FSx, scheduled; Transfer Family: SFTP/FTPS/FTP endpoints into S3 |
| on-premises apps that need cloud storage | AWS Storage Gateway | file, volume and tape gateways with a local cache |
| terabytes to petabytes offline | AWS Snowball | a device shipped to you |
A typical data lake: producers -> Kinesis or Firehose -> raw S3 (JSON) -> a Glue job (or Firehose conversion) -> curated S3 (Parquet, partitioned by date) -> the Glue Data Catalog -> Athena / Redshift Spectrum / EMR -> Quick dashboards, with Lake Formation deciding who sees which table. Secure the ingestion points: IAM for producers (no long-lived keys in devices - Cognito or IoT credentials), VPC endpoints for Kinesis and Firehose, encryption with KMS on the stream and the bucket.
On the exam (SAA-C03 3.5 "high-performing data ingestion and transformation"; CLF-C02 3.8): Kinesis Data Streams for real-time with replay and multiple consumers, Firehose for near-real-time delivery to S3 / Redshift / OpenSearch with no code, Glue for ETL and the catalog, Athena for ad-hoc SQL on S3 (Parquet + partitions = less scanned = cheaper), EMR for Spark / Hadoop, Lake Formation for data-lake permissions, DataSync / Storage Gateway / Snowball for transfers.
In an interview: "Kinesis Data Streams or Firehose?" - "Data Streams when consumers need the records in real time, in order per key, several of them, with replay - I own the consumers and the shard capacity. Firehose when the goal is to land the data somewhere, like S3 as Parquet, with a minute of buffering: no consumer code, it scales itself. They combine: a stream for the real-time consumers, with Firehose as one more consumer that archives it."
You can now: create a stream, explain shards, partition keys and their limits, write and read records with iterators, spot a hot partition and iterator age, archive records to S3 with Firehose buffering and know its newline and latency traps, and place Glue, Athena, EMR, Lake Formation, Quick, Redshift, DataSync, Storage Gateway, Transfer Family and Snowball on the map.