Redis 8 Streams and Time Series
Redis 8 streams time series capabilities make Redis a compelling choice for real-time data pipelines. Redis Streams provide a durable, append-only log structure similar to Kafka but with sub-millisecond latency. Combined with the TimeSeries module, you can ingest, process, and query time-stamped data entirely within Redis. Notably, Redis 8 ships TimeSeries, Bloom, JSON, and the query engine as core components rather than separately installed modules, which simplifies deployment considerably.
This guide covers production patterns for building real-time pipelines — from event ingestion with consumer groups to windowed aggregations and alerting. If your use case demands sub-millisecond latency and you are already running Redis, streams and time series eliminate the need for separate message brokers and TSDB systems. Consequently, you reduce both operational surface area and the network hops that erode tail latency.
Redis Streams Fundamentals
A Redis Stream is an append-only log where each entry has an auto-generated ID (timestamp-based) and a set of field-value pairs. Consumer groups enable multiple consumers to process the stream in parallel, with each message delivered to exactly one consumer within the group. Importantly, this is not the same as Kafka’s exactly-once semantics across systems — Redis guarantees at-least-once delivery within a group, and your handler must be idempotent to tolerate redelivery after a crash.
# Add events to a stream
XADD sensor:readings * device_id sensor-42 temperature 23.5 humidity 68.2
XADD sensor:readings * device_id sensor-43 temperature 24.1 humidity 65.8
# Read latest entries
XRANGE sensor:readings - + COUNT 10
# Create a consumer group
XGROUP CREATE sensor:readings processing-group $ MKSTREAM
# Read as consumer in group (blocks until data available)
XREADGROUP GROUP processing-group consumer-1 COUNT 10 BLOCK 5000 STREAMS sensor:readings >
# Acknowledge processed messages
XACK sensor:readings processing-group 1679012345678-0
Building a Consumer Group Pipeline
import redis.asyncio as redis
import asyncio
import json
class StreamProcessor:
def __init__(self, redis_url: str, stream: str, group: str, consumer: str):
self.client = redis.from_url(redis_url)
self.stream = stream
self.group = group
self.consumer = consumer
async def setup(self):
try:
await self.client.xgroup_create(self.stream, self.group, id="$", mkstream=True)
except redis.ResponseError as e:
if "BUSYGROUP" not in str(e):
raise
async def process(self):
await self.setup()
while True:
# Read new messages (> means undelivered only)
messages = await self.client.xreadgroup(
groupname=self.group,
consumername=self.consumer,
streams={self.stream: ">"},
count=100,
block=5000,
)
if not messages:
continue
for stream_name, entries in messages:
for entry_id, data in entries:
try:
await self.handle_event(entry_id, data)
await self.client.xack(self.stream, self.group, entry_id)
except Exception as e:
print(f"Error processing {entry_id}: {e}")
# Message stays pending, will be reclaimed
async def handle_event(self, entry_id: str, data: dict):
device_id = data.get(b"device_id", b"").decode()
temperature = float(data.get(b"temperature", 0))
# Store in time series for aggregation
await self.client.execute_command(
"TS.ADD", f"ts:temp:{device_id}",
"*", temperature,
"RETENTION", 86400000,
"LABELS", "device", device_id, "metric", "temperature"
)
# Alert on threshold
if temperature > 35.0:
await self.client.publish("alerts", json.dumps({
"device": device_id,
"temperature": temperature,
"severity": "high"
}))
# Run multiple consumers for horizontal scaling
async def main():
processor = StreamProcessor(
"redis://localhost:6379",
"sensor:readings",
"processing-group",
f"consumer-{os.getpid()}"
)
await processor.process()
Handling the Pending Entries List and Poison Messages
The consumer-group model above has a subtle failure mode that catches many teams in production. When a consumer reads a message with XREADGROUP but crashes before calling XACK, that message does not disappear — it sits in the Pending Entries List (PEL) attached to the consumer that read it. If that consumer never comes back, the message is orphaned forever. Therefore, a robust pipeline needs a reclaim loop that periodically scans for stale pending messages and reassigns them to a healthy worker.
Since Redis 6.2 the idiomatic tool for this is XAUTOCLAIM, which atomically finds and transfers messages whose idle time exceeds a threshold. Crucially, you must also track the delivery count so that a genuinely broken “poison” message — one that fails every time it is processed — does not loop forever. The docs recommend routing messages past a retry ceiling to a dead-letter stream for offline inspection.
# Inspect pending messages for the group (summary view)
XPENDING sensor:readings processing-group
# Detailed view: ranges, idle time, delivery counts per message
XPENDING sensor:readings processing-group IDLE 60000 - + 100
# Atomically claim messages idle for >60s, hand them to consumer-2
# Returns a cursor; loop until it returns "0-0"
XAUTOCLAIM sensor:readings processing-group consumer-2 60000 0 COUNT 50
# Move a confirmed poison message to a dead-letter stream, then ack it
XADD sensor:readings:dlq * original_id 1679012345678-0 reason "parse_failure"
XACK sensor:readings processing-group 1679012345678-0
In production, teams typically run the reclaim loop on a 30-to-60-second cadence and treat any message with a delivery count above three or four as poison. As a result, transient failures recover automatically while persistently bad payloads are quarantined instead of stalling the whole group.
Time Series Aggregations
# Create time series with retention and labels
TS.CREATE ts:temp:sensor-42 RETENTION 604800000 LABELS device sensor-42 metric temperature
TS.CREATE ts:humidity:sensor-42 RETENTION 604800000 LABELS device sensor-42 metric humidity
# Add data points
TS.ADD ts:temp:sensor-42 * 23.5
TS.ADD ts:temp:sensor-42 * 24.1
# Create aggregation rules (automatic downsampling)
TS.CREATERULE ts:temp:sensor-42 ts:temp:sensor-42:avg_5m AGGREGATION avg 300000
TS.CREATERULE ts:temp:sensor-42 ts:temp:sensor-42:max_1h AGGREGATION max 3600000
# Query with aggregation
TS.RANGE ts:temp:sensor-42 - + AGGREGATION avg 60000 # 1-minute averages
TS.MRANGE - + FILTER metric=temperature AGGREGATION avg 300000 # All devices
Memory, Retention, and Downsampling Strategy
Because every raw sample lives in RAM, retention and downsampling are not optional tuning knobs — they are the difference between a stable cluster and an out-of-memory crash. A single raw sample in Redis TimeSeries consumes roughly a handful of bytes after Gorilla-style compression, but at high cardinality the totals add up fast. Consider ten thousand devices each emitting one reading per second: that is 864 million points per day, and even at a conservative compressed size per point you are looking at multiple gigabytes of raw data daily before any downsampled copies.
The standard pattern, accordingly, is a retention pyramid. You keep raw data for a short window — say one day — and let compaction rules feed progressively coarser series with longer retention. For instance, raw at one day, five-minute averages at thirty days, and one-hour rollups for a full year. The key edge case to watch is the DUPLICATE_POLICY: by default a duplicate timestamp raises an error, so for idempotent ingestion you should set ON_DUPLICATE LAST or create the series with an explicit policy. Otherwise a stream redelivery will fail your TS.ADD and push the message back into the pending list — turning a benign retry into an infinite loop.
# Idempotent ingestion: last write wins on duplicate timestamps
TS.CREATE ts:temp:sensor-42 RETENTION 86400000 DUPLICATE_POLICY LAST \
LABELS device sensor-42 metric temperature tier raw
# Retention pyramid: 5-min avg kept 30 days, hourly max kept 1 year
TS.CREATE ts:temp:sensor-42:avg_5m RETENTION 2592000000 LABELS tier rollup
TS.CREATE ts:temp:sensor-42:max_1h RETENTION 31536000000 LABELS tier rollup
TS.CREATERULE ts:temp:sensor-42 ts:temp:sensor-42:avg_5m AGGREGATION avg 300000
TS.CREATERULE ts:temp:sensor-42 ts:temp:sensor-42:max_1h AGGREGATION max 3600000
# Inspect memory and chunk stats for capacity planning
TS.INFO ts:temp:sensor-42Windowed Queries for Dashboards
# Real-time dashboard data
async def get_dashboard_data(device_id: str):
now = int(time.time() * 1000)
hour_ago = now - 3600000
# Last hour, 1-minute resolution
temps = await redis_client.execute_command(
"TS.RANGE", f"ts:temp:{device_id}",
hour_ago, now,
"AGGREGATION", "avg", 60000
)
# Multi-device comparison
all_temps = await redis_client.execute_command(
"TS.MRANGE", hour_ago, now,
"AGGREGATION", "avg", 300000,
"FILTER", "metric=temperature"
)
return {"device": temps, "all_devices": all_temps}
When dashboards query across many series, lean on labels and TS.MRANGE with a GROUPBY reducer rather than fanning out one TS.RANGE per device from the application. Server-side grouping keeps the round-trip count flat as device counts grow, which matters because the network round trip, not Redis itself, is usually the bottleneck for a live dashboard.
Streams Versus a Dedicated Broker: Honest Trade-offs
It helps to be precise about where Redis Streams sit relative to alternatives. Compared to Kafka, Streams trade durability and horizontal capacity for latency and simplicity. Kafka persists to disk across a partitioned, replicated log and comfortably handles terabytes of retained history; Redis Streams live in memory, so retention is bounded by RAM unless you aggressively trim with XADD ... MAXLEN or XTRIM. On the other hand, for a moderate-volume pipeline where consumers and producers share a Redis you already operate, the saved operational overhead is substantial — there is no separate ZooKeeper or KRaft quorum, no broker fleet, and no schema registry to run. For a deeper look at the engine itself, see the companion piece on Redis 8 vs the Valkey fork.
When NOT to Use Redis Streams
Redis Streams have real limitations compared to Kafka. Therefore, avoid them when you need: multi-datacenter replication with strict ordering guarantees, message retention beyond RAM capacity, schema evolution and exactly-once processing across systems, or sustained throughput well above a few hundred thousand messages per second on a single shard. Moreover, if your consumers are slow and bursty, an unbounded stream can quietly exhaust memory before you notice. As a result, Redis Streams are ideal for low-latency pipelines under moderate volume, while Kafka handles planetary-scale event streaming. If you are weighing a full event-driven design, the patterns in Event-Driven Architecture with Kafka map out the broker-centric alternative.
Key Takeaways
Redis 8 streams combined with time series provide a complete real-time data pipeline within a single system. Consumer groups enable horizontal scaling, while XAUTOCLAIM and a dead-letter stream keep that scaling reliable. Automatic aggregation rules and a retention pyramid handle downsampling and keep memory bounded. For use cases requiring sub-millisecond latency at moderate volume, Redis eliminates the complexity of running separate message brokers and time series databases — provided you respect the RAM ceiling and design for at-least-once, idempotent processing from day one.
Related Reading
- Redis 8 vs Valkey Fork Comparison
- Redis Caching with Spring Boot
- Event-Driven Architecture with Kafka
External Resources
In conclusion, Redis 8 streams time series is an essential topic for modern software development. By applying the patterns and practices covered in this guide — idempotent consumers, pending-list reclaim, a retention pyramid, and honest capacity planning — you can build more robust, scalable, and maintainable systems. Start with the fundamentals, iterate on your implementation, and continuously measure results to ensure you are getting the most value from these approaches.