Event Streaming
Definition
Real-time processing of data events as they happen. Like a live news feed that processes and responds to events as they occur.
Use Cases
- Netflix: Real-time personalization and operational monitoring based on user playback events — Netflix publishes high-volume user and system events to an internal event streaming platform (commonly described publicly as Kafka-based) and processes them with stream processing to drive near-real-time recommendations, experimentation, and monitoring. (Faster feedback loops for personalization and A/B tests, improved reliability through real-time anomaly detection, and quicker operational response to issues.)
- LinkedIn: Activity feed updates and real-time data pipelines across many internal systems — LinkedIn created and open-sourced Apache Kafka to move event data between services and analytics systems, using topics and consumer groups to distribute events to multiple downstream applications. (Scalable, decoupled data distribution that supports multiple consumers (feeds, analytics, monitoring) with low latency and high throughput.)
- Uber: Real-time trip, driver, and marketplace event processing for dispatch and monitoring — Uber has described using Kafka as a central event bus to stream events from microservices into real-time processing and storage systems for operational visibility and near-real-time decisioning. (Improved real-time observability and faster processing of marketplace signals, supporting time-sensitive operational workflows.)
Provider Equivalents
- AWS: Amazon Kinesis Data Streams
- Azure: Azure Event Hubs
- GCP: Pub/Sub
- OCI: OCI Streaming
Frequently Asked Questions
- What's the difference between event streaming and message queues?
- Message queues are usually for point-to-point work distribution: a message is processed and then removed. Event streaming is for continuous event data where many consumers may read the same events, often with ordering per partition and the ability to replay events for a period of time (useful for analytics, auditing, and rebuilding state).
- When should I use event streaming?
- Use it when you need near-real-time reactions to data (seconds or less), multiple downstream consumers (analytics, alerts, personalization), or the ability to replay events. Common cases include clickstream analytics, fraud detection, IoT telemetry, real-time dashboards, and microservice event-driven architectures.
- How much does event streaming cost?
- Costs typically depend on throughput (ingress/egress), retention duration, number of partitions/shards, and any processing you add (stream processing, connectors, storage). Managed services often charge for capacity (e.g., shards/partitions or throughput units), data volume, and retention; networking and cross-region replication can add significant cost.
Category: data
Difficulty: advanced
Related Terms
See Also