Assignment 18: Real-Time Data Pipeline

01 Assignment Overview

This assignment focuses on building a resilient data ingestion pipeline capable of handling real-time IoT sensor streams. The core objective was to demonstrate proficiency in distributed systems, message queuing, and stateful processing while maintaining strict SLA requirements for latency and data integrity.

Key deliverables included a production-ready ingestion service, a processing layer with exactly-once semantics, and a comprehensive observability stack integrated via OpenTelemetry.

02 Requirements

• Ingest up to 50,000 events/sec with p99 latency < 200ms
• Implement idempotent processing to prevent duplicate aggregation
• Ensure crash recovery with at-least-once delivery guarantees
• Provide structured logging, metrics, and distributed tracing
• Document deployment configuration and scaling strategies

All requirements were validated through load testing using k6 and synthetic event generators mimicking real-world network jitter and partial failures.

03 Technical Implementation

The pipeline leverages Rust for the ingestion layer to maximize throughput and memory safety, while Python handles the complex transformation logic due to its rich data processing ecosystem.

// Ingestion worker handling async event streams
pub async fn process_batch(&mut self, batch: Vec<Event>) -> Result<(), Error> {
    let start = Instant::now();
    for event in batch {
        if self.validate_checksum(&event).is_err() {
            metrics!.increment("dropped_invalid");
            continue;
        }
        self.wal.append(&event).await?;
        self.publisher.send(event).await?;
    }
    let elapsed = start.elapsed();
    metrics!.histogram("batch_processing_ms", elapsed.as_millis() as f64);
    Ok(())
}

Write-ahead logging (WAL) ensures durability before events are forwarded to the processing queue. The publisher uses a backpressure-aware channel to prevent memory exhaustion during downstream slowdowns.

04 System Architecture

The architecture follows a modular pipeline pattern:

• Ingestion Edge: TLS-secured HTTP/2 endpoints with connection pooling and automatic retry logic.
• Message Broker: Kafka cluster with exactly-once semantics enabled, partitioned by device ID for ordered processing.
• Processing Workers: Horizontally scalable consumers running windowed aggregations (5s tumbling windows).
• Storage: TimescaleDB for time-series data, Redis for hot cache, S3 for cold archival.

All components communicate via typed protobuf schemas to ensure forward/backward compatibility across rolling deployments.

05 Testing & Validation

Quality assurance followed a three-tier strategy:

• Unit Tests: 92% coverage focusing on edge cases in checksum validation and serialization.
• Integration Tests: Docker Compose spin-ups of Kafka and DB to verify end-to-end flow.
• Load Tests: k6 scripts simulating 3x peak traffic with 10% packet loss to validate resilience.

Results showed stable p99 latency of 142ms under load, with graceful degradation rather than cascade failure when workers scaled down.

06 Reflection & Lessons

The most significant challenge was tuning the backpressure thresholds to balance throughput with memory safety. Initial benchmarks showed GC pauses in the Python layer during high-volume bursts, which we resolved by switching to streaming parquet chunks instead of in-memory DataFrames.

This assignment reinforced the importance of observability by design. Instrumenting the pipeline early allowed us to pinpoint a hot partition issue that would have taken days to diagnose in production. The experience also highlighted the value of chaos engineering in validating theoretical guarantees against real-world network conditions.

07 Submission Checklist

Submission locked. Grade and rubric feedback available in the portal.