Retail and point of sale
Real-time retail analytics, from till to warehouse
A serverless, event-driven ingestion path that moved a retail business from overnight batch reporting to real-time analytics.
Millions of files per pipeline
Context
A retail business was making operational decisions on data that was, at best, a day old. Store performance, stock movement and e-commerce activity all arrived through batch processes that ran overnight, which meant every intraday question had to wait until tomorrow.
The problem
The point-of-sale system emitted an enormous number of small CSV and JSON files, and exposed a REST API that was never designed as an analytics source. Polling it on a fixed schedule was both wasteful and slow. The volume also made a conventional always-on ingestion service expensive for a workload that is inherently bursty.
What we did
- 01Engineered a serverless ETL process in Python to extract from the POS REST API directly into the warehouse.
- 02Designed and deployed a containerised, event-driven ingestion service in Python, triggered by events from the POS system rather than by a schedule.
- 03Scaled that service to handle millions of CSV and JSON files without provisioning idle capacity.
- 04Set up and managed a managed message queue for real-time streaming, so critical events were available immediately rather than at the next batch window.
Outcome
- The business moved from overnight batch reporting to real-time analytics across stores and e-commerce.
- Ingestion cost scaled with actual activity instead of with provisioned capacity.
- Immediate data availability for the operations that needed to respond within the trading day.