Piping Rain in Python - Adventures in a minute latency Radar Pipeline
| Start time | 10:25 |
|---|---|
| End time | 10:50 |
| Countdown link | Open timer |
This talk covers the devops for a data pipeline channeling images from a self-hosted Passive Rain Radar in Nairobi to WebApps, Mobile apps and bots in Twitter and WhatsApp. The idea was put together in a restaurant and hastily deployed by 2 engineers. It continues run (and to fail in weird ways) 6 months in. Shoestrings hold together a service with 1000's of users with around 70% uptime.
In Australia we are used to live radar images provided by Bureau on Meterology (BoM). This agency provides forecasting and myriads of other services and has around 500 million in Annual budget. This is not the case in developing countries like Kenya where the last operational weather radar instrument was seen some 30 years ago.
Frustrated with the lack of live radar while in Nairobi a couple of engineers put together a passive radar system with an SDR and an Intel NUC. The data from this system travels via S3 and DynamoDB from screen capture tools on the endpoint device. Eventually gets exposed via API's to web and mobile apps running on some EC2 boxes ( no fancy kubernetes cluster to see here) . There are also a couple of Cron Jobs acting as Twitter and WhatsApp bots for followers. All this scratching your own itch infrastructure has slowly improved in uptime to 70% and supports around 5000 regular users. The entire data pipeline and web applications are written as set of small Python Services using Flask, OpenCV and ImageMagick.
This talk will describe how the pipeline was put together incrementally with intermediate outputs and rapidly deployed using a basic git pull based workflow. Along the way we will enumerate the numerous gotchas including the excessive S3 List usage, power failures on sensor site and dogpiling Twitter bots. The budget for the operations remains 100's of dollars per month.
Tish is an expert at building software for large-scale data processing particularly satellite and airborne sensor data. He has been awarded innovation and commercialization award for building high resolution 3D city capture product and has worked for various space agencies including NASA, scientific research organizations and in commercial research. Tisham originally grew up in Kenya and left for his studies, eventually returning to work on a logistics startup and other projects. Tish is a regular speaker at open-source technology conferences addressing the intersection of software design and hardware design. Currently he wrangles containers and Kubernetes clusters for Digital Earth Australia and Africa making analysis ready satellite imagery accessible to everyone.