What I now know about HAproxied Django DB connections, and wish I'd known sooner
| Start time | 13:55 |
|---|---|
| End time | 14:20 |
| Countdown link | Open timer |
Asynchronous Celery tasks that manipulate a MySQL/Galera database from a Django application can produce very interesting behavior when HAProxy is involved. This talk discusses these issues, and proposes remedies available to both application developers and infrastructure operators.
When you’re running a Django application, the following things are all pretty commonplace:
- You use MySQL or MariaDB as your Django database backend.
- You don’t run a single standalone MySQL/MariaDB instance, but a Galera cluster.
- You run asynchronous tasks in Celery.
Now suppose your application doesn’t talk to your Galera cluster directly, but via HAProxy. That’s not exactly unheard of; in fact it’s an officially documented HA option for Galera. And, to complicate things further, this may be a feature of your setup that you, the application developer, don't necessarily have control over.
In such a scenario, you may be dealing with very "interesting" features of HAproxy which — if you are unfamiliar with them — can throw you very nasty curveballs.
Having been in the position where, together with my team, I was indeed unfamiliar with some of HAProxy's/Galera's intricacies and we were thus on he receiving end of those curveballs, I am taking this opportunity to share our findings so that our pain is someone else's gain. You have multiple options at your disposal — some in your infrastructure setup, and some in your Django application code. This talk covers both those angles.
I run the Education business unit at City Network, and help people learn to use, understand, and deploy OpenStack. Together with my Education team I run academy.citycloud.com, a learning platform based on the open source Open edX software stack. I've worked exclusively with open source software since about 2002. I ran hastexo, a small professional services company specializing in OpenStack, Ceph, and Open edX, until our acquisition by City Network in October 2017. I have presented at several OpenStack/OpenInfra Summits, LinuxCnns, linux.conf.au's, OSCONs, and numerous other conferences and meetups