Scaling celery infrastructure on AWS with python

If you use celery and struggle with scaling your aws infrastructure for your application queue on AWS, this talk will share the solution that works and scales the application based on the queue.

Celery is a distributed system that allows application to process tasks asyncrhronously and allows task scheduling as well. Along with AMQP brokers and result store, celery allows the task to be processed in real-time.

But if your message queues have variable amount of load and needs a scaling solution to help you scale the celery workers based on the load instead of keeping bigger instance running and save cost as well, this talk can be insightful to you.

In this talk, I will share the solution that works on AWS infrastructure with celery and reds to scale your workers for variable load. How we solve the scaling problem to keep our system process all task almost in real time irrespective of the number of tasks in the message queue and keep the cost of our infrastructure as low as possible using python.

Pre-requisites:

• Understanding about how distributed system works with messaging queue
• Understanding about Celery and redis
• Little bit knowledge on AWS (specifically ECS)

The talk will illustrate the working setup for Django environment with celery and Redis as broker.

The solution is the one that works for us and this talk intends to share the solution to help other developers trying to solve the similar problem. There may be better approach or other ways to solve this problem better and I hope to learn those as well by delivering this Brief about solution: * Custom solution to monitor redis queues (can be extended to other brokers such as Rabbitmq) and scale the celery workers (by creating new instances/containers) as per the load and reduce the container count when the load is below the threshold to keep the cost as minimum as possible.

Note: This talk can be considered as Proof-of-concept for the solution that can be adapted to the user needs. If there is more support to this solution, I hope to release the solution as an opensource library.