Bringing Background Jobs to the Foreground: An Exposé

Sidekiq and ActiveJob are great ways to get up and running with a robust background job runner in rails - but they’re easily abused! Come find out about the pitfalls of background processing and how to avoid them in this deep dive into background workers and how we deal with them at Gitlab.

Background job processing is an important part of any Rails app. At GitLab it allows us to run important slow tasks like sending emails and handling uploads without impacting users’ browsing experience - thus reducing the ever-important page load times!

At GitLab we use both Sidekiq and Activejob for background processing with extensive applications for everything from generating notifications to dealing with git pushes from millions of users. This scale has allowed us to learn where the pitfalls are and we’d love to share what we’ve learned with you.

This talk will explore:

• How to do background job processing the right way using both ActiveJob and Sidekiq.
• What it truly means to have an idempotent worker, why workers should be fast and small, and how to deal with workers running at least once.
• What horizontal scaling is and why you want it
• Tips and tricks to ensure background worker infrastructure is easy to maintain, even for self-hosted users and applications.

We’ll also talk about differences between ActiveJob and using Sidekiq directly which can be a critical decision point for many organizations as you’re making a trade-off between portability and performance.

Attendees can expect to leave the talk with a strong basis to make the best decisions for background jobs in their Ruby app at work. We will also be available for any questions afterwards via Twitter and email.

I’ve had tons of chances to work with Sidekiq during my tenure at GitLab, which has hundreds of thousands of background workers running every day to deal with things like launching CI jobs, generating notifications and emails, dealing with incoming git pushes, project imports, etc.

During this time I’ve learned about the many pitfalls you can fall in if you’re not careful! I’ve also soaked in best practices crowdsourced from the entire company and learned not only about the development side, but also the management and maintenance required from a background job cluster.

To add a hard-mode to all this, we have to make sure our background worker infrastructure is easy to maintain for our self-hosted users who run our software on their own infrastructure. We’ve had to make very certain that our approach is not only sensible, but easily understood. I hope to share the same knowledge with the Ruby conference attendees and start an interesting dialogue.