At work earlier today I ran across an issue where one of our application queues got backed up and it got me to thinking about how queues are organized in general. The TLDR answer: use urgency and intensity.
// Share.on([hacker news,
linkedin,
twitter,
facebook,
reddit])
If you've ever had to scale an application with a lot of traffic, you'll find that one of your first tasks is reducing web requests to only handle exactly what the user needs to get a response while offloading everything else to background workers via some type of queueing system.
There are many, options out there for such systems but the common bit of functionality that matters for sake of this article is naming the queues. Inevitably, as you start putting more and more things in your queue you start naming them instead of putting them all in the same pipe so that you can handle them differently.
Sometimes, your workers will all listen to each queue with a level of priority assigned to each. Other times you'll have some workers that ONLY listen to specific queues that need dedicated resources allocated to them.
This is easy to identify when you know everything that's being backgrounded. Unfortunately, when your application is being iterated this tends to be something that just sort of evolves over time so with that in mind, here's a simple way to name your background queues from the very beginning so that you'll be properly ready to deal with resource constraints later.
Urgency is basically just another way of saying priority, but in reality this should mean "user urgency". Will users notice and care if this takes a little while to run? Most things that you can put in the background at all are not urgent, otherwise you would have handled it immediately. Sending an email is a good example. If the user doesn't get the email within a few minutes it's not a big deal. If they don't get it after a few hours, it's a problem.
Intensity is a measure of resource usage but NOT speed. A background job that takes a long time to complete because of contacting 3rd party systems is slow but it's effect on your server is negligible. A background job that runs reporting queries on your database, could lock tables, or involves a lot of processing like video...that is intense. The distinction is important because your application can handle thousands of slow but not intensive jobs running at the same time. It might only be able to handle one intensive job at a time.
Clearly, every application is different and the bigger things get, the more specific queues will become in their focus. Here's the structure that I tend to use for organizing queues that handles growth pretty smoothly.
// Share.on([hacker news,
linkedin,
twitter,
facebook,
reddit])