What is Solid Queue?
By Abhishek Kanojia on February 16, 2024
Solid Queue is DB-based queuing backend for ActiveJob, designed with simplicity and performance in mind. Besides regular enqueuing and processing, Solid queue supports delayed jobs, concurrency control, priortising jobs, pausing and resuming queues, priorities per queue order, bulk processing of jobs.
#What Makes Solid Queue a Good Choice?
Why do we need Solid when we already have Resque, Sidekiq and Delayed Job in place? In general implementation of jobs, the workers are interdependent on each other in a sense that they can only pick jobs when the other worker instance releases the lock. This prevents the concurrent operation of jobs.
With the implementation of SELECT ... FOR UPDATE SKIP LOCKED
which allows a session to query the queue table, skip rows currently locked by other sessions, select the next unlocked row, and lock it for processing.
This was previously not available in MySQL, with the introduction of this clause in MySQL 8, Solid Queue becomes the first to implement this approach in its segment. This gives the concurrency that you would expect while working with similar system that utilizes Redis to do the same.
#Three things that make up the Solid Queue
-
Workers
Workers are in charge of running the jobs and picking them up fromsolid_queue_ready_executions
table created by solid queue gem when you install it and runbin/rails solid_queue:install:migrations
-
Dispatchers
Dispatchers are in charge of selecting jobs which are scheduled to run in future and making them available for workers, i.e moving jobs fromsolid_queue_scheduled_executions
table tosolid_queue_ready_executions
. Concurrency control is another responsiblity of Dispatchers. -
Supervisors
Supervisors forks workers and dispatchers according to the configuration provided inconfig/solid-queue.yml
. Example config:
# config/solid-queue.yml
production:
dispatchers:
- polling_interval: 1
batch_size: 500
concurrency_maintenance_interval: 300
workers:
- queues: "*"
threads: 3
polling_interval: 2
- queues: [ real_time, background ]
threads: 5
polling_interval: 0.1
processes: 3
# What makes it a great choice for Rails 8
The idea based on the discussion here is to make it hassle free and so that user does not have to configure it whatsoever. Following the Rails convention over configuration pattern.
In summary, Rails 8 brings performance enhancements, improved developer experience, modern technology integrations, and enhanced security measures, making it a significant milestone in the evolution of the Rails framework. As we await its official release, Rails developers can look forward to building even more robust and scalable web applications with ease.
💡 Do You Know?
Rails 8.0 adds security tool Brakeman to new apps as default