Queue – Deferred execution in CakePHP

At the beginning of 2.x I ported the Queue plugin from 1.x to 2.0 and started to integrate it into a few apps of mine.
Since then it was running OK, but neither with fixed tests, nor with any real documentation.
I fixed all that up – the plugin is available at github.

Background

Since this whole topic isn’t really covered by the book at all, we might start at the very beginning.

What is deferred execution – and why do we need it?

Most of us have at least a few typical examples in our applications where this would be either handy or a really good idea.
Let’s say your website is sending notification emails after a comment has been posted to one of your blog posts. You are probably doing this as a afterSave()
callback and trigger maybe 4-9 emails to you (the author) and maybe some previous commenters who wish to be notified on updates, as well.
Now, with basic CakePHP coding, you would probably also need a sleep(1) throttling timeout between the email sending processes to avoid
hitting a flood timeout. All in all you make the commenting user wait at least a few seconds before the page will refresh and he will finally see that his comment went though.
Imagine the email sending would need a little bit longer due to network issues. And the user hits F5 several times as nothing happens for him.

An annoying situation that could easily be avoided if unrelated tasks like notifications are not directly processed by the same thread (in this case the browser request of this commenting user). We delegate this task to another thread in the background which then processes this and many other tasks in the order they come in. This is called deferred execution. In our example above the main thread just puts those 9 emails to be sent out into the "queue" and immediately responds to the user within less than a second.
This can be combined with a priority flag to make tasks around email sending being processed before cleanup jobs for example.

There are many other scenarios where you should use a queue to handle background-tasks. Examples are:

• Image processing/resizing
• PDF (or other larger/demanding file) generation/extraction
• Notification or larger update jobs
• (Regular) cleanup or data monitoring

The Queue plugin is a good way to get to know background-processing and how to reduce response-time for heavy-processing tasks.
It also doesn’t have any other dependencies – except for basic CLI access.

Setup and configuration

As always, we need to load the plugin using CakePlugin::load('Queue') or CakePlugin::loadAll().

You can set any configuration values using your APP config file:

$config['Queue'] = array(
    ...
);

The available options are explained in the README file.

The most basic setup – for local testing for example – would be to use the default "workermaxruntime" with 0 (running forever) and manually
starting a single worker cake Queue.Queue runworker. But once a worker dies in case of an error or server restart it won’t restart without another manual
trigger. Bear that in mind. I recommend the following approach using scheduled restarts for productive systems.

If you plan on having multiple workers or a fail-safe restart every few minutes you need a cronjob scheduler.
For linux the tool "crontab" is usually already available and used for other cronjob activities in CakePHP anyway.
With this you can simply create a new worker every x seconds – note that the lifetime of a worker should also be around that time.
Open the crontab editor for the www-data user (assuming you are using apache): crontab -u www-data -e

*/30 * * * * cd /srv/path/to/app && ../lib/Cake/Console/cake Queue.Queue runworker

If you have a Console folder inside your app, you can simplify the cake command ../lib/Cake/Console/cake to ./Console/cake.

Example usage

As explained above you should have always at least one worker up and running – either via crontab every x seconds or manually started.

Then you can start adding tasks to the queue. To play around you can use the command line. This might also be useful for some tasks you only
want to trigger once in a while – manually. cake Queue.Queue will list you all available queue tasks.

cake Queue.Queue add Example

This will add the Example task to the queue and should be executed and finished immediately.

If you want to see how the queue processes multiple tasks in order, try the LongExample task and while it is running add some Example tasks.
You can also play with more than a single worker at once.

Real usage

Enough of playing around, let’s get down to business.

Image generation

Let’s say we have an upload form and after save we want to trigger the rendering in the afterSave() callback of the model.

I created a task QueueImageTask inside my APP/Console/Command/Task folder. This way it will be recognized by the Queue plugin as "Image".
Then I could schedule this rendering task after each new upload using the "QueuedTask" model:

// In the afterSave() callback of the model, for example
$QueuedTask = ClassRegistry::init('Queue.QueuedTask');
$QueuedTask->createJob('Image', array('id' => $id));

The queue when idle will run in the background and check for scheduled tasks. It will see the new Image task and trigger its execution.
So when the worker processes it we can retrieve the record via the passed id and do all the magic we need to.
We can also can put a callback at the end of the task method before returning true. This way we can for example then update the status of the image from "inactive (not rendered)" to "active" (fully rendered and ready to be used).
This is an example of a two way communication task, reporting back to the app on success.
If it is a long running task (minutes to hours), one could also leverage the "progress" feature of the plugin.

Sending emails

For some projects I just send out SMTP mails without the queue. But if I want to switch, I simply have to modify the email settings.
Using my EmailLib, I can just switch transport from Smtp to Queue.Queue with 4 lines of adjustment:

public $default = array(
    ...
    'transport' => 'Queue.Queue',
    'logTrace' => false, // Detailed trace log
    'logReport' => true, // Report log entry
    'queue' => array('transport' => 'Smtp', 'logTrace' => true), // Overwrite for inside the queue

This is the code to create a new email:

App::uses('EmailLib', 'Tools.Lib');

$Email = new EmailLib();
$Email->to(...);
$Email->subject('Testing Message');
$Email->domain(...); // If generated in a shell/task
$result = $Email->send('Foo');

This would now send the email directly into the queue – where it then will be processed and actually send through a worker.
With logReport => true we can see a log entry for passing the task on to the queue.

Using logTrace => true we get a detailed trace report in the log file after the worker processed the task. It would not make sense for the initial creation (as there is no trace from the SmtpTransport yet).

Emails are usually just sent out. They are one-way communication.

Debugging

With background tasks debugging is a little bit more difficult than with synchronous execution.
That’s why, by default, the Queue plugin will store the last execution time of all currently running workers in the TMP/queue/ folder (last modified date).
This gives you a basic idea of how many workers currently are running and how they can be identified.

You can, accessing the CLI, also identify and display all currently running workers using top. They have the command "php" in the last column.
If you have to kill all at once (especially when you are experimenting with lifetime 0), use killall php then.

Sugar

As noted above, I do have a little admin backend of its own for a quick look on how many workers are currently running and a glance at the stats/settings.
Just navigate to /admin/queue and it should display all of that. Note that it might need the Tools plugin as dependency here, though.

Vista

The above approach is a "CakePHP only" minimalistic tool for small and medium sized applications.
If you need a more robost approach, you can also look at complete tools like php-resque, djjob, celery or gearman which also usually have a CakePHP plugin these days.
So I will most likely look at Gearman and CO in the near future. For resque there is a nice integration article available.

Additionally, this article is more about raising awareness of background jobs in every day apps than actually proclaiming the one perfect solution for this.
But with more and more in the community actually going down that path, the existing tools around queuing in CakePHP sure will become more sophisticated. Check the awesome list for Queue plugins.