Backgrounding high load tasks in Symfony2 using Beanstalk

You all have had this problem at one time or maybe you are having this problem now. You have a big ass cpu intensive task to do and you’re all doing it in a webserver request.

Of course this is insanity. Why would you want your webserver to spend 20 seconds on resizing all the 20 sizes of pictures you need, Or generating and emailing that pdf invoice after a customer has paid.

What if you could background this task and have it be processed on a place more suitable for boring and intensive tasks like the console. This saves on timeouts. There may be a solution!

A solution

Recently I open sourced a bundle for Symfony2 called Pheanstalk Task Queue Bundle  via a donation of code from Mijndomein.nl (for whom I work at the time of writing of this post).  It is a bundle that simplifies the hassle that is backgrounding tasks. It builds upon a great library for communicating with beanstalkd called pda/pheanstalk and It provides a complete package for backgrounding your tasks without having to implement this yourself. Just add,use and enjoy. It has a lot going for it :

  • A worker tender that takes care of spawning workers.
  • A worker that actually starts commands
  • A base class for workers to extend from.
  • A base class for tasks to extend from
  • A service to add tasks to the queue
  • Easy to offload your code. Move it to a command and have the queue take care of it.

How do I get this sweetness in my project?

Lets say you have an image resizing service that does all the resizing for your site’s content. And that service does not need to immediately return a resized version. (It could return a “i’m processing” type response ). This would be an ideal candidate for backgrounding.

Add the latest and greatest version to your composer.json by requiring and installing it. Then add the bundle to your appkernel. Make sure you have an up and running beanstalkd on your server. You will need to configure it in your symfony2 project by adding this to your config.yml

webdevvie_pheanstalk_task_queue:
    default_tube: testtube
    server: localhost

Then you create a console command for Symfony 2. You can extend this from the AbstractWorker class. You put the call to the actual resizing in there. (See the example classes under Command/Example). Please make sure that you limit the amount of data sent to this via the command line. Use ids to point the worker to its workload and don’t send user input as arguments or options for security reasons.

Following that create a class extended from the AbstractTaskDescription class. You need to fill in the $command and possibly the $commandArguments and $commandOptions params. Then you can send that object to the TaskQueueService using the public “queueTask” method on the place where your original resizing happened. This could be another service or a controller or even another command.  The queueTask method will return an identifier that you can use to request the status again later (handy for if you want to keep your user up to date but not block a webserver process)

On the backend you can start a task worker using:

app/console taskqueue:task-worker –verbose 

(verbose is to make sure you can see the worker’s output)

Now call the thing you were backgrounding again and see .. Your webserver is not spending time resizing anymore. (more time for actual visitors)

Example workers and backgrounded tasks

You can try out this straight out of the box after adding the bundle to your composer and app kernel and installing beanstalk on your server.

Open up two console screens .

In one start a single worker

app/console taskqueue:task-worker –verbose 

In the other :

app/console taskqueue:add-example-task 10

Your console with the worker will start working on the 10 very simple tasks added to the queue

Where can I find this?

Github or Packagist

Would you like to use it? Or are you using it. Please poke me on twitter @webdevvie or on google plus (The button on the top left)

Leave a Reply

Your email address will not be published. Required fields are marked *