Distributed Job Processing with Celery

Recently I got a task to create a job processing application. The application will fetch the data from an API, parse the data, and create jobs for the data.

The job is to record a livestream using ffmpeg. There will be multiple jobs running simultaneously, so the idea is to create a distributed job processing application.

We have selected Celery as it is an asynchronous distributed task queue written in Python. Also we have used RabbitMQ as the message broker.

We have designed the infrastructure with the application running on multiple instances. One instance will be the Celery master where we execute the script to fetch data from API, parse the data, add jobs to the queue. Other instances will be running the Celery worker with the same RabbitMQ server as message broker.

app = Celery('tasks', backend='amqp://guest@', 

We are using a cron job in the Celery master to execute the script periodically. The Celery master and the Celery workers will be pointing to the same RabbitMQ server, so whenever a new job is added to the queue it will be assigned to a worker.

We are also using Flower: a celery monitoring tool in Celery master to check the status of the jobs and for other monitoring.


Front End App CORS issue in development environment

When you are developing a Front End Application or Single page application using Backbone.js, Angular.js or any other MV* frameworks, you might have faced the issue with CORS. You will face this issue if you are accessing the hosted web services (API) from your development environment because the domains are different.

We can solve the CORS issue in different ways. First method is JSONP. To implement JSONP you need the access to modify the web services. Because to implement the JSONP the web service has to return the json data wrapped in a function. You can do this only if you have the rights to change the web service.

If you don’t have the right to change the web services, the next method to solve CORS issue is to set the Header Access Control Allow Origin in the server. You can set the Origin to multiple domains to support access from multiple domains. For this you need access to server to modify the apache/nginx virtual host configuration. But what if you don’t have access to the server at all?

The next and the best solution is to implement reverse proxy in the development environment. Here is a sample virtual host file which has reverse proxy implemented:

<VirtualHost *:80> 
    ServerName front_end_app.abhilash-desktop.com 
    DocumentRoot /home/abhilash/Projects/front_end_app/public 
    <Directory /home/abhilash/Projects/front_end_app/public> 
       AllowOverride all    
       Options -MultiViews  
    LoadModule proxy_module modules/mod_proxy.so 
    ProxyRequests Off 
    <Proxy *> 
      Order deny,allow 
      Allow from all 
    ProxyPass /api http://abhidsm.com/api 
    ProxyPassReverse /api http://abhidsm.com/api 
ErrorLog /var/log/apache2/error.log 
# Possible values include: debug, info, notice, warn, error, crit, 
# alert, emerg. 
LogLevel warn 
CustomLog /var/log/apache2/access.log combined 

From your app if you start accessing the /api URL it will be accessed from the proxy URL. I think this is the easiest and best method to solve the CORS issue in development environment.


Hacknight @ Multunus

We had a very successful hacknight here at multunus yesterday (Jan 25, 2013). This was our 5th or 6th hacknight. And why it is very successful because we could complete all the 3 applications which we planned to do. Last time our team couldn’t finish the app completely.

We started by 10:30 PM, and there were 8 participants. We came up with ideas and selected 3 out of them, 2 team with 3 members and one team with 2 members. The ideas were:

  1. An obstacle detection app based on Arduino board [Akshay, Anup, Vimal]
  2. A Chrome extension for broadcasting URL [Abhilash, Sandeep, Manoj]
  3. Scrollee - an app which scrolls the content across the screens (Multiple monitors) [Midhun, Ernest]

So I form a pair with Sandeep to work on the BroadcastIt app. We came up with the tasks and spikes. We have selected node.js to implement the server part. We worked on the spikes and we were confident that it can be done. By 1:30 AM we have done with the basic app. That is the server part and client using the normal page. So we were able to show the demo of broadcasting URL to others. The next task was to integrate the client with the extension.

Then we faced the first problem, that is the extension is not able load the socket.io.js file. The chrome extension can load js files only from inside the package or from an external URL which is secured(https). We have tried to include the socket.io.js onside the extension package, that didn’t work. Then searched for socket.io cdn URLs. We found a cdn URL for socket.io, and that problem got fixed.

So almost done with the app. On click of the extension it will popup a small dialog with Broadcast button, on cllick of that will broadcast the current tab URL to all other browsers which installed this extension. When we have done with this it was around 2:30 AM. But then the next problem, the extension will work only when the popup is showing. When the popup is closed the extension can’t listen to the server event.

For chrome extension we can have background scripts and pages. We spend around 1 hr learning these and found a solution to fix this problem. We removed showing the popup and started running the script in background. So it will listen to the server in background. So now when we install the extension, it can broadcast the current tab URL to other browsers which are installed the extension. It was 4:30 AM.

Then we thought of extending the app by introducing channel, by adding options to the extension. The options page will ask the user to enter a channel, and the URL will broadcast only to those browsers who are selected the same channel. We were able to create the options page and saving the channel for a browser. But we couldn’t complete the broadcast only to the same channel browsers, because it was morning 6 Am and we had to create a final video of the demo.

So it was a very nice experience having successfully completing an app that we started with the idea, implementation and finally showing the demo. It was very fun too, we didn’t feel asleep at all. Once I get the demo videos link, I will add it here :) Here is the demo videos :)