Recently I got a task to create a job processing application. The application will fetch the data from an API, parse the data, and create jobs for the data.
The job is to record a livestream using ffmpeg. There will be multiple jobs running simultaneously, so the idea is to create a distributed job processing application.
We have selected Celery as it is an asynchronous distributed task queue written in Python. Also we have used RabbitMQ as the message broker.
We have designed the infrastructure with the application running on multiple instances. One instance will be the Celery master where we execute the script to fetch data from API, parse the data, add jobs to the queue. Other instances will be running the Celery worker with the same RabbitMQ server as message broker.
app = Celery('tasks', backend='amqp://firstname.lastname@example.org//',
We are using a cron job in the Celery master to execute the script periodically. The Celery master and the Celery workers will be pointing to the same RabbitMQ server, so whenever a new job is added to the queue it will be assigned to a worker.
We are also using Flower: a celery monitoring tool in Celery master to check the status of the jobs and for other monitoring.
When you are developing a Front End Application or Single page
application using Backbone.js, Angular.js or any other MV* frameworks,
you might have faced the issue with CORS. You will face this issue if
you are accessing the hosted web services (API) from your development
environment because the domains are different.
We can solve the CORS issue in different ways. First method is
JSONP. To implement JSONP you
need the access to modify the web services. Because to implement the
JSONP the web service has to return the json data wrapped in a
function. You can do this only if you have the rights to change the
If you don’t have the right to change the web services, the next
method to solve CORS issue is to set the Header Access Control Allow
Origin in the server. You
can set the Origin to multiple domains to support access from multiple
domains. For this you need access to server to modify the apache/nginx
virtual host configuration. But what if you don’t have access to the server at all?
The next and the best solution is to implement reverse
proxy in the
development environment. Here is a sample virtual host file which has
reverse proxy implemented:
LoadModule proxy_module modules/mod_proxy.so
Allow from all
ProxyPass /api http://abhidsm.com/api
ProxyPassReverse /api http://abhidsm.com/api
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
CustomLog /var/log/apache2/access.log combined
From your app if you start accessing the /api URL it will be accessed
from the proxy URL. I think this is the easiest and best method to
solve the CORS issue in development environment.
We had a very successful hacknight here at
multunus yesterday (Jan 25, 2013). This was
our 5th or 6th hacknight. And why it is very successful because we
could complete all the 3 applications which we planned to do. Last
time our team couldn’t finish the app completely.
We started by 10:30 PM, and there were 8 participants. We came up with
ideas and selected 3 out of them, 2 team with 3 members and one team
with 2 members. The ideas were:
- An obstacle detection app based on Arduino board [Akshay, Anup, Vimal]
- A Chrome extension for broadcasting URL [Abhilash, Sandeep, Manoj]
- Scrollee - an app which scrolls the content across the screens
(Multiple monitors) [Midhun, Ernest]
So I form a pair with Sandeep to work on the
BroadcastIt app. We came up
with the tasks and spikes. We have selected node.js to implement the
server part. We worked on the spikes and we were confident that it can be done. By 1:30 AM we have done with the basic
app. That is the server part and client using the normal page. So we
were able to show the demo of broadcasting URL to others. The next
task was to integrate the client with the extension.
Then we faced the first problem, that is the extension is not able
load the socket.io.js file. The chrome extension can load js files
only from inside the package or from an external URL which is
secured(https). We have tried to include the socket.io.js onside the
extension package, that didn’t work. Then searched for socket.io cdn
URLs. We found a cdn URL for socket.io, and that problem got fixed.
So almost done with the app. On click of the extension it will popup a small dialog with
Broadcast button, on cllick of that will broadcast the current tab URL
to all other browsers which installed this extension. When we have
done with this it was around 2:30 AM. But then the
next problem, the extension will work only when the
popup is showing. When the popup is closed the extension can’t listen
to the server event.
For chrome extension we can have background scripts and pages. We
spend around 1 hr learning these and found a solution to fix this
problem. We removed showing the popup and started running the script
in background. So it will listen to the server in background. So now
when we install the extension, it can broadcast the current tab URL to
other browsers which are installed the extension. It was 4:30 AM.
Then we thought of extending the app by introducing channel, by adding
options to the extension. The options page will ask the user to enter
a channel, and the URL will broadcast only to those browsers who are
selected the same channel. We were able to create the options page and
saving the channel for a browser. But we couldn’t complete the
broadcast only to the same channel browsers, because it was morning 6
Am and we had to create a final video of the demo.
So it was a very nice experience having successfully completing an app
that we started with the idea, implementation and finally showing the
demo. It was very fun too, we didn’t feel asleep at all.
Once I get
the demo videos link, I will add it here :) Here is the demo videos :)