Getting started with Marconi, the message queue for OpenStack

OpenStack’s potential is continuously being tested in contexts that need support for bigger and complex applications, which often require a reliable, robust and efficient way to exchange data.

This can be done using local messaging infrastructures, but this approach adds an extra overhead due to unused capacity intended to meet peak demands, human resources required to maintain those structures and applications idle time waiting for the required data.

Is there a way to do this better? Of course there is!

Queue as a Service

Message queues provide an asynchronous communication protocol – the sender doesn’t need to interact with the queue at the same time that the receiver – originally intended for message passing within an operative system – interprocess communication – or within an application.

But this idea has been extended and other implementations allow the passing of messages between different computer systems, potentially connecting multiple applications and multiple operating systems.

Furthermore, new implementations provide this resources as a service. This is, Queue as a Service (QaaS).

Using this approach the access to messaging resources is simplified and the integration with different application is easier. Along this, costs are reduced, performance and reliability are enhanced, and scalability is improved.

How do message queues work?

There are a few simple facts that summarize the way that queues work – in its simpler implementation -.

  • A queue can be created anytime by the administrator.
  • Those interested in exchanging messages subscribe to the desired queue.
  • Messages can be sent, retrieved or deleted.
  • Messages placed onto the queue are stored until the recipient retrieves it or until a preconfigured time period ends.

This can be extended with extra functionality like batch management, claiming – get specific messages from the queue -, and health checks.

Marconi, the OpenStack Queuing and Notification Service

Marconi is a new OpenStack project that aims to provide applications running on OpenStack clouds an open Simple Queue Service (SQS) and Simple Notification Service (SNS).

It defines a clean HTTP-based RESTful API, uses a modular architecture and supports unified publish-subscribers and job-queuing semantics. Also, users will be able to customize Marconi to fit their needs.

Marconi can be used to distribute tasks among multiple workers (transactional job queues), forward events to data collectors (transactional event queues), publish events to any number of subscribers (publish-subscriber), send commands to one or more agents (point-to-point or publish subscribers) and request actions or get information from an agent (RPC).

A better and more complete summary of what Marconi is and which is its current state can be seen in the following videos.

Allan and Kurt quickly explain at the OpenStack Summit in Portland the basics about Marconi, its architecture, its operation modes, the project status and what is being planned for future implementations. Plus, they show us Marconi in action!

Flavio let us know what Marconi is, what it is not, its architecture and details about the API v1 during EuroPython 2013. Also, there are great questions answered during the QA. +∞!

Alejandro talks about why Marconi is needed, the difference with other services like Amazon SQS, Celery or RabbitMQ, the current code structure, and… some client features! Really cool.

Marconi test drive

Currently Marconi is under heavy development, but you can get a taste of it following this simple steps.

  1. Install MongoDB
  2. Clone the Marconi repo with git clone
  3. Go to your local copy of the Marconi repo cd marconi/
  4. Copy the configuration files to the folder ~/.marconi cp marconi/etc/*.conf-sample ~/.marconi/
  5. Rename those configuration files to *.conf
    mv logging.conf-sample logging.conf
    mv marconi.conf-sample marconi.conf
  6. Find the [drivers:storage:mongodb] section in ~/.marconi/marconi.conf and modify the URI to point to your local mongodb instance. It should look like this:
    #uri = mongodb://,
    uri = mongodb://localhost
    database = marconi
  7. (Optional) Create an isolated Python environment with virtualenv. Make sure you have it installed and then just run
    virtualenv marconi
    source marconi/bin/activate
  8. Run python develop to perform an installation using local development files
  9. Start Marconi server with marconi-server

You should get something like this…

Once you have the Marconi server running, you can start creating queues and sending messages to see it in action. This can be easily done by hand or using any REST client. I’m currently using a plugin for Chromium, Postman.

Contributing to Marconi

As in every OpenStack project, there is a lot to do and every extra hand is always welcome. You can join us in #openstack-marconi on and help making this project even better.

Currently there is much work devoted to the API and we are about to start coding the client.

Check the code at Github, and learn more about what is going on with Marconi at Launchpad.

Keep learning

Before finishing this post, I really want to say thanks to Malini (malini), Flavio (flaper87), Kurt (kgriffs), Alejandro (cppcabrera) and Allan (amets) for their guidance and for helping me to be part of this awesome project. o/.

More information

Check this great Python For Beginners article explaining what is a virtual environment and how to use them in How to use virtualenv in Python.
Here is some extra information about Marconi in the OpenStack’s wiki entry for Marconi.
flaper87 let us know Why MongoDB for Marconi?.
ladquin goes technical and teach us about RESTful APIs in Let’s get technical: RESTful APIs.

Comments (5)

  1. Pingback: OpenStack Community Weekly Newsletter (July 12 – 19) » The OpenStack Blog

Leave a Comment

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>