Routing HTTP requests with Nginx+Mruby

NOTE:

This article was first published in 2016. While most of it is still of some relevance, since then many new options have become available that makes this easier, such as Caddy. However, we're leaving this article in place as Nginx+Mruby is still also a very flexible alternative, but beware that some details may need to be adjusted.

The Challenge

How to route A.com and B.com (and so on) based on host name to any one of a set of backend servers, and have full flexibility in programmatically adding and removing backends at runtme?

Introduction

An ongoing issue with containerisation is that so many applications are not written with stateless containers in mind. In fact, so many applications are not written with modern operations in mind at all.

We see this when it comes to e.g. routing inbound traffic by hostname. It's tremendously simple if every hostname is hosted on a single web server. If you can fit everything in a single Apache or Nginx instance, you're good to go. Load balancing between multiple identical instances is also "trivial".

But if you for operational reasons or capacity reasons wants to segregate services in separate containers, you suddenly face more challenges, as it is oddly a fairly new consideration for most of the available tools.

Haproxy is an excellent contender, but not all its configuration settings can asily be updated at runtime, and so people resort to all kinds of "tricks" involving musical chairs with haproxy instances. Some of them may fit your needs, but there's complexity involved.

The commercial version of Nginx - Nginx Plus now offers DNS SRV support, which is greatnews for their customers, as long as their model fits serving up SRV records, and as long as they're ok with the configuration options it offers.

Another alternative may be Hipache.

However the alternative we will look at in this article is Nginx + MRuby.

To be clear: This article presents a "work in progress". You can use it as a basis for setting up your own HTTP routing (or contract us to..), but it is not a finished "product". It does, however, give you a very flexible toolbox.

A (very) brief introduction to Nginx + MRuby

Matz (MATSUMOTO, Ryosuke; the creator of Ruby ) has also created MRuby. MRuby is a light-weight, embeddable version of Ruby.

He's also created a Nginx module for MRuby, that lets us embedd Ruby code in Nginx.

There are similar solutions for Lua, and so if you are familiar with Lua, you may want to check out OpenResty, which is a Nginx version with lots of Lua support baked in.

But my preferred language is Ruby, so I was intrigued to see Matz' release of ngx_mruby.

Putting together a trivial Ngix + Mruby HTTP router

Our requirements are extremely simple:

For a production quality system, we would want to look at handling retries, what to do about caching etc., but as a simple proof of concept, this will do.

First we need to pick a means of accessing the routing data. I first picked Redis as used by Hipache, but it struck me that I should simplify. Redis is an excellent piece of software, but it introduces an additional dependency for no good reason given the simple requirements above.

Ngx_mruby comes with Mruby-vedis support. Vedis is an in-process "database engine" offering various primitives similar to Redis. It (or its Ruby binding - I've not used the C-version) has some peculiar limitations but they are easily dealt with.

Combined this gives a simple building-block for a router, but leaves the specifics of how the routes are set to you. The important part is as a simple demo of how easy this makes it to make the routes 100% updateable at runtime. One could easily extend this to e.g. use Vedis to cache slower or more complicated mechanisms like Redis, DNS SRV or even Etcd or Consul.

You are also free to go the opposite way and push configuration changes into one or more Nginx+MRuby+Vedis containers. Personally I like this option (with the caveat if you somehow have too many hosts to keep the full mappings up-to-date on each frontend) because it means that your canonical data store can go down and the frontends can still keep serving traffic as long as their mappings don't get too stale.

The starting point

Matz has thoughtfully put together a simple Docker container with ONBUILD directives that will copy Ruby and Nginx configuration into a new container with very minimal effort. So lets start with writing a Dockerfile:

FROM matsumotory/ngx-mruby:latest
MAINTAINER [email protected]

VOLUME /config

# matsumotory/ngx-mruby image supports ONBUILD for below commands.
#
# ONBUILD ADD docker/hook /usr/local/nginx/hook
# ONBUILD ADD docker/conf /usr/local/nginx/conf
# ONBUILD ADD docker/conf/nginx.conf /usr/local/nginx/conf/nginx.conf

As you can see there's almost nothing to it. I add a volume to hold the Vedis config database, but that's it. I do this becaue I like to set my systems up to wipe the containers on stop/restart and only keep whatever data is explicitly put into volumes. It forces you to account for all state from the outset, rather than get surprised down the line...

Configuring Nginx

We'll provide a very bare-bones config file. First a warning: ngx_mruby is still rough around the edges, and so is the documentation and there's likely to be better/cleaner ways of doing things, so keep an eye on the docs and don't just blindly copy this test setup:

user daemon;
daemon off;
master_process off;
worker_processes 1;
error_log stderr notice;

events {
    worker_connections  1024;
}

http {
  server {
    listen 80;
    resolver 8.8.8.8;

    location / {
      mruby_set $backend /usr/local/nginx/hook/backend.rb;
      if ($backend = "404") {
             mruby_content_handler_code '
                  Nginx.echo "Not Found."
             ';
        }
        if ($backend != "404") {
             proxy_pass  http://$backend;
           }
}
    }
}

Consider most of this boilerplate suitable for the Docker container. The only bits worth paying attention to are the resolver line and the Mruby directives. The resolver line is there to specify what DNS settings Nginx should use. This could point to your own internal resolver, but in this case I've pointed it to one of Google's public DNS addresses.

Next, you can see we tell Nginx to assign the result of the bakend.rb script to $backend. Then we crudely compare $backend to 404 and use that as an indicator of whether or not the hostname was found. I'm sure there are cleaner ways of achieving the same. What is certain is that you should provide a better result than the mruby handler we call in that case, and return more than just "Not Found"... But it'll do for our test.

More importantly: When a backend is found, we tell Nginx to simply proxy the request to that backend.

We're free to modify the Nginx config pretty much as we see fit - it gets the address of a backend returns, but that is also all the Ruby script in question does for us.

Though you could add all kinds of other things, like e.g. logging, stats-gathering, rate controlling - whatever you want.

backend.rb

So here's a first stab at the backend:

r = Nginx::Request.new
v = Vedis.new "/config/backend.db"

# We're going to assume that you don't have
# enough backends for a single site for it
# to be worthwhile complicating this.
backends = v.exec("SMEMBERS site_#{r.hostname}")
v.close
if backends && !backends.empty?
  backends[rand(backends.size)]
else
  Nginx.log(Nginx::LOG_ERR,"No entry for #{r.hostname.inspect}")
  "404"
end

It really is that simple: We open a Vedis file, we retrieve a whole set of bbackends, and pick one. The beauty of having this in a simple Mruby file is of course that you can adapt it as you see fit. Let's say you want to include e.g. load data in order to send traffic to the least loaded servers - you can easily do that.

Packaging it up and configuration

Here's a simple Makefile:

NAME=nginx-mruby-vedis-virtualhosts
IMAGE=vidarh/${NAME}
VEDISCFG=`pwd`/config

build: Dockerfile
    docker build -t ${IMAGE} .

rundev: build
    -mkdir -p ${VEDISCFG}
    -docker kill ${NAME}
    -docker rm ${NAME}
    echo ${VEDISCFg}
    docker run --rm -p 8080:80 -v ${VEDISCFG}:/config --name ${NAME} ${IMAGE}

mirb:
    echo "NOTE: Requires running dev container"
    docker exec -t -i ${NAME} /usr/local/src/ngx_mruby/mruby/build/host/bin/mirb

"make rundev" will build and start a container attached to the current terminal.

"make mirb" will (assuming you've done "make rundev" in another terminal) throw you into a "mirb" session in the container. You can use this to update the Vedis database. E.g:

    $ make mirb
    echo "NOTE: Requires running dev container"
    NOTE: Requires running dev container 
    docker exec -t -i nginx-mruby-vedis-virtualhosts /usr/local/src/ngx_mruby/mruby/build/host/bin/mirb
    mirb - Embeddable Interactive Ruby Shell
    > v = Vedis.new("/config/backend.db")
    => #<Vedis:0x1c4a8c0>
    > v.exec("SADD site_foo.com a.example.com b.example.com")
    => nil
    > v.close
    => #<Vedis:0x1c4a8c0>
    >

You can similarly use docker exec like make mirb does to execute mruby and run scripts you add to the container (or put in the /config volume) to let you manage the contents of the Vedis database trivially. Or you can extend the Ruby handler to implement a REST API to change the settings (but beware security) - the choice is up to you.

The code in this post can be found on Github

Closing words

We currently host a number of sites behind nginx+mruby on an experimental basis, and it's worked very well. We initially used Redis, and one caveat worth mentioning is that at least as of when we tested it, the Mruby Redis driver had significant bugs that caused us problems, to the extent that we implemented a (very) basic Redis driver we could just inline in the handler script.

If you are looking for someone to set up and manage Nginx + MRuby - or other services - or have development requirements, contact us regarding our consulting services