Category Archives: Uncategorized

Reflective monitoring

We’re using Consul for our cloud deployment. Consul makes it very easy and cheap to run health checks on every node, validating its own services. It does all the hard work of monitoring things that you used to rely on e.g. Nagios to do. The obvious drawback is that it runs locally, so you’re not validating that your service actually works across the network.

I spent a bit of time wondering how I could get node X to monitor node Y in a way that

  • fits neatly with Consul’s data model (service checks running on node A can’t easily report failures that are really on node B), and
  • doesn’t require a lot of orchestration when you have tens or hundreds of thousands of nodes that all need to keep an eye on themselves and (some subset of) each other.

I realised that I didn’t actually need another node to perform the monitoring. I just needed to have the network be a factor in the check. After learning that the MIRROR target in iptables went the way of the dodo over a decade ago, I hacked together a little script called reflector.

You simply install it on a host on your network, run e.g. “reflector –return-port 22 10022″ and any connection to port 10022 will be reflected back to the connecting node.

In other words, any node connecting to 10022 on the reflector node will actually connect to itself on port 22, except it will have traversed the network, thus ensuring that the service functions remotely.

It is availabe on pypi and Github.

168,000 instances in *how many hours*?

The ever awesome James Page posted an article on an experiment conducted by the Ubuntu Server team:

It’s a fine post and I’m happy that they did this experiment. However, I find myself shaking my head in disbelief at these findings. I don’t question their accuracy, I’m just disappointed with the results.

First of all, 4 years into a project whose original intent was a service provider focused platform for running public clouds, a 640 node cluster (1536 cores) shouldn’t be a rarity. If anything, it should be an unusually *small* OpenStack cloud. It’s thought provoking to me that they had to disable Neutron, tweak RabbitMQ settings, etc. for it to work at all.

Let’s look at the numbers, though. The 168,000 instances were spawned on a cluster that grew halfway through the experiment, so I’m going to ignore that particular experiment. I’m going to guess the numbers aren’t going to be prettier at that larger scale anyway.

So, apparently, they got 75,000 instances running on 374 compute nodes in 6 hours and 33 minutes (= 393 minutes). That’s an average of 191 instances launched per minute.

They got 100,000 instance running in 10 hours and 49 minutes (= 649 minutes). That’s an average of 154 instances launched per minute. That’s a rather significant drop from 191. From 6 hours and 33 minutes to 10 hours and 49 minutes is 4 hours, 16 minutes = 256 minutes. Those last 25,000 instances were launched at an average rate of 98 per minute. That’s almost half the launch rate of the first 75,000. Wow.

Looking at it another way, 374 nodes each with 4 cores gives us a total of 1496 cores. 649 minutes, 1496 cores = 970,904 core minutes. With 100,000 instances launched, that’s an average of 9.7 core minutes per instance launch.

9.7 minutes. That’s embarassing. 30 seconds would be acceptable, but something like 10 seconds should be perfectly possible measured from when the launch request is sent until its state is RUNNING, and then another 20 seconds to finish the boot sequence and start listening for SSH connections.

Depressing meeting calculations

I just did some really rather depressing calculations on meeting time.

10 people, 7 projects, weekly meeting of one hour.

Let’s pretend that 25 minutes are spent on general announcements that are genuinely useful to everyone.

The remaining 35 minutes are spent on the 7 different projects. That’s 5 minutes each. The project you’re on is obviously important to you, so that’s 5 minutes more of useful stuff.

The remaining 30 minutes are spent on 6 projects that you’re not working on. Sure, it may be interesting, but on average proably not very useful. Let’s be generous and say one minute of each of the other projects’s time is useful to you. That gives us 36 minutes (or 60%) of useful time. That’s 40% of the hour that is wasted.

Multiplied by 10 people, that’s 4 hours that’ll never come back.

Ok, let’s say the team grows: Five more people, two more projects and half an hour.

We keep the 25 minutes of general announcements.

Then there’ll be some introductory stuff. Let’s say 11 minutes. This is useful to the 5 new people and not at all to the 10 old people.

So now we have 54 minutes left to be divided across 9 projects. That’s 6 minutes each. I.e. 6 useful minutes from your own project, 8*1 useful minutes for other people’s projects and 8*5 useless minutes from other people’s projects.

Useful time:
10 old people * 39 minutes of useful time = 6:30 (43%)
5 new people * 50 minutes of useful time = 4:10 (56%)

That’s a total of 10:40 (10 hours, 40 minutes) of useful time, but 22 and a half hours spent. That’s translates into an efficiency of 48% and it’ll only get worse as the team grows, the project list grows and the meeting gets longer.

Why do we keep doing this?

Openstack Nova in Maverick

Ubuntu Maverick was released yesterday. Big congrats to the Ubuntu team for another release well out the door.

As you may know, both Openstack storage (Swift) and compute (Nova) are available in the Ubuntu repositories. We haven’t made a proper release of Nova yet, so that’s a development snapshot, but it’s in reasonably good shape. Swift, on the other hand, should be in very good shape and be production ready. I’ve worked mostly on Nova, so that’s what I’ll focus on.

So, to get to play with Nova in Maverick on a single machine, here are the instructions:

sudo apt-get install rabbitmq-server redis-server
sudo apt-get install nova-api nova-objectstore nova-compute \
                nova-scheduler nova-network euca2ools unzip

rabbitmq-server and redis-server are not stated as dependencies of Nova in the packages, because they don’t need to live on the same host. In fact, as soon as you add the next compute node (or API node or whatever), you’ll want to use a remote rabbitmq server and a remote database, too. But, for our small experiment here, we need a rabbitmq server and a redis server (it’s very likely that the final release of Nova will not require Redis, but for now, we need it).

A quick explanation of the different components:

is a messaging system the implements AMQP.  Basically, it’s a server that passes messages around between the other components that make up Nova.
is the API server (I was schocked to learn this, too!) . It implements a subset of the Amazon EC2. We’re working on adding the rest, but it takes time. It also implements a subset of the Rackspace API.
stores objects. It implements the S3 API. It’s quite crude. If you’re serious about storing objects, Swift is what you want. Really.
the component that runs virtual machines.
the network worker. Depending on configuration, it may just assign IP’s or it could work as the gateway for a bunch of NAT’ed VM’s.
the scheduler (another schocker). When a user wants to run a virtual machine, they send a request to the API server. The API server asks the network worker for an IP and then passes off handling to the scheduler. The scheduler decides which host gets to run the VM.

Once it’s done installing (which should be a breeze), you can create an admin user (I name mine “soren” for obvious reasons):

sudo nova-manage user admin soren

and create a project (also named soren) with the above user as the project admin:

sudo nova-manage project create soren soren

Now, you’ll want to get a hold of your credentials:

sudo nova-manage project zipfile soren soren

This yields a in the current working directory. Unzip it..


and source the rc file:

. novarc

And now you’re ready to go!

Let’s just repeat all that in one go, shall we?

sudo apt-get install rabbitmq-server redis-server
sudo apt-get install nova-api nova-objectstore nova-compute \
                nova-scheduler nova-network euca2ools unzip
sudo nova-manage user admin soren
sudo nova-manage project create soren soren
sudo nova-manage project zipfile soren soren
. novarc

That’s pretty much it. Now your cloud is up and running, you’ve created an admin user and retrieved the corresponding credentials and put them in your environment.
This is not much fun without any VM’s to run, so you need to add some images. We have some small images we use for testing that you can download here:


Extract that file:

tar xvzf images.tgz

This gives you a directory tree like this:

|-- aki-lucid
|   |-- image
|   `-- info.json
|-- ami-tiny
|   |-- image
|   `-- info.json
`-- ari-lucid
    |-- image
    `-- info.json

As a shortcut, you could just extract this directly in /var/lib/nova and change the permisssions appropriately, but to get the full experience, we’ll use euca-* to get these images uploaded.

euca-bundle-image -i images/aki-lucid/image -p kernel --kernel true
euca-bundle-image -i images/ari-lucid/image -p ramdisk --ramdisk true
euca-upload-bundle -m /tmp/kernel.manifest.xml -b mybucket
euca-upload-bundle -m /tmp/ramdisk.manifest.xml -b mybucket
out=$(euca-register mybucket/kernel.manifest.xml)
[ $? -eq 0 ] && kernel=$(echo $out | awk -- '{ print $2 }') || echo $out

out=$(euca-register mybucket/ramdisk.manifest.xml)
[ $? -eq 0 ] && ramdisk=$(echo $out | awk -- '{ print $2 }') || echo $out

euca-bundle-image -i images/ami-tiny/image -p machine  --kernel $kernel --ramdisk $ramdisk
euca-upload-bundle -m /tmp/machine.manifest.xml -b mybucket
out=$(euca-register mybucket/machine.manifest.xml)
[ $? -eq 0 ] && machine=$(echo $out | awk -- '{ print $2 }') || echo $out
echo kernel: $kernel, ramdisk: $ramdisk, machine: $machine

Alright, so we have images!

Now, we just need a keypair:

euca-add-keypair mykey > mykey.priv
chmod 600 mykey.priv

Let’s run a VM!

euca-run-instances $machine --kernel $kernel --ramdisk $ramdisk -k mykey

This should respond with some info about the VM, among other things, the IP.

In my case, it was

ssh -i mykey.priv root@


I’ll leave it to someone else to provide similar instructions for Swift