MCollective using a vhost other than / with RabbitMQ

We'd recently began using MCollective along with Puppet in our setup. MCollective is a framework to build server orchestration or parallel job execution systems. We had MCollective running with ActiveMQ as MOM for management of our EC2 infrastructure, however for our setup at Softlayer we decided to go ahead and make use of RabbitMQ as our MOM.

Thanks to R.I. Pienaar we were quickly able to get solution to some of our issues and very recently one of our developers suggested that we should be making use of a separate vhost for use of mcollective. Seems like a great idea, we all thought ! However after discussions on mcollective user group we found that stomp connector provided with mcollective, by default doesn't allow for passing of host header information to RabbitMQ.

R.I. Pienaar once again came as a saviour and suggested some changes to our stomp connector which now allow us to make use of /mcollective as a vhost while communicating with RabbitMQ.

So here is a change we made to /usr/libexec/mcollective/mcollective/connector/stomp.rb (This location can change with distribution).


@connection = connector.new(user, password, host, port, true, 5, {"accept-version" => '1.0,1.1', "host" => "/mcollective\"})

User-Agent based redirect in haproxy

We've been using haproxy as our loadbalancer for quite sometime now  and I'd say it's been more than wonderful. We've been using it for serving around 2 million page-views per day and it's been pretty good at handling that kind of HTTP traffic. Recently we decided to launch our mobile site and the idea was to allow users to visit our website through normal URL that we use for our website, however upon detection of user-agent we wanted to rewrite the URL to something else. We knew that haproxy was able to parse and identify HTTP Headers and hence we were pretty positive about the idea of using haproxy to do the redirection, however to our surprise, haproxy 1.3 doesn't allow for conditional rewrites and the worst aspect of it is that it doesn't even complain about implementing conditional rewrites, instead it silently ignores the condition and implements the rewrite to every request.

 So we finally decided to migrate to haproxy 1.4 and we use following configuration to setup conditional redirect:

1) Create new ACL to identifiy user-agent:

acl mobile_user_agent hdr_sub(User-Agent) -i iphone

2) We didn't want to redirect any requests for .css, .js, .bmp, .jpg, .png, .jpeg, .gif and .ico so created a new ACL

acl is_static_file url_reg .*\.(css|js)\?[0-9.]+

acl is_static_img url_reg .*\.(png|bmp|jpg|jpeg|gif|ico)\?[0-9a-z.]+

acl is_static url_reg .*\.(css|js|png|bmp|jpg|jpeg|gif|ico)

3) If request was for /mobile/* we didn't want to redirect so yet another ACL :)

acl is_mobile url_reg ^\/mobile.*

4) Do a URL rewrite as follows:

reqrep ^([^\ ]*)\ /(.*)   \1\ /mobile/\2 if iphone !is_static_file !is_static_img !is_static !is_mobile

So now any request coming from a mobile device for any thing other than .css, .js, .png, .gif, .bmp, .jpg, .jpeg, ico will be redirected to /mobile/.

So a request like www.abc.com/pqr will become www.abc.com/mobile/pqr !

Access mongodb collection through shell script

We've been using mongodb for quite some time for various purpose, like newsfeed, analytics etc. Recently a need was found to setup a monitoring for some queries to mongodb and unlike MySQL I was not able to find any means to query the database collection. However a quick look at help of mongo yielded a nice solution. We can put series of statements that we execute at mongo shell inside a .js file and finally execute :

mongo .js

Regex handling is indeed black magic

Mathematicians Vs. Computer Scientists

I always had this confusion in my mind about what Computer Scientists do. We've mathematicians who are usually associated with design of algorithms and than we have electronics wizards who are designing computers of today and tommorrow. This thinking always confused about relevance of Computer Science as a subject. However while reading an article about Parsing, I came across this note by Authors (Dick Grune and Ceriel J.H. Jacobs).

"There is a considerable difference between a mathematician's view of world and a computer scientist's. To a mathematician all structures are static: they have always been and always be; the only time dependence is that we just haven't discovered all of them yet. The computer scientist is concerned with (and fascinated by) the continuous creation, combination, separation and destruction of structures: time is of essence. In the hands of a mathematician, the Peano axioms create the integers without reference to time, but if a computer scientist uses them to implement integer addition, he finds that they describe a slow process, which is why he'll be looking for a efficient approach. In this respect the computer scientist has more in common with physicist and chemist; like these, he cannot do without a solid basis in several branches of applied mathematics, but, like these, he is willing (and often virtually obliged) to take on faith certain theorems handed over to him by mathematicians. Without rigor of mathematics all science would collapse, but not all inhabitants of a building need to know all the spars and griders that keep it upright. Factoring off certain knowledge to specialists reduces intellectual complexity of a task, which is one of the things computer science is about."

Rules about Roman Numerals

While reading about regular expressions, found few interesting things about Roman Numerals today.

In Roman Numerals there are 7 characters which are repeated and combined in various different ways to represent numbers.

• I = 1
• V = 5
• X = 10
• L = 50
• C = 100
• D = 500
• M = 1000

Following are general rules for constructing Roman numerals:

• Characters are additive. I is 1, II is 2 and III is 3. VI is 6, VII is 7 and VIII is 8.
• Tens characters (I, X, C and M) can be repeated upto 3 times.
• At 9, we need to subtract from the next highest tens character.
• The five characters can not be repeated.
• Roman numerals are always written highest to lowest, and read left to right, so the order of characters matters very much.

Puppetizing Amazon EC2

For those who aren't there yet ?

Amazon EC2 is a cloud computing service from Amazon. Using various mechanisms like API, Web Console, Command Line tools, one can reap benefits of IAAS facility provided by Amazon.

Puppet is systems configuration management software developed by Puppet Labs (Formally Reductive Labs). Puppet provides a framework to simplify the majority of technical tasks that System Administrators perform. It provides a declarative language, which can be used to express system configuration.

Why bother about Puppet? Didn't we have ghost do same job for us in real server scenario. And since I learnt about EC2, so there is concept of AMIs as well !

Yeah, AMIs are great way to package a group of software and configuration files on top of an Operating System, but that's all that's there about them. With development in today's circumstances going Agile, we see a lot of changes in configurations and software installed on a system. Re creation of AMIs at regular intervals is definitely not the coolest thing one would like to do. Imagine running 10 webservers on EC2, which were spawned using a private custom AMI with httpd package pre-installed and configuration built into the AMI. During the course of day, developers made changes to the code base and also added some features, which required 10 changes to the configuration of httpd. This situation is nothing less than a nightmare for system administrators when deployments go out in Agile fashion. Our imaginary circumstance would require 10 changes to AMIs and we'd be required to propagate these changes to either 10 existing servers, 10 times or re spawn 10 servers every time a deployment went out to production.

Puppet comes to our rescue !

With puppet these changes are done using it's declarative language in a file called recipe. So in our setup, we make required changes to the recipe and push the code change to puppetmaster server and puppet client on the webservers connect to puppetmaster after a specified interval (default=30 minutes) and pull the new configuration. Puppet allows designing a setup where in machines are classified under various roles and we can write down configuration for these roles using the declarative language of Puppet.

Cloud and Puppet

EC2 instances can be spawned using command line tools (provided by ec2-api-tools) and we've created a set of shell scripts wrappers around these commands to start of new instances on EC2 and assign them some role through which the instances will get configured using appropriate puppet recipe.

How did we do it ?

Slideshare has been using EC2 for building up it’s conversion stack and as most sysadmins will agree, performance of machines tend to degrade after they’ve been operational for a certain amount of time. We’ve tried to kill this problem by designing conversion stack where new machines on EC2 are spawned at regular interval and old instances are killed.

We've setup a puppet server on one EC2 instance and used Apache proxy to proxy through multiple instances of puppetmasterd running on different ports. We've also reconfigured init script for puppetmasterd to allow for stop and start of puppetmasterd service on different ports using well known /etc/init.d/ start/stop syntax.

Puppet client by default tries to connect to a host named puppet on port 8140 and hence we've setup firewall settings for this host to allow connection to be established on 8140 port and setup apache to listen on 8140 port. Apache than proxies the request to puppetmasterd server running on different ports in backend. This allows for easy scaling of the setup.

Puppet server is configured to automatically self sign the certificate request by the client and this allows new clients to connect directly to puppetmaster and get required configuration.

We’ve also created a separate set of private AMIs which we use to fire up adhoc instances in times of high load on our regular instances. These instances are meant to be running for a small time duration and have all the software and configuration baked into the AMI itself.

We’ve created our own home grown scripts which spawn instances on EC2 at regular intervals and code on these machines take care of killing the machine after a fixed time.

Following is excerpt of the script which we use to spawn different instances. Role of the instance is passed on as argument to the script:

#!/bin/bash

...

...

...

#Place request for spot instance

ec2-request-spot-instances $AMI -n $NUM_INSTANCES -p $PRICE -t $TYPE -k $KEY_PAIR --group $SECURITY_GROUP > /tmp/ec2_spot_instance_request

SIR_REQUEST=`cat /tmp/ec2_spot_instance_request | cut -f 2`

rm -f /tmp/ec2_spot_instance_request

#Capture status of request. Initially request has STATUS=open and we need it to be active in order to continue

STATUS=`ec2-describe-spot-instance-requests | grep $SIR_REQUEST | cut -f 6`

#We won't want to wait till infinity for instance to spawn up. Our threshold is 5 minutes.

COUNT=1

#This variable checks if instance is spot or regular

IS_SPOT=1

#Wait for spot instance request to succeed.

REQUIRED_STATUS="active"

while [ $STATUS != $REQUIRED_STATUS ]

do

sleep 60

STATUS=`ec2-describe-spot-instance-requests | grep $SIR_REQUEST | cut -f 6`

if [ $COUNT -gt 5 ]

then

IS_SPOT=0

break

fi

COUNT=`expr $COUNT + 1`

done

if [ $IS_SPOT -eq 1 ]

then

INSTANCE_ID=`ec2-describe-spot-instance-requests | grep $SIR_REQUEST | cut -f 12`

else

#Kill spot instance request we made earlier

ec2-cancel-spot-instance-requests $SIR_REQUEST

#Spawn up a regular instance

ec2-run-instances $AMI -n $NUM_INSTANCES -t $TYPE -k $KEY_PAIR --group $SECURITY_GROUP > /tmp/ec2_instance_request

INSTANCE_ID=`cat /tmp/ec2_instance_request | tail -1 | cut -f2`

STATUS=`cat /tmp/ec2_instance_request | tail -1 | cut -f6`

rm -f /tmp/ec2_instance_request

REQUIRED_STATUS="running"

while [ $STATUS != $REQUIRED_STATUS ]

do

sleep 60

STATUS=`ec2-describe-instances $INSTANCE_ID | tail -1 | cut -f6`

done

fi

sleep 120

#Instance is now active. Capture data associated with instance like instance-id, external and internal dns.

INSTANCE_EXTERNAL_DNS=`ec2-describe-instances $INSTANCE_ID | tail -1 | cut -f 4`

INSTANCE_INTERNAL_HOSTNAME=`ec2-describe-instances $INSTANCE_ID | tail -1 | cut -f 5 | cut -f 1 -d'.' `

#We need to do the record keeping

echo "`date`: $INSTANCE_ID: $INSTANCE_EXTERNAL_DNS: $INSTANCE_INTERNAL_HOSTNAME: PDF2SWF" >> $DB_FILE

#If we are not able to get internal hostname within next 1 minutes for some reason then quit

while [ -z $INSTANCE_INTERNAL_HOSTNAME ]

do

COUNT=`expr $COUNT + 1 `

sleep 10

INSTANCE_INTERNAL_HOSTNAME=`ec2-describe-instances $INSTANCE_ID | tail -1 | cut -f 5 | cut -f 1 -d'.' `

INSTANCE_EXTERNAL_DNS=`ec2-describe-instances $INSTANCE_ID | tail -1 | cut -f 4`

if [ $COUNT -ge 6 ]

then

exit 1

fi

done

#Configure puppetmaster to associate relevant class with node

$SCP root@$PUPPET_MASTER:/etc/puppet/manifests/nodes.pp /tmp/nodes.pp

grep $INSTANCE_INTERNAL_HOSTNAME /tmp/nodes.pp

if [ $? -eq 0 ]

then

sed "/$INSTANCE_INTERNAL_HOSTNAME/d" /tmp/nodes.pp > /tmp/newnodes.pp

mv /tmp/newnodes.pp /tmp/nodes.pp

fi

echo "node $INSTANCE_INTERNAL_HOSTNAME { include $1 }" >> /tmp/nodes.pp

$SCP /tmp/nodes.pp root@$PUPPET_MASTER:/tmp

$SSH root@$PUPPET_MASTER "mv /tmp/nodes.pp /etc/puppet/manifests/nodes.pp"

rm -f /tmp/nodes.pp

$SSH root@$INSTANCE_EXTERNAL_DNS "cat /tmp/puppet_host >> /etc/hosts"

...

...

...

#Install puppet on newly built ec2 host

$SSH root@$INSTANCE_EXTERNAL_DNS "apt-get -y install puppet"

This script tries to spawn spot instances on EC2 (for more details about spot instances, refer to this wonderful article by Jonathan Boutelle, CTO, Slideshare). Since availability of spot instances is not certain and we wanted our script to spawn up a instance in stipulated time, so we decided to set a time limit of 5 minutes, to wait for ascertaining availability of spot instance. If we don't get a spot instance in 5 minutes, we spawn up a regular instance instead.

Once an instance has been spawned we do some record keeping tasks and add an entry for puppet server in /etc/hosts file of the node. Puppet client by default tries to connect to a host named puppet and hence our entry in /etc/hosts looks like follows:

XXX.XXX.XXX.XXX puppet

Finally we make an entry for this newly spawned node in nodes.pp on puppet master. This is a tricky part as DNS on EC2 reassigns previously assigned hostname to a new node and this can cause duplicate entries in nodes.pp also many a times it so happens that we are able to spawn off an instance however the script is unable to get hostname using ec2-api-tools and this can cause an invalid entry to appear in nodes.pp. We've tried to circumvent these errors by implementing few error checks.

There are several improvements that can be made to existing setup. We are relying on DNS service of EC2 and this causes lots of issues while identifying instances in cases of troubleshooting errors. This can be resolved by setting up a home grown DNS server on EC2.

We’ve tried to setup a nagios monitoring system on EC2 as well to monitor our services running on cloud. This can be very useful as most of the monitoring facilities provided by Amazon are pretty basic and are more oriented towards system health checks rather than service health checks.