Vladimir Vuksan's blog

This is a "recipe" on how to bootstrap your whole environment in case of a disaster ie. your data center goes dark or if you are migrating from one environment to another. This guide differs from others in that it uses mcollective and DNS to provide you with greater flexibility in deploying and bootstraping environments. Some of the alternate ways are ec2-boot-init by R.I. Pienaar or Grig Gheorghiu's Bootstrapping EC2 images as Puppet clients.

Intro

You will need two disk images, your code repository and your DB backup and you can rebuild your whole environment from scratch in a relatively short period of time. This could be adapted to generic cloud provisioning however use case I'm trying to address is disaster recovery. We are using DNS so that we can keep hostnames consistent between environments ie. mail01 will be a mail server in all environments instead of domU-1-2-3-4 in one, rack-2345 in other etc.

Set up a master node image

Master node is the node that controls all the other nodes. Most importantly it contains all your configuration management data. You will need to install following

• mcollective with ActiveMQ
• DnsMasq
• Puppet from Puppet Labs

1.  You will need to get a DNS name from a dynamic DNS provider such as DynDNS. Once you have that you will need to write a shell script that runs at boot and sets your EC2 private IP to that DNS name. Let's say we want our controller station to be known as controller.ec2.domain.com we can do something like this

IP=`facter ipaddress`
change_my_dns_ip controller.ec2.domain.com
# Delete any entries from hosts
sed -i "/controller.ec2.domain.com/d" /etc/hosts
echo "${IP}     controller.ec2.domain.com" >> /etc/hosts

2. Set up ActiveMQ to be used with mcollective http://code.google.com/p/mcollective/wiki/GettingStarted
3. Set up mcollective

Configure controller.ec2.domain.com as the stomp host in your mcollective configuration for both client and server configuration.

4.Install dnsmasq. You don't need to configure anything since by default dnsmasq will read /etc/hosts and serve those names over DNS

5. Install puppetmaster, configure it anyway you want

6. Image it

Set up a generic/worker node image

You will need to Install following

• Mcollective
• puppet agent

1. On the worker node you need to configure the server piece of mcollective and make sure the stomp.host is pointed to the master ie.  controller.ec2.domain.com.

2. Create a reboot agent (we'll discuss later how to use it). Please visit http://code.google.com/p/mcollective/wiki/SimpleRPCIntroduction for an example. Create a new file ie. reboot.rb. Paste this code in it

module MCollective
 module Agent
  class Reboot<RPC::Agent
    def reboot_action
     `/sbin/shutdown -r now`
    end
  end
 end
end

Copy the resulting file to the mcollective agents directory

3. Add following script to the bootup

MASTER=`host controller.ec2.domain.com | grep address | cut -f4 -d" "`
IS_ALREADY_SET=`grep -c ec2.domain.com /etc/resolv.conf`
if [ $IS_ALREADY_SET -lt 1 ]; then   
sed -i "s/^search .*/search ec2.domain.com/g" /etc/resolv.conf
sed -i "s/^nameserver/nameserver ${MASTER}\nnameserver/g" /etc/resolv.conf
fi
# Set Hostname
IP=`facter ipaddress`
MY_HOST=`/bin/ipcalc --silent --hostname ${IP} | cut -f2 -d=`
hostname ${MY_HOST}

What that does is point tells your worker nodes to use controller DNS for resolving names as well as setting your hostname.

4. Get the mcollective puppet plugin from github

5. Image it

Bringing up the environment

You will need to start the master instance first since that's the instance that everyone will be talking to. As soon as it's up you can start up as many instances as you'd like.

While you wait rsync your puppet manifests and configurations to the master node

To find out what nodes are up and available issue mc-ping from the master and you should get a response similar to this

# mc-ping
controller.ec2.domain.com               time=77.21 ms
domu-12-31-55-11-22-18.compute-1.internal time=188.76 ms

Trouble is that hostnames on the worker nodes are set to Amazon names. We want to make them recognizable e.g. mail01.

To do so simply add the IP of the worker instance and it's name into /etc/hosts on the master e.g.

echo "10.1.2.3      mail01.ec2.domain.com" >> /etc/hosts

Reload dnsmasq configuration ie.

/etc/init.d/dnsmasq reload

What this has bought you is reverse DNS resolution of the node.  To take effect you will need to reboot the worker node. We already have the reboot agent on the worker nodes so all we have to do is run following command on the master node

./mc-rpc -F hostname=domu-12-31-55-11-22-18 reboot reboot

This will seek out the domU-1-2-3-4 host and reboot it (--arg is irrelevant so put anything). Once the machine is up it will advertise it's new name ie. running mc-ping will show you this

# mc-ping
controller.ec2.domain.com           time=47.59 ms
mail01.ec2.domain.com               time=80.71 ms

Now let's activate puppet. From master node run

# mc-puppetd -F hostname=mail01 runonce

 * [ ============================================================> ] 1 / 1

Finished processing 1 / 1 hosts in 1051.23 ms

Once that is done puppetca should give you this

# puppetca --list
mail01.ec2.domain.com

Sign it

# puppetca –sign mail01.ec2.domain.com

Now you can simply run

# mc-puppetd -F hostname=mail01 enable

and off you go. Now lather, rinse, repeat to get the rest of the instances going. You would certainly want to automate this further but I leave that exercise to you .

If you are looking for an easy cross-cloud API check out my "Provision to cloud in 5 minutes using fog".

Most recently I have been working on disaster recovery project where we are assembling documentation, processes and code to be able to fire up our whole environment in the cloud in case of a major disaster. At Velocity Conference I met Wesley Beary who is the main developer for fog, a Ruby cloud computing library. What appealed to me about fog is that it has varying support for different clouds so that we are not stuck using a provider due to our non-portable code. Now off to couple quick example to get you going.

To install fog you will need to install Ruby Gems. If you have them type

  sudo gem install fog

The install may fail if you don't have the libxslt and libxml2 dev libraries. On my Ubuntu laptop I resolved it by doing

  sudo apt-get install libxslt1-dev libxml2-dev

On Centos/RHEL 5 I had to do

   yum install libxslt-devel libxml2-devel

Create a file called config.rb which contains your credentials e.g.

#!/usr/bin/ruby

@aws_access_key_id = "XXXXXXXXXXXXXXXXXX"
@aws_secret_access_key = "AXXZZZZZZZZZZZZZZZZZZ"
@aws_region = "us-east-1"

Let's start with the basics. Let's get our currently running instances and what images are available

#!/usr/bin/ruby

require 'rubygems'
require 'fog'

# Import EC2 credentials e.g. @aws_access_key_id and @aws_access_key_id
require './config.rb'

# Set up a connection
connection = Fog::AWS::EC2.new(
    :aws_access_key_id => @aws_access_key_id,
    :aws_secret_access_key => @aws_secret_access_key )

# Get a list of all the running servers/instances
instance_list = connection.servers.all

num_instances = instance_list.length
puts "We have " + num_instances.to_s()  + " servers"

# Print out a table of instances with choice columns
instance_list.table([:id, :flavor_id, :ip_address, :private_ip_address, :image_id ])

###################################################################
# Get a list of our images
###################################################################
my_images_raw = connection.describe_images('Owner' => 'self')
my_images = my_images_raw.body["imagesSet"]

puts "\n###################################################################################"
puts "Following images are available for deployment"
puts "\nImage ID\tArch\t\tImage Location"

#  List image ID, architecture and location
for key in 0...my_images.length
  print my_images[key]["imageId"], "\t" , my_images[key]["architecture"] , "\t\t" , my_images[key]["imageLocation"],  "\n";
end

Let's spin up a m1.large instance

#!/usr/bin/ruby
require 'rubygems'
require 'fog'
# Import EC2 credentials e.g. @aws_access_key_id and @aws_access_key_id
require './config.rb'

# Set up a connection
connection = Fog::AWS::EC2.new(
 :aws_access_key_id => @aws_access_key_id,
 :aws_secret_access_key => @aws_secret_access_key )

server = connection.servers.create(:image_id => 'ami-1234567',
 :flavor_id =>  'm1.large')

# wait for it to be ready to do stuff
server.wait_for { print "."; ready? }

puts "Public IP Address: #{server.ip_address}"
puts "Private IP Address: #{server.private_ip_address}"

This may take a while so please be patient.  You could obviously spin up a number of these instances without waiting for any of them to be available then use connection.servers.all to get a list of running instances.

Now let's destroy a running instance

#!/usr/bin/ruby
require 'rubygems'
require 'fog'
# Import EC2 credentials e.g. @aws_access_key_id and @aws_access_key_id
require './config.rb'

# Set up a connection
connection = Fog::AWS::EC2.new(
    :aws_access_key_id => @aws_access_key_id,
    :aws_secret_access_key => @aws_secret_access_key )

instance_id = "1-123456"

server = connection.servers.get(instance_id)

puts "Flavor: #{server.flavor_id}"
puts "Public IP Address: #{server.ip_address}"
puts "Private IP Address: #{server.private_ip_address}"

server.destroy

There is tons more out there although this gets me going . Now off to playing with R.I. Pienaar's ec2-boot-init.

Thanks to Wesley Beary for answering questions about fog and Ian Meyer for pointing out Chef Fog code.

Couple weeks ago I read this post on the memcached mailing list. Key quote

We currently run a cluster of aproximately 40 memcache servers with about 6.5 gb of ram each machine using m1.medium ec2 instances. I was in the process of reducing the number of servers while increasing the memory size for each from 6 to about 30gb. Now i've started noticing that some servers seem to hit certain bandwidth limitations not consistenly though since i have some servers pushing 6mb/sec and some at 4mb having packet los and tcp timeouts.

.....

I've replaced the instances hoping this will give me an instance on a better area or on a less congested switch but i still have the issue on the same server.

This surprised me at first since my understanding was that as EC2 instances get bigger there are less and less "neighbors" on compute nodes you have to deal with and in theory less chance they may impact your performance. Yet this person was having more issues with bigger instances than smaller. Thinking about it some more I realized that the reality may be exactly opposite ie. bigger the size the likelihood is that the instance is going to be used for a "big" workload e.g. a busy relational database. This could lead to inconsistent node performance. Inconsistent node performance is a bad thing since it makes troubleshooting problems much harder and also provides poor end user experience. Providing slow/substandard performance to fraction of your visitors may not seem like much but if you are a retailer it's lost sales.

Another thing to note is that  lots of performance problems are subtle. Just the other day we had an issue when upgrading our F5 load balancers. We upgraded from version 9.4.3 to 10.1. Upgrade was mostly uneventful and everything seemed to work however after about a day we observed a fall in traffic going to our web caching tier. It looked like this

We also had another graph from a different source that "corroborated" this behavior. We spent a lot of time trying to identify what the problem was since F5 wouldn't believe there was a problem since the only evidence were couple graphs. To cut the long story short the upgrade "fixed" a behavior where certain objects were served out of F5 memory instead of being passed onto our web caching tier. It was apparently broken in the previous release and we didn't even know it. There are plenty of other cases where things have "broken" and only by observing metrics we were able to determine that there is an actual problem. Having inconsistent behavior makes that job extremely difficult if not impossible since it may be much harder to isolate problems.

Getting back to the initial problem one the obvious strategies is to keep cycling machines until you get more performance but as evidenced by the poster that was less than successful. Also what happens if after you have filled up your 30 GB memcache your performance degrades. What then ? You could try and launch another machine but that may spike up the load on your database server. Not a pleasant set of options.

Instead what you could do is following

1. Find cloud providers that don't use virtualization (I have heard that they exist even though they are like the Bigfoot, hard to find) but deploy directly onto raw hardware. This will eliminate most of inconsistent node performance issues. The downside is it may be more expensive.
2. Stick with virtualization but implement a private cloud where you have more insight into load on underlying hardware and control how images are deployed onto host machines. More on this point later.
3. Hybrid between approach 1. and 2.

I personally think hybrid approach may be best since some workloads are best handled without going through the virtualization layer. As far as virtualization is concerned it is best to strategically place services based on resource utilization. Network services will fit into 3 broad levels of utilization ie. high resource utilization services such as relational databases, application servers, medium resource utilization services such as web servers and low resource utilization services such as DNS, monitoring, memcache etc. Trouble with public clouds is that you have no insight or say on what type of workloads are run. Instead if you had deployment control you could pair a relational DB image and memcache image on the same physical piece of hardware. That would likely work fine. If the performance of one degraded you could take appropriate action  ie. move memcache image or look for the root cause of performance degradation. Since you have access to the underlying hardware you can isolate problems which will surely help in getting down to the root of the problem. The cons of the approach are increased complexity, cost and additional management overhead.

Even if you choose to adopt the above approach you still could use public clouds for things like static content storage, image resizing, development and QA systems etc. For really critical operations I would stick with raw hardware and/or private clouds.

I read with interest a post about measuring disk I/O performance on EC2.

http://stu.mp/2009/12/disk-io-and-throughput-benchmarks-on-amazons-ec2.html

It is a good test however results were not unexpected. The problem with shared infrastructure is not that it provides subpar performance but the fact that in any infrastructure which can be "modified" by a customer you will run into "abuse" where one or couple customers will use infrastructure unevenly and will affect other customers. I have blogged about virtualization stress points before ie.

http://vuksan.com/blog/2009/12/04/cloud-cartography-load-co-residence-detection/

http://vuksan.com/blog/2009/09/01/cloud_computings_achilles_heel/

I have also in the past been in charge of a operations for an e-commerce SaaS startup and we would see this issue quite often. For instance we had two customers that did about the same amount of yearly sales yet one of the customers' infrastructure utilization (number of disk ops, DB bandwidth etc.) was 3-4 times higher than the other customer. At times they would "abuse" a shared database so much that it affected everyone else. We resolved it when a collegue figured out that we could QoS traffic to the database. That way only the abusing customer would be affected if they did anything crazy. It also helped that we ran the infrastructure and the application so we could quickly determine what is normal and what is not.  I suspect this problem becomes much trickier in clouds since you have very little idea what applications are running and what is normal.

One other thing to point out is that some of the "abuse" may be inadvertent. Coders are sloppy and occasionally (more often than one would hope) things start leaking memory and machine will start thrashing on the disk. Add to that impromper monitoring and if you are on the lucky duckies to be on the same piece of physical hardware as them your performance will go down the drain. I recall a tweet some time ago where the person was scratching his head that untarring a tarball on one EC2 instance took 15 minutes and on another 45 minutes.

There are certainly solutions to these problems however they require a lot more work. I think clouds are great and use them extensively however you should be aware of some of the drawbacks. It also helps if you are designing your app in such a way that it doesn't rely on a centralized relational database (often a bottleneck).

Some weeks ago @krishnan and I had a tweet conversation regardinga claim he heard at an Amazon webcast where the speaker claimed that cloud cartography attacks are impossible due to Amazon's use of virtual interfaces to separate customers traffic. I responded that any such claim should make anyone sceptical (not in those words ). Specifically I cited that the paper addresses other ways of detection ie.

Section 8.2 - Load-based co-residence detection

I have written in the past about Cloud Computing's Achilles Heel which dealt with performance degradation in case there is misbehaving instance running on the same piece of hardware as your own instance. I did not think of cartography in those cases but today while making a large back up of a virtual instance I thought let's try the load-based co-residence detection so on a different virtual instance running on the same machine I typed

dd if=/dev/zero of=testfile bs=1M count=15000

This simply creates a 15G file with zeroes in it. Check out what happens to the network performance of the machine that was being backed up

Performance dives from about average of 15 Mbytes/s to between 0 and 2 Mbytes/sec. For completeness here is the CPU utilization graph

I was actually quite surprised at the magnitude of degradation. I'd say this may be even a more successful co-residence detection attack than network probing since you could generate legitimate HTTP traffic to a site of interest (or a node of interest), throw tons of load at it and see if you notice response degradation.

There are obviously ways to mitigate some of these issues ie. control tightly who can connect to your instances within the cloud, cycle your own instances so that they keep "moving around", etc. Unfortunately it does come at a price of additional complexity and work.

There was quite a discussion on Twitter about the BitBucket outage which initially appeared to be failure of Amazon EC2/EBS. More about the outage can be found here. Brett Piatt was kind enough to write up his view of the situation

http://www.bretpiatt.com/blog/2009/10/03/availability-is-a-fundamental-design-concept/

In principal I do agree with his suggestions and his conclusion ie. that availability is a fundamental design concept. I do however disagree that "warm" redundancy is cheap. In my own view and experience redundancy is extremely expensive if you are going to do it right. Redundancy is not just being able to add more hardware, systems and monitoring software and failover policies but a matter of process where you continuously have to make sure that the redundancy works. For instance successful backup strategy doesn't consist of simply getting a backup device yet never testing the backups by doing an actual restore. As many organizations have discovered backups do break, media gets corrupted, etc. and you can suffer a devastating blow. So if you want to do redundancy right you have to invest lots and lots of time practicing. For example running fire drills is a useful tool or doing periodic site failovers ie. run on site A for two weeks, then during low traffic times failover to site B, run for two week then back to site A and on and on. That certainly ain't cheap.

I'd also point out that "warm" redundancy is in lots of instances riskier than "hot" redundancy since you may discover that redundancy doesn't work when you have to failover whereas in "hot" redundancy issues may crop up much earlier allowing you to stay on top more readily.

That said the discussions over how you are responsible for your own availability reminds of "individual responsibility" (for my international readers this is something that is a hot topic in the United States). Sure you should "own" your redundancy however that may often be impractical or too expensive. Not everyone is blessed with copious resources.

I have perhaps been overly critical of clouds. I do not think clouds are useless. They certainly have their place and usefulness. I just dislike the hype around the clouds since I find it completely misplaced. I see clouds primarily as a way of easily "creating" and "disposing" of hardware ie. you need extra couple machines you can at a press of a button create them. When you are done you can dispose of them. However they also have some major drawbacks which I have alluded in the past ie. in Cloud Computing's Achilles Heel and Trouble with Cloud Computing.

That said there are a number of ways clouds can be improved. Some may be impractical, some may be expensive and some may be overly complicated but they are certainly options. Shall we go through the list

1. Don't use virtualization - you could certainly implement a cloud where an instance you get runs on raw hardware. That way you are guaranteed to have a dedicated piece of hardware that is not affected by other users' (mis)use. This would be a lot more expensive but some people may be willing to pay for it
2. Intelligent web traffic load balancing - one of the big issues in general is that load balancing methods such as round-robin, server with least connections are imperfect since a server may get slow for a number of reasons and a portion of your clients may receive "substandard" response. Chances of that happening in shared environments is greater. Thus devising a load balancer which can "intelligently" figure out which server is slow and deal with it accordingly by either taking it out of the pool or sending less traffic to it.
3. Similar to 2. for your relational DB traffic you would have to use a DB cluster and devise a way of preferring "faster" DB servers. This part in some ways "scares" me the most since if you start using synchronous replication you have to wait until all members of the cluster have commited the change. If you start doing asynchronous replication you will run the risk of DB inconsistencies which you will have to resolve.

There are likely other options but you can see from above that this gets real complicated, real quick. It can be done it is just a lot of work and a lot of QA. Hopefully someone comes up with a generic solution for problems 2. and 3. and those become non-issues.