One of the things I like is the ability to get a full backup of all my project data, repositories, etc in a single tarball. You can even ask them to keep a copy in your own S3 account.

To create a backup do the following

1) log into Unfuddle and goto the Project page.
2) Click the ‘settings’ tab then
3) Scroll down till you see link that says ‘Request a backup of this project now’ link. Click it.

In a few moments you’ll get an email, and you’ll see a new link on the right hand side of your project settings page that includes a timestamped backup. This backup is a tarball that contains all the GIT repositories and some other files like a backup.xml file which looks like all your tickets.

To use the git dumps run the following

mkdir reponame
cd reponame
git init
git fast-import < ../my-unfuddle-backup.git.dmp
git checkout master

You’re done!

If you’re using subversion repositories there is documentation on how to use these repo backups on Unfuddles website.

There are economic concepts that dictate how you use services and hosting on the internet.

Do tell.

Computing is really cheap. Think about all the email that Gmail handles in a day. It’s so cheap that advertising can pay for it. But the “Network is the computer” after all, so we have to think about what it takes to get that information in and our of these clusters of cheap computing, and that’s the rub. Amazon charges $0.17/gig to get your data out of EC2, which is equivalent to almost two hours of their cheapest computing instance. This is a good scenario if the task you send to your cheap compute cluster can be defined in a very small package, and yields a relatively small result but typical web services and applications don’t work this way. The point is: Its cheaper to move the computing than it is to move the data.

So what?

This all clicked for me when I messed around with Freebases development environment, “Acre”. Acre is great. It lets you create, edit, and host your applications through a browser. Not only had I been messing around with Acre, but I’d also been toying with the idea of using Freebase as a mechanism for validating and normalizing data. The problem is asking Freebase for a bunch of information on say, “every city on the planet” is pretty expensive. Not only do you incurr a network transfer cost, but you then have to process the information. Not exactly ideal. But what if I could pose a question to an application running at Freebase? What if, instead of pulling out all the information about every movie and creating your own Freebase-based IMDB, you could host it right next to the data source. You get all the benefits of transferring the ‘heavy stuff’ over the WAN, and the browser gets the good stuff, but only when it asks for it.

This is why server-side javascript is perfect

Hosting Ruby, PHP, Python, etc is kind of a pain in the ass. Well its easier than it used to be but it could be a lot better. If I had to choose something relatively lightweight to interface to my data-source and create that rich browsing experience, you’d probably pick Javascript. My initial impression is that depending on your data-source, scaling it would be easy, too. Running computing close (as in LAN close) to the data-set means a few things

1. You can create cheaper mashups

2. You can eliminate all the cruft from your data before it gets sent over the wire

3. You can create nifty applications and ask them short questions that yield short answers but require huge amounts of data to determine

ZOMG How do I start?

You’ll have to learn javascript, and as a hosting or service operator you’ll have to choose an application for running it server-side. There are a few options. Trusty Wikipedia has a lengthy list of Server-side Javascript implementations. I’d recommend checking out the following:

Rhino

Spidermonkey

V8

AppJet

Jaxer

-Trevor

There are a few monitoring solutions for ActiveMQ that will let you know when its broken, but I needed to grab usage patterns over time, so I would be able to automatically spin up more workers. I didn’t see anything quickly available so I threw this together:

#queuemonitor.rb

# queries the activeMQ status XML file and returns relevant data.
# I'm sure there's a more elegant way of creating method directly out of xml.
 
# Rather than have each queuemonitor as a seprate class with one name, we
# might want to be able to parse multiple queues on the same server. So we have
# an array of queue names (@queues) that contain the names of queues we want to monitor.
# if you want to monitor more than one queue then you'd have to create a new QueueMonitor
# instance.
 
class QueueMonitor
 
  attr_accessor :url, :queues
 
  require 'net/http'
  require 'rexml/document'
 
  def initialize
    @url="http://127.0.0.1:8161/admin/xml/queues.jsp"
    @queues = []
  end
 
  #add a queue to the list of queues that we'll pay attention to
  def addqueue(queue)
    @queues.push(queue) unless @queues.include?(queue)
  end
 
  def delqueue(queue)
    @names.reject!{|q| q == queue}
  end
 
  def query
    results = []
    date = Time.now
    xml_data = Net::HTTP.get_response(URI.parse(@url)).body
    doc = REXML::Document.new(xml_data)
 
    doc.elements.each('queues/queue') do |queue|
      name = queue.attributes["name"]
      # only list from Queues listed in the '@names'
      if @queues.member?(name) then
 
        queue.elements.each('stats') do |ele|
          size = ele.attributes["size"]
          consumers = ele.attributes["consumerCount"]
          enqueue = ele.attributes["enqueueCount"]
          dequeue = ele.attributes["dequeueCount"]
 
          queue = { 'name' => name,
                    'size' => size,
                    'consumers' => consumers,
                    'enqueue' => enqueue,
                    'dequeue' => dequeue,
                    'date' => date
          }
 
          results < < queue
        end
      end
    end
    return results
  end
 
end

You can then start monitoring your queue servers with something like this:

#!/usr/bin/env ruby
 
require 'queuemonitor'
 
mfreq = 30
@monitor = QueueMonitor.new
@monitor.url = "http://67.202.41.64:8161/admin/xml/queues.jsp"
@monitor.addqueue "pubsub.pings.spider"
 
#do something interesting like write to a file
#screen or database
 
def query
  date = Time.now
  results = @monitor.query
  results.each do |q|
    puts "#{date} #{q["name"]} #{q["size"]} #{q["consumers"]}"
  end
end
 
#You'd want to use daemonize here...
 
loop do
  query
  sleep mfreq
end

It works pretty well. Right now I’m just writing the results to a file to parse later, and so you can setup other parts of a program that would automatically spin up new workers based on a set of circumstances. The algorithm for that is the hard part and will depend on a bunch of your own rules.

Take Amazon as an example. Amazon operates their environment as a bunch of different groups, each running different services within the same company. S3, EC2, Payment Services. They’re all independent, highly scalable functions, tied together in the application we call Amazon.com.

Companies and startups are starting to break this operational model open, and putting those individual functions online for everyone. They’re building services that do something really well – or rather that do one thing really really well. They’re companies that focus on a specific function or feature and are open enough so creative people can say “I’m going to take this, this, and this – mix it in a pot and voila!”.

Do you want to build your own Twitter? Find an SMS gateway, Cloud Computing Host and XMPP service provider.

Do you want to build an interesting RSS/ATOM service? Find an RSS aggregator service and pour on some glue – see what sticks.

It’s Feature as a Service world (to use an already overused description). Eventually cloud companies will realize that doing one thing really *really* well is tremendously valuable. Why does everyone have to build their own DNS service? Why does everyone have to build their own hosting system? What about Enterprise Storage, Authentication, SMS Gateways, Massively scalable XMPP services? How come I have to do that myself? Can 10,000 messages sent through a jabber server be worth a dollar? I think it can (maybe the math needs adjusting but you get my point). We’re all really just building a massive computer called the internet, only with each big trend we replace ‘The Internet’ with something else. First it was ‘The Web’, then it was ‘Web 2.0′, and now its ‘The Cloud’. The fact of the matter remains – the further along we go the more tightly knit the internet becomes, and that means that theres opportunity for programmable white label services to propel us further and faster.

Application Integration

Just like everyone thought WSDL would automagically make applications work together, people think the new ubiquitous ‘Cloud’ will make applications work together. Unless your application was designed from the beginning to talk to and operate with other cloud stacks, then you’re going to have an application integration issue on your hands. This is why behemoths like SAP exist. Everything is one universal and translatable data format, all controlled by one vendor. If you want to be able to move data, or integrate applications that run in different clouds, then you’re going to have to do it the old way. Sit down, translate the data yourself, and emulate it where you can’t. Cloud != Application / Data Babelfish

Eliminate Monitoring

Monitoring is either a dirty word or a multi-million dollar business depending on who you talk to. As a sysadmin for several high-availability environments, I can’t stress it enough. Building your application and environment to be redundant is always step number one, but it’s useless if you don’t know the state of your environment at all times – especially before and after changes. Environment monitoring is like Behaviour Driven Development (see rspec) for your application. You know what’s supposed to happen, and how things are supposed to look – and if something changes you find out about it. You still need this kind of functionality, and just putting your stuff in the cloud doesn’t make issues go away. Cloud != Perfectly Reliable Environment

Eliminate Lock-in

Oh boy I said a dirty word. If you write your application for AppEngine, then you’re using AppEngine until you refactor. Period. The AppDrop application lets you “run AppEngine on EC2″ but thats kind of masking the issue. AppDrop is just the AppEngine developer tools running in an AMI – it doesn’t do anything like emulate all the necessary infrastructure that makes AppEngine appealing in the first place and the authors acknowledge this. Don’t get me wrong – AppEngine is great. Its a highly abstracted environment for writing web applications, but it’s not appropriate for a lot of different services. If you want to be able to treat the cloud like you treat a “standard” machine -> LAMP stack then you should be aware of all these factors. Cloud != Universal Magical Computer

Eliminate Jobs

Sysadmins (and I can say this) are an odd bunch. Most of them treat their environments like children, rearing them to a specific age, but always keeping a watchful eye. Others tend to automate everything necessary in order to move on to the next best thing. The people that horde data, knowledge, useful tools, and expertise will fit right in to closed cultures, but if you want to make it in the new reality, you’re going to have to open your mind. The cloud isn’t going to make you obsolete unless you try stopping progress. If you didn’t have to worry about making system images and building back-ends all day think about all the other amazing things you could do! There are a slew of bad analogies I can use to drive this point home, but I’ll use the most cliche: Be a reed in the wind. If you’re clinging onto a job because you’re easily replaceable, then its just a matter of time. Learn a new skill. Cloud != Massive Layoffs

Great Scott! An aquaintance of mine just forwarded this to me today and my jaw dropped. A team working out of University of California, Santa Barbara led by Rich Wolski has reverse engineered the EC2 framework and (apparently) released it under a FreeBSD style license.

They gave a demo at the Open Source Grid and Cluster Conference, which was a paid conference (tsk tsk tsk)

From the website:

Overall, the goal of the EUCALYPTUS project is to foster community research and development of Elastic/Utility/Cloud service implementation technologies, resource allocation strategies, service level agreement (SLA) mechanisms and policies, and usage models. The current release is version 1.0 and it includes the following features:

• Interface compatibility with EC2
• Simple installation and deployment using Rocks cluster-management tools
• Simple set of extensible cloud allocation policies
• Overlay functionality requiring no modification to the target Linux environment
• Basic “Cloud Administrator” tools for system management and user accounting
• The ability to configure multiple clusters, each with private internal network addresses, into a single Cloud.

The initial version of EUCALYPTUS requires Xen to be installed on all nodes that can be allocated, but no modifications to the “dom0″ installation or to the hypervisor itself.

This is fantastic and *exactly* what the industry needs right now. In fact, it falls in line nicely with what we’re working on at LayerBoom. I’m extremely interested to see how this works. I’m actually booting it up as I type this.

[link] [link]

OpenSolaris 2008.05 came out the other day, and its pretty nice as far as OS’s go. One of the things that’s great about OpenSolaris is by coupling Xen with ZFS you can get a really powerful virtualization system.

Unfortunately because of space constraints in a liveCD, you can’t squeeze all those ‘nice to have’ packages into one spot, so you have to download and setup xVM manually. A few things are broken, too so some minor tweaking is required. Here’s a list of steps required to get xVM setup as a Dom0 in OpenSolaris

1) Become root and install the required packages. This will install the xen.gz kernel, as well as all the other utilities and services necessary for running a Dom0

pkg install SUNWxvmhvm
pkg install SUNWvirtinst
pkg install SUNWlibvirt
pkg install SUNWurlgrabber

2) The package installation doesn’t support adding xvm to the default boot.lst in grub. Additionally, OpenSolaris 2008.05 has ZFS as the default filesystem, so a few things have been moved around. Specifically, the boot.lst file used by grub and bootadm has been placed on its own zfs filesystem. Unfortunately it seems that bootadm has been compiled, or at least the distribution has been put together, without the ability to gracefully handle having the boot.lst on a separate partition.

A workaround is to move the /boot/grub/menu.lst file (which has some explanatory text in it) to  /boot/grub/menu.lst-old, and to symlink the /rpool/boot/grub/menu.lst file to /boot/grub/menu.lst.

cd /boot/grub
mv menu.lst menu.lst-old
ln -s /rpool/boot/grub/menu.lst menu.lst

Now we can run bootadm without any errors

bootadm list-menu

Time to add the boot entry for our xVM kernel

Update: Kevin Elliot said that running “bootadm -m upgrade”  added the xVM entry automatically after performing the linkage steps. You will still need to add the ‘boofs rpool/ROOT/opensolaris lines however. [link] [link]

open up the /boot/grub/menu.lst file  and add the following after the ‘bootadm’ section

vim /boot/grub/menu.lst

title OpenSolaris 2008.05 snv_86_rc3 X86 xVM
bootfs rpool/ROOT/opensolaris
kernel$ /boot/$ISADIR/xen.gz
module$ /platform/i86xpv/kernel/$ISADIR/unix /platform/i86xpv/kernel/$ISADIR/unix -B $ZFS-BOOTFS
module$ /platform/i86pc/$ISADIR/boot_archive

Note that if you’ve done a pkg upgrade, your ZFS root will more than likely be different. You’ll have to choose the ZFS root partition that you were using when you installed the packages earlier, otherwise they won’t be available

Note: Make sure to change the default selection to the appropriate placeholder of your xVM entry. The numbering for grub starts at 0. This means that to boot the first entry in your menu.lst file, your ‘default’ value in menu.lst should be 0, and so on and so forth.

Let’s run bootadm and see what our changes look like

bootadm list-menu

The location for the active GRUB menu is: /boot/grub/menu.lst
default 1
timeout 10
0 OpenSolaris 2008.05 snv_86_rc3 X86 i86pc
1 OpenSolaris 2008.05 snv_86_rc3 X86 xVM

Pretty. Now its time to try it out. In order to switch kernels you have to reboot your system

shutdown -y -g0 -i6

Your system should reboot using the xvm capable kernel. If your sytem could not boot properly then select the old kernel from the boot menu as your system starts.

To double-check that you’re running the proper kernel, run uname

uname -a

it should give you something like this

SunOS pluto 5.11 snv_86 i86pc i386 <strong>i86xpv</strong> Solaris

The last step is to enable the proper xVM services

svcadm enable store
svcadm enable xend
svcadm enable console
svcadm enable domains
svcadm enable virtd

Running

svcs | grep xvm

should print the following

online          2:22:12 svc:/system/xvm/store:default
online          2:22:27 svc:/system/xvm/xend:default
online          2:22:27 svc:/system/xvm/console:default
online          2:22:27 svc:/system/xvm/domains:default

One quick test

xm top

Should work properly.

Hopefully that worked for you. I appreciate any feedback!

Update #2

Some people are reporting problems creating DomU’s. If you’re running into problems, try running the script below

BASEDIR=${BASEDIR:-/}
/usr/sbin/syseventadm list -R $BASEDIR -c EC_xendev > /dev/null 2>&1
if [ $? -ne 0 ]
then
/usr/sbin/syseventadm add -R $BASEDIR -c EC_xendev \
/usr/lib/xen/scripts/xpvd-event 'action=$subclass' \
'domain=$domain' 'vdev=$vdev' 'device=$device' \
'devclass=$devclass' 'febe=$fob'
fi
/usr/sbin/syseventadm list -R $BASEDIR -c EC_xpvsys > /dev/null 2>&1
if [ $? -ne 0 ]
then
/usr/sbin/syseventadm add -R $BASEDIR -c EC_xpvsys \
/usr/lib/xen/scripts/xpvsys-event 'subclass=$subclass' \
'shutdown=$shutdown'
fi
# restart daemon if the package is being added to the running system
if [ "$BASEDIR" = "/" -a $? -eq 0 ]
then
/usr/sbin/syseventadm restart
fi

The Open Virtual Machine Format, or OVF is a proposed universal format that aims to create a secure, extensible method of describing and packaging virtual containers. Because the standard is open, it means any environment supporting the standard can import and export those virtual machines between different hypervisor platforms. The current OVF specification includes definitions ranging from virtual machine metadata and disk format, all the way to detailed hardware specifications and logical network information. It also provides an ability for the virtual machine itself to get information from the hypervisor host, meaning that if you’re creative you could create some really nifty automated integration and deployment tools.

If that doesn’t mean much to you, then consider this: Industry heavy-weights like Dell, HP, IBM, Microsoft, VMware, and XenSource all took part in drafting the specification. As far support tools go, VMware has published what appears to be the first OVF container creation tool, available here.

There IS a big problem with OVF right now, and a lot of bloggers and analysts out there are getting it wrong. OVF is not and does not define a new virtual disk format, simply a wrapper around them. This means that OVF support doesn’t enable you to drag and drop virtual machines between Xen and VMware. Some formats can be converted externally using tools, however most of the current techniques involve booting up a system, and running a migration tool to be able to convert the image – not exactly ideal. OVF does include the ability to describe your specification in an HREF, which means that you could publish your spec, and create a system that could modify containers on the fly.

If Vmware, Xen, and Parallels are technolgical islands, then OVF may one day be the bridge that will allow you to travel between them.

Update It looks like OVF will be announced formally at the Catalyst 2008 conference. More information here.