sysadmin


2
Jul 08

What The Cloud Isn’t

Cloudcamp and Structure 08 had a lot of people talking about how the cloud was going to magically solve a lot of problems, and this stemmed from a major issue – we don’t really know what the cloud is yet. In its current form the young cloud can’t solve certain issues.

Application Integration

Spaghetti Integration. MMmm Spaghetti.

Just like everyone thought WSDL would automagically make applications work together, people think the new ubiquitous ‘Cloud’ will make applications work together. Unless your application was designed from the beginning to talk to and operate with other cloud stacks, then you’re going to have an application integration issue on your hands. This is why behemoths like SAP exist. Everything is one universal and translatable data format, all controlled by one vendor. If you want to be able to move data, or integrate applications that run in different clouds, then you’re going to have to do it the old way. Sit down, translate the data yourself, and emulate it where you can’t. Cloud != Application / Data Babelfish

Eliminate Monitoring

Monitoring is either a dirty word or a multi-million dollar business depending on who you talk to. As a sysadmin for several high-availability environments, I can’t stress it enough. Building your application and environment to be redundant is always step number one, but it’s useless if you don’t know the state of your environment at all times – especially before and after changes. Environment monitoring is like Behaviour Driven Development (see rspec) for your application. You know what’s supposed to happen, and how things are supposed to look – and if something changes you find out about it. You still need this kind of functionality, and just putting your stuff in the cloud doesn’t make issues go away. Cloud != Perfectly Reliable Environment

Eliminate Lock-in

Oh boy I said a dirty word. If you write your application for AppEngine, then you’re using AppEngine until you refactor. Period. The AppDrop application lets you “run AppEngine on EC2″ but thats kind of masking the issue. AppDrop is just the AppEngine developer tools running in an AMI – it doesn’t do anything like emulate all the necessary infrastructure that makes AppEngine appealing in the first place and the authors acknowledge this. Don’t get me wrong – AppEngine is great. Its a highly abstracted environment for writing web applications, but it’s not appropriate for a lot of different services. If you want to be able to treat the cloud like you treat a “standard” machine -> LAMP stack then you should be aware of all these factors. Cloud != Universal Magical Computer

Eliminate Jobs

Sysadmins (and I can say this) are an odd bunch. Most of them treat their environments like children, rearing them to a specific age, but always keeping a watchful eye. Others tend to automate everything necessary in order to move on to the next best thing. The people that horde data, knowledge, useful tools, and expertise will fit right in to closed cultures, but if you want to make it in the new reality, you’re going to have to open your mind. The cloud isn’t going to make you obsolete unless you try stopping progress. If you didn’t have to worry about making system images and building back-ends all day think about all the other amazing things you could do! There are a slew of bad analogies I can use to drive this point home, but I’ll use the most cliche: Be a reed in the wind. If you’re clinging onto a job because you’re easily replaceable, then its just a matter of time. Learn a new skill. Cloud != Massive Layoffs


4
Jun 08

Eucalyptus – Roll your own EC2

eucalyptus logo

Great Scott! An aquaintance of mine just forwarded this to me today and my jaw dropped. A team working out of University of California, Santa Barbara led by Rich Wolski has reverse engineered the EC2 framework and (apparently) released it under a FreeBSD style license.

They gave a demo at the Open Source Grid and Cluster Conference, which was a paid conference (tsk tsk tsk)

From the website:

Overall, the goal of the EUCALYPTUS project is to foster community research and development of Elastic/Utility/Cloud service implementation technologies, resource allocation strategies, service level agreement (SLA) mechanisms and policies, and usage models. The current release is version 1.0 and it includes the following features:

  • Interface compatibility with EC2
  • Simple installation and deployment using Rocks cluster-management tools
  • Simple set of extensible cloud allocation policies
  • Overlay functionality requiring no modification to the target Linux environment
  • Basic “Cloud Administrator” tools for system management and user accounting
  • The ability to configure multiple clusters, each with private internal network addresses, into a single Cloud.

The initial version of EUCALYPTUS requires Xen to be installed on all nodes that can be allocated, but no modifications to the “dom0″ installation or to the hypervisor itself.

This is fantastic and *exactly* what the industry needs right now. In fact, it falls in line nicely with what we’re working on at LayerBoom. I’m extremely interested to see how this works. I’m actually booting it up as I type this.

[link] [link]


7
May 08

Getting xVM to work in Opensolaris 2008.05

os_com_logo.jpg

OpenSolaris 2008.05 came out the other day, and its pretty nice as far as OS’s go. One of the things that’s great about OpenSolaris is by coupling Xen with ZFS you can get a really powerful virtualization system.

Unfortunately because of space constraints in a liveCD, you can’t squeeze all those ‘nice to have’ packages into one spot, so you have to download and setup xVM manually. A few things are broken, too so some minor tweaking is required. Here’s a list of steps required to get xVM setup as a Dom0 in OpenSolaris

1) Become root and install the required packages. This will install the xen.gz kernel, as well as all the other utilities and services necessary for running a Dom0

pkg install SUNWxvmhvm
pkg install SUNWvirtinst
pkg install SUNWlibvirt
pkg install SUNWurlgrabber

2) The package installation doesn’t support adding xvm to the default boot.lst in grub. Additionally, OpenSolaris 2008.05 has ZFS as the default filesystem, so a few things have been moved around. Specifically, the boot.lst file used by grub and bootadm has been placed on its own zfs filesystem. Unfortunately it seems that bootadm has been compiled, or at least the distribution has been put together, without the ability to gracefully handle having the boot.lst on a separate partition.

A workaround is to move the /boot/grub/menu.lst file (which has some explanatory text in it) to  /boot/grub/menu.lst-old, and to symlink the /rpool/boot/grub/menu.lst file to /boot/grub/menu.lst.

cd /boot/grub
mv menu.lst menu.lst-old
ln -s /rpool/boot/grub/menu.lst menu.lst

Now we can run bootadm without any errors

bootadm list-menu

Time to add the boot entry for our xVM kernel

Update: Kevin Elliot said that running “bootadm -m upgrade”  added the xVM entry automatically after performing the linkage steps. You will still need to add the ‘boofs rpool/ROOT/opensolaris lines however. [link] [link]

open up the /boot/grub/menu.lst file  and add the following after the ‘bootadm’ section

vim /boot/grub/menu.lst
title OpenSolaris 2008.05 snv_86_rc3 X86 xVM
bootfs rpool/ROOT/opensolaris
kernel$ /boot/$ISADIR/xen.gz
module$ /platform/i86xpv/kernel/$ISADIR/unix /platform/i86xpv/kernel/$ISADIR/unix -B $ZFS-BOOTFS
module$ /platform/i86pc/$ISADIR/boot_archive

Note that if you’ve done a pkg upgrade, your ZFS root will more than likely be different. You’ll have to choose the ZFS root partition that you were using when you installed the packages earlier, otherwise they won’t be available

Note: Make sure to change the default selection to the appropriate placeholder of your xVM entry. The numbering for grub starts at 0. This means that to boot the first entry in your menu.lst file, your ‘default’ value in menu.lst should be 0, and so on and so forth.

Let’s run bootadm and see what our changes look like

bootadm list-menu
The location for the active GRUB menu is: /boot/grub/menu.lst
default 1
timeout 10
0 OpenSolaris 2008.05 snv_86_rc3 X86 i86pc
1 OpenSolaris 2008.05 snv_86_rc3 X86 xVM

Pretty. Now its time to try it out. In order to switch kernels you have to reboot your system

shutdown -y -g0 -i6

Your system should reboot using the xvm capable kernel. If your sytem could not boot properly then select the old kernel from the boot menu as your system starts.

To double-check that you’re running the proper kernel, run uname

uname -a

it should give you something like this

SunOS pluto 5.11 snv_86 i86pc i386 <strong>i86xpv</strong> Solaris

The last step is to enable the proper xVM services

svcadm enable store
svcadm enable xend
svcadm enable console
svcadm enable domains
svcadm enable virtd

Running

svcs | grep xvm

should print the following

online          2:22:12 svc:/system/xvm/store:default
online          2:22:27 svc:/system/xvm/xend:default
online          2:22:27 svc:/system/xvm/console:default
online          2:22:27 svc:/system/xvm/domains:default

One quick test

xm top

Should work properly.

Hopefully that worked for you. I appreciate any feedback!

Update #2

Some people are reporting problems creating DomU’s. If you’re running into problems, try running the script below

BASEDIR=${BASEDIR:-/}
/usr/sbin/syseventadm list -R $BASEDIR -c EC_xendev &gt; /dev/null 2&gt;&amp;1
if [ $? -ne 0 ]
then
/usr/sbin/syseventadm add -R $BASEDIR -c EC_xendev \
/usr/lib/xen/scripts/xpvd-event 'action=$subclass' \
'domain=$domain' 'vdev=$vdev' 'device=$device' \
'devclass=$devclass' 'febe=$fob'
fi
/usr/sbin/syseventadm list -R $BASEDIR -c EC_xpvsys &gt; /dev/null 2&gt;&amp;1
if [ $? -ne 0 ]
then
/usr/sbin/syseventadm add -R $BASEDIR -c EC_xpvsys \
/usr/lib/xen/scripts/xpvsys-event 'subclass=$subclass' \
'shutdown=$shutdown'
fi
# restart daemon if the package is being added to the running system
if [ "$BASEDIR" = "/" -a $? -eq 0 ]
then
/usr/sbin/syseventadm restart
fi

15
Mar 08

Open Virtual Machine Format

Open standardized protocols are what made the Web possible. We have standards all the way up the computational stack, from agreeing on which pins mean what in a wire, to what an X button means in a user-interface. Companies who don’t embrace them are destined to isolate themselves on tiny technological islands.  Specific implementations, however, don’t have to be shared and open. Huge markets with tiny verticals of implementation lock out competition, but they also prevent innovation. But sometimes something beautiful happens, and people get together to support a new kind of standard. An open, extensible standard that can be written and read by anyone. One place where this is just starting to happen is with computer virtualization.

The Open Virtual Machine Format, or OVF is a proposed universal format that aims to create a secure, extensible method of describing and packaging virtual containers. Because the standard is open, it means any environment supporting the standard can import and export those virtual machines between different hypervisor platforms. The current OVF specification includes definitions ranging from virtual machine metadata and disk format, all the way to detailed hardware specifications and logical network information. It also provides an ability for the virtual machine itself to get information from the hypervisor host, meaning that if you’re creative you could create some really nifty automated integration and deployment tools.

If that doesn’t mean much to you, then consider this: Industry heavy-weights like Dell, HP, IBM, Microsoft, VMware, and XenSource all took part in drafting the specification. As far support tools go, VMware has published what appears to be the first OVF container creation tool, available here.

There IS a big problem with OVF right now, and a lot of bloggers and analysts out there are getting it wrong. OVF is not and does not define a new virtual disk format, simply a wrapper around them. This means that OVF support doesn’t enable you to drag and drop virtual machines between Xen and VMware. Some formats can be converted externally using tools, however most of the current techniques involve booting up a system, and running a migration tool to be able to convert the image – not exactly ideal. OVF does include the ability to describe your specification in an HREF, which means that you could publish your spec, and create a system that could modify containers on the fly.

If Vmware, Xen, and Parallels are technolgical islands, then OVF may one day be the bridge that will allow you to travel between them.

Update It looks like OVF will be announced formally at the Catalyst 2008 conference. More information here.