howto

Development / Geek / General / howto / sysadmin — 1 Comment
12
Feb 10

Unfuddle Git Backups – How to Actually Use Them

I really like Unfuddle. The service is easy to use, and there are a lot of great features in there. The documentation is… lacking, however.

One of the things I like is the ability to get a full backup of all my project data, repositories, etc in a single tarball. You can even ask them to keep a copy in your own S3 account.

To create a backup do the following

1) log into Unfuddle and goto the Project page.
2) Click the ‘settings’ tab then
3) Scroll down till you see link that says ‘Request a backup of this project now’ link. Click it.

In a few moments you’ll get an email, and you’ll see a new link on the right hand side of your project settings page that includes a timestamped backup. This backup is a tarball that contains all the GIT repositories and some other files like a backup.xml file which looks like all your tickets.

To use the git dumps run the following

mkdir reponame
cd reponame
git init
git fast-import < ../my-unfuddle-backup.git.dmp
git checkout master

You’re done!

If you’re using subversion repositories there is documentation on how to use these repo backups on Unfuddles website.

Analysis / Development / Geek / General / howto / Twitter — 3 Comments
22
Apr 09

URL Shortener Statistics on Twitter

URL Shortening services have gotten a lot of attention because of Twitter. The reason is that most significant links on the internet are longer than 140 characters and that doesn’t leave you any room to actually say anything when you tweet. Twitter will shorten some URL’s for you using TinyURL, but what about all of the other services? What’s the state of the URL shortening service world on Twitter?

Of course, the best people to answer this question is Twitter themselves, but since they haven’t published anything, and I doubt they’re interested in spending time answering my almost academic question I decided to write a quick app to get the stats myself.

The Twitter API is gorgeous. I mean. It’s simple, easy to use, and it’s fast. I grabbed a list of all the popular URL shortening services from a few locations, cross-referenced them and came up with 36 “popular” services. The actual number of URL shorteners is pretty incredible. (Some of them even have long names like “shortna.me” WTF are you thinking?)

The process is simple – Search for the urls of the shortening services using the search API, then count how many hits you get. Repeat every 30 seconds or so. (There are other nuances – look at the code).  This sampling was done over a 3 hour period on Wednesday April 22nd between the hours of 8AM and 11AM Pacific.

Results

Twitter URL Shortening Statistics - April 22 2009

Twitter URL Shortening Statistics - April 22 2009

Issues

You can only return 100 search results at a time, which means if you see 100 results for your term, there are probably way more of them. The only way to get around this would be to search more frequently for that specific term. I tried doing that and was quickly throttled by Twitter. Suffice to say, TinyURL is still the dominant force here, but probably because of their Twitter integration. This really throws off the numbers. There’s a fine line between searching too often, and getting accurate results. Tests in the middle of the night showed that TinyURL had about 30% of the URL Shortening Service market on Twitter. I’m 99% positive that TinyURL numbers are way higher.

Things to do

Optimize the searching algorithm so it displays more accurate results.

Record the time, and see statistics over the course of hours, days, weeks. I can tell you off the top of my head that TinyURL usage is high all times of the day, where as bit.ly usage is pretty much a “waking hours” service, as most others.

Better Reporting

GitHub

The scripts are on GitHub. Go bananas. http://github.com/Trevoro/urihz/tree/master

General / howto / monitoring / sysadmin — No Comments
16
Nov 08

How to Monitor Usage Patterns in ActiveMQ

ActiveMQ is an enterprise message bus that’s completely open source. It’s great if you want to tie together a bunch of different services, or act as your own personal Simple Message Queue (SQS). It supports a few interface methods such as JMS, Stomp, XMPP and plain REST. You can learn more about ActiveMQ here.

There are a few monitoring solutions for ActiveMQ that will let you know when its broken, but I needed to grab usage patterns over time, so I would be able to automatically spin up more workers. I didn’t see anything quickly available so I threw this together:

#queuemonitor.rb

# queries the activeMQ status XML file and returns relevant data.
# I'm sure there's a more elegant way of creating method directly out of xml.
 
# Rather than have each queuemonitor as a seprate class with one name, we
# might want to be able to parse multiple queues on the same server. So we have
# an array of queue names (@queues) that contain the names of queues we want to monitor.
# if you want to monitor more than one queue then you'd have to create a new QueueMonitor
# instance.
 
class QueueMonitor
 
  attr_accessor :url, :queues
 
  require 'net/http'
  require 'rexml/document'
 
  def initialize
    @url="http://127.0.0.1:8161/admin/xml/queues.jsp"
    @queues = []
  end
 
  #add a queue to the list of queues that we'll pay attention to
  def addqueue(queue)
    @queues.push(queue) unless @queues.include?(queue)
  end
 
  def delqueue(queue)
    @names.reject!{|q| q == queue}
  end
 
  def query
    results = []
    date = Time.now
    xml_data = Net::HTTP.get_response(URI.parse(@url)).body
    doc = REXML::Document.new(xml_data)
 
    doc.elements.each('queues/queue') do |queue|
      name = queue.attributes["name"]
      # only list from Queues listed in the '@names'
      if @queues.member?(name) then
 
        queue.elements.each('stats') do |ele|
          size = ele.attributes["size"]
          consumers = ele.attributes["consumerCount"]
          enqueue = ele.attributes["enqueueCount"]
          dequeue = ele.attributes["dequeueCount"]
 
          queue = { 'name' => name,
                    'size' => size,
                    'consumers' => consumers,
                    'enqueue' => enqueue,
                    'dequeue' => dequeue,
                    'date' => date
          }
 
          results < < queue
        end
      end
    end
    return results
  end
 
end

You can then start monitoring your queue servers with something like this:

#!/usr/bin/env ruby
 
require 'queuemonitor'
 
mfreq = 30
@monitor = QueueMonitor.new
@monitor.url = "http://67.202.41.64:8161/admin/xml/queues.jsp"
@monitor.addqueue "pubsub.pings.spider"
 
#do something interesting like write to a file
#screen or database
 
def query
  date = Time.now
  results = @monitor.query
  results.each do |q|
    puts "#{date} #{q["name"]} #{q["size"]} #{q["consumers"]}"
  end
end
 
#You'd want to use daemonize here...
 
loop do
  query
  sleep mfreq
end

It works pretty well. Right now I’m just writing the results to a file to parse later, and so you can setup other parts of a program that would automatically spin up new workers based on a set of circumstances. The algorithm for that is the hard part and will depend on a bunch of your own rules.

Geek / howto / opensolaris / sysadmin — 25 Comments
7
May 08

Getting xVM to work in Opensolaris 2008.05

os_com_logo.jpg

OpenSolaris 2008.05 came out the other day, and its pretty nice as far as OS’s go. One of the things that’s great about OpenSolaris is by coupling Xen with ZFS you can get a really powerful virtualization system.

Unfortunately because of space constraints in a liveCD, you can’t squeeze all those ‘nice to have’ packages into one spot, so you have to download and setup xVM manually. A few things are broken, too so some minor tweaking is required. Here’s a list of steps required to get xVM setup as a Dom0 in OpenSolaris

1) Become root and install the required packages. This will install the xen.gz kernel, as well as all the other utilities and services necessary for running a Dom0

pkg install SUNWxvmhvm
pkg install SUNWvirtinst
pkg install SUNWlibvirt
pkg install SUNWurlgrabber

2) The package installation doesn’t support adding xvm to the default boot.lst in grub. Additionally, OpenSolaris 2008.05 has ZFS as the default filesystem, so a few things have been moved around. Specifically, the boot.lst file used by grub and bootadm has been placed on its own zfs filesystem. Unfortunately it seems that bootadm has been compiled, or at least the distribution has been put together, without the ability to gracefully handle having the boot.lst on a separate partition.

A workaround is to move the /boot/grub/menu.lst file (which has some explanatory text in it) to  /boot/grub/menu.lst-old, and to symlink the /rpool/boot/grub/menu.lst file to /boot/grub/menu.lst.

cd /boot/grub
mv menu.lst menu.lst-old
ln -s /rpool/boot/grub/menu.lst menu.lst

Now we can run bootadm without any errors

bootadm list-menu

Time to add the boot entry for our xVM kernel

Update: Kevin Elliot said that running “bootadm -m upgrade”  added the xVM entry automatically after performing the linkage steps. You will still need to add the ‘boofs rpool/ROOT/opensolaris lines however. [link] [link]

open up the /boot/grub/menu.lst file  and add the following after the ‘bootadm’ section

vim /boot/grub/menu.lst
title OpenSolaris 2008.05 snv_86_rc3 X86 xVM
bootfs rpool/ROOT/opensolaris
kernel$ /boot/$ISADIR/xen.gz
module$ /platform/i86xpv/kernel/$ISADIR/unix /platform/i86xpv/kernel/$ISADIR/unix -B $ZFS-BOOTFS
module$ /platform/i86pc/$ISADIR/boot_archive

Note that if you’ve done a pkg upgrade, your ZFS root will more than likely be different. You’ll have to choose the ZFS root partition that you were using when you installed the packages earlier, otherwise they won’t be available

Note: Make sure to change the default selection to the appropriate placeholder of your xVM entry. The numbering for grub starts at 0. This means that to boot the first entry in your menu.lst file, your ‘default’ value in menu.lst should be 0, and so on and so forth.

Let’s run bootadm and see what our changes look like

bootadm list-menu
The location for the active GRUB menu is: /boot/grub/menu.lst
default 1
timeout 10
0 OpenSolaris 2008.05 snv_86_rc3 X86 i86pc
1 OpenSolaris 2008.05 snv_86_rc3 X86 xVM

Pretty. Now its time to try it out. In order to switch kernels you have to reboot your system

shutdown -y -g0 -i6

Your system should reboot using the xvm capable kernel. If your sytem could not boot properly then select the old kernel from the boot menu as your system starts.

To double-check that you’re running the proper kernel, run uname

uname -a

it should give you something like this

SunOS pluto 5.11 snv_86 i86pc i386 <strong>i86xpv</strong> Solaris

The last step is to enable the proper xVM services

svcadm enable store
svcadm enable xend
svcadm enable console
svcadm enable domains
svcadm enable virtd

Running

svcs | grep xvm

should print the following

online          2:22:12 svc:/system/xvm/store:default
online          2:22:27 svc:/system/xvm/xend:default
online          2:22:27 svc:/system/xvm/console:default
online          2:22:27 svc:/system/xvm/domains:default

One quick test

xm top

Should work properly.

Hopefully that worked for you. I appreciate any feedback!

Update #2

Some people are reporting problems creating DomU’s. If you’re running into problems, try running the script below

BASEDIR=${BASEDIR:-/}
/usr/sbin/syseventadm list -R $BASEDIR -c EC_xendev &gt; /dev/null 2&gt;&amp;1
if [ $? -ne 0 ]
then
/usr/sbin/syseventadm add -R $BASEDIR -c EC_xendev \
/usr/lib/xen/scripts/xpvd-event 'action=$subclass' \
'domain=$domain' 'vdev=$vdev' 'device=$device' \
'devclass=$devclass' 'febe=$fob'
fi
/usr/sbin/syseventadm list -R $BASEDIR -c EC_xpvsys &gt; /dev/null 2&gt;&amp;1
if [ $? -ne 0 ]
then
/usr/sbin/syseventadm add -R $BASEDIR -c EC_xpvsys \
/usr/lib/xen/scripts/xpvsys-event 'subclass=$subclass' \
'shutdown=$shutdown'
fi
# restart daemon if the package is being added to the running system
if [ "$BASEDIR" = "/" -a $? -eq 0 ]
then
/usr/sbin/syseventadm restart
fi