Derailment

Two posts in a week... hold on to your hats.

The purpose of this post is two-fold. First, I'm curious who is wandering by to peruse my screed. Second, I've got a hankering for a top-ten sort of list, a la High Fidelity. My topic is iconic voices of 80's hard rock and metal. Twenty years ago saw the height of glam metal, but for all its ridiculous excess, there is a soft spot in my heart (and my ears) for a good, honest, bigger-than-life metal masterpiece. There were also some damn good bands that managed to stay good despite the post-80's decline in popularity.

So anyway, enough cheesy reminiscence. I'm splitting my list into hard rock and metal, because, well, I feel like it. It makes room for more entries. These are also in no strict order, except maybe the order in which I pull them from my brain. I'm not out to list the best, or the most influential, only the most memorable or recognizable. The voices that, when you hear them, instantly identify both the singer and the band. In other words, this is totally subjective, so make up your own mind. I'm also sure that I've left out some notable names, but non one's perfect.

On to the lists:

Hard Rock

• Sammy Hagar, Van Halen


• Bret Michaels, Poison


• Brian Johnson, AC/DC


• Stephen Pearcy, Ratt


• Sebastian Bach, Skid Row


• Vince Neil, Motley Crue


• Axl Rose, Guns N' Roses


• Steven Tyler, Aerosmith


• Jon Bon Jovi, Bon Jovi


• Joe Elliott, Def Leppard

Metal

• Bruce Dickinson, Iron Maiden


• Rob Halford, Judas Priest


• James Hetfield, Metallica


• Dave Mustaine, Megadeth


• Geoff Tate, Queensryche


• Lemmy, Motorhead


• Ozzy Osbourne, Black Sabbath/solo


• Klaus Meine, Scorpions


• Ronnie James Dio, Dio


• Glenn Danzig, Misfits/Samhain/Danzig

Drop a line with your thoughts.

I'm fortunate to work at OmniTI for a multitude of reasons, not the least of which is the incredible group of people with whom I interact every day.

Geeks love to express themselves, and there's no better place to wear your creds than on your bumper.

Ben Franklin's famous quote about liberty versus safety frames Bruce Schneier's latest column for Wired, in which he sums up post-9/11 politics as a false dichotomy between security and privacy. As is typical for Bruce's writing, it is full of links to supporting information and is well worth your time to explore.

We do indeed have a reason to be terrified in this country-- we should be terrified that our republic may well be undone by the very government that exists to safeguard it. As long as we're quoting 18th-century luminaries, I'll add another favorite. In a 1783 speech to the House of Commons, Prime Minister William Pitt said:

"Necessity is the plea for every infringement of human freedom. It is the argument of tyrants; it is the creed of slaves."

I couldn't agree more.

Where's my jetpack?

Happy 2008. We're a little behind, you know. We're supposed to be going to Jupiter by 2010. As it stands, we might make it to Mars in my lifetime. Apparently we're too busy with all the other pressing matters before us.

Computer technology is a fast-paced industry. Even though I'm directly involved in it, I'm still sometimes unaware of just how quickly it evolves. Today I was thinking back to my first semester of college, which began 15 years ago this month. Yikes. I already feel some perspective coming on, the kind where you find yourself saying things that begin with, "In my day..."

In techie circles, whenever you reminisce about some old computer you used to own, or what life was like at 9600 baud, someone invariably counters with their own, even older, or slower, computer/modem/whatever. "Oh yeah? In my day, we carved our own circuit boards out of WOOD!" I'm not out to win any competitions; I just enjoy looking back and appreciating how far we've come. What follows is some anecdotes about how I remember encountering the technologies that would one day become my profession.

I went to a a small liberal arts college that was certainly not at the cutting edge of technology, but had what I think was an average set of computing resources for the time. In 1992, there was no wifi. Very few, if any, people had laptops. Not all students even had a computer. If you wanted to use a computer that was networked, you had to go a computer lab. No one had heard of MP3. The portable music player you were most likely to see would be playing cassettes.

My computer was a 286 that ran at a cool 16 MHz, had 4MB of RAM and a 40MB hard drive. I remember that the CPU didn't even have a heatsink. A CDROM drive would have been a luxury. I had no modem (and no money to pay for an online service even if I had one). It ran Windows 3.1, and I used it mainly for writing papers (and playing Wolfenstein 3D.) I had a dot-matrix printer that was so slow, I had to budget printing time when working down to the wire on an assignment. Professors had started insisting on true-type fonts, because the built-in printer fonts were too hard to read. My printer took anywhere from 30-60 minutes to print out a reasonably-sized term paper.

My first exposure to email was about mid-way through the year, when a friend told me about it. There were a number of computer labs around campus, but the one I used most was in the library. Access was mainly through VT220 text terminals, which we used to log into the university's main system, which was an HP of some sort running HPUX. At the time, I had only been using a PC for a couple of years, and understood nothing of Unix or non-PC hardware. I learned by example-- how to log in, how to start programs to read email, browse Usenet (back when it was actually useful) and Gopher. Only a handful of people I knew had email accounts. Most professors didn't use it. I would go days without checking my inbox. Spam was unheard of. There was no instant messenger (unless you count the Unix 'talk' program).

The World Wide Web was in its infancy. At first, my only exposure to it was via a Gopher gateway. There was very little content. If I'd had a graphical browser (which I didn't see for a couple more years), the whole web would have looked like this. Most people outside of universities and large corporations had never heard of the Web, or even the Internet, for that matter.

W&L had a single 56 Kbps line for internet access, for the whole school. Of course, when almost all applications were text-only, it wasn't that bad. I think it was my third year (1994-95) that they upgraded to a T1. The fiber-optic internet service that I recently got is 10 times faster than a T1.

My freshman dorm room had a single analog phone line, direct from the local phone company, that was not paid for by the university. By my second year, the university had a real phone system. It was a ROLM system (don't ask me how I remember that.) The phones had a serial port that could be attached to your computer to provide a Unix login on your PC, via a DOS terminal emulator that you could pick up for free on a floppy disk at the campus bookstore. Now I didn't have to walk to the library to check my email or do anything else online, and it was faster-- a whopping 19200 bps. Ethernet in dorm rooms didn't arrive until the fall of 1996, after I'd graduated. Bummer. Not that I would have been able to use it anyway-- by the time I graduated, I was still using a 386 and Windows 3.1. I didn't even get a modem until after I got out of school. My next system was the first one I ever had with an ethernet port.

These days, we can hardly imagine a world without constant email, web sites, IM, wifi, MP3s and iPods. Laptops are standard for college students, and universities are much more wired than ever. Even the cheapest laptops come standard with wireless networking, a 10/100 ethernet port, and a screen larger and with higher resolution than my old 14" CRT, and they weigh only a few pounds. The printer around the corner in my office can spit out several pages of text in the time it takes me to walk over to it. We think nothing of downloading gigabytes of files that would have completely overwhelmed the storage capacity of even a high-end system 10 years ago.

What state-of-the-art stuff from 2007 will we look back on fondly from 2022, I wonder?

Recently I saw one of Shell's Real Energy TV ads. I have TiVo, so I'm used to zipping through ads, but this one caught my eye because it was a good bit longer than the usual ad. This spot featured a Shell engineer working in Southeast Asia, who was being interviewed by a local reporter about the work Shell is doing there. I won't go through the whole story, but to summarize, after the interview, the engineer is sitting in a restaurant with his teenage son, who is drinking a milkshake. The kid slurps the last bit from the glass and his dad says, "Do that again." He has a "eureka moment" which leads to a design for a flexible drill that can snake into otherwise unreachable places to extract oil.

On the surface, the spot (and Shell's overall campaign) is about human creativity and how Shell is employing such creativity to meet the world's energy demands. The protagonist in the ad speaks of how "The challenge is to get to the world's reserves that we know about, but haven't yet been able to reach. Reserves that would otherwise go to waste." I got quite the opposite impression. I immediately thought about how increasingly expensive it is to go after more marginal oil reserves, and how Shell's prodigious wealth (and apparently boundless creativity) might be better spent devising a way to reduce the amount of carbon released into our environment. Untapped oil reserves are not a waste. They are a savings account. They represent millions of years of carbon sequestration, in the form of plant and animal matter trapped underground, pressed and heated until only concentrated hydrocarbons remain. Now we've managed to release these millions of years' worth of carbon in just a couple of centuries.

Humans can be incredibly creative when they have to survive, but we are also tragically short-sighted. Our brains are still largely wired for getting out of the way of predators, not assessing long-term risk and making the choices required to mitigate them. What we need are different solutions to the world's energy needs, not simply more of what got us into this mess in the first place. I fear that we won't really commit to finding these solutions until we've slurped every last drop out of the ground.

As any admin knows, OS upgrades can be painful. Despite the best intentions and efforts by the vendor, bad things happen. This is especially true with kernel upgrades. It's one thing to run a blanket "yum update" or "apt-get upgrade" on your average Linux desktop, but even the tamest patch can ascend to truly ulcer-inducing levels when it applies to a critical system that ill afford any extended downtime.

It was with such trepidation in our hearts today that we at the $DAYJOB faced the necessity of patching a critical server running Solaris 10 11/06. The impending patch was for the kernel, and it was a dependency for almost all the other outstanding patches. It had a big, fat, hairy disclaimer that it might do anything and everything short of setting the system on fire, just so we were aware. Gee, thanks. Granted, this was not really uber-critical, because the system in question isn't in production yet, but its current state represents many man-hours of careful application setup and tuning, not to mention data import (its primary function is serving a half-terabyte Oracle instance.) Having to start over from scratch would be a significant setback. Furthermore, once it is in production, you can bet we'll one day be in the same position, and it really will be critical to minimize downtime.

Enter Live Upgrade. Very briefly, LU enables the administrator to either copy the existing boot environment, or install a completely new environment, to an alternate location, all without disturbing the running system. We knew about LU and had saved a slice during jumpstart that matched the size of /. This would give us the chance to duplicate our boot environment, apply the necessary patches, and do a test boot on the updated copy, while retaining the unpatched version, in case the proverbial mass hit the cooling device.

Our filesystem layout did not leave enough free slices to duplicate /var and /opt, however. Not to worry, these directories actually had far less data in them than we originally budgeted for, so we could merge all three into our one LU slice for the test. What followed was very nearly a textbook application of LU. It worked like a charm, and the patches turned out to work just fine. Ulcer avoided!

We chose not to make this the official layout of the system, because /var was still part of /, which is sub-optimal. Another stroke of fortune (or incredible foresight... yeah, that's it) was to make all four slices (/, /var, /opt, LU) the same size. We could stand having /opt folded back in, because we'd already installed all the significant apps and knew it wasn't going to grow much larger. So, giddy with success, we devised a clever scheme to redo the layout with two slices for the OS and two identical slices for future LU use. It involved another round of LU to get rid of the old /, /var, and /opt slices and constitute a version that combined / and /opt on the first slice and put /var on the second, leaving the third to become a future LU location. When the dust settled, we had the layout we wanted, with two fresh LU slices for future acid-reflux relief, and the total downtime was between 20-30 minutes. Very cool.

Looking to the future, the prospects for LU seem even brighter with the advent of root-on-ZFS (currently available as of Solaris Express b62). If you have everything on ZFS, then you don't need spare slices for anything-- a snapshot becomes your LU environment. Break it, and you just roll back. Even if it's not quite that simple, it's still a quantum leap beyond what we can do today in mainline Solaris. I can't wait.

At the $DAYJOB we use a lot of open source code, as do the vast majority of our clients. While not always the best solution, it often permits us to build excellent solutions at a low cost. This week, we launched OmniTI Labs as a place to collect various bits of code developed in the course of our business and share them with the community. We have some phenomenally talented people on staff, so expect good things.

I'm proud to have had a small part to play in one Labs project to this point, Zetaback. Zetaback is a backup utility for ZFS. It's the brainchild of Theo Schlossnagle. It's still in alpha, but we're running it internally with good results.

OmniTI Labs will be a great place to find lots of other useful tools that may be just what you're looking for to solve your problem.

They say a picture is worth a thousand words. But the lack of a picture can speak as much, if not more. I came across this observation today. I don't need to say anything else, really.

I consider myself a pretty good speller. I recognize that not everyone puts the same degree of emphasis on correct spelling as I do. Nevertheless, I think that if you're going to write something that will be viewed or used by the public, then you should make sure it's correct. This applies especially to program code. Read on for some fun.

I've been trying to get Cacti to graph ZFS statistics like read/write bytes and ops, and space usage. I've got a custom script that gets the stats, indexed by zpool name, similar to what you get with an SNMP query. To integrate this into Cacti, I have to provide an XML description of how my script query works and what its output looks like. No biggie, right? I thought so. I got all the pieces in place and looked at the results. There were three indexes for the host, corresponding to the three ZFS pools configured there. But the names were missing.

Well, as it turns out, there's a wrinkle. One of the standard pieces of the script query XML syntax is the output delimiter. Since the script query uses colons, I did

<output_delimiter>:</output_delimiter>

Yes, there are two "i"s in "delimiter", as you can see in the dictionary. The root of the word is "limit". I am not aware of any variations in its spelling. Note how it is spelled in the manual. That's not just a typo in the docs... it's in the code too:

if (preg_match("/(.*)" . preg_quote($script_queries["output_delimeter"])

That's an excerpt from lib/data_query.php in version 0.6.8h.

The end result is that my well-formed XML document was nevertheless being misread by the code, failing to find the output delim-E-ter, and not splitting the output properly. Once I "fixed" my XML, everybody was happy. But that's 30 minutes of my life I'll never get back.

Spelling is doubly important if you're writing code to match what others will write!

At the $DAYJOB, we have a cluster of build systems that we use to test source trees on various platforms, including FreeBSD, Linux and Solaris. Some of these machines are fairly old, and have relatively puny hard drives by current standards, yet they continue to do their job just fine. We've simply begun to run low on local storage, as the source trees grow and developers need to work with multiple branch and tag checkouts.

Rather than try to boost local storage with extra drives (not a surefire solution everywhere), we focused our attention on a shared storage solution. The idea was to reuse some decommissioned hardware and create enough storage for every build system to be able to mount its home directories and give developers the space they need to do their work. We'd been anticipating the availability of Sun's new ZFS filesystem in the 6/06 release of Solaris 10. It's a promising solution, offering superior performance and reliability without the need for expensive RAID hardware. Oh, and we get built-in compression too, which maximizes our use of available space. All was not roses, however, as we ran into a pretty significant performance issue, but overall, it's been a success. More on that below.

The hardware we had on hand was a modest 1U dual-Xeon server with a single internal ATA disk, and an empty Sun D1000 SCSI shelf with 12 drive bays. The D1000 has two differential SCSI channels and can be configured as a split bus, with six devices on each channel. We wanted to maximize the available I/O bandwidth, so we went with the split bus setup. Having only one PCI slot in our 1U server, we needed a dual-channel differential SCSI card. An LSI U40HVD was just the ticket-- $50 on Ebay. Also on Ebay, we picked up a couple of VHDCI-to-HD68 cables for $28 each to complete the connection. Lastly, it was back to Ebay for a lot of 12 36GB, 10K rpm SCA SCSI drives for $420. Total outlay for hardware: $526.

After installing Solaris on the internal ATA disk, we had 12 disks at our disposal. To maximize available space while retaining fault tolerance, we chose the RAID-Z model. Setting aside one disk on each channel as a spare, we were left with 10 disks, which should yield somewhere around 300GB of usable space.

The zpool creation involved two raidz vdevs, since the ZFS Administration Guide recommends that a raidz vdev contain fewer than 10 disks for optimal performance.

# zpool create homepool \

     raidz c0t0d0 c1t0d0 c0t1d0 c1t1d0 c0t2d0 \

     raidz c1t3d0 c0t3d0 c1t4d0 c0t4d0 c1t5d0

The final tally was 332 GB of available space. Cool!

Making the filesystem hierarchy was as easy as running a shell loop:

# for i in 1 2 3 5 6 7 9 10 11 12 15 16 17 18 20 21 22; \

    do \

    zfs create homepool/buildbox-${i}; \

    zfs set compression=on homepool/buildbox-${i}; \

    zfs set sharenfs=anon=0 homepool/buildbox-${i}; \

    done

I don't recall exactly how long that took, but I'll wager that I would not have run out of fingers if I had been counting.

The whole process of creating ZFS storage was delightfully easy. Each filesystem will employ compression and will be automatically mounted at boot and shared via NFS. We pass the "anon=0" option to sharenfs so that the root user can access files on the mount (developers need to "sudo make install" in their source trees, for example).

Mounting each export was not terribly difficult either, but there were some pitfalls. I quickly ran into a performance problem with NFS on ZFS, in the ZFS Intent Log (ZIL). As explained in the post linked below, the ZIL tracks file modifications that have immediate integrity requirements (such as fsync). Each write operation has a sequence number, and when a synchronous command comes in, the ZIL commits all I/O blocks up to that sequence number. However, as noted in this excellent blog post from Sun kernel engineer Roch Bourbonnais, the current state of the code is such that an fsync issued by one process causes all pending data for the filesystem to be flushed to disk before the fsync can return. When copying a large number of relatively small files (like a source checkout) over NFS, the sheer number of fsyncs causes a lot of blocking, making the overall operation painfully slow. I had to resort to tarring up the home directory contents, copying the tarball to the server and unpacking it locally.

During the process of converting the build machines to use the NFS exports, the server inexplicably locked up, seemingly due to problems with the ZFS filesystems. No clients could access the mounts, and even locally on the server, a 'df' would hang on the ZFS filsystems. There were no messages on the console, and since I did not have the foresight to launch the kernel debugger at boot time, I was stuck. I ended up having to do a warm reset to recover. The silver lining was that I had no data corruption as a result. Still, it was disturbing and I was unable to come up with an explanation.

A couple of non-ZFS related pitfalls also cropped up, namely that I had to synchronize all the UIDs with the server, and that on Solaris hosts, I had to tweak domain search order settings as noted in this blog post. Also on Solaris hosts, I had to disable the "auto_home" feature in /etc/auto_master, otherwise the /home or /export/home mountpoint would be occupied "invisibly", preventing the NFS mount from succeeding after boot.

So, in the end, we have an operational home directory server that cost us only around $500 to set up, which provides enough storage for the forseeable future, and is easily manageable (I can use ZFS to reserve space for some filesystems at the expense of under-utilized ones, for example). I'm sure that future updates to ZFS will take care of the performance problems, and I'm now booting with kmdb in the background in case that mysterious hang manifests again.

I'd like to thank the folks on the #opensolaris IRC channel for helping track down the cause of the NFS performance problem.

We've all seen and heard the response to the UK terror plot exposed last week. Apparently I'm not the only one who thinks we're now closing the barn door after the horse is long gone. Bruce Schneier's recent op-ed for the Minneapolis Star-Tribune reminds us that real security doesn't focus on terrorist tactics, but on the terrorists themselves. "Last week's arrests demonstrate how real security doesn't focus on possible terrorist tactics, but on the terrorists themselves. It's a victory for intelligence and investigation, and a dramatic demonstration of how investments in these areas pay off."

I'm a big fan of the BOFH, otherwise known as the Bastard Operator from Hell. It's a fun column about a sysadmin who is assaulted daily by clueless users and an even more clueless manager. He delights in exacting poetic revenge through a range of suitably bastardly deeds.

This clever stunt should rate a smile from any aspiring BOFH. This fellow's neighbors were "borrowing" his internet access via wireless. Instead of merely blocking them, he decided to have a little fun with iptables, a transparent proxy, and some image manipulation. There are pictures of the result.

Clever.

Just read a fascinating post which describes an AP report on how fake IDs actually help people survive sectarian violence in Iraq. Since Iraqi surnames refer to tribe and clan, you can be targeted as a Sunni or Shiite by name alone. Being able to travel under an assumed name makes your daily life safer. It's an interesting real-life example of the law of unintended consequences.

Security is a topic I frequently think about. Computer system security is part of my job as a sysadmin, but I enjoy thinking about it as a larger socioeconomic issue. Good security reading can always be found at Bruce Schneier's blog.

You'll find some links to other blogs in the Bookmark section to the left. Been a bit too busy to post anything for a while, but I've got some neat projects in the cooker, and my birthday is coming up. Whee!