This interview was prompted by QRZ.com‘s recent move into “the cloud”. QRZ means “Who is calling me?” or “You are being called by ___”, which is very appropriate for what is widely considered to be the largest online community for amateur (“ham”) radio in the world. Moving this resource from traditional hosting into the cloud is an interesting comment on the readiness of the cloud to actually deliver for a community that has come to depend on this resource.
The computer and ham communities have a long history together. The original “hacker” community originally had quite a few ties to ham radio and computers, as all were involved with experimenting, especially with electronics. In fact, one possible origin for the term “hacker” its use by the amateur radio community from the 1950s to mean “creative tinkering to improve performance”. This continuing curiosity and desire to build and improve is a hallmark of these communities.
I’ve encountered a few system and network administrators who are hams, and vice versa. QRZ’s founder and publisher, Fred Lloyd, is no exception. Fred spent much of his career on the cutting edge of Internet adoption, working for Sun and other companies in Silicon Valley and other locations. As it turns out, he’s been a ham radio operator about as long.
Fred was kind enough to do an email interview with me earlier this week to discuss system administration, QRZ, ham radio, the Internet and his experiences in moving to the cloud.
How did you get involved in QRZ.com? When did QRZ.com go live? What led you to create an Internet presence for radio amateurs?
QRZ began as a hobby for me, first on a dial-up bulletin board in 1992. We went live online (qrz.com) in 1994. Additional info is at http://www.qrz.com/i/about.html
When QRZ launched, how many users did you have, and how did people find out about it? What kind of hardware did you use, which software, and how were you connected to the ‘net?
QRZ was started back when the internet was relatively new. We were one of the first 25,000 websites. We’ve simply been around for so long that we’re in every search engine by default.
Our original hardware (1994) was a SparcStation 1, 16mhz with 4mb of RAM and a 40MB hard drive. We were originally co-located with a small ISP who had a T1 line. In 1995 we got our own T1. In 2000 we had to go with dual T1’s. In 2003 we went to 10mb in a co-location datacenter. Today we’re at Amazon.com with huge bandwidth.
The original software was custom written in C. Later we moved to Informix and then to MySQL. Today most of the programming is done in Perl, with some PHP.
How many people does it take to run the site and what are their roles? Administrators, moderators, editors?
I am the technical manager. There are half a dozen application administrators and about 20 moderators. All are volunteers.
I saw these statistics today (29 jan 2011):
- Threads 236,966
- Posts 1,537,047
- Members 428,791
- Active Members 39,415
Did you ever think you’d get to over 400K users? Has the growth rate been faster or slower over the past few years?
No, I never really imagined more than a few thousand. The growth rate has been steady over the past few years, perhaps slowing down a bit as we capture nearly 100% of amateurs that are on line.
Which features of QRZ have been the most popular? The least?
The callsign database is #1. The Forums are #2, and Practice Tests are very popular.
What’s your background? When did you become a ham, and when did you first get on the Internet?
I became a ham in 1986 while living in Silicon Valley. My first call was N6UFT, then KJ6RK, then AA7BQ when I moved to Arizona in 1988. I had my first email account in 1986. Since I was a computer/software engineer, I’ve always been on the cutting edge.
What are your favorite aspects of ham radio? Of the Internet?
I like HF [High Frequency] however don’t get on the air very much. I’m on the internet, however, almost continuously.
Do you have any background in system administration? In programming?
Yes, I was in a senior technical position at Sun Microsystems for 18 years.
What were your roles at Sun?
I started with them in 1988 as a software engineer in SunOS. I wrote parts of the Suninstall program (tape, back then).
Then, I transferred to Phoenix in 1989 and was a Field systems engineer for about 10 years. Then I went into Sun Professional Services and became a district manager. Later, I went into IT and became their worldwide datacenter manager, in charge of 2000+ servers.
You mentioned starting on Sun HW with SunOS. Which Sun hardware and OSes have you used over the years?
I started on a sun 2/60 back in 1985. At Sun I used or worked on almost every model up until 2007 when I was laid off. My own Sun hardware (recently retired) includes a V20z, a V40z, 2 X4100 and an X2100. I’ve also owned a bunch of others such as the 440R, the 280R, SS-1’s, SS2’s, SS-10, Ultra 2’s, and a V210. Most of them were bought used from various leasing companies at a substantial discount, less then the employee rate. Just about every year I’d add at least one new machine and retire an older one. My best find ever was the v40z with 4xCPU, 16GB Ram, 4x146gb drives for $800. It ran the Forums until last week.
How was the transition from SunOS/Solaris to Linux? Any special hurdles you recall?
It was nothing, really. I just had to learn a few new names for the package utilities. QRZ always ran on Solaris, and most recently an S-10 (2007) but I used a lot of Linux, mostly Fedora, on other projects. The new QRZ runs the Ubuntu 64-bit server. I’ve been happy with that and I like the Debian package utilities. I really don’t miss Solaris at all.
At Sun, I was the Solaris Ambassador for about 6 years, and was one of the first to evangelize Solaris 2. I have a special-edition, gold CD and Box set signed by Bill Joy to commemorate the launch. My wife thinks it belongs in the junk box, or thrown out.
What was your primary motivation for moving into the cloud?
I went into the cloud so I would no longer be tethered to a physical location. I also like the virtualization as well. When something goes wrong, just start another server [instance] and toss the old one.
Where there any special challenges in moving from hosting on physical hardware to virtual cloud instances?
The most awful thing was trying to understand how images worked, and what they actually consist of. The idea of ephemeral storage is still a bit fuzzy to me. I had to rewrite a lot of the QRZ server code to take advantage of S3, which has helped performance dramatically. I’m still a bit fuzzy about the relationship between a volume, snapshot, AMI, etc., and how they are interchangeable. I’m an old “copy the disk using ‘dd’ guy”, so this takes some study time to figure out. Fortunately, I think I have it now, at least to the point to which the system actually runs. I wish I had a mentor for AWS. Their website and forums are a bit less than informative sometimes.
With the latest upgrade, you’ve put everything in the cloud. How did you come to that decision? Was it a pure technical decision, or were there support or economic factors?
I like to travel and don’t like the idea of owning hardware that might break. Since I’m the only technical person at QRZ, it gets scary when I’m out of town, or even out of the country. I’ve bought a lot of servers over the years, at least a new one every year. The economics however, favor renting over owning and paying for a co-location space.
What does you cloud footprint look like in terms of instance counts and types of instances? What software are you using for the overall site and the forums? Databases?
I’m using 1 AWS Relational Database instance and 2 EC2 instances. The RDB is an m1.xlarge, plus one EC2 m1.large (forums w/PHP) and an EC2 m1.xlarge (callsign lookup w/Perl) Everything is MySQL 5.1. The Forum software is vBulletin – latest edition.
Is there any automation in the administration of the site? I’m looking more for information on puppet, cfengine, kickstart, or other system administration tools than on the automation within the application itself.
I don’t use any of those and instead have a collection of roll-your-own scripts that do several things for me.
What have been the biggest technical challenges over the years? Hardware, network bandwidth, software?
Software has always been the most time consuming. Performance and reducing the memory footprint is always a concern. When an EC2 system runs out of memory, it just hangs. Not good.
What kind of uptime have you had and what kind do you expect to see in the cloud?
My self-hosted uptime has been remarkable. Some of my servers have gone two years between reboots. We haven’t had an actual hardware failure that I can recall. We used Sun servers, mostly 1 and 2U sizes. 3 machines total in the mix.
How do you monitor the health of the site? Cacti, or Nagios or ???
vmstat, iostat mostly. Hey, I’m a sysadmin, I just login and look!
What do you see as the future of QRZ?
QRZ should continue to grow with a more modern look, faster response times, and more subscribers.
How have you seen the Internet change ham radio? For the better or worse? How do you see ham radio in the future, and does that vision involve the ‘net?
Cell phones changed ham radio. Kids are no longer excited by two-way radio communications. The Internet is the perfect tie-in for ham radio and it has changed the hobby by a great deal. Most of what I know, however, is from the QRZ vantage point. I don’t think that I can answer the question objectively.
What questions have I not asked? What things would you like your users (and the article readers) to know?
That QRZ is now poised to be even more reliable than before and that I will continue to develop new features as long as I’m able, which should be for many more years.
Thuktun (Message) 