The sysadmin's journey: A series of unexpected events

"It's not magic. It's talent and sweat." -- Gilfoyle

The path to becoming a sysadmin is different for everyone. There is no single university degree that takes you directly to the job. In most cases, experience forges each sysadmin's character, with self-learning as one of the essential skills during the journey.

Twenty one years have passed since I began my sysadmin journey. Over the years, I have found four pillars to guide me. These are not laws written in stone, but rather ideas to develop that have always helped me improve my sysadmin knowledge. What started as a hobby involving computers and open-source became a lifestyle, evolving as I gained more experience and knowledge.

As part of the 21st annual System Administrator Appreciation Day celebration, I want to share these four pillars to help you improve your skills, just as they did with me.

Backup

Always back up. Also, make sure that the backup job works for a restore. It seems logical, but experience shows us that many recent and historical failures could have had a different outcome if there had been a useful backup.

Challenge: storage

At the beginning of my career, storage was a challenge. This was true in general, not only for backups. The situation has changed and has even been reversed. In some projects I have found many terabytes of unused disk space. In these cases, we can take advantage of new technology trends and use software-defined storage (SDS) to better manage storage.

Proposed solution:

Consider using GlusterFS or Ceph to help you manage storage space.

Console

Console : Terminal : CLI : Shell

No matter what you call it, fluency at the command-line is an often-neglected skill. It is sometimes considered archaic. Still, command line expertise enhances your flexibility and productivity as a sysadmin in both obvious and subtle ways.

My good friend Kaz defines the following four essential sysadmin console skills:

1. Use the console: Type commands, string them, concatenate, make scripts.
2. Know the text editors: nano, vim, emacs.
3. Use regular expressions: Put the most stilted patterns into the most optimized scripts.
4. Ask the one who knows: Use resources in the following order: --help → man → info → /usr/share/doc → Google. It is very common, especially for newbies, to search for help in the opposite direction. The best way to learn is to consult the console for command options, by using --help, man, or info. If you have the doc package installed, review the package documentation. If you don't find the right answer, then it is time to search on the Internet.

Challenge: practice

Tip: Start with your day-to-day tasks.

For example: purge the directory with the most used disk space in the file system.

• List files and directories.
• Estimate the disk space used by each of them.
• Sort by disk space used (from lowest to highest).
• Create the list in human format (e.g., 1K, 234M, 2G).

ls | xargs du -sk | sort -n | awk '{ print $2 }' | xargs du -sh

The mother of skill is practice. The only way to master commands is to practice, practice, and practice.

Study

Study time seems like a joke, especially with the sysadmin's usual workload. However, there is a keyword for this: automate.

It is said that the lazy sysadmin is the best sysadmin. The concept of laziness is considered a virtue in our field. A lazy sysadmin has processes automated by tools like cron and scripts. It seems as if the sysadmin is never doing any work themselves.

Actually, the best sysadmin is the one that can take advantage of the time saved by automating tasks. This is achieved by studying how to develop or improve a skill.

Challenge: Configuration management

For example, assume that the tasks of the day are to configure some hypervisors for KVM:

• Register hosts to RHN.
• Set up repos.
• Install packages and apply OS updates.
• Reduce /home.
• Configure LVM for /var/lib/libvirt/images.

Proposed solution: Ansible

Create an Ansible playbook to save you time:

• Use ansible-vault to encrypt your RHN username and password.
• Use Ansible modules to perform tasks: redhat_subscription, shell, yum, command, lvol, filesystem, mount

And with the free time you generate, study. Learn and practice.

Tolerance to frustration

On more occasions than we would like to count, we will be guilty of failure or will be overworked. The pressure to avoid downtime or deliver a product can be too much when the stars line up on the opposite side of our bed.

Those moments make us consider taking over the family business or partnering with our entrepreneurial friend.

Unfortunately, in those frustrating cases, we don't have a magic wand to help us disappear.

Challenge: Frustration management

Finding ways to manage frustration can be very difficult for many sysadmins.

Proposed solution: This is fine

• Breathe. Take deep breaths and continue.
• Cake and pie. Get up and walk around, get a piece of pie, clear your mind a little.
• Accept and learn. Maybe it's your fault. Accept it and learn from it.
• Commit yourself. Make sure it doesn't happen again.
• Respect your schedule. Respect your time and your family's time, just as you respect time for work.
• Rest. A rested mind thinks and executes better.
• Be your own fan. Take control, know your limits, but acknowledge your progress. No one knows you better, so help yourself get better all the time. Encourage yourself.

Wrap up

Remember the four pillars:

• Backup
• Console
• Study (create time by automating your tasks)
• Tolerance to frustration

It's not magic, it's effort and a lot of sweat.

I hope these pillars serve you well.

Have a read-only Happy System Administrator Appreciation Day!

[ Want to test your sysadmin skills? Take a skills assessment today. ]