Wednesday, October 7, 2015

Linux Security: Lock Down a New System Immediately

PCWorld recently published an article about Linux botnets launching DDoS attacks. The attackers find and exploit poorly secured Linux systems. Some Linux users have a fairly cavalier attitude about security, assuming the supposedly superior design of the OS somehow protects them. It doesn't. Now that Chromebooks outsell Windows laptops and Amdroid devices are ubiquitous the days when Linux was a secondary target for malware are long gone. Linux' prominence in both the server room and on consumer devices make it a prime target.

How quickly will a newly commissioned system be attacked? Here's my recent experience. I did a build of a webserver with CentOS 7.1 and Drupal 8. That went smoothly. Then I installed all the latest patches. On to configuring and hardening the server. The first time I issued a su command to allow me to configure sudo I got a message about failed logins. I looked at the logs and sure enough, someone was already trying to break in with a scripted brute force attack. I immediately configured ssh to disallow root login and to disconnect after three failed attempts. That slowed but did not stop the attacks. OK, I checked the IP address they were coming from, knowing full well it might be spoofed, and found the owner was a French ISP. To satisfy my curiosity I put in a temorary firewall rule to disallow that IP range and the attack stopped and did not resume. So... either the IP address was not spoofed and some script kiddie was trying to break in from their actual connection or else the attack came through a proxy server with an IP address in that range. Cute.

Anyway, I finished my usual tasks for hardening a server and Apache, uploaded my backed up files and put them in the appropriate places with the appropriate permissions. My idea of hardening Apache includes both mod_evasive and mod_security with a current rule set. My idea of securing a server includes blocking pretty much all inbound connections not needed by either the web server, software used on the website or SSH.

I should also add that IP address blocking, up to and including geolocking, is generally not an effective defense. Knowledgeable hackers use proxies. Many change their apparent IP addresses frequently. Blocking IPs, even at the firewall level, is a lot like playing Whac-a-Mole. The attack simply continues from a different IP in short order. The brute force attacks I'm still experiencing are trying to login as root and since remote root logins are disabled that simply won't work. However, the excess traffic to my server continues and there is relatively little I can do about it.

How long between my rebuild and the start of the attacks? A few minutes. Such is life on the Internet today. The first thing anyone should do with a new system, any system, is lock it down.

15 comments:

  1. Whenever possible you should be doing the initial install on a private IP so inbound connections can't happen yet you can connect outbound do download the updates.

    ReplyDelete
    Replies
    1. Most of my work is remote systems administration and security. It's all done over the 'net, making your suggestion impossible, unfortunately.

      Delete
    2. We all work remotely. And yes there are methods to help you too. Disable root login, set firewall rules to accept ssh from particular IP (range), use alternative ssh port (37562 for ex.), use limited user for ssh connection and elevate upon successful connection, use ssh keys with passphrase rather than password. Disable sshd password authentication... And even more.

      Delete
    3. Everything you say is true; none of them will stop the sort of attacks I saw from continuing. Indeed, most of the steps you list are done on that server. Note that those attacks all failed. I wasn't writing a security primer or a system hardening guide. There are many excellent ones out there. There are also steps that neither you nor I listed which can help lock down a server hard. For example, the server I refer to has SELinux enabled in enforcing mode.

      The purpose of the article is to point out just how quickly attacks on a server can happen and to make clear that a new machine needs to be locked down immediately.

      Delete
    4. Hi Caitlyn,

      So I am using Linux (Debian Jessie) on a laptop and only connect to the internet using wireless access and/or temporary via a LAN and internet facing rooter. So my IP address is unlikely to remain the same for long. Still, I assume some hardening is needed all the same. How much will this situation differ from yours (typical Linux set up as a permanently online server with remote log-in enabled)? Most security guides I have seen assume that Linux will be used as an online server. Or is the steps to hardening pretty much the same for a desktop/laptop Linux system?

      Maybe related: How secure are most live Linux distro's default installations (if you install from a live CD/DVD)?

      Thanks!

      Delete
    5. B.t.w. thanks for making us aware of hoe quickly attacks can start!

      Delete
    6. A botnet like the one I referred to in the article doesn't care if your system is a server, a desktop or a laptop. Any system that can be hijacked will do. They all need to be locked down. The changing IP address really doesn't matter. Also, once malware is on your system it may have the ability to "phone home". Everything, and I do mean everything, needs to be locked down. The only difference will be what ports, applications and services you need.

      As far as live CDs go, the OS is read only so it can't be hacked directly. That doesn't mean malware can't be placed on a read/write device (hard drive, USB stick) and executed. Some live distros are terribly insecure - logging in as root without a password and leaving your data wide open. Most Linux default installations are not properly secured and need to be locked down.

      Delete
  2. Nice article, Caitlyn. Thank you for writing this.

    Can you refer any of the guides that you mentioned in your October 8, 2015 comment? "I wasn't writing a security primer or a system hardening guide. There are many excellent ones out there."

    Thanks again

    ReplyDelete
    Replies
    1. Yes, of course. There is a nearly identical comment below and I'm adding my links as a reply there. Thanks for your kind words.

      Delete
  3. Great article to remind Admins of what they should be doing from day one. On top of that, I use CloudFlare for my website as well as a few other recommendations that aren't mentioned here. I rely on a product called Lynis to scan my systems and relay to me where items are weak. I then go to work resolving these issues as much as possible - sometimes it's impossible to do it all as some software just doesn't work with a completely hardened system. Another great tool is fail2ban, This implements quite an auto blocking feature of IP addresses after X amount of hits against the rule - also extremely useful!

    https://cisofy.com/lynis/

    ReplyDelete
  4. Thanks for the nice article.

    "I wasn't writing a security primer or a system hardening guide. There are many excellent ones out there.."

    Can you refer a couple of these?

    ReplyDelete
    Replies
    1. First, since I was writing about CentOS, a clone of Red Hat Enterprise Linux 7.1, I'll start by pointing out that Red Hat has excellent security documentation that is freely available:
      https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0CCkQFjAAahUKEwjv6bylhbTIAhWTgA0KHdUxCLs&url=https%3A%2F%2Faccess.redhat.com%2Fdocumentation%2Fen-US%2FRed_Hat_Enterprise_Linux%2F7%2Fpdf%2FSecurity_Guide%2FRed_Hat_Enterprise_Linux-7-Security_Guide-en-US.pdf&usg=AFQjCNFfpL7FONkJstSPmj2bAZ1358YClw&sig2=UPcX_zHvHBnA2SbmuOVUdQ&bvm=bv.104615367,d.eXY
      https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Security_Guide/chap-Hardening_Your_System_with_Tools_and_Services.html

      The University of Texas has also published a checklist for hardening RHEL which also applies well to CentOS, Scientific Linux, Springdale Linux and Oracle Linux: https://wikis.utexas.edu/display/ISO/Red+Hat+Enterprise+Linux+7+Hardening+Checklist

      The CIS security benchmarks are another good guide for Red Hat and related systems: http://benchmarks.cisecurity.org/downloads/browse/index.cfm?category=benchmarks.os.linux.redhat

      Debian also has excellent documents, but they need to be updated a bit for Debian 8 and systemd. Still, an excellent document: https://www.debian.org/doc/manuals/securing-debian-howto/
      There is also additional information in the Debian wiki: https://wiki.debian.org/Hardening

      IBM has an introductory document for server hardening. It's best section are the links at the end, which give you more generic Linux and Ubuntu-specific sources: https://www.ibm.com/developerworks/linux/tutorials/l-harden-server/

      SUSE also has excellent documentation for their enterprise product. A lot of this can also be applied to OpenSUSE: https://www.suse.com/documentation/sles11/singlehtml/book_hardening/book_hardening.html

      I hope this helps.

      Delete
    2. This helps A LOT. Kudos and thanks.

      Further question: What about a post-installation script that automates the hardening process, i.e., for someone that needed to stage multiple systems at once, say in a site roll-out?

      I am but a Grasshopper... :-)

      Delete
    3. Once upon a time there was Bastille Linux but that hasn't been updated in quite some time. There are enough differences between Linux distributions to make me believe that any such script would have to be distro specific. I'm not aware of any off the shelf scripts that I would trust for this for the distros I use.

      Delete
  5. This is one reason i wrote a script that adds an iptables rule based on failed login attempts recorded in last log. Its not perfect but i figure if a system is being use to brute force my little old login machine its not a system i ever want to communicate with ever.

    ReplyDelete