kasia in a nutshell

My server was hacked last week.

It was not a super-smart uber hacker who did it. There isn't enough interesting information on there to tempt one of those guys. It was just a script. One of undoubtedly hundreds currently perusing the net in search of easy targets. The weak point was allowing password logins with an account that uses a person's first name for a username. Once the bot is in the system it's really trivial to root a linux box, especially one running an older version of redhat like mine was.

The real kicker? I know better. I would never ever advice anyone allow password logins on a publicly available server. Why did I allow it? Convenience, laziness, thinking all the passwords should be strong enough (yah right).

I feel like an idiot (and rightfully so), but at least I spotted it rather quickly - in a couple of hours. I monitor my logs regularly and keep on top of my traffic statistics, but what really gave it away is my email stopped.

I logged onto the system to check the logs and see what's going on. The email log (/var/log/maillog on most linux systems) stopped recording about an hour earlier. That's not normal. Checked netstat to see all the usual suspects happily running and listening on all the right ports. No unusual ports opened.

This is typical of what is up on my machine:

[root@cygnus log]# netstat -an | grep LISTEN
tcp        0      0 127.0.0.1:10023             0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:783               0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:25                  0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:26                  0.0.0.0:*                   LISTEN      
tcp        0      0 :::110                      :::*                        LISTEN      
tcp        0      0 :::143                      :::*                        LISTEN      
tcp        0      0 :::22                       :::*                        LISTEN      

At any other time I'd have thought nothing was wrong. But my maillog was not recording any traffic and that's not normal. I checked my secure log. Nothing unusual. Current logins, all normal. As far as the system was concerned all looked normal.

I checked recent logins and noticed that an hour earlier an account was used that hasn't been in a while. That's unusual, checked the IP. Romania. That's when the real alarms started going off. First check, syslog. It's not running, won't start and strace hangs. Checked /dev/ for unusual files (that's where historically hackers like to put their binaries).

 find /dev -type f -print

Nothing unusual there. Next thing to check is the date of binaries in /usr/bin and /usr/sbin. Bingo. All important binaries (netstat, lsof, etc) have a modification date of right now. That's the point where you know the system cannot be trusted. Sure, netstat is showing no unusual activity, but that's not what the system is really doing. lsof hangs, strace hangs, all the important tools that sysadmins depend on to know the state of the system can no longer be trusted.

Once a linux system is compromised like that, you can't recover without replacing the entire OS. So that's what I did. I shut the system down and asked the web hosting company to put a fresh install of whatever linux they have handy on it. As far as I can tell from the little evidence I gathered before shutting the box down, the system was compromised by a script using an account that had no sudo access. Then the box was rooted. Considering that the postfix binary was replaced and it stopped receiving external email for any of the configured domains (my postfix configuration is unorthodox in that its based in mysql), I would imagine the goal was to turn it into a spambox.

The real value to in a publicly connected linux machine is that it's trusted by other systems (until alarms go off anyway), it's not blacklisted and can serve as the perfect spambot. This is why shutting it down as soon as I confirmed it was hacked was so important.

Lack of burst in network traffic after it was compromised confirms that it was not someone interested in what information the server contained. A hacker with intentions other than turning the server into a zombie would have copied as much information as he could as fast as he could before the sysadmin was alerted to something not being kosher.

The system is back and running now, I restored my data from a backup (I have a daily rotating one) and configured it in a much smarter way. No more password based logins. Public/Private key authentication only. I had all the other important points down already, but here's the list in case it's helpful:

• No password logins, private/public key authentication only
• No root password, strictly monitored sudo access
• Strict firewall rules based on knowing what to allow and why. Deny first, allow second.
• SSH protocol 2 only, 1 is not secure
• Monitor, monitor, monitor, monitor. There's nothing that can replace a vigilant sysadmin. Tools like logwatch and mrtg are incredibly useful.
• If you think your server is hacked, shut it down. You cannot trust anything the system tells you if binaries have been replaced. It's imperative to take the system offline as soon as possible.
• Don't try to recover a hacked system, if you don't know when it was hacked, don't trust backups either.
• Password protect any private keys stored on a publicly accessible server.
• Only install application you actually use and need. The less stuff running on the system, the lower the security risk from bugs in applications.

My one reason for allowing password logins was the ability to login from anywhere into my linux box and then use my keys to login to any system I need to that requires key authentication. The solution? Generate a new, password-protected key and keep it in a usb keyfob. Use that key to login to the system.

How often do scripts target public servers? Here's some extracts from my system logs:

Over last 24 hours:

 Received disconnect:

    11: Bye Bye : 1617 Time(s)

    11: No supported authentication methods available : 1 Time(s)

    14: No supported authentication methods available : 1 Time(s)

That's 1617 of these in 24-hour period:

May 19 14:05:17 cygnus sshd[24358]: input_userauth_request: invalid user testuser

May 19 14:05:18 cygnus sshd[24358]: Received disconnect from 218.210.68.166: 11: Bye Bye

May 19 14:05:19 cygnus sshd[24359]: Invalid user tester from 218.210.68.166

May 19 14:05:19 cygnus sshd[24359]: reverse mapping checking getaddrinfo for 

adsl-218-210-68-166.pc.sparqnet.net failed - POSSIBLE BREAK-IN ATTEMPT!

May 19 14:05:19 cygnus sshd[24360]: input_userauth_request: invalid user tester

May 19 14:05:19 cygnus sshd[24360]: Received disconnect from 218.210.68.166: 11: Bye Bye

May 19 14:05:20 cygnus sshd[24361]: reverse mapping checking getaddrinfo for 

adsl-218-210-68-166.pc.sparqnet.net failed - POSSIBLE BREAK-IN ATTEMPT!

May 19 14:05:20 cygnus sshd[24362]: Received disconnect from 218.210.68.166: 11: Bye Bye

May 19 14:05:21 cygnus sshd[24363]: reverse mapping checking getaddrinfo for

adsl-218-210-68-166.pc.sparqnet.net failed - POSSIBLE BREAK-IN ATTEMPT!

May 19 14:05:21 cygnus sshd[24364]: Received disconnect from 218.210.68.166: 11: Bye Bye

May 19 14:05:23 cygnus sshd[24365]: reverse mapping checking getaddrinfo for

adsl-218-210-68-166.pc.sparqnet.net failed - POSSIBLE BREAK-IN ATTEMPT!

May 19 14:05:23 cygnus sshd[24366]: Received disconnect from 218.210.68.166: 11: Bye Bye

May 19 14:05:24 cygnus sshd[24367]: Invalid user knoppix from 218.210.68.166

May 19 14:05:24 cygnus sshd[24367]: reverse mapping checking getaddrinfo for

adsl-218-210-68-166.pc.sparqnet.net failed - POSSIBLE BREAK-IN ATTEMPT!

This is why not allowing password authentication is so important.