Arrived there from a google search, error was gone a minute later.
]]>Looking through the recent resumes two things stand out:
1. More than half of the candidates have a Master's in Computer Science.
2. Less than half (out of those only one had a Master's) did well enough in a phone screen to progress to an in-person interview.
The obvious possible conclusions are:
1. A Master's in CS dumbs a person down.
--- or ---
2. The kind of person who chooses to pursue a Master's degree is not the kind of person we'd like to hire.
The second choice leads to a conclusion that people who are not exactly the greatest in their field will move on to pursue higher degrees in order to showcase better qualifications on their resumes. Pretty much every single person we interviewed with a Master's would barely qualify for a junior-level position. It's really stunning how little interest these candidates appear to have in a field they chose to pursue graduate work in.
And here I am about to become one of them. Yikes. Pray for my brain.
Saving my pennies for this expensive one..
Handbook of Theoretical Computer Science : Formal Models and Semantics
Grad school starts in two months!
It was not a super-smart uber hacker who did it. There isn't enough interesting information on there to tempt one of those guys. It was just a script. One of undoubtedly hundreds currently perusing the net in search of easy targets. The weak point was allowing password logins with an account that uses a person's first name for a username. Once the bot is in the system it's really trivial to root a linux box, especially one running an older version of redhat like mine was.
The real kicker? I know better. I would never ever advice anyone allow password logins on a publicly available server. Why did I allow it? Convenience, laziness, thinking all the passwords should be strong enough (yah right).
I feel like an idiot (and rightfully so), but at least I spotted it rather quickly - in a couple of hours. I monitor my logs regularly and keep on top of my traffic statistics, but what really gave it away is my email stopped.
I logged onto the system to check the logs and see what's going on. The email log (/var/log/maillog on most linux systems) stopped recording about an hour earlier. That's not normal. Checked netstat to see all the usual suspects happily running and listening on all the right ports. No unusual ports opened.
This is typical of what is up on my machine:
In order: postgrey, mysql, spamd, apache, postfix, postfix, pop, imap, ssh.[root@cygnus log]# netstat -an | grep LISTEN tcp 0 0 127.0.0.1:10023 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:783 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:26 0.0.0.0:* LISTEN tcp 0 0 :::110 :::* LISTEN tcp 0 0 :::143 :::* LISTEN tcp 0 0 :::22 :::* LISTEN
At any other time I'd have thought nothing was wrong. But my maillog was not recording any traffic and that's not normal. I checked my secure log. Nothing unusual. Current logins, all normal. As far as the system was concerned all looked normal.
I checked recent logins and noticed that an hour earlier an account was used that hasn't been in a while. That's unusual, checked the IP. Romania. That's when the real alarms started going off. First check, syslog. It's not running, won't start and strace hangs. Checked /dev/ for unusual files (that's where historically hackers like to put their binaries).
find /dev -type f -print
Nothing unusual there. Next thing to check is the date of binaries in /usr/bin and /usr/sbin. Bingo. All important binaries (netstat, lsof, etc) have a modification date of right now. That's the point where you know the system cannot be trusted. Sure, netstat is showing no unusual activity, but that's not what the system is really doing. lsof hangs, strace hangs, all the important tools that sysadmins depend on to know the state of the system can no longer be trusted.
Once a linux system is compromised like that, you can't recover without replacing the entire OS. So that's what I did. I shut the system down and asked the web hosting company to put a fresh install of whatever linux they have handy on it. As far as I can tell from the little evidence I gathered before shutting the box down, the system was compromised by a script using an account that had no sudo access. Then the box was rooted. Considering that the postfix binary was replaced and it stopped receiving external email for any of the configured domains (my postfix configuration is unorthodox in that its based in mysql), I would imagine the goal was to turn it into a spambox.
The real value to in a publicly connected linux machine is that it's trusted by other systems (until alarms go off anyway), it's not blacklisted and can serve as the perfect spambot. This is why shutting it down as soon as I confirmed it was hacked was so important.
Lack of burst in network traffic after it was compromised confirms that it was not someone interested in what information the server contained. A hacker with intentions other than turning the server into a zombie would have copied as much information as he could as fast as he could before the sysadmin was alerted to something not being kosher.
The system is back and running now, I restored my data from a backup (I have a daily rotating one) and configured it in a much smarter way. No more password based logins. Public/Private key authentication only. I had all the other important points down already, but here's the list in case it's helpful:
My one reason for allowing password logins was the ability to login from anywhere into my linux box and then use my keys to login to any system I need to that requires key authentication. The solution? Generate a new, password-protected key and keep it in a usb keyfob. Use that key to login to the system.
How often do scripts target public servers? Here's some extracts from my system logs:
Over last 24 hours:
Received disconnect:
11: Bye Bye : 1617 Time(s)
11: No supported authentication methods available : 1 Time(s)
14: No supported authentication methods available : 1 Time(s)
That's 1617 of these in 24-hour period:
May 19 14:05:17 cygnus sshd[24358]: input_userauth_request: invalid user testuser
May 19 14:05:18 cygnus sshd[24358]: Received disconnect from 218.210.68.166: 11: Bye Bye
May 19 14:05:19 cygnus sshd[24359]: Invalid user tester from 218.210.68.166
May 19 14:05:19 cygnus sshd[24359]: reverse mapping checking getaddrinfo for
adsl-218-210-68-166.pc.sparqnet.net failed - POSSIBLE BREAK-IN ATTEMPT!
May 19 14:05:19 cygnus sshd[24360]: input_userauth_request: invalid user tester
May 19 14:05:19 cygnus sshd[24360]: Received disconnect from 218.210.68.166: 11: Bye Bye
May 19 14:05:20 cygnus sshd[24361]: reverse mapping checking getaddrinfo for
adsl-218-210-68-166.pc.sparqnet.net failed - POSSIBLE BREAK-IN ATTEMPT!
May 19 14:05:20 cygnus sshd[24362]: Received disconnect from 218.210.68.166: 11: Bye Bye
May 19 14:05:21 cygnus sshd[24363]: reverse mapping checking getaddrinfo for
adsl-218-210-68-166.pc.sparqnet.net failed - POSSIBLE BREAK-IN ATTEMPT!
May 19 14:05:21 cygnus sshd[24364]: Received disconnect from 218.210.68.166: 11: Bye Bye
May 19 14:05:23 cygnus sshd[24365]: reverse mapping checking getaddrinfo for
adsl-218-210-68-166.pc.sparqnet.net failed - POSSIBLE BREAK-IN ATTEMPT!
May 19 14:05:23 cygnus sshd[24366]: Received disconnect from 218.210.68.166: 11: Bye Bye
May 19 14:05:24 cygnus sshd[24367]: Invalid user knoppix from 218.210.68.166
May 19 14:05:24 cygnus sshd[24367]: reverse mapping checking getaddrinfo for
adsl-218-210-68-166.pc.sparqnet.net failed - POSSIBLE BREAK-IN ATTEMPT!
This is why not allowing password authentication is so important.
]]>
Arrived there from a google search, error was gone a minute later.
]]>Since it's been (a) long and (b) slow and (c) old and well, I hate having to wait 10+ minutes for a build to finish when working from home.. You can guess what I just did, right? I bought a brand-spanking new macbook pro. It's fast, it's shiny, it has a scrolling trackpad. That's not what this post is really about though.
I'm most impressed with apple's 'puter transfer tool. It feels as if I haven't really switched machines. Particularly since this one looks nearly identical to the old one. Except for that creepy camera lens staring at me from the top. It's really starting to creep me out. What if it's on and I don't realize it? What if someone out there is watching me stick the tip of my tongue out as I desperately try to make fewer typos? (Can you just visualize that? See, scary part is, maybe someone doesn't have to visualize it!). But again, that is not what this post is about.
Despite the really wonderful transfer of files, settings, browser cache, cookies and other goodies, one thing that did not move along to my new machine is my iTunes purchased music. I had to enter my password and authorize this machine.
So now.. by doing nothing other than upgrading computers, Apple is telling me I only have 3 more authorizations left before I'm no longer allowed to listen to the music i legally paid for. In other words, unless I never upgrade machines again (yah right), this music will no longer be available to me in a few years. Did I mention I paid for it legally, unlike many other users? Based on a statistic I just made up, close to 70% of iPod users don't bother paying for the music. Someone please tell me one good reason why I should, seeing as I'm being treated like a thief even when I'm trying to be good.
To hell with that. Screw you, music industry.
]]>When I got home I noticed there was something odd with my vision. Couldn't quite put my finger on it, but it was definitely not right. In about fives minutes, it deteriorated to serious blurriness, then it got weird. Jaggedy, colourful images dancing in front of my eyes with surrounded blurriness. I was practically blind for about 5-10 minutes and had serious spatial disorientation when I could see my surroundings. Closing eyes did not help, could still see the same hallucinations and no blackness. The feeling was very much like after a small dose of hallucinogenic drugs.
The whole episode ended in about 15 minutes and via the magic of google, I found the symptoms were very consistent with a migraine aura. I have had migraines before, but the precursor was usually blurry vision and nausea. I didn't even know migraines could cause hallucinations.
More google magic and turns out that air fresheners are pretty nasty stuff and one of the known triggers of migraines. Tomorrow, i'll have a conversation with my co-workers, but in the meantime, if you get migraines, something to think about.
]]>While removing the lonely, unused constants and getting rid of some duplicates (user_id vs userid vs user_identifier that all point to the same string.. and so on..) I decided to remove a constants that will probably return to hunt me at some point. I've decided to live with the consequences of the change. It was either change mucho code to use the constant and be (that dirty word..) consistent or remove it. So I decided to remove it. If we have to change the value in the future, it'll be a lot of work, I admit it, but I'm willing to live with that result of my actions.
Today, I removed..
public static final int ZERO = 0;
RIP, leftovers of NSBCs.
]]>A table with mucho data and a date column. The date column has an index on it and its type is just pure old date. A query using jdbc and java.sql.Timestamp type on a prepared statement generates a full table scan.. and the associated slllloowwwwwnnnessssssss.
Identical query using java.sql.Date on a prepared statement or to_date(String, format) in the SQL uses the index.
Am I missing some obvious Oracle clue here? Cause google isn't helping!
Edit: I realize we can hint to Oracle which indexes to use, but it would be nice to have a more general solution.
]]>
Note: these were all taken with a 50mm prime lense, so "zoom" is created with photoshop.
]]>In the first general session, we were told it's the biggest JavaOne conference to date and I can certainly believe that looking at the lines to the sessions and crowds everywhere. What surprise me (pleasantly) was the amount of women this year. Nearly made me cry to see an actual line to the ladies room yesterday.
I'm sitting in the alumni lounge right now waiting for my next session ("Using Service Component Architecture to Wire BPEL Processes to Java™ Technology-Based Components" at noon). Only one other woman at the lounge beside me right now, but I'll bet it'll look better next year.
It's good to see that Java is pretty healthy and conference is doing well. Now if only Sun didn't come up with that hideous, orange color for the backpacks. You can spot the geeks all over town from half a mile away thanks to those. Pictures coming up soon!
PS: If you're in SF right now, awesome cream puffs in a little place accross the street from Yerba Buena Gardens (behind Moscone North). Worth the 2min walk!
PPS: To the geek who accidentally wondered to the ladies room and ran out horribly embarassed once he realized it: Sorry I laughed so hard dude!
]]>As my own example from last night shows..
My most recent nightmare?
Suddenly realizing it's the last day of JavaOne and I missed all sessions.
Yes, I really had that nightmare last night.
[0] I doubt he had computer programmers in mind, but, in his own way, he was a geek for his own time. Sort of. That really explains the whole ren-faire thing though, doesn't it?
PS: yep, still alive, just busy!
]]>