There’s no getting around that security incidents will happen. The best I can hope for is that they don’t completely ruin my day and that they aren’t specific to my organization, so that I can write about them and possibly help my readers through similar situations.
What happened last week was one of those.
It started when a system administrator, newly hired as a contractor in our India development center, asked me after just a couple of days on the job for administrative access to several key development servers in order to manage accounts. He told me that the password he had for those servers didn’t work. That raised a couple of questions in my mind. Why wasn’t he asking his local supervisor to get that straightened out? And why was he going straight for administrative access when all he really needed were plain-vanilla passwords?
I decided to get his manager’s take on the situation. He agreed; this contractor didn’t need an administrative level of access to those servers. And he thought it odd that the guy was saying that his credentials didn’t work.
With that in mind, I looked at some logs from one of the development servers the contractor wanted admin access for. I saw many unsuccessful login attempts from a Linux server on the India subnet. I had root access to that particular server, so I logged in and took a look at the contractor’s home directory. It was disconcerting to find that it was world readable, which is not the standard for such user accounts. That prompted me to dig deeper, and I discovered two files of interest.
The first, named “password_list.xls,” was password-protected and appeared to be encrypted. Well, that was good, right? But the other file, called “password.txt,” was a text file with one entry. You guessed it: It was the password to the encrypted Excel file.
I opened the spreadsheet to find a plethora of server names, IP addresses, server purposes and the administrative user IDs and associated passwords. Not good at all. The file contained account information for resources that were long ago retired, so it hadn’t been compiled by this new hire. In fact, I learned, the file had been prepared by a former employee and passed around. This was a sickening discovery, alleviated only by the fact that it was no longer hidden from me.
But I had to consider all of those resources to be compromised. My reaction was swift. First, I removed the files and reconfigured the contractor’s home directory. I then called a meeting with the general manager for our India office and our head of IT. We went through the list of accounts and rank-stacked them according to risk. For example, anything in the DMZ that was Internet-facing and any resources containing sensitive data got a priority of 1, meaning that the password had to be changed within 24 hours. The lowest priority that we assigned called for the account passwords to be changed within 72 hours. The extra time was needed in some cases to ensure that we didn’t impact any business processes.
I also ordered a discovery scan to find all locations that contained those password files. Because we don’t have a robust data loss prevention infrastructure, I had to resort to conducting broad file-storage searches, both on the internal network and in the cloud, for files containing the word “password.” (We use Google Docs for collaboration, and there are third-party tools that allow for expanded reporting and searching against all files. I found one, installed an eval copy and was able to conduct my searches.)
I also put a data filter rule in our firewall to trigger an event if someone tried to transmit a file called “password.” Then I sent a companywide message warning against the use of spreadsheets to maintain passwords. We have a corporate-sanctioned password vault that allows for the secure storage, management and sharing of passwords, so there is (generally) no reason to keep passwords on spreadsheets. I say “generally” because certain passwords, such as encryption key passphrases and passwords for critical resources, are printed out and kept in a safe, because we’re just too paranoid to store them in an online, digital password vault.
There was just one more thing to do to put this incident behind us: Our general manager in India terminated the contractor.
This week's journal is written by a real security manager, "Mathias Thurman," whose name and employer have been disguised for obvious reasons. Contact him at firstname.lastname@example.org.
Click here for more security articles.