I was working on an ESX upgrade project for a client last week, and had an incident (lower case i) that I thought might be interesting to our readers.
I had just finished ugrading the vCenter server (vCenter is the management application for vSphere environments), everything looked good, and I was on my way home. Thats when it happened - I got the call. If youre a consultant, or have employed a consultant, you know the call Im talking about. The vCenter server seems to be *really* slow my client said, just since you upgraded it. Oh Darn! I said to myself (ok, maybe I didnt say exactly that, but you get my drift). I re-checked they hardware requirements for 5.1 as compared to 4.1, and the VM I had this on seemed to be OK on that front, and after a quick check the CPU and memory also looked fine. OK, over to the event log we go.
Ah-ha! Whats this?
Now dont that look suspicious? Who would try logging into SQL with a userid like hd673qyz? A brute force bit of malware maybe? And after a quick check, the IP in question was still live on the network, but doesnt resolve on my clients internal DNS server. So that means its not a server, and its not a client in Active Directory - this thing is not one of ours as they say. Now things are getting interesting!
Lets dig just a bit more, this time on the switching infrastructure - getting the MAC address and identifying the switch port its on:
At this point, I call my client back, and ask if he might know what this offending device is, and if he maybe wants to shut that switch port down until he can deal with it.
This brings some new information into the mix - he asks me I wonder if thats our XYZ scanner? (insert a name-brand security scanner here, often used for PCI scans). This question was followed immediately by a face-palm moment - because the scanner - with its logo prominently displayed - had been on the desk beside me the entire day while I was there! He was doing the absolute right thing, something I wish more folks did, he was scanning both his internal and perimeter network periodically for changes and security issues.
Sometimes, the change you just made isnt the cause of the problem thats just come up. OK, usually the problem is related to your change, but sometimes, as they say, coincidences happen - ok, maybe they dont say *exactly* that, but its close. When were engaged to do penetration tests and security scans, we always caution clients that the act of scanning can cause performance issues and service interruptions. But when you run your own scans internally, just keep in mind that this caveat is still in play. Its very easy to DOS internal services by changing one tick-box in your scanner.
In the end, this incident had a positive outcome. Well be changing the Windows Firewall settings on the vCenter server, restricting SQL access to local access only (the vCenter server itself), denying network access to SQL. Because theres no reason at all to offer up SQL services to everyone on the network if only local services need it. We likely would have gotten there anyway, the vSphere Hardening Guide calls this out, in the guideline restrict-network-access. This doesnt specifically mention the SQL ports, but the hardening Guide does recommend using the host firewall on the vCenter Server to block ports that dont need a network presence.
After all was said and done, I find my taste for irony isnt what it used to be .. when clients take your advice (in this case, scheduled scans of the internal network), you dont have to look far when it comes back to bite you !
If youve had a success story, where youve implemented a scheduled scanning process and found an unexpected issue that needed a resolution, please let us know in our comment form. Alternatively, if youve accidentally DOSd a production service, that also makes a great comment!
(c) SANS Internet Storm Center. http://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.