It seems that every time I do a security assessment or pentest, the findings include problems with access controls for things like routers, switches, fiber channel switches, bladecenters and the like. And when I say every time, I really mean EVERY TIME. So, what kind of problems are common, and what can you do to prevent them?
People love default credentials. I've actually had a customer tell me if we forget the password, we can just google for it - like somehow that's a good thing. Even for gear that comes with default credentials that force you to change them on login (for instance, fiber channel switches), I've seen other engineers then change the password back to the default - their rationalle being that it reduces their support calls later.
The number of bladecenters that I've seen that have changed the default admin user and password I think I can count without taking my shoes off (ie - it's a small number - almost nobody changes their bladecenter admin password). This access gives the right to power off servers, decommission servers, unpatch servers, hack windows or linux passwords, almost everything is possible - the bladecenter admin access gives you the near-equivalent of physical access, and we all know that physical access trumps almost every control in the book.
People - default credentials are BAD. If you are using default userid and password, and someone compromises you, we don't call that hacking, we call it logging in. You don't need an uber-hacker to target you in this situation, ANYONE WITH GOOGLE can compromise you. Change the admin password on your gear, and if possible, change the admin userid. Better yet, back end admin logins with another directory (read on ...)
Prevent Access in the First Place - Access Classes
Many products will allow you to restrict administrative access in the configuration. For instance, Cisco gear has the access-class config statement. HP, Brocade, Juniper and others all have an equivalent construct. The commands you'd put in your switch or router config might be:
first, define the subnet or ip addresses of authorized administrative workstations
ip access-list standard ACL-ACCESS-CLASS
permit host 192.168.72.10
permit 192.168.68.0 0.0.0.255
next, apply the ACL to your access
line vty 0 15
access-class ACL-ACCESS-CLASS in
Prevent Access in the First Place, Reloaded - Define a Management Network
Even with all the other controls we'll talk about, implementing a management network is a good move. It zones all infrastructure admin into one place, you can control access to this netowrk using VPN controls or a jump box. The VPN approach is a neat one - it means that if you are an authorized admin, you can use an IPSEC or SSL VPN solution to get an IP that has access to admin your network from anywhere in the company. This is really handy for admins that are mobile within the company or have to provide support from home (or the cottage, the beach, or anywhere else they can find you).
Encrypt all Administrative Accesses
I still see LOTS of admin access over standard HTTP and Telnet. And there are LOTS of tools that will strip passwords out of this type of traffic - you can do this with CAIN if you want a GUI, but really even doing it with wireshark or tcpdump is pretty simple. Be sure to force SSH (Version 2 if you can, Version 1 can be decrypted), or HTTPS for administration of critical network infrastructure.
Even what you might consider non-critical infrastructure should see the same protections. I've done a pentest on a company that used telnet to administer their UPS gear (to monitor temperature and humidity as well as power characteristics and remote control of power etc). Unfortunately for them, they used their AD admin password to login to their UPS, which they accessed using plaintext HTTP and Telnet. Doubly unfortunate, as a standard procedure they logged in each morning to check logs etc.
Checking logs daily would normally be a really good thing if they had other controls in place (for instance, using HTTPS or SSH for administration), but since they were being pentested, I had their admin password within 10 minutes of starting the actual process !
On a cisco router or switch, the commands you need to set up SSH are:
first, define a hostname and domain name
ip domain-name yourcompany.com
next, generate the key
(use some number equal or greater than 1024 for stronger key strength)
crypto key generate rsa general-keys modulus 1024
next, force SSH version 2 and force SSH for access
ip ssh version 2
line vty 0 15
transport input ssh
If you allow web administrative access (that's a whole different discussion), forcing https on network instrastructure is generally even easier:
disable plaintext HTTP
no ip http server
enable SSL encrypted HTTPS
entering this last line will generate the (self-signed) SSL cert to
encrypt the webmin sessions.
ip http secure-server
To go one better, I'd also suggest that you replace the self-signed certs that is used by default for HTTPS admin on most gear, using certs on SSH is also a really good mechanism. Without replacing the default certificates, tools like ettercap can still be used to mount a man-in-the-middle attack and recover passwords.
Back-end Authentication and Change Logging
After everything else is said and done, there's still way too much gear out there that has a single administrative account, or no account at all (only access passwords). This plays hob with managing change - since every config update is done using the single admin account, if a change goes bad everyone in your team can deny making that change. (Does your team have Ida Know as an honorary member?) If you don't take a stab at non-repudiation of changes, this WILL bite you eventually.
So, what to do? Should you define a userid for each and every user on all your infrastructure gear? Well, only as a lastresort. Most network infrastructure has the capability of back-ending authentication and access controls using some external source. Popular back-ends are what you'd expect - RADIUS, TACACS, Kerberos, LDAP and Active Directory. I'd say pick one and go with it. I often go with RADIUS back-ended with AD(IAS, now NPS)because it's simple, easy to troubleshoot, and supported by almost everything. Mind you, it's also likely that if you go with RADIUS you are then susceptible to other attacks, but you can mitigate that by setting your RADIUS server up in a private vlan or by using other intelligent design decisions to implement security controls.
Basic definition of AAA, as well as the definition of the RADIUSserver
(Note that the RADIUSserver config will need to match this)
radius-server host 126.96.36.199
auth-port 1645 acct-port 1646 key randomcharkey
this line forces radius authentication for login by default. If radius is down (ie - no response is received from the radius server, or the radius keepalive is missed), then local authentication still works. Note that on one hand this leaves you open to attacks that involve DOSing your RADIUS server. On the other hand, you still have access to your network gear if your RADIUS servers or domain controllers are offline.
aaa authentication login default group radius local
defining the source interface is important, since the ip of the device is normally hard-coded on the radius server
ip radius source-interface Vlan1
on some gear you may also need to force authentication on individual lines:
line vty 0 4
login authentication radius
line console 0
login authentication radius
line aux 0
login authentication radius
Don't forget to LOG ALL ACCESSES (this is built into RADIUS and TACACS) and LOG ALL CONFIG CHANGES (lots of tools will do this for you - syslog will log that a change occurred and who made it, CATTOOLS and RANCID(thanks to our reader Bmac for this correction)are 2 that come to mind for more complete change logging, I've also written shell scripts to do this. Feel free to suggest others that you use as a comment to this diary). In some cases, you may also want to log all commands as well (most gear will let you do this in syslog).
What does this give you? As changes occur, you are notified that a change happened, who made it, and what it was. You can then compare this to the CHANGE REQUEST FORM that you have in your CHANGE CONTROL SYSTEM, to be sure that:
the change made was both requested and approved
the change happened during the change window
the person making the change was the one authorized to do it
if it was an unauthorized change, you have the culprit identified and can have, shall we say, a discussion that's appropriate to the situation.
Basic commands for simple logging and NTPtime sync are, well, pretty simple (as with most examples in this diary, it can get more complicated)
Basic Syslog logging is a one-liner
Similarly, setting a target host to get time from is also very simple
ntp server 192.168.5.6
I've phrased this discussion in the context of network infrastructure gear, but really many of these points extend to other datacenter infrastructure components as well. Replacing default certificates used for RDP to critical Windows servers is a good move, as is certificate updates for things like VMware vSphere (both vCenter servers and ESX Hosts). Using a Management network is an important part of designing a virtual infrastructure from any vendor, as well as bladecenters (for the same reasons). Management networks can also be used to protect things like RADIUS authentication, syslog, NTP and SNMP based network management, all of which are sent in cleartext. Back-end authentication to an enterprise directory (often AD) is a common solution for authentication, is useful as we discussed for managing and auditing change for both Linux and VMware servers, as well as for lots of other gear besides just routers and switches.
I'm hoping that you find these suggestions to be a helpful starting point. There's lots more that can be done in this area, please use the comment feature to let us know if you've found this useful, have your own stories in this area, or if I've missed (or messed up) anything in this diary.
=============== Rob VandenBrink, Metafore ===============
(c) SANS Internet Storm Center. http://isc.sans.org Creative Commons Attribution-Noncommercial 3.0 United States License.