In a lot of ways, our job in IT and Information Security is implementing change. But as we all know, every change involves risk, and changes gone bad can be your worst nightmare. Ive seen the number of business system service interruptions due to changes in infrastructure pegged at anywhere from 50 to 70%, and Id lean towards the high side of that. If its not a hardware failure, its probably a mistake. Why is this number so high?'
Testing takes time, time that we often don't have. Managers will often consider every change to be simple, so getting testing time is often a challenge. We'll often be working in complex environments, environments that involve lots of people who don't care about your change until it breaks something of theirs. We often have system components in the mix that we don't know about until something breaks. We may have legacy apps that simple cannot be known - stuff that was written 20 years ago by people who have long since retired for instance.. Compound this with the pervasiveness of the GUI interface , which in a lot of ways encourages people to stumble through the menu until they find a setting that works (yes this happens, every day).
So what to do? Its our job to patch and update systems, to take projects from a set of ideas to working systems. All of this involves change, every day. Most of us have some method of change control. This generally involves forms and meetings, but dont run for the hills, its not all bad really most of it is common sense:
Figure out what youre doing
Make sure its going to work
If it messes up, have a plan
Clear it with others in your team to make sure it wont mess anyone else up (remember, theyre all making changes too)
Make the change
Tell your team mates that its done.
You can do all of this with a few emails, but as I mentioned, most of us have forms to make sure we dont forget steps in this, and we have meetings to make sure that everyone who needs to know gets told (and doesnt have the but nobody told me excuse later).
Just the facts
So lets start with the form. Its simple really. You need to describe your change in terms that your audience will understand. Your audience will usually include folks are technical, but dont always understand your job. Sometimes non-technical people will be on the list business unit people who may be affected for instance. Whoever is on the list, be sure that youre accurately describing the change, but arent talking down to them.
Youll need a plan. This needs to describe both the change and how you are going to test it. I almost always use a Gantt or Pert chart to describe my plan. Not only does everyone understand a picture (remember your audience?), but at 2am when Im doing it for real, its REALLY easy to miss a step. And an hour into it, what you DONT want to do is undo steps 4 through 15 so that you can do the step 3 that you missed the first time.
You need a backout plan. I dont know about you, but lots of things that I do fail in spectacular ways this is what makes my job so much fun. The important thing is to have a plan to deal with it.
Put a time estimate on your plan, and be sure that the change window you are requesting includes the time to make the change, test it, then back out. If you have a backout plan, then problems become what they should be - an opportunity to learn something new, rather than the political firestorm that they can become if you don't have that plan ready to go.
What is a change? Especially when change control is new in an organization, the topic of what change is will be a hot one - when do you need to fill out a change control request? How often have we had a problem that was an obvious result of a change last night, but the next morning nobody owns up to making the change? Or dealing with a change related issue that's been a problem for about two weeks, with no record of what changes were made in that timeframe? I've been in organizations where unplugging an unused ethernet cable is a change, or plugging in a power cable. On the face of it, you might think that these are both going overboard. But if you unplug that dead cable you may find out later that it's not really dead - it's only used to run that legacy accounting report at month end. Or when you plug in the new IPS unit you may trip the breaker on the PDUthat the main firewall or fileserver is plugged in to. The level of change that needs to hit the process will vary from company to company, if you're just starting out in this, no worries, just be prepared for some healthy back-and-forth as you find the level between too much paper work and enough process to get the job done
The key in determining the scope of the change management process is to make sure that it catches the changes you need it to catch, but isn't such an administrative problem that everyone hates the process. If people hate it, they won't use it and there's no point. It needs to be easy enough and useful enough so that the people in IT making system changes see it as a clear benefit instead of an obstacle.
Meetings, Meetings and more meetings OK, just one meeting, but BE THERE.
After youve got the change all mapped out, youll need to make sure that everyone who needs to know, knows. The whole point of change control is to avoid conflicting changes. Just to pick a few examples - If youre doing a firewall upgrade, it cant be when the DBA needs to VPN in to do a schema update. Or if youre doing a router upgrade, it cant be during a server migration thats on the other side of the router (yes, Ive seen both at one company or other).
The meeting before the meeting
So you need a meeting, but not immediately. In most cases, change control forms need to be submitted a few days before the meeting. This is so everyone who needs to has a chance to review the change and ask any questions. By the time you get to the meeting, you should have informal approval.
Finally the meeting
The point of a change control meeting is to get formal approval and signoff. You need to know that everyone is ok with the change, and that it wont conflict with anyone elses change. This means a few things you need a quorum, and you need the right people there. If your DBA boycotts your meetings, then theres no point in having them. Really everyone in IT needs to buy into change control, because if they dont, its their changes that will bite your company in the tender parts!
Change control meetings should be SHORT. I normally see discussion limited to 5 minutes per change discuss any last minute details, be sure that change A and Change B dont conflict, then its a yes or a no. If you go over 5 minutes, either you didnt do your homework (remember the meeting before the meeting?), or someone is hijacking your meeting.
Speaking of hijacking, change control meetings need a chair. In many companies, the chair rotates between everyone who is NOT a manager. This is a good thing for the team, as it gives people who otherwise wouldnt speak during a meeting practice in talking to actual people.It alsoemphasizes that the meeting is to make the job ofthe doerseasier - at its root change control is not supposed to be a heirarchal management function at all. On the other hand, in lots of other companies the IT manager chairs the meetings. In either case, the main job of the meeting chair is to keep it moving, to be sure that changes are approved or nixed quickly so everyone can get back to work.
Finally, any completed changes should be discussed. The point of this is to continually make the process better. Reviewing completed changes is all about making sure that the testing process is good, and that people are writing good implementation and backout plans.
Doing the Deed
When you are making your change, follow the plan. Sneaking in little changes here and there will only cause you grief. If you have an hour approved, the backout plan is 10 minutes and you are at 50 minutes and still typing, you should bail. Going over on changes is an unscheduled outage, which everyone takes a dim view of. If you needed 70 minutes instead of 60, or even 2 hours, ask for that up front. If you've got a plan and a process, this all get simple, and you get to make that change (aka - real work) in peace!
Iwrote this diary because Iend up talking to one client or other about change control processes a few times per month - it's a pretty popular topic (especially when you're convincing people who don't have a process that they need one). I've attached a a sample change control form below. If you find them useful, feel free to modify this and use it in your organization (or not). Over the years, I've seen change control implemented as Word documents in a shared folder or paper binder (this doc goes back almost 20 years), small access databases, as functions within Helpdesk systems, or recently everyone seems to havea need to write these in Sharepoint. The mechanics of how you implement your process is not important, it's the fact that you implement a process that will make your changes successful !
If you have any thoughts, or if I've missed anything, please feel free to use the comment feature on this diary. I'll update this article as the day goes on, based on your input.
=============== Rob VandenBrink Metafore ===============
(c) SANS Internet Storm Center. http://isc.sans.org Creative Commons Attribution-Noncommercial 3.0 United States License.