the Phoenix project fifth edition by Gene Kim Kevin bear and George Spofford with new audio material including the first three chapters of the devops handbook and a new afterward by Gene Kim original text narrated by Chris ruin devops handbook chapters and afterward narrated by Ron Butler published by it Revolution press find us at itrevolution.com in the books and free resources section press release by Kelly Lawrence Chief industry analyst Nestor Myers for immediate release company Parts Unlimited stock Trader symbol p-a-u-d rating cell price Target eight dollars current thirteen dollars effective immediately Parts Unlimited CEO Steve Masters is stepping down from his role as chairman after eight years of holding that position board director Bob Strauss who served as company chairman and CEO two decades ago is returning from retirement to assume the role of chairman Parts Unlimited stock has tumbled 19 in the last 30 days under heavy trading down 52 percent from its peak three years ago the company continues to be outmaneuvered by its arch rival famous for its ability to anticipate and instantly react to customer needs Parts Unlimited now Trails the competition in sales growth inventory turns and profitability the company has long promised that its Phoenix Program will restore profitability and close the gap by tightly integrating its retailing and e-commerce channels already years late many expect the company to announce another program delay in its analyst earnings call next month we believe that institutional investors such as Wayne Yokohama pressured Strauss to reconfigure the board as the first of many actions to write the ship in Elkhart Grove a growing number of investors are pushing for more significant leadership changes and strategic options such as splitting up the company despite Masters past achievements that transformed Parts Unlimited into one of the top Automotive Parts manufacturers and retailers We Believe splitting up the chairman and CEO roles is long overdue Parts Unlimited needs fresh leadership either from the outside or from within We Believe Sarah Moulton senior vice president of retail operations and a rising star at the company could just be what the company needs according to our sources the board has given Strauss and Masters six months to make dramatic improvements if they can't pull this off expect more changes and turbulent times Kelly Lawrence Chief industry analyst Nestor Myers part one chapter 1 Tuesday September 2nd Bill Palmer here I say answering my cell phone on the first ring I'm late so I'm driving 10 miles per hour over the speed limit instead of my usual five I spent the morning at the doctor's office with my three-year-old son trying to keep the other toddlers from coughing on us constantly being interrupted by my vibrating phone the problem of the day is intermittent Network outages as the Director of mid-range technology operations I'm responsible for the availability and smooth functioning of a relatively small I.T group at Parts Unlimited a four billion dollar per year manufacturing and retail company based in Elkhart Grove even in the technology backwaters I've chosen to make my turf I need to track network issues closely because these issues disrupt the services my group provides people will blame the outages on me hi Bill this is Laura Beck from Human Resources she's not the person I usually deal with from HR but her name and voice sound familiar holy crap I try not to swear out loud when I remember who she is from the monthly company meetings she's the VP in charge of HR good morning Laura I say with forced cheer what can I do for you she responds when will you be in the office I'd like to meet as soon as possible I hate vague requests to meet I only do them when I'm trying to schedule a time to choose someone out or fire them wait is Laura calling because someone wants to fire me was there an outage I didn't respond to quickly enough as an I.T operations guy the career ending outage is the joke my peers and I tell one another daily we agreed to meet at her desk in a half hour but when she doesn't share any more details I say in my most cajoling boys Laura what's this all about is there a problem in my group or am I the one in trouble I laugh extra loudly so she hears it over the phone no it's nothing like that she says breezily you could even say this is good news thanks Bill when she hangs up I try to think of what good news would even look like these days when I can't I turn the radio back on and immediately hear a commercial from our largest retailing competitor they're talking about their unparalleled customer service and a breathtaking new offering that allows people to customize their cars with their friends online the ad is brilliant I'd use the service in a second if I weren't such a loyal company man how do they keep bringing such incredible new capabilities to Market while we remain stuck in the mud I turn the radio off despite all our hard work and late nights the competition keeps leapfrogging us when our marketing people hear this ad they'll go ballistic because they're likely art or music Majors not people with a technology background they'll publicly promise The Impossible and it will have to figure out how to deliver each year it gets harder we have to do more with less to simultaneously maintain competitiveness and reduce costs some days I think that it can't be done maybe I spent too much time as a sergeant in the Marines you learn that you argue your case as best as you can with your officer but sometimes you have to say yes sir and then go take that Hill I pull into the parking lot three years ago finding an empty parking spot was impossible now after all the layoffs parking is rarely a problem when I walk into building 5 where Laura and her staff reside I immediately notice how nicely furnished it is I can smell the new carpeting and there's even classy wood paneling on the walls suddenly the paint and carpet in my building seem decades overdue for replacement that's it's lot in life at least we're not in a dingy dimly lit dank basement like in the British TV show the IT Crowd when I get to Laura's office she looks up and smiles good seeing you again bill she extends her hand which I shake have a seat while I see whether Steve Masters is available to meet Steve Masters our CEO she picks up and dials her phone while I sit down looking around the last time I was here was a couple of years ago when HR notified us that we needed to dedicate a room for nursing mothers we were critically short of office and meeting space and we had big project deadlines looming we merely wanted to use a conference room in a different building however Wes made it sound like we were a bunch of 1950s Mad Men neanderthals shortly afterward we were both summoned here for a half day of political Rehabilitation and sensitivity training thanks Wes among other things Wes is in charge of the networks which is why I track Network outages so closely Laura thanks the person on the other end of the phone and turns back to me thanks for coming down on short notice how is your family doing these days she asks my brow furrows if I wanted to chit chat there are many people I'd rather talk to than someone in HR I forced myself to Banter about our families and kids trying not to think about my other pressing commitments eventually I say without much grace so what can I do for you this morning of course she pauses and then says effective as of this morning Luke and Damon are no longer with the company this went all the way to the top with Steve getting involved he's chosen you to be the VP of it operations she smiles broadly holding out her hand again you're our newest VP in the company bill I think some congratulations are in order holy crap I numbly shake her hand no no no the last thing I want is a promotion Luke was our CIO or Chief Information officer Damon worked for him and was my boss in charge of it operations across the entire company both gone just like that I didn't see this coming there wasn't any chatter on the Subspace radio nothing for the last decade like clockwork new cios would come and go every two years they stay just long enough to understand the acronyms learn where the bathrooms are Implement a bunch of programs and initiatives to upset the apple cart and then they're gone CIO stands for career is over and VPS of it operations don't last much longer I've figured out that the trick to a long career in it operations management is to get enough seniority to get good things done but to keep your head low enough to avoid the political battles that make you inherently vulnerable I have absolutely no interest in becoming one of the VPS who just give each other power points all day long fishing for more information I joke two Executives leaving at the same time were they stealing money from the stores late at night she laughs but quickly returns to her HR trained deadpan they both chose to pursue other interests more than that you'll have to find out from them as the saying goes if your colleague tells you they've decided to quit it was voluntary but when someone else tells you they've decided to quit it was mandatory Ergo my boss and his boss were just whacked this is exactly why I don't want a promotion I'm extremely proud of the team I've built over the last 10 years it's not the largest group but we're the most organized and dependable by far especially compared to Wes I groan at the thought of managing Wes he doesn't manage a team he's barely one step ahead of a chaotic mob as I break out in a cold sweat I know I will never accept this promotion all this time Laura has been talking and I haven't heard a single word and so we'll obviously need to talk about how we're going to announce this transition and Steve wants to see you as soon as possible look thanks for the opportunity I'm honored but I don't want this role why would I I love my current job and there are tons of important things that still need to be done I don't think this is optional she says looking sympathetic this came straight from Steve he chose you personally so you'll have to talk with him I stand up and reiterate firmly no really thanks for thinking of me but I've already got a great job good luck finding someone else minutes later Laura is walking me to building two the tallest building on campus I'm angry at myself for getting sucked into this insanity if I run now I'm pretty sure she wouldn't be able to catch me but then what Steve would just send a whole Squad of H.R goons to fetch me I don't say anything definitely not feeling like small talk anymore Laura doesn't seem to care walking briskly beside me nose buried in her phone occasionally gesturing directions she finds Steve's office without ever looking up obviously having made this walk many times before this floor is warm and inviting furnished just like it was in the 1920s when the building was constructed with dark hardwood floors and stained glass windows it's from an era when everyone wore suits and smoked cigars in their offices the company was booming then Parts Unlimited made various widgets inside almost every make of automobile when horses were being vanquished from daily life Steve has a corner office where a no-nonsense woman is keeping guard she's about 40. radiating cheerfulness and a sense of organization and Order her desk is tidy with Post-it notes everywhere on the wall there's a coffee mug with the words don't mess with Stacy by her keyboard hi Laura she says looking up from her computer busy day huh so this is Bill yep In the Flesh Laura replies smiling to me she says Stacy keeps Steve in line You'll grow to know her well I suspect you and I can finish up later then she leaves Stacy Smiles at me pleasure I've heard a lot about you already Steve is expecting you she points to his door I immediately like her and I think about what I've just learned it's been a busy day for Laura Stacy and Laura are on very familiar terms Steve has HR on speed dial apparently people who work for Steve don't last long great walking in I'm a little surprised to find Steve's office looks just like Laura's it's the same size as my boss's office or rather my ex-boss's office and potentially my new office if I'm stupid which I am not maybe I was expecting Persian rugs water fountains and large sculptures everywhere instead there are photos on the wall of a small propeller airplane his smiling family and to my surprise one of him in a U.S army uniform on a Runway somewhere tropical I know with surprise the Insignia visible on his lapels so Steve was a major he is sitting behind his desk scrutinizing would appear to be paper spreadsheets there's a laptop open behind him displaying a browser full of stock graphs Bill good to see you again he says standing and shaking my hand it's been a long time about five years right it was after you pulled off that amazing project to integrate one of the manufacturing Acquisitions I trust life has been treating you well I'm surprised and a bit flattered that he remembered our brief interaction especially when it was so long ago I smile in return saying yes very well thank you I'm amazed you remember something so far back you think we give out Awards like that to just anyone he says earnestly that was an important project to make that acquisition pay off we needed to nail it which you and your team did superbly I'm sure Laura has told you a bit about the organizational changes I've made you know Luke and Damon are no longer with the company I intend to fill the CIO position eventually but in the meantime all of it will report to me he continues brisk and business-like however with Damon's departure I have an organizational hole I need to fill based on our research you're clearly the best candidate to take over as VP of it operations as if he just remembered he says you were a marine when and where I announce automatically 22nd Marine Expeditionary Unit Sergeant I was in for six years but never saw combat remembering how I joined the Marines as a cocky 18 year old I say with a smile the core really straightened me out I owe them a lot but I sure hope neither of my sons join under the same conditions I did I bet Steve laughs I was in the Army for eight years myself slightly longer than I was obligated to but I didn't mind ROTC was the only way I could pay for college and they treated me well he adds they didn't cuddle us like they did you Marines but I still can't complain I laugh finding myself liking him this is the longest interaction we've had I suddenly wonder if this is what politicians are like I try to stay focused on why he summoned me here he's going to ask me to undertake some Kamikaze mission here's the situation he says motioning me to have a seat at his conference table as I'm sure you're aware we must regain profitability to do that we need to increase our market share and average order sizes our retail competitors are kicking our ass the whole world knows this which is why our stock price is half what it was three years ago he continues project Phoenix is essential to closing the Gap with the competition so we can finally do what the competition has been doing for years customers need to be able to buy from us from wherever they want whether it's on the Internet or in our retail stores otherwise we'll soon have no customers at all I nod I might be in the technology backwaters but my team has been involved with Phoenix for years everyone knows how important it is we're years late delivering he continues our investors and Wall Street are howling and now my board is losing confidence in our ability to hit our commitments I'll be blunt he says the way things are going I'll be out of a job in six months as of last week Bob Strauss my old boss is the new chairman of the company there's a vocal group of shareholders trying to split up the company and I don't know how much longer we can fend them off what's at stake here is not just my job but the nearly 4 000 employees who work here at Parts Unlimited suddenly Steve looks much older than the early 50s I had guessed him to be looking right at me he says as acting CIO Chris ahlers our VP of application development will report to me and so will you he stands up and starts to Pace I need you to keep all the things that are supposed to be up well up I need someone reliable who isn't afraid to tell me bad news above all I need someone I can trust to do the right thing that integration project had many challenges but you always kept a cool head you've built a reputation as someone who is Dependable pragmatic and willing to say what you really think he's been candid with me so I reply with the same sir with all due respect it seems very difficult for senior I.T leadership to succeed here any request for Budget or staff is always shot down and executives are replaced so quickly some never even get a chance to fully unpack with finality I say mid-range operations is critical to getting Phoenix done too I need to stay there to see those things through to completion I appreciate you thinking of me but I can't accept however I promise I'll keep my eyes open for any good candidates Steve looks at me appraisingly his expression surprisingly grave we've had to cut budgets across the entire company that edict came straight from my board my hands were tied I won't make promises I can't keep but I can promise you I'll do whatever it takes to support you and your mission bill I know you didn't ask for this job but the company's survival is at stake here I need you to help me save this great company can I count on you oh come on before I can politely decline again I suddenly hear myself saying yes sir You Can Count On Me I panic realizing that Steve somehow used some Jedi mind trick on me I forced myself to stop talking before I make some more dumb promises congratulations Steve says standing up and shaking my hand firmly he clasps my shoulder I knew you'd do the right thing on behalf of the entire executive team we're grateful for you stepping up I look at his hand grasping mine wondering if I can backpedal my way out not a chance in hell I decide swearing to myself I say I'll do my best sir and could you at least explain why no one who accepts this position lasts very long what do you want most from me and what don't you want with a resigned half smile I add if I fail I'll try to make sure it's in a new and novel way I like that Steve says laughing loudly what I want is for it to keep the lights on it should be like using the toilet I use the toilet and hell I don't ever worry about it not working what I don't want is to have the toilets back up and flood the entire building he smiles broadly at his own joke great in his mind I'm a glorified janitor he continues you have a reputation of running the tightest ship in the it organization so I'm giving you the entire fleet I expect you to make them all run the same way I need Chris focused on Phoenix execution anything in your area of responsibility that takes Focus away from Phoenix is unacceptable that applies not just to you and Chris but everyone else in this company is that clear absolutely I say nodding you want the I.T systems to be reliable and available and for the business to be able to depend on them you want disruptions to normal operations kept to an absolute minimum so that the business can focus on getting Phoenix done looking surprised Steve nodds exactly yes well put whatever you said that's exactly what I want he hands me an email printout from dick Landry the CFO from dick Landry to Steve Masters date September 2nd 8 27 am priority highest subject action needed payroll run is failing hey Steve we've got serious issues with this week's payroll we're trying to figure out if the problem is with the numbers or in the payroll system either way thousands of employees have paychecks stuck in system and are at risk of not getting paid seriously bad news we must fix this before payroll window closes at 5 pm today please advise on how to escalate this given the new it org dick I wince employees not getting paychecks means families not being able to pay their mortgages or put food on the table suddenly I realize that my family's mortgage payment is due in four days and we could be one of the families affected a late mortgage payment could screw up our credit rating even more which we spent years repairing after we put Paige's student loans on my credit card you want me to jump on this and manage the incident to conclusion Steve nodds giving me the thumbs up keep me posted on the progress please his expression turns great every responsible company takes care of its employees many of our Factory workers live from paycheck to paycheck do not create hardship for their families you hear this could get us in trouble with the Union maybe even triggering a work stoppage creating some very bad press for us I nod automatically restore critical business operations and keep us out of the front page news got it thanks why exactly I'm thanking him is not clear chapter 2. Tuesday September 2nd how'd it go in there Stacy asks kindly looking up from her keyboard I just shake my head I can't believe it he just talked me into taking a new job I don't want how did that happen he can be very persuasive she says for what it's worth he's one of a kind I've worked for him for nearly 10 years and I'll follow him anywhere anything I can help with to make your job easier thinking for a moment I ask there's an urgent payroll issue that needs to be fixed dick Landry is on floor three right here you go she says before I've finished asking my question handing me a Post-It note with all of Dick's contact information office location phone numbers and everything grateful I smile at her thanks a lot you are fantastic I dialed Dick's cell phone on my way to the elevator dick here he answers roughly still typing in the background this is Bill Palmer Steve just made me the new VP of it operations and he asked me to congratulations he interrupts now look my people found a huge payroll irregularity when can you get to my office right away I reply I hear the click of him ending the call I've had warmer welcomes on the third floor I walk through Finance and Accounting surrounded by pinstriped shirts and starched collars I find Dick at his desk still on the phone with someone when he sees me he puts his hand over the mouthpiece you from I.T he asks roughly when I nod he says into the phone look I gotta run someone who's supposedly going to help is finally here I'll call you back without waiting for an answer he hangs up the phone I've never actually seen someone who routinely hangs up on people I brace myself for a conversation that is likely to be short on any comforting let's get to know each other foreplay as though in a hostage situation I slowly raise my hands showing dick the printed email Steve just told me about the payroll outage what's the best way for me to get some situational awareness here we're in deep kimchi dick responds in yesterday's payroll run all of the records for the hourly employees went missing we're pretty sure it's an I.T issue this screw-up is preventing us from paying our employees violating countless State labor laws and no doubt the union is going to scream bloody murder he mutters under his breath for a moment let's go see Anne my operations manager she's been pulling her hair out since yesterday afternoon walking quickly to keep up I nearly run into him when he stops and peers through a conference room window he opens the door how's it going in here Anne there are two well-dressed women in the room one around 45 years old studies the Whiteboard filled with flow charts and a lot of tabulated numbers and the other in her early 30s Types on a laptop spreadsheets are strewn all over the large conference room table the older woman gestures with an open marker at what appears to be a list of potential failure causes something about the way they dress and they are concerned and irritated Expressions makes me think they were recruited from a local accounting firm ex-auditors good to have them on our side I suppose and shakes her head in exhausted frustration not much progress I'm afraid we're almost certain it's an I.T system's failure in one of the Upstream timekeeping systems all of the hourly factory worker records got screwed up in the last upload dick interrupts her this is Bill from it he's been assigned to fix this mess or die trying is what I think he said I say hi guys I've been made the new head of IT operations can you start from the beginning and tell me what you know about the problem and walks over to the flowchart on the Whiteboard let's start with the information flow our financial system gets payroll data from our various divisions in different ways we roll up all the numbers for salaried and hourly personnel which includes wages and taxes sounds easy but it's extremely complex because each state has different tax tables labor laws and so forth to make sure something doesn't get screwed up she continues we make sure the summarized numbers match the detailed numbers from each division as I hurriedly jot down some notes she continues it's a pretty clunky and manual process it works most of the time but yesterday we discovered that the general ledger upload for hourly production staff didn't come through all of the hourlys had zeros for their hours worked and amount due we've had so many problems with this particular upload she says obviously frustrated that it gave us a program that we use to do manual Corrections so we don't have to bother them anymore I wince I don't like Finance Personnel manually changing payroll data outside the payroll application it's Error prone and dangerous someone could copy the data onto a USB drive or email it outside of the organization which is how organizations lose sensitive data did you say all the numbers for salaried employees are okay I ask that's right she replies but hourly employees are all zeros I confirm yep she again replies interesting I ask why do you think the payroll run failed when it was working before have you had problems like this in the past she shrugs nothing like this has happened before I have no idea what could have caused it no major changes were scheduled for this pay period I've been asking the same questions but until we hear from the it guys we're stuck dead in the water what is our backup plan I ask if things are so hosed that we can't get the hourly employee data in time for crying out loud dick says it's in that email you're holding the deadline for electronic payments is 5 pm today if we can't hit that window we may have to FedEx bales of paper checks to each of our facilities for them to distribute to employees I frown at this scenario and so does the rest of the finance team that won't work and says clicking a marker on her teeth we've outsourced our payroll processing each pay period we upload the payroll data to them which they then process in the worst case maybe we download the previous payroll run modified in a spreadsheet and then re-upload it but because we don't know how many hours each employee worked we don't know how much to pay them she continues we don't want to overpay anyone but that's better than accidentally underpaying them it's obvious that plan B is fraught with problems we'd basically be guessing at people's paychecks as well as paying people who were terminated and not paying people who were newly hired to get Finance the data they need we may have to Cobble together some custom reports which means bringing in the application developers or database people but that's like throwing gasoline on the fire developers are even worse than networking people show me a developer who isn't crashing production systems and I'll show you one who can't fog a mirror or more likely is on vacation dick says these are two lousy options we could delay our payroll run until we have the correct data but we can't do this even if we're only a day late we'll have the union stepping in so that leaves Anne's proposal of paying our employees something even if it's the incorrect amount we'd have to just adjust everyone's paycheck in the next pay period but now we have a financial reporting error that we've got to go back and fix he pinches the bridge of his nose and continues to ramble we'll have a bunch of odd journal entries in our general lecture just when our Auditors are here for our Sox 404 audits when they see this they'll never leave oh man a financial reporting error dick mutters will need approval from Steve we're going to have Auditors camped out here until the cows come home no one will ever get any real work done again Sox 404 is short for the sarbanes-oxley act of 2002 which Congress enacted in response to the accounting failures at Enron Worldcom and Tycho it means the CEO and CFO have to personally sign their names attesting that their company's financial statements are accurate everyone Longs for the days when we didn't spend half our time talking to Auditors complying with each new regulatory requirement du jour I look at my notes and then at the clock time is running out Dick based on what I've heard I recommend that you continue to plan for the worst and we fully document Plan B so we can pull it off without further complications furthermore I request that we wait until 3 pm before making a decision we may still be able to get all the systems and data back when Anne nods dick says okay you've got four hours I say rest assured that we understand the urgency of the situation and that you'll be appraised of how it's going as soon as I find out myself thanks Bill and says dick remains silent as I turn around and walk out the door I feel better now that I've seen the problem from the business perspective it's now time to get under the covers and find out what broke the complex payroll machinery while walking down the stairs I dig out my phone and scan my emails my feeling of calm Focus disappears when I see that Steve hasn't sent out an announcement of my promotion Wes Davis and Paddy McKee who until today were my peers still have no idea that I'm now their new boss thanks Steve when I enter Building 7 it hits me our building is the ghetto of the entire Parts on limited campus it was built in the 1950s and last remodeled in the 1970s obviously built for utility not Aesthetics Building 7 used to be our large brake pad manufacturing Factory until it was converted into Data Center and office space it looks old and neglected the security guard says cheerfully hello Mr Palmer how is the morning going so far for a moment I'm tempted to ask him to wish me luck so he can get paid the correct amount this week of course I merely return his friendly greeting I'm headed toward the network operations center or as we call it the knock where Wes and Paddy are most likely to be they're now my two primary managers Wes is director of distributed technology operations he has a technical responsibility for over a thousand Windows servers as well as the database and networking teams Paddy is the director of IT service support she owns all the level 1 and 2 help desk technicians who answer the phones around the clock handling brake fix issues and support requests from the business she also owns some of the key processes and tools that the entire it operations organization relies upon like the trouble ticketing system monitoring and running the change management meetings I walk past rows upon rows of cubicles the same as every other building however unlike buildings two and five I see peeling paint and dark stains seeping through the carpet this part of the building was built on top of what used to be the main assembly floor when they converted it they couldn't get all the machine oil cleaned up no matter how much sealant we put down to coat the floors oil still has a tendency to seep through the carpet I make a note to put in a budget request to replace the carpets and paint the walls in the Marines keeping the barracks neat and tidy was not only for Aesthetics but also for safety old habits die hard I hear the knock before I see it it's a large Bullpen area with long tables set up along one wall displaying the status of all the various I.T services on large monitors the level 1 and 2 help desk people sit at the three rows of workstations it's not exactly like Mission Control in the movie Apollo 13 but that's how I explain it to my relatives when something hits the fan you need all the various stakeholders and Technology managers to communicate and coordinate until the problem is resolved like now at the conference table 15 people are in the midst of a loud and heated discussion huddled around one of the classic gray speaker phones that resembles a UFO Wes and Patty are sitting next to each other at the conference table so I walk behind them to listen in Wes leans back in his chair with his arms crossed over his stomach they don't get all the way across at six feet three inches tall and over 250 pounds he casts a shadow on most people he seems to be in constant motion and has a reputation of saying whatever is on his mind Patty is the complete opposite where Wes is loud outspoken and shoots from the hip Paddy is thoughtful analytical and a stickler for processes and procedures where Wes's large combative and sometimes even quarrelsome Patty is Elfin logical and level-headed she has a reputation for loving processes more than people and is often in the position of trying to impose order on the chaos in it she's the face of the entire I.T organization when things go wrong in it people call Patty she's our professional apologist whether it's service is crashing web pages taking too long to load or as in today's case missing or corrupted data they also call Patty when they need their work done like upgrading a computer changing a phone number or deploying a new application she does all of the scheduling so people are always lobbying her to get their work done first she'll then hand it off to people who do the work for the most part they live in either my old group or in Wes's group Wes pounds the table saying just get the vendor on the phone and tell them that unless they get a tech down here Pronto we're going to the competition we're one of their largest customers we should probably have abandoned that pile of crap by now come to think of it he looks around and jokes you know the saying right the way you can tell a vendor is lying is when their lips are moving one of the engineers across from West says we have them on the phone right now they say it'll be at least four hours before their sand field engineer is on site I frown why are they talking about the SAN storage area networks provide centralized storage to many of our most critical systems so failures are typically Global it won't be just one server that goes down it'll be hundreds of servers that go down all at once while West starts arguing with the engineer I try to think nothing about this payroll run failure sounds like a San issue Anne suggested that it was probably something in the timekeeping applications supporting each plant but after we tried to roll back the San it stopped serving data entirely another engineer says then the display started displaying everything in kanji well we think it was kanji whatever it was we couldn't make heads or tails of those little pictures that's when we knew we needed to get the vendor involved although I'm joining late I'm convinced we're totally on the wrong track I lean in to whisper to Wes and Patty can I get a minute with you guys in private Wes turns and without giving me his full attention says loudly can't it wait in case you haven't noticed we're in the middle of a huge issue here I put my hand firmly on his shoulder Wes this is really important it's about the payroll failure and concerns a conversation I just had with Steve Masters and Dick Landry he looks surprised Paddy is already out of her chair let's use my office she says leading the way following Paddy into her office I see a photo on her wall of her daughter who I'd guess is 11 years old I'm amazed at how much she looks like Patty Fearless incredibly smart and formidable in a way that is a bit scary and such a cute little girl in a Gruff voice Wes says okay Bill what's so important that you think is worth interrupting a Sev one outage in progress that's not a bad question severity one outages are serious business impacting incidents that are so disruptive we typically drop everything to resolve them I take a deep breath I don't know if you've heard but Luke and Damon are no longer with the company the official word is that they've decided to take some time off more than that I don't know the surprised Expressions on their faces confirm my suspicions they didn't know I quickly relate the events of the morning Patty shakes her head uttering a tisk tisk in disapproval Wes looks angry he worked with Damon for many years his face redens so now we're supposed to take orders from you look no offense pal but aren't you a little out of your league you've managed the mid-range systems which are basically antiques for years you created a nice cushy little job for yourself up there and you know what you have absolutely no idea how to run modern distributed systems to you the 1990s is still the future quite frankly he says I think your head would explode if you had to live with the Relentless pace and complexity of what I deal with every day I exhale while counting to three you want to talk to Steve about how you want my job be my guest let's get the business what they need first and make sure that everyone gets paid on time Paddy responds quickly I know you weren't asking me but I agree that the payroll incident needs to be our Focus she pauses and then says I think Steve made a good choice congratulations bill when can we talk about a bigger budget I flash her a small smile and a nod of thanks returning my gaze to Wes a couple moments go by and expressions I can't quite decipher across his face finally he relents yeah fine and I will take you up on your offer to talk to Steve he's got a lot of explaining to do I nod thinking about my own experience with Steve I genuinely wish West luck if he actually decides to have a showdown with him thank you for your support guys I appreciate it now what do we know about the failure or failures what's all this about some San upgrade yesterday are they related we don't know Wes shakes his head we were trying to figure that out when you walked in we were in the middle of a sand firmware upgrade yesterday when the payroll run failed Brent thought the San was corrupting data so he suggested we back out the changes it made sense to me but as you know they ended up bricking it up until now I've only heard bricking something in reference to Breaking something small like when a cell phone update goes bad using it to refer to a million dollar piece of equipment where all our Irreplaceable corporate data are stored makes me feel physically ill Brent works for Wes he's always in the middle of the important projects that it is working on I've worked with him many times he's definitely a smart guy but can be intimidating because of how much he knows what makes it worse is that he's right most of the time you heard them Wes says gesturing toward the conference table where the outage meeting continues unabated the sand won't boot won't serve data and our guys can't even read any of the error messages on the display because they're in some weird language now we've got a bunch of databases down including of course payroll to work the San issue we had to pull Brent off of a phoenix job we promised to get done for Sarah Paddy says ominously there's going to be hell to pay uh oh what exactly did we promise her I ask alarmed Sarah is the senior vice president of retail operations and she also works for Steve she has an uncanny knack for blaming other people for her screw-ups especially I.T people for years she's been able to escape any sort of real accountability although I've heard rumors that Steve is grooming her as his replacement I've always discounted that as being totally impossible I'm certain that Steve can't be blind to her machinations Sarah heard from someone that we were late getting a bunch of virtual machines over to Chris she replies we dropped everything to get on it that is until we had to drop everything to fix the San Chris ahlers our VP of application development is responsible for developing the applications and code that the business needs which then get turned over to us to operate and maintain Chris's life is currently dominated by Phoenix I scratch my head as a company we've made a huge investment in virtualization although it looks uncannily like the Mainframe operating environment from the 1960s virtualization changed the game in Wes's world suddenly you don't have to manage thousands of physical servers anymore they're now logical instances inside of one big Iron server or maybe even residing somewhere in the cloud building a new server is now a right click inside of an application cabling it's now a configuration setting but despite the promise that virtualization was going to solve all our problems here we are still late in delivering a virtual machine to Chris if we need Brent to work the San issue keep him there I'll handle Sarah I say but if the payroll failure was caused by the San why didn't we see more widespread outages and failures Sarah is definitely going to be one unhappy camper you know suddenly I don't want your job anymore Wes says with a loud laugh don't get yourself fired on your first day they'll probably come for me next Wes pauses to think you know you have a good point about the sand Brent is working the issue right now let's go to his desk and see what he thinks Patty and I both nod it's a good idea we need to establish an accurate timeline of relevant events and so far we're basing everything on hearsay that doesn't work for solving crimes and it definitely doesn't work for solving outages chapter 3 Tuesday September 2nd I follow Patty and Wes as they walk past the knock into the sea of cubicles we end up in a giant workspace created by combining six cubicles a large table sits against one wall with a keyboard and four LCD monitors like a Wall Street trading desk there are piles of servers everywhere all with blinking lights each portion of the desk is covered by more monitors showing graphs login Windows code editors Word documents and countless applications I don't recognize Brent types away in a window oblivious to everything around him from his phone I hear the knock conference line he obviously doesn't seem worried that the loud speakerphone might bother his neighbors hey Brent you got a minute Wes asks loudly putting a hand on his shoulder can it wait Brent replies without even looking up I'm actually kind of busy right now working the San issue you know Wes grabs a chair yeah that's what we're here to talk about when Brent turns around Wes continues tell me again about last night what made you conclude that the San upgrade caused the payroll run failure Brent rolls his eyes I was helping one of the San Engineers perform the firmware upgrade after everybody went home it took way longer than we thought nothing went according to the tech note it got pretty hairy but we finally finished around seven o'clock we rebooted the sand but then all the self-tests started failing we worked it for about 15 minutes trying to figure out what went wrong that's when we got the emails about the payroll run failing that's when I said game over we were just too many versions behind the sand vendor probably never tested the upgrade path we were going down I called you telling you I wanted to pull the plug when you gave me the knot we started the rollback that's when the sand crashed he says slumping in his chair it not only took down payroll but a bunch of other servers too we've been meaning to upgrade the sand firmware for years but we never got around to it Wes explains turning to me we came close once but then we couldn't get a big enough maintenance window performance has been getting worse and worse to the point where a bunch of critical apps were being impacted so finally last night we decided to just bite the bullet and do the upgrade I nod then my phone rings it's Anne so I put her on speakerphone as you suggested we looked at the data we pulled from the payroll database yesterday the last pay period was fine but for this pay period all the social security numbers for the factory hourlys are complete gibberish and all their hours worked and wage fields are zeros too no one has ever seen anything like this before just one field is gibberish I ask raising my eyebrows in Surprise what do you mean by gibberish what's in the fields she tries to describe what she's seeing on her screen well they're not numbers or letters there's some hearts and Spades and some squiggly characters and there's a bunch of foreign characters with umlots and there are no spaces is that important when Brent Snickers as he hears Anne trying to read line noise aloud I give him a Stern glance I think we've got the picture I say this is a very important clue can you send the spreadsheet with the corrupted data to me she agrees by the way are a bunch of databases down now that's funny they were up last night Wes mutter something under his breath silencing Brent before he can say anything um yes we're aware of the problem and we're working it too I deadpan when we hang up I breathe a sigh of relief taking a moment to thank whatever deity protects people who fight fires and fix outages only one field corrupted in the database come on guys that definitely doesn't sound like a sand failure I say Brent what else was going on yesterday besides the San upgrade that could have caused the payroll run to fail Brent slouches in his chair spinning it around while he thinks well now that you mention it a developer for the time keeping application called me yesterday with a strange question about the database table structure I was in the middle of working on that Phoenix test VM so I gave him a really quick answer so I could get back to work you don't suppose he did something to break the app do you Wes turns quickly to the speakerphone dialed into the knock conference call that has been on this whole time and unmutes the phone hey guys it's Wes here I'm with Brent and Patty as well as with our new boss Bill Palmer Steve Masters has put him in charge of all it Ops so listen up guys my desire for an orderly announcement of my new role seems less and less likely Wes continues does anyone know anything about a developer making any changes to the timekeeping application in the factories Brent says he got a call from someone who asked about changing some database tables from the speakerphone a voice pipes up yeah I was helping someone who was having some connectivity issues with the plants I'm pretty sure he was a developer maintaining the timekeeping app he was installing some Security application that John needed to get up and running this week I think his name was Max I still have his contact information around here somewhere he said he was going on vacation today which is why the work was so urgent now we're getting somewhere a developer jamming in an urgent change so he could go on vacation probably as part of some urgent project being driven by John Pesci our chief information security officer situations like this only reinforce my deep suspicion of developers they're often carelessly breaking things and then disappearing leaving operations to clean up the mess the only thing more dangerous than a developer is a developer conspiring with security the two working together gives us means motive and opportunity I'm guessing our chief information security officer or ciso probably strong-armed a development manager to do something which resulted in a developer doing something else which broke the payroll run information security is always flashing their badges at people and making urgent demands regardless of the consequences to the rest of the organization which is why we don't invite them to many meetings the best way to make sure something doesn't get done is to have them in the room they're always coming up with a million reasons why anything we do will create a security hole that alien space hackers will exploit to pillage our entire organization and steal all our code intellectual property credit card numbers and pictures of our loved ones these are potentially valid risks but I often can't connect the dots between their shrill hysterical and self-righteous demands and actually improving the defensibility of our environment okay guys I say decisively the payroll run failure is like a crime scene and we're a Scotland Yard the San is no longer a suspect but unfortunately we've accidentally maimed it during our investigation Brent you keep working on the injured sand obviously we've got to get it up and running soon Wes and Patty are new Persons of Interest are Max and his manager I say do whatever it takes to find them detain them and figure out what they did I don't care if Max is on vacation I'm guessing he probably messed up something and we need to fix it by 3 pm I think for a moment I'm going to find John either if you want to join me Wes and Patty argue over who will help interrogate Jon Paddy says adamantly it should be me I've been helping to keep John's people in line for years they never follow our process and it always causes problems I'd love to see Steve and Dick rake him over the coals for pulling a stunt like this it is apparently a convincing argument as Wes says okay he's all yours I almost feel sorry for him now I suddenly regret my choice of words this isn't a Witch Hunt and I'm not looking for Retribution we still need a timeline of all relevant events leading up to the failure jumping to inappropriate conclusions caused the sand failure last night we won't make these kinds of mistakes again not on my watch as Paddy and I call John I squint at the phone number on Paddy's screen wondering if it's time to heed my wife's advice to get glasses yet another reminder that 40 is just around the corner I dial the number and a voice answers in one ring John here I quickly tell him about the payroll and sand failure and then ask did you make any changes to the time keeping application yesterday he says that sounds bad but I can assure you that we didn't make any changes to your mid-range systems sorry I can't be of more help I sigh I thought that by now either Steve or Laura would have sent out the announcement of my promotion I seem destined to explain my new role in every interaction I have I wonder if it would be easier if I just sent out the announcement myself I repeat The Abridged story of my Hasty promotion yet again Wes Paddy and I heard that you were working with Max to deploy something urgent yesterday what was it Luke and Damon are gone John sounds surprised I never thought that Steve would actually fire both of them over a compliance audit finding but who knows maybe things are finally starting to change around here let this be a lesson to you Bill you operations people can't keep dragging your feet on security issues anymore just some friendly advice speaking of which I'm suspicious about how the competition keeps getting the jump on us he continues as they say once is coincidence twice is happenstance third must be enemy action maybe our sales people's email systems have been hacked that would sure explain why we're losing so many deals John continues to talk but my mind is still stuck at his suggestion that Luke and Damon may have been fired over something security related it's possible Jon routinely deals with some pretty powerful people like Steve and the board as well as the internal and external auditors however I'm certain Steve didn't mention either John or information security as reasons for their departure only the need to focus on Phoenix I look at paddy questioningly she just rolls her eyes and then Twirls her finger around her ear lier she thinks Jon's theory is crazy has Steve given you any insights on the new org structure I ask out of genuine curiosity John is always complaining that information security was always prioritized too low he's been lobbying to become a peer of the CIO saying it would resolve an inherent conflict of interest to my knowledge he hasn't succeeded it's no secret that Luke and Damon sideline Jon as much as possible so he couldn't interfere with people who did real work Jon still managed to show up at meetings despite their best efforts what I have no clue what's going on he says in an aggrieved tone my question apparently striking a nerve I'm being kept in the dark like usual I'll probably be the last to find out too if history is any guide until you told me I thought I was still reporting to Luke and now that he's gone I don't know who I'm reporting to you got a call from Steve this is all above my pay grade I'm as much in the dark as you are I respond playing it dumb quickly changing the subject I ask what can you tell us about the timekeeping app change I'll call Steve and find out what's going on he's probably forgotten Information Security even exists he continues making me wonder whether we'll ever be able to talk about payroll to my relief he finally says okay yeah you were asking about Max we had an urgent audit issue around storage of pii that is personally identifiable information like ssns that's Social Security numbers obviously birthdays and so forth European Union law and now many U.S state laws prohibit us from storing that kind of data we got a huge audit finding around this I knew it was up to my team to save this company from itself and prevent us from getting dinged again that would be Front Page News you know he continues we found a product that tokenized this information so we no longer have to store the ssns it was supposed to be deployed almost a year ago but it never got done despite all my badgering now we're out of time the payment card industry Auditors that's PCI for short are here later this month so I fast tracked the work with the timekeeping team to get it done I stare at my phone speechless on the one hand I'm ecstatic because we found the Smoking Gun in John's hand John's mention of the SSN field matches Anne's description of the corrupted data on the other hand let me see if I've got this right I say slowly you deployed this tokenization application to fix an audit finding which caused the payroll run failure which has Dick and Steve climbing the walls John responds hotly first I am quite certain the tokenization security product didn't cause the issue it's inconceivable the vendor assured us that it's safe and we checked all their references second dick and Steve have every reason to be climbing the walls compliance is not optional it's the law my job is to keep them out of orange jumpsuits and so I did what I had to do orange jumpsuits like what you wear in prison he says my job is to keep Management in compliance with all relevant laws regulations and contractual obligations Luke and Damon were Reckless they cut Corners that severely affected our audit and security posture if it weren't for my actions we'd probably all be in jail by now I thought we were talking about a payroll failure not being thrown in jail by some imaginary police force John we have processes and procedures for how you introduce changes into production Paddy says you went around them and once again you've caused a big problem that we're having to repair why didn't you follow the process ha good one patty John snortz I did follow the process you know what your people told me that the next possible deployment window was in four months hello the Auditors are on site next week he says adamantly getting trapped in your bureaucratic process was simply not an option if you were in my shoes you do the same thing Patty redden's I say calmly according to Dick we have fewer than four hours to get the timekeeping app up now that we know there was a change that affected ssn's I think we have what we need I continue Max who helped with the deployment is on vacation today Wes or Brent will be contacting you to learn more about this tokenization product you deployed I know you'll provide them with whatever help they need this is important when John agrees I thank him for his time wait one more question why do you believe that this product didn't cause the failure did you test the change there's a short silence on the other phone before John replies no we couldn't test the change there's no test environment apparently you guys requested a budget years ago but I should have known well that's good news Paddy says after John hangs up it may not be easy to fix but at least we finally know what's going on was John's tokenization change in the change schedule I ask she laughs humorously that's what I've been trying to tell you John rarely goes through our change process nor do most people for that matter it's like the wild west out here we're mostly shooting from the hip she says defensively we need more process around here and better support from the top including it process tooling and training everyone thinks that the real way to get work done is to just do it that makes my job nearly impossible in my old group we were always disciplined about doing changes no one made changes without telling everyone else and we'd bend over backward to make sure our changes wouldn't screw someone else up I'm not used to Flying this blind we don't have time to do interrogations every time something goes wrong I say exasperated give me a list of all the changes made in the past say three days without an accurate timeline we won't be able to establish cause and effect and we'll probably end up causing another outage good idea she nods if necessary I'll email everyone in it to find out what they were doing to catch things that weren't on our schedule what do you mean email everyone there's no system where people put in their changes what about our ticketing system or the change authorization system I ask stunned this is like Scotland Yard emailing everyone in London to find out who was near the scene of a crime dream on she says looking at me like I'm a newbie which I suppose I am for years I've been trying to get people to use our change management process and tools but just like John no one uses it same with our ticketing system it's pretty hit or miss too things are far worse than I thought okay do what you need to do I finally say unable to hide my frustration make sure you hit all the developers supporting the timekeeping system as well as all the system administrators and networking people call their managers and tell them it's important that we know about any changes regardless of how unimportant they may seem don't forget John's people too when Patty nods I say look you're the change manager we've got to do better than this we need better situational awareness and that means we need some sort of functional change management process get everyone to bring in their changes so we can build a picture of what is actually going on out there to my surprise Paddy looks dejected look I've tried this before I'll tell you what will happen the change Advisory board or cab will get together once or twice and within a couple of weeks people will stop attending saying they're too busy or they'll just make the changes without waiting for authorization because of deadline pressures either way it'll fizzle out within a month not this time I say adamantly send out a meeting notice to all the technology leads and announce that attendance is not optional if they can't make it they need to send a delegate when is the next meeting tomorrow she says excellent I say with genuine enthusiasm I'm looking forward to it when I finally get home it's after midnight after a long day of disappointments I'm exhausted balloons are on the floor and a half empty bottle of wine sits on the kitchen table on the wall is a cram poster saying congratulations Daddy when I called my wife Paige this afternoon telling her about my promotion she was Far happier than I was she insisted on inviting the neighbors over to throw a little celebration coming home so late I missed my own party at 2PM Paddy had successfully argued that of the 27 changes made in the past three days only John's tokenization change and the San upgrade could be reasonably linked to the payroll failure however Wes and his team were still unable to restore San operations at 3 pm I had to tell Anne and Dick the bad news that we had no choice but to execute Plan B their frustration and disappointment were all too evident it wasn't until 7pm when the timekeeping application was back up and 11 p.m when the San was finally brought back online not a great performance on my first day as VP of it operations before I left work I emailed Steve dick and Ann a quick status report promising to do whatever it takes to prevent this type of failure from happening again I go upstairs finish brushing my teeth and check my phone one last time before going to bed being careful not to wake up Paige I curse when I see an email from our company PR manager with a subject of bad news we may be on the front page tomorrow I sit on the bed squinting to read the accompanying news story Elkhart Grove Herald Times unlimited flubs paychecks local union leader calls failure unconscionable automotive parts supplier Parts Unlimited has failed to adequately compensate its workers with some employees receiving no pay at all according to a Parts Unlimited internal memo the locally headquartered company admitted that it had failed to issue correct paychecks to some of its hourly Factory workers and that others hadn't received any compensation for their work Parts Unlimited denies that the issue is connected to cash flow problems and instead attributes the error to a payroll system failure the once high-flying four billion dollar company has been plagued by flagging revenue and growing losses in recent quarters these Financial woes which some blame on a failure of upper management have led to rampant job insecurity among local workers struggling to support their families according to the memo whatever the cause of the payroll failure employees might have to wait days or weeks to be compensated this is just the latest in a long string of management execution missteps taken by the company in recent years according to Nestor Myers Chief industry analyst Kelly Lawrence Parts Unlimited CFO dick Landry did not return phone calls from The Herald Times requesting comment on the payroll issue accounting errors and questions of managerial competency in a statement issued on behalf of Parts Unlimited Landry expressed regret at the glitch and vowed that the mistake would not be repeated The Herald times will continue to post updates as the story progresses too tired to do anything more I turn off the lights making a mental note to myself to find Dick tomorrow to apologize in person I close my eyes and try to sleep an hour later I'm still staring at the ceiling very much awake chapter 4 Wednesday September 3rd I drink my coffee as I open up my laptop at 7 30 am hoping to get through my emails and voicemails before my 8 am meeting I stare at the screen in the 22 hours since I was promoted 526 new emails have arrived in my inbox holy crap I skip all the messages about yesterday's failure and am startled by all the congratulatory notes from vendors wanting to meet for lunch how did they find out I'm pretty sure most of my organization still doesn't know I read an email from Ellen my former boss's assistant who is now assigned to support me congratulating me and asking when we can meet I reply telling her I'd like to take her out for coffee this morning I send a note to the IT service desk requesting that Ellen be granted access to my calendar a blinking red light on my desk phone catches my attention it reads 7 50 am 62 new voicemails my jaw drops it would take an hour I don't have just to listen to them I email Ellen again asking her to go through all my voicemails transcribing any that require action before I hit send I quickly add if there are any messages from Steve or dick please call me right away on my cell phone grabbing my clipboard I hurry toward my first meeting when my phone vibrates it's an urgent email from Sarah Moulton to Bill Palmer CC Steve Masters date September 3rd 7 58 am priority highest subject latest Phoenix slip bill as you know project Phoenix is the most important project this company is undertaking I've heard disturbing rumors that you are holding up the release I don't need to remind you that our competition isn't standing still each day that goes by our market share goes down I need everyone to have a sense of urgency especially from you Bill we have an emergency project management meeting at 10 A.M today please join us and be prepared to explain these unacceptable delays Steve I know how important this project is for you given the commitments you've made to the board please feel free to attend we'd love your perspective regards Sarah oh no I forward the email to Wes and Patty flagging it as high priority something seems wrong in a world where half the email messages sent are urgent can everything really be that important I call Wes's cell phone I just got your email from Sarah he says what utter what's this all about I ask he says I'm pretty sure it's about Brent not finishing up that configuration work for the Phoenix developers everyone is chasing their tails because the developers can't actually tell us what the test environment should look like we're doing our best but every time we deliver something they tell us we did it wrong when did they tell us about it I ask two weeks ago it's the typical with development but worse they're so freaked out about hitting their deadlines they're only now starting to think about how to test and deploy it apparently they're making it our problem I hope you're wearing your asbestos underwear like me Sarah is going to be at that meeting with torches wanting to throw us onto the bonfire it's amazing to me how handoffs between development and it operations always get screwed up but given the Perpetual tribal Warfare between the two groups maybe I shouldn't be surprised I reply I get the picture look make sure you dig into this Dev specification issue personally we've got to get this nailed down grab everyone involved whether they're in Dev or Ops and lock them in a room until they come up with a written specification Phoenix is so important we can't afford to screw this up Wes says he's on it and I ask is there anything else Sarah could pop on us he pauses to think and finally says no I don't think so we have a pretty valid reason with the payroll run failure for why Brent wasn't able to complete his work I agree feeling like our asses are sufficiently covered I say see you at 10. less than an hour later I'm walking to building nine in the hot Sunshine where many of the marketing folks call home to my surprise I join a small army of I.T people walking the same way why then it hits me the majority of our marketing projects can't be done without it high touch marketing requires high tech but if there's so many of us assigned to these marketing projects shouldn't they be coming to us I imagine that Sarah likes it this way the spider sitting back enjoying seeing all the company minions making their way to her Lair I arrive and immediately see Kirsten fingle who runs the project management office sitting at the head of the table I am a big fan of hers she is organized level-headed and a stickler for accountability when she first joined the company five years ago she brought a whole new level of professionalism to our organization at her right Sarah leans back in her chair tapping away on her iPhone oblivious to the rest of us Sarah is my age 39. she's very guarded about her age always saying things in a way that would lead one to conclude she's much older but never actually lying yet another maddening thing about Sarah there are about 25 people in the room many of the business line owners are here some of whom work for Sarah Chris alllers is also here Chris is a little older than me and looks lean and fit he'll just as often be seen joking with someone as kicking their ass about missing a deadline he has a reputation as a capable and no-nonsense manager with nearly 200 developers working for him he needs to be to help with Phoenix his team has grown by 50 people in the last two years many through offshore development shops Chris is constantly asked to deliver more features and do it in less time with less money several of his managers are in the room too Wes is also here sitting right next to Chris as I start to look for an open chair I note how everyone seems unusually tense and then I see why there sitting right next to the only open seat at the table is Steve everyone seems to be going to Great Lengths not to stare at him as I casually take my seat next to Steve my phone vibrates it's a text message from Wes Steve has never attended a project management meeting we are totally screwed Kirsten clears her throat first on our agenda is Phoenix the news isn't good this project went from yellow to Red about four weeks ago and it's my personal assessment that the deadline is in grave Jeopardy she continues in her professional voice to refresh your memory last week there were 12 tasks in the critical path of Phoenix phase one only three of those tasks were completed there is a collective grown in the room and several people mutter to one another Steve turns to look at me well I explain the critical resource in question is Brent who has been 100 utilized helping to recover from the payroll failure which we all know about this was a totally unforeseen emergency but obviously one that we had to handle everyone knows how important Phoenix is and we are doing everything we can to make sure Brent can stay focused thanks for that super creative explanation Bill Sarah immediately responds the real issue here is that your people don't seem to grasp how important Phoenix is to the company our competition is killing us in the market you've all seen and heard the commercials about their new Services they're beating us on Innovation both in the retail stores and online they've already lured away some of our biggest partners and our sales force is starting to panic I'm not the type to say I told you so but their latest product announcement shows why we can't be acting as if this is just business as usual she continues see Bill in order for us to increase market share we must ship Phoenix but for some reason you and your team keep dragging your feet maybe you're not prioritizing correctly or maybe you're just not used to supporting a project this important despite all my mental preparation I feel my face get hot with anger maybe it was the condescending way she was parroting Steve to me or how she wasn't even looking at me while she was addressing me instead looking at Steve to see how he reacts or the way she basically called me out of touch and incompetent everyone is silent as I force myself to take a deep breath my anger dissipates this is all just corporate theater I don't like it but accept it for what it is I almost made the Marines my career when I was up for promotion to staff sergeant you don't become a senior NCO in the Marines without being able to play politics interesting I say to Sarah you tell me which is more important getting our Factory employees paid or getting the Phoenix tasks done Steve told me to resolve the payroll failure how would you have prioritized this differently than Steve at my mention of Steve Sarah's expression changes well maybe if it didn't cause the failure in the first place you wouldn't have blown your commitments to us I don't think we can depend on you and your team I nod slowly not taking the bait I look forward to any suggestions you have to offer Sarah she looks at me then at Steve apparently deciding there are no more points to be gained here she rolls her eyes I see Wes shaking his head in disbelief at this discussion staying uncharacteristically quiet Sarah continues we've spent over 20 million on Phoenix and we're nearly two years late we must get to Market looking over at Chris she asks given the delays from Bill's group when is the soonest we can go live Chris looks up from his papers I've looked into this since we talked last week if we expedite some things and if the virtualized environments from Bill's team work as expected we can go into production one week from Friday I gape at Chris he just made up an arbitrary date to go into production with complete disregard for all the things we need to do before deployment I have a sudden flashback in the Marines we had a ritual for all the senior ncos we'd hang out with beers and watch Star Wars Return of the Jedi every time Admiral Ackbar would cry it's a trap we'd all laugh uproariously yelling for a replay this time I'm not laughing now just wait a minute here Wes interjects pounding the table what the hell are you trying to pull we just found out two weeks ago about the specifics of the Phoenix deployment your guys still haven't told us what sort of infrastructure we need so we can't even order the necessary server and networking gear and by the way the vendors are already quoting us three-week delivery times he is now facing Chris pointing at him angrily and I've heard that the performance of your code is so shitty we're going to need the hottest fastest gear out there you're supposed to support 250 transactions per second and you're barely doing even four we're going to need so much Hardware that we'll need another chassis to put it all in and probably have to pay a Custom Manufacturing fee to get it in time God knows what this will do to the budget Chris wants to respond but Wes is relentless we still don't have a concrete specification of how the production and test system should be configured oh do you guys not need a test environment anymore you haven't even done any real testing of your code yet because that fell off the schedule too my heart lurches as all the implications sink in I've seen this movie before the plot is simple first you take an urgent date driven project where the shipment date cannot be delayed because of external commitments made to Wall Street or customers then you add a bunch of developers who use up all the time in the schedule leaving no time for testing or operations deployment and because no one is willing to slip the deployment date everyone after development has to take outrages and unacceptable shortcuts to hit the date the results are never pretty usually the software product is so unstable and unusable that even the people who were screaming for it end up saying that it's not worth shipping and it's always it operations who still has to stay up all night rebooting servers hourly to compensate for crappy code doing whatever heroics are required to hide from the rest of the world just how bad things really are guys I understand the desire to get Phoenix into production as quickly as possible I say to Steve and Chris as calmly as I can but based on what we've heard from Wes I think it's incredibly premature to deploy we still don't know what equipment we need to hit the performance objectives nor have we done any capacity testing to confirm our guesses it's unlikely we have adequate documentation to run this thing in production let alone get everything monitored and backed up in my most persuasive voice I continue I want Phoenix in the market as badly as anyone else but if the user experience is bad enough we'll end up driving our customers to the competition I turn to Chris you can't just throw the pig over the wall to us and then high-five each other in the parking lot congratulating yourselves on how you made the deadline Wes is telling us that the pig will probably break its leg and it'll be my guys who work all-nighters and weekends to keep that pig alive Chris replies hotly don't give me that about throwing the pig over the wall we invited your people to our architecture and planning meetings but I can count on one hand the number of times you guys actually showed up we routinely have had to wait days or even weeks to get anything we need from you guys then he just holds up his hands as if everything is outside his control look I'd like more time too but from the very beginning we all knew that this was a date driven project that was a business decision we all made exactly Sarah exclaims before I can respond this just shows how bill and his team lack the necessary sense of urgency Perfection is the enemy of good bill we simply do not have the luxury of time to polish this to whatever gold standard you're proposing we need to create positive cash flow and we cannot do that without taking back market share and to do that we need to deploy Phoenix she looks over at Steve we understand risk don't we Steve you've been doing an absolutely amazing job selling this to analysts and even the guys on CNBC I don't think we want egg on our face by shipping even later than we already are Steve nods his head and rubs his chin rocking back and forth in his chair as he thinks agreed he finally says leaning forward we've made commitments to our investors and analysts that we were going to launch Phoenix this quarter my jaw drops Sarah has blunted all my arguments leading Steve down a reckless destructive path exasperated I say does anyone think this is really odd I've been in this room when we discussed installing new water fountains in the front of every store we gave that team nine months to plan the rollout nine months and all of us agreed that was reasonable now we're talking about Phoenix which impacts thousands of point-of-sale systems and all of the back office order entry systems this is at least 10 000 times more complex than rolling out new water fountains with way more risk to the business and you're only giving us one week to plan and execute the rollout I throw my hands up imploring Steve doesn't this seem a bit Reckless and unfair Kirsten nods but Sarah says dismissively bill that's a touching story but we're not discussing water fountains we're discussing Phoenix besides I believe the decision has already been made Steve says yes it has thank you for sharing what you view as the risks bill he turns to Sarah when is the launch date Sarah replies quickly marketing launches next Saturday September 13th Phoenix will deploy at 5 PM the previous day Steve writes the date in the back of his notebook and says good keep me posted on progress and let me know if there's anything I can do to help I look over at Wes who Minds with his hands an airplane crashing into the table in front of him and bursting into flames in the hallway West says I thought that went pretty well boss I'm not laughing what the hell happened in there how did we get into this position does anyone know what's required from us to support this launch no one has a clue he says shaking his head in disgust we haven't even agreed on how to do the handoff with development in the past they've just pointed to a network folder and said deploy that there's newborn babies dropped off at church doorsteps with more operating instructions than what they're giving us I shake my head at his awful imagery but he's right we've got a serious problem here he continues we're going to have to assemble a huge team including Chris's guys to figure out how we're going to pull this off we have problems at every layer networking servers databases operating systems applications layer 7 switching the whole wad of crap it's going to be late nights for all of us for the next nine days I nod unhappily this type of all hands effort is just another part of life in I.T but it makes me angry when we need to make some heroic diving catch because of someone else's lack of planning I say get your team assembled and ask Chris to assemble his respective team as well stop trying to do this by email and in the ticketing system we need everyone in the same room speaking of commitments I say what was Chris referring to when he said that our guys never showed up to the Phoenix architecture and planning meetings is that true Wes rolls his eyes in frustration yeah it's true that his people would invite us at the last minute seriously who can clear their calendar on less than a day's notice although In fairness he says after a moment we did get ample notice on a couple of the big planning meetings and one of the most critical people who needed to be there wasn't able to make it due to escalations you can probably guess who that is I groan Brent Wes nods yep he's the guy we need at those meetings to tell those developers how things work in the real world and what type of things keep breaking in production the irony of course is that he can't tell the developers because he's too busy repairing the things that are already broken he's right unless we can break this cycle we'll stay in our terrible downward spiral Brent needs to work with developers to fix issues at the source so we can stop fighting fires but Brent can't attend because he's too busy fighting fires I say we need our best Minds to prepare for this deployment so make sure Brent is there West looks sheepish for a moment I ask him what I think he's working a network outage right now he replies not anymore I say they're going to have to fix it without him if someone has a problem with that send them to me okay whatever you want boss he says shrugging his shoulders after the project management meeting I'm in no mood to talk to anyone I sit at my desk and Grumble when my laptop doesn't wake up the disk drive light just keeps blinking when nothing shows on the screen I grab my empty mug that I keep on my desk by the picture of Paige and my two sons and walk to the coffee machine in the corner when I get back to my desk a window on the screen tells me that it's going to install some critical new updates I sit down click OK and watch the status bar crawl across the screen Suddenly I See the dreaded blue screen of death my laptop is now completely locked up and unusable it happens again even after I reboot I mutter in frustration you've got to be kidding me just then Ellen my new assistant pokes her head around the corner holding out her hand she says good morning congratulations on the promotion bill noticing my blue screened laptop she says sympathetically oh that doesn't look good um thanks I say reaching out to shake her hand yeah about this laptop can you get a hold of someone in desktop support there's some serious crap headed our way from Phoenix and I'm going to need it no problem she says nodding with a smile I'll tell them our new VP is hopping mad demanding that his laptop get fixed of all people you need a working computer right you know she adds I've heard that a bunch of other people are having problems like this today I'll make sure you get to the top of the list you can't afford to wait in line more bricked laptops this is surely evidence that the universe is out to get me today by the way I need some help coordinating some emergency Phoenix meetings has anyone granted you access to my calendar yet I ask she rolls her eyes no that's actually why I came down here I wanted to see if you could print out your next couple of days obviously that's out of the question I'll have the desktop support person do that while he's here sometimes it takes weeks for the email administrators to get around to stuff like this weeks that's unacceptable I quickly look at my watch and realize I'll have to tackle this later I'm already late do your best I say I'm off to Paddy's Enterprise change management meeting call me if you need anything okay being 10 minutes late to Paddy's meeting I hurry into the room expecting to see either a bunch of people waiting for me impatiently or perhaps a meeting already underway instead I see only Patty sitting at the conference table typing away on her laptop welcome to the cab bill I hope you can find an empty chair she says where is everybody I ask I'm baffled when I ran the mid-range group my team would never miss our change management meetings it was where we coordinated and organized all our work to make sure we didn't shoot ourselves in the foot I told you yesterday that change management around here is hit or miss Patty says sighing some groups have their own local change management process like yours but most groups do nothing at all yesterday's outage is just proof that we need to have something at the Enterprise level right now the left hand rarely knows what the right hand is doing so what's the problem I ask she purses her lip I don't know we sent a bunch of people to ITIL training so they could get up to speed on all the best practices we brought in some Consultants who helped us replace our ticketing system with an idle compliant change management tool people were supposed to put change requests into it where it would get routed for approvals but even after two years all we have is a great process on paper that no one follows and a tool that no one uses when I pester people to use them all I get are complaints and excuses I nod Idol stands for it infrastructure Library Which documents many it best practices and processes and the idol program has had a reputation of spending years merely walking in circles I'm bothered that Wes isn't here I know he's busy but if he's not here why would any of his people bother to show up efforts like this must start and be continually maintained from the top well they can bring their complaints and excuses to me I say adamantly we're rebooting the change management process with my total support Steve's told me to make sure people can stay focused on Phoenix screw-ups like the sand failure made us miss a phoenix deliverable and now we're paying for it if someone wants to skip a change management meeting they obviously are in need of some special compassionate coaching from me at Paddy's puzzled expression at my Phoenix reference I tell her about how Wes and I spent our morning being run over by the bus Sarah and Chris were at the wheel but Steve was in back cheering them on to floor it not good she says disapprovingly they even ran over Kirsten ha I nod silently but refused to say more I always liked the phrase in Saving Private Ryan there's a chain of command gripes go up not down instead I asked her to walk me through the current change process and the way it's been automated in the tools it all sounds good there's only one way to see if the process works I say schedule another cab meeting for the same time Friday I'll send out an email to all the cab members letting them know that this is mandatory when I get back to my cubicle Ellen is at my desk bending over my laptop writing a note everything working I hope I ask she startles at the sound of my voice oh my God you scared me she says laughing support left you a replacement laptop because they couldn't get your laptop to boot even after a half hour of trying she points at the far side of my desk and I do a double take my replacement laptop appears to be almost 10 years old it's twice as large as my old one and looks three times as heavy the battery has been taped on and half the keyboard lettering is worn off from heavy use for a moment I wonder if this is a practical joke I sit down and bring up my email but everything is so slow that several times I thought it had locked up Ellen has a sympathetic expression on her face the support guy said that this is all they have available today over 200 people are having similar problems and many aren't getting replacements apparently people with your laptop model also have had theirs break because of some security patch I forgot it's Patch Tuesday when Jon and his team roll out all their security patches from our major vendors once again John is causing huge issues and disruptions for my team and me I merely nod and thank her for the help after she's gone I sit down and type out an email to all the cab members my keystrokes often taking 10 seconds to show up on the screen from Bill Palmer two Wes Davis Patty McKee I.T operations management date September 3rd 2 43 PM priority highest subject mandatory cab meeting Friday 2PM today I attended the weekly cab meeting I was extremely disappointed that I was the only one there besides Patty especially given the totally avoidable change-related failure yesterday effective immediately managers or their assigned delegates are required to attend all scheduled cab meetings and to perform their assigned duties we are resurrecting the Parts Unlimited change management process and it will be followed to the letter any persons caught circumventing change management will be subject to disciplinary action there will be a mandatory cab meeting Friday at 2PM see you there call me if you have any questions or concerns thanks for your support bill I hit send waiting 15 seconds for the email to finally leave my outbox almost immediately my cell phone rings it's Wes I say I was just about to call you about the laptops we've got to get Replacements to our managers and employees so they can do their jobs you hear yeah we're on it but I'm not calling about that and I'm not calling about Phoenix either he says sounding irritated look about your memo on change management I know you're the boss but you better know that the last time we did one of these change management kumbayas we ran I.T straight into the ground no one and I mean absolutely no one could get a single thing done Patty insisted on having everyone take a number and wait for her pointy heads to authorize and schedule our changes it was absolutely ridiculous and a total waste of time he's Unstoppable that software application she made us use is a total piece of crap it takes 20 minutes to fill out all those fields for a simple five minute change I don't know who designed the process but I think they assume that we all get paid by the hour and want to talk about doing work instead of actually doing work eventually the networking and server team staged a rebellion refusing to use Patty's tool he continues heatedly but John waved an audit finding around and went to Luke our old CIO and just like you did Luke said that following policies was a condition of employment threatening to fire anybody who didn't follow them my guys were spending half their time doing paperwork and sitting in that damned cab meeting he continues luckily the effort finally died and John was too clueless to catch on that no one was actually going to the meetings anymore even John hasn't gone to one of those meetings in over a year interesting I hear you I say we can't repeat that but we also can't have another payroll disaster Wes I need you there and I need you to help create the solution otherwise you're part of the problem can I count on you I hear him sigh loudly yeah sure but you can also count on me calling if I see Patty trying to create some sort of bureaucracy that sucks out everybody's will to live I sigh before I was merely worried that it operations was under attack by development information security audit and The Business now I'm starting to realize that my primary managers seem to be at war with each other as well what will it take for us to get along chapter 5. Thursday September 4th I wake up with a jolt when the alarm clock goes off at 6 15 am my jaw still hurts from clenching it all night the Dismal prospects of the upcoming Phoenix launch were never far from my mind as usual before climbing out of bed I quickly scan my phone for any bad news usually I would spend about 10 minutes replying to emails it always feels good to lob a couple of balls off my side of the Court I see something that makes me bolt upright so abruptly that I wake up Paige oh my God what what she asks frantically not fully awake it's another email from Steve hang on darling I say to her while I squint to read it from Steve Masters to Bill Palmer CeCe Nancy mailer dick Landry date September 4th 605 am priority highest subject urgent Sox 404 it audit findings review bill please look into this ASAP I don't need to tell you how critical it is to have a clean Sox 404 audit Nancy please work with Bill Palmer who is now in charge of it operations Steve begin forwarded message we just concluded our quarter 3 internal audit in preparation for the upcoming Sox 404 external audit we discovered some very concerning deficiencies that we need to discuss with you due to the severity and urgency of the findings we need to meet with it this morning Nancy indeed there's a two-hour meeting scheduled for 8 am on my calendar set up by Nancy mailer Chief audit executive holy crap she is incredibly smart and formidable years ago during the retail acquisition integration I watched her grill a manager from the business we were acquiring he was presenting their financial performance when she started a rapid fire interrogation like a cross between Colombo Matlock and Scarface he quickly broke admitting that he was exaggerating his division's performance recalling that meeting my armpits feel damp I haven't done anything wrong but given the tone of the email she is clearly hot on the trail of something important and Steve just threw me in her path I've always run a very tight ship in my mid-range Technology Group this kept audit from interfering too much sure there would still be a lot of questions and documentation requests requiring us to spend a few weeks collecting data and preparing responses occasionally they would find something but we would quickly fix it I like to think that we built a mutually respectful working relationship however this email portends something more ominous I look at my watch the meeting is in 90 minutes and I don't have a clue about what she wants to talk about she I exclaim as I jostle Paige's shoulder honey can you drive the kids to school today something really bad just came up involving a chief audit executive and Steve I need to make some phone calls and get to the office right away annoyed she says for two years you've always taken the kids on Thursdays I have an early start today too I'm sorry honey this is really important the CEO of the company asked me to handle this Steve Masters you know the guy on TV and who gives the big speeches at the company holiday party I can't drop another ball after a day like yesterday and the newspaper headline the night before that without a word she storms down the stairs when I finally find the conference room for the 8 A.M meeting I immediately noticed how silent it is devoid of the usual small talk that fills the time while attendees trickle in Nancy sits at the head of the table with four other people sitting around her sitting next to her is John along with his ever-present black three-ring binder as always I'm surprised by how young he is he's probably in his mid-30s with thick curly black hair John has a Haggard look about him and like many college students has continually gained weight in the three years he's been here at Parts Unlimited most likely from all the stress associated with his failing moral Crusade John actually reminds me more of Brent than anyone else in the room however unlike Brent who normally wears a Linux t-shirt Jon wears a starched collared shirt that's slightly too large Wes is conspicuously underdressed compared to everyone in the room but he obviously doesn't care the last person in the room is a young man who I don't recognize presumably the IT Auditor Nancy begins we have just concluded our quarter 3 internal audit in preparation for the upcoming external Sox 404 audits we have a grave situation Tim RIT auditor found an eye-opening number of it control issues worse many are repeat findings going into the third year left unresolved these findings May Force us to conclude that the company no longer has sufficient controls to assert the accuracy of its financial statements this could result in an adverse footnote from the external Auditors in the company 10K filings with the U.S Securities and Exchange Commission although these are only preliminary findings due to the gravity of the situation I have already verbally informed the audit committee I Blanche although I don't understand all the audit jargon I know enough that this could ruin Dick's day and mean potentially more bad Front Page News satisfied that I understand the severity of the situation Nancy nods Tim please walk us through your conclusions he takes out a huge stack of stapled papers handing one out to everyone assembled we have just concluded our audit of the it General controls at Parts Unlimited for all of the critical Financial systems it took a team of four people over eight weeks to create this Consolidated report holy crap I lift the 2-inch thick stack of papers in my hand where did they find a stapler this big it's a printed Excel spreadsheet with 20 rows per page in tiny eight point type the last page is numbered page 189 there must be a thousand issues here I say in disbelief unfortunately yes he responds not entirely able to hide his smug satisfaction we found 952 it General control deficiencies of which 16 are significant deficiencies and two are potential material weaknesses obviously we're very alarmed given how soon the external audit starts we need your remediation plan as soon as possible Wes is hunched over the table one hand on his forehead the other hand flipping through Page after page what kind of horse is this he holds up one page Issue 127 insecure Windows operating system Max sin cookie setting is this a joke in case you haven't heard we've got a real business to run sorry if that interferes with this full-time audit employment racket you've got going on here trust Wes to say what people are thinking but are too smart to actually say aloud Nancy responds Gravely unfortunately at this point the phase of control review and testing is over what we require from you now is the management response letter you need to investigate each of these findings confirm them and then create a remediation plan we'll review it and then present to the audit committee and the board of directors normally you would have months to prepare your response letter and execute your remediation plan she continues suddenly looking apologetic unfortunately the way the audit testing calendar worked out we only have three weeks until the external Auditors arrive that's regrettable we'll make sure to give it more time in the next audit cycle but this time around we require your response by she looks at her calendar one week from Monday at the very latest do you think you can make it oh that's just six working days away we'll need half that time just to read the entire document our Auditors who I've long believed are a force for justice and objectivity are crapping on me too I pick up the huge stack of papers again and look at a couple of random pages there are many entries like Wes red but others have references to inadequate security settings presence of ghost login accounts Change Control issues and segregation of Duties issues John flips his three-ring binder open and says officiously bill I brought up many of the same issues with Wes and your predecessor they convinced the CIO to sign a management waiver stating that he accepted the risk and do nothing given that some of these are now repeat audit findings I don't think we'll be able to talk our way out of it this time he turns to Nancy during the previous management regime it controls clearly weren't a priority but now that all the security chickens are coming home to roost I'm sure bill will be more prudent Wes looks at John with contempt I can't believe John is Grand standing in front of the Auditors it's times like this that make me wonder whose side he's really on oblivious to Wes in me John says to Nancy my department has been remediating some other controls which I think we should be given credit for for starters we've completed the tokenization of the pii on our critical Financial systems so at least we dodged that bullet that finding is now closed Nancy says dryly interesting the presence of pii is not in the scope of the Sox 404 audit so from that perspective focusing on the it General controls might have been a better use of time wait John's Urgent tokenization change was for nothing if that's true John and I need to talk later I say slowly Nancy I genuinely don't know what we can get to you by Friday we're buried in recovery work and are scrambling to support the upcoming Phoenix rollout which of these findings are the most important for us to respond to Nancy nods to Tim who says certainly the first issue is the potential material weakness which is outlined on page seven this finding states that an unauthorized or untested change to an application supporting financial reporting could have been put into production this could potentially result in an undetected material error due to fraud or otherwise management does not have any control that would prevent or detect such a change furthermore your group was also unable to produce any change management meeting minutes which is supposed to meet weekly according to your policy I try not to wince visibly recalling that no one even showed up at the cab meeting yesterday and during the payroll incident we were so oblivious to Jon's tokenization change that we ended up bricking the San if we were clueless about those changes I sincerely doubt that we'd notice if someone disabled a control that would enable a minor say 100 million dollar fraudulent transaction really that's unbelievable I'll look into that I say with what I hope is the right amount of surprise and moral outrage after I pretend to take detailed notes on my clipboard circling and underlining random words I nod prompting Tim to continue next we found numerous instances where developers have administrative access to production applications and databases this violates the required segregation of Duty required to prevent risk for fraud I look over to John really you don't say developers making changes to an application without an approved change order that certainly sounds like a security risk what would happen if someone coerced a developer say Max into doing something unauthorized we've got to do something about that right John John turns bright red but says politely yes of course I agree and would be happy to help Tim says good let's move on to the 16 significant deficiencies a half hour later Tim is still droning on I stare glumly at the huge stack of findings most of these issues are just like the huge useless reports we get from information security which is another reason why John has such a bad reputation it's the never-ending hamster wheel of pain information security fills up people's inboxes with never-ending lists of critical security remediation work quarter after quarter when Tim finally finishes John volunteers we must get these vulnerable systems patched my team has a lot of experience patching systems if you require assistance these audit findings are an awesome opportunity to close some big security holes look both of you guys have no idea what you're asking for Wes says to John and Tim clearly exasperated some of the servers that those manufacturing Erp systems run on are over 20 years old half the company will grind you a halt if they go down and the vendor went out of business decades ago these things are so fragile that if you even look at them at the wrong time of day they'll crash and require all sorts of voodoo to get them to successfully reboot they'll never survive the changes you're proposing he leans over the table putting his finger in John's face you want to patch it yourself fine but I want a signed piece of paper from you saying that if you push the button and the entire business grinds to a halt you'll fly around and grovel to all the plant managers explaining to them why they didn't hit their production targets deal my eyes widen with amazement when John actually leans forward into Wes's finger and says angrily oh yeah how about when we're on the front page of the news because we lost consumer data that were responsible for protecting you'll personally apologize to the thousands or millions of families whose data are now being sold by the Russian mafia I say settle down everyone we all want to do what's right for the company the trick is figuring out what we have time to do and what systems can actually be patched I look at the stack of papers Wes Paddy and I can assign people the task of investigating each issue but who will actually do the work we're already buried with Phoenix and I fear that this new massive project might be the straw that breaks the camel's back I say to Nancy I'll get with my team right away and we'll come up with a plan I can't promise you that we'll have our response letter completed by then but I can promise you that we'll get you everything we can will that be adequate so Nancy says amicably going through the preliminary audit findings and identifying next steps were the only objectives for this meeting as the meeting adjourns I ask Wes to stay behind noticing this John remains behind as well this is a disaster all my objectives and bonuses are tied to getting a clean compliance report for the Sox 404 and PCI audits I'm going to fail because you Ops guys can't get your together join the club I say to get him off my back I say Sarah and Steve decided to move up the Phoenix deployment date to next Friday they want to skip all the security reviews you probably should talk to Chris and Sarah right away predictably John swears and storms out slamming the door behind him exhausted I lean back in my chair and say to Wes this is just not our week Wes laughs humorously I told you that the pace of things around here would make your head explode I gesture at the audit findings we're supposed to protect all our key resources for Phoenix but that's sucking in everybody we don't have a bunch of people just sitting on the bench we can throw at the audit findings right Wes shakes his head his face uncharacteristically pinched with tension he flips through his stack of papers again we're definitely going to need to bring the technology leads into this but as you said they're already assigned to the Phoenix team should we reassign them here I honestly don't know West Air is at one of the pages for a moment by the way I think a bunch of these will require Brent oh come on I mutter Brent Brent Brent Brent can't we do anything without him look at us we're trying to have a management discussion about commitments and resources and all we do is talk about one guy I don't care how talented he is if you're telling me that our organization can't do anything without him we've got a big problem Wes shrugs slightly embarrassed he's undoubtedly one of our best guys he's really smart and he knows a lot about almost everything we have in this shop he's one of the few people who really understand how all the applications talk together at an Enterprise level heck the guy may know more about how this company Works than I do you're a senior manager this should be as unacceptable to you as it is to me I say firmly how many more brents do you need one ten or a hundred I'm going to need Steve to prioritize all this work what I need from you is what resources we need if I ask Steve for more resources I don't want to have to crawl back begging for more later he rolls his eyes look I'll tell you right now what's going to happen we'll go to management and present our case not only will they say no they'll cut our budget by another five percent that's what they've done for the past five years in the meantime everyone will continue to want everything at the same time and keep adding to our list of things to do exasperated he continues and just so you know I have tried to hire more brents because I never got the budget I eliminated a bunch of people just so I could hire four more very senior engineers at the same level of experience as Brent and you know what happened I merely raise my eyebrows West says half quit within a year and I'm not getting anywhere near the productivity I need from the ones who stayed although I don't have data to prove it I'm guessing Brent is even more behind than ever he complains that he had to spend a bunch of time training and helping the new guys and is now stretched thinner than ever and he's still in the middle of everything I respond you said that people add stuff to our list what does The List look like right now where can I get a copy who owns the list replies slowly well there are the business projects and the various I.T infrastructure projects but a lot of the commitments just aren't written down how many business projects how many infrastructure projects I ask Wes shakes his head I don't know offhand I can get the list of business projects from Kirsten but I'm not sure if anyone knows the answer to your second question those don't go through the project management office I have a sinking feeling in the pit of my stomach how can we manage production if we don't know what the demand priorities and status of work in process and resource availability are suddenly I'm kicking myself that I didn't ask these questions on my first day finally I'm thinking like a manager I call Patty Wes and I just got hammered by audit and they need a response one week from Monday I need your help to figure out what all our work commitments are so I can have an intelligent discussion with Steve about resourcing can you talk she says that's right up my alley come on over after Wes briefs Paddy on the implications of the mammoth audit report that he thumped down on the table she whistles you know I really wish you were at that meeting with the Auditors I say most of the biggest issues were around the absence of a functional change management process I think you could end up being the auditor's best friend Auditors have friends she laughs I need you to help Wes estimate the work to fix the audit findings by Monday I say but right now let's talk about a higher level issue I'm trying to get the list of what all our commitments to the organization are how big is that list and how do things get on it after hearing what Wes told me Patty replies Wes is right Kirsten owns the official business project list almost all of which have something that we're on the hook for we have our own it operations projects which are typically managed by the technology budget owner there is no centralized list of those projects Patty continues we also have all the calls going into the service desk whether it's requests for something new or asking to fix something but that list will be incomplete too because so many people in the business just go to their favorite I.T person all that work is completely off the books I ask slowly so you're saying that we have no idea what the list of our commitments is really Wes says defensively until now no one ever asked we've always hired smart people and tasked them with certain areas of responsibility we've never had to manage things beyond that well we need to start we can't make new commitments to other people when we don't even know what our commitments are now I say at the very least get me the work estimate to fix the audit findings then for each of those resources tell me what their other commitments are that we're going to be pulling them off of thinking for a moment I add for that matter do the same thing for every person assigned to Phoenix I'm guessing we're overloaded so I want to know by how much I want to proactively tell people whose projects have been bumped so they're not surprised when we don't deliver what we promised both Wes and Patty look surprised Wes speaks up first but but we'd have to talk with almost everyone Patty may have fun grilling people on what changes they're making but we can't go around wasting the time of our best people they've got real work to do yes I know they have real work to do I say adamantly I merely want a one-line description about what all that work is and how long they think it will take realizing how this might come across I add make sure you tell people that we're doing this so we can get more resources I don't want anyone thinking that we're Outsourcing or firing anyone okay Patty nods we should have done this a long time ago we bump up the priorities of things all the time but we never really know what just got bumped down that is until someone screams at us demanding to know why we haven't delivered something she Types on her laptop you just want a list of organizational commitments for our key resources with a one-liner on what they're working on and how long it will take we'll start with all Phoenix and audit remediation resources first but will eventually cover the entire it operations organization do I have it right I smile genuinely happy that Patty has framed it so succinctly I know she's going to do a great job exactly bonus points if you and West can determine which resources are most over utilized and how many new resources we need that would be the basis of an ask to Steve for more Staffing Paddy says to Wes this should be pretty straightforward we can put together 15-minute interviews pull data from our service desk and ticketing system get Kirsten's project list surprisingly West agrees adding we could also look in our budgeting tools to see how we've coded personnel and Hardware requests I stand up great thinking guys get a meeting set up for us to go over what you find no later than Friday I want to have a meeting with Steve on Monday armed with some real data she gives me the thumbs up now we're getting somewhere chapter six Friday September 5th in another one of the Endless Phoenix status meetings I realize that the developers are even more behind than we feared as Wes had predicted more and more work is being deferred to the next release including almost all of the testing this means that we'll be the ones finding the problems when they blow up in production instead of the quality assurance QA Department great during a lull in the discussion I looked down at my phone and see an email from Patty she wants to meet about resourcing promising some eye-opening surprises I open the attached spreadsheet seeing an encouraging level of detail but on my minuscule phone screen I can't make heads or tails of it I reply to Patty that I'm on the way and ask her to have Wes meet me there when I arrive I'm surprised to see that Wes has set up a projector displaying a spreadsheet on the wall I'm excited that we're meeting to analyze the situation instead of just reacting to the Daily fires I grab a seat okay what you got for me Wes starts Paddy did a great job putting this together what we found was well it was interesting Paddy explains we did our interviews collected the data and then did our analysis right now these numbers are only for our key resources we're already seeing something troubling she points at a row in the spreadsheet first we have a lot of projects Kirsten says she's officially managing about 35 Major business projects Each of which we have commitments to internal to it operations we've already identified over 70 projects and that number keeps growing with each person we interview wait I say genuinely startled sitting upright in my chair we have 150 it operations people right if you've already found over 105 projects that's 1.5 people per project doesn't that seem like a lot to you Wes replies totally and we know that the project count is low so by the end it'll probably be more like one person per project that's insane I ask how big are these internal projects Wes switches tabs on the spreadsheet showing the list of projects they've inventoried along with the estimated number of man weeks consolidate and upgrade email server upgrade 35 instances of Oracle databases install supported lemming database server virtualize and migrate primary business applications and so on I groan while some projects are small most seem like major undertakings estimated at three man years or more when Patty sees the expression on my face she says that was my reaction too we're on the hook for a huge number of projects so let's look at what our capacity is this is a little harder since we can't just assign random people to Any Given project she continues when we looked at who was assigned to each project and what their other commitments and availability were here's what we found when Wes flips to another spreadsheet tab my heart drops Grim ha says Wes most of our resources are going to Phoenix and look at the next line compliance is the next largest project and even if we only worked on compliance it would still consume most of our key resources for an entire year and that includes Brent by the way incredulous I say you're kidding if we put all our projects on hold except for the audit findings our key resources would be tied up for an entire year yep Paddy says nodding it's hard to believe but it just shows you how much work is in that stack of audit findings I looked down at the table speechless if someone had shown me these figures during my first conversation with Steve I would have run from the room screaming like a little boy it's not too late I think smiling at the image with practiced calm I say okay knowing is always better than not knowing keep going Wes looks back at the spreadsheet the third largest item is incident and break fix work right now it's probably consuming 75 percent of our staff's time and because these often involve critical Business Systems incidents will take priority over everything else including Phoenix and fixing audit findings by the way did you know that yesterday when we were talking with Brent we had to reschedule the interview twice because he had to go help fix an outage so there we were interrupting him from Phoenix work only to be interrupted by an outage he says laughing I start to laugh but then stop abruptly wait what outage why didn't I hear about it we can't keep running our organization like this well it was another San issue but nothing critical Wes replies a drive went bad a couple of months ago so the San was running with no redundancy when another Drive failed the entire volume went down Brent had to help restore some of the databases when we got the sand back up exasperated I shout dang it Wes that was completely preventable get one of your Junior guys to look at the logs every day for Drive failures maybe even have him visually inspect the drives and count all the blinking lights it's called preventative maintenance for a reason we need Brent on Phoenix not piddly like this Wes says defensively hey it's actually a little more complicated than that we put in the order for replacement drives but they've been stuck in procurement for weeks we had to get one of our vendors to give it to us on credit this wasn't our fault I lose my temper Wes listen to me I don't care I don't care about procurement I don't care how nice your inept vendors are I need you to do your job make sure this doesn't happen again I take a deep breath I realize my frustration is not because of the drive failure but because we're continually unable to stay focused on the things that matter most to the company look let's put this aside for now I say looking back at Wes I'm serious about getting someone to look at that sand daily though set up a meeting sometime next week for you me and Patty to get to the bottom of these outages we've got to figure out how to bring down the amount of brake fix work so we can get Project work done if we can't get Phoenix work done it's jeopardizing the company yeah I got it I'll try to get it in before the Phoenix roll out west says nodding sullenly and I'll get on that San issue this afternoon okay back to the spreadsheet I say Patty observes glumly you're right the one consistent theme in the interviews was that everyone struggles to get their project worked on even when they do have time they struggle to prioritize all their commitments people in the business constantly ask our staff to do things for them especially marketing Sarah I ask sure but it's not only her she replies practically every executive in the company is guilty of going directly to their favorite I.T person either asking a favor or pressuring them to get something done how do we change the game here and get resource to do all these projects properly I ask what should we be asking Steve for Wes Scrolls down his spreadsheet based on our rough numbers we'll probably need to hire seven people three database administrators two server Engineers one network engineer and one virtualization engineer of course you know that it'll take time to find these people and then another 6 to 12 months before they're fully productive of course I knew that new hires aren't productive right away but it was still dispiriting to hear Wes point out that real help was still a long way off even if Steve approved the headcount later that day as I'm walking to our second cab meeting I feel hopeful if we can get our old change process going we might be able to quickly resolve one of the largest audit issues and get some operational wins as well I'm also pleased at how well Paddy and Wes are working together as I near the conference room I hear loud voices arguing then Patty got that engineer fired for doing his job he was one of our best networking people that wasn't your call to make no mistake that's Wes hollering then I hear Patty reply heatedly what you signed off on that termination why is this suddenly my fault I knew it was too good to be true then I hear John say that was the right call we're going into our third year of repeat audit finding around change controls that goes in front of the audit committee next time around it probably won't be just an engineer getting fired if you get my drift wait who invited John to this meeting before John can make things any worse I quickly stepped through the door and say cheerfully good afternoon everyone are we ready to review some changes 14 people turned to look at me most of the technical leads from the various groups are sitting at the table Wes is standing up behind his chair fuming while Patty is standing in the front of the room arms crossed John sits in the back of the room with his three-ring binder open very much an unwanted guest using both hands I set down my antique laptop it hits the table with a thud and a clatter as the battery falls off the tape no longer holding it in place and then I hear a scratching sound as the disk drive spins down Wes's angry expression disappears momentarily wow boss nice gear what is that a k-pro 2 I haven't seen one of those in about 30 years let me know if you need an 8-inch floppy to load CPM on it I've got one in my attic at home two of the engineers Snicker and point I smile briefly at Wes grateful for the comic relief remaining standing I say to everyone let me tell you why I assembled all of you here given the urgency of Phoenix you can bet your ass that I wouldn't waste your time if I didn't think this was important I continue first the events that led to the San and payroll failure on Tuesday must not happen again what started off as a medium-sized payroll failure snowballed into a massive friendly fire san incident why because we are not talking to one another about what changes we're planning or implementing this is not acceptable second John is right we spent yesterday morning with our Auditors discussing a bunch of deficiencies they found I continue dick Landry is already crapping bricks because it could impact our quarterly financial statements we need to tighten up our change controls and as managers and Technical leads we must figure out how we can create a sustainable process that will prevent Friendly Fire incidents and get the Auditors off our back while still being able to get work done we are not leaving this room until we've created a plan to get there understood when I'm satisfied that everyone has been properly cowed I open it up for discussion so what's preventing us from getting there one of the technical leads quickly says I'll start that change management tool is impossible to use there's a million mandatory fields and most of the time the drop down boxes for the applications affected don't even have what I need it's why I've stopped even putting in change requests another lead howlers out he's not kidding to follow Paddy's rules I have to manually type in hundreds of server names in one of the text boxes most of the time there's not enough room in the field a hundred server names are supposed to fit in a 64 character text box what idiot built that form again more unkind laughter Patty is bright red she shouts we need to use drop down boxes so we can maintain data integrity and I'd love to keep the application list up to date but I don't have the resources who's going to keep the application catalog and change management database current you think it just magically updates itself it's not just the tool Patty it's the entire broken process Wes asserts when my guys put in change requests they have to wait a lifetime to get approvals let alone get on the schedule we have the business breathing down our neck to get crap done we can't wait for you to ham and haw complaining that we didn't fill out the form right Patty snaps that's crap and you know it your people routinely Break The Rules like say when everyone marks all their change requests as an urgent or emergency change that field is only for actual emergencies Wes retorts we have to do that because marking them urgent is the only way to get your team to look at it who can wait three weeks for an approval one of the lead Engineers suggests maybe we can make another field called extremely urgent I wait until the Uproar quiets down at this rate we're getting nowhere fast thinking furiously I finally say let's take a 10 minute break when we reconvene the meeting I say we are not leaving this meeting without a list of authorized and scheduled changes that we're implementing in the next 30 days as you can see my assistant has brought in a pile of blank index cards I want each group to write down every change they're planning one change per index card I want three pieces of information who is planning the change the system being changed and a one sentence summary I've drawn a calendar on the Whiteboard where we will eventually post approved changes according to their scheduled implementation I continue those are the rules short and simple West picks up a pack of cards looking at them dubiously really paper cards in this day and age how about we use that laptop of yours which probably even predates paper everyone laughs but not Patty she looks angry obviously not pleased with the direction things are going this isn't like any change management process I've ever seen John says but I'll put my changes on the board like the upcoming firewall updates and monitoring changes that are scheduled for the next couple of days surprisingly John's willingness to jump in inspires others who begin writing their plan changes on their cards finally Wes says okay let's try it anything is better than using that busted change management tool one of the leads holds up a handful of cards I'm done with all the database changes we're planning to make when I nod for him to proceed he quickly reads one of the cards execute the vendor recommended database maintenance script on octave server xz577 to fix retail store POS performance issues this affects the order entry database and applications we'd like to do this next Friday evening at 8 30 pm I nod pleased with the clarity of his proposed change but West says that's not a change that's just running a database script if you were changing the script then we'd have something to talk about next the lead replies quickly no it's definitely a change it temporarily changes some database settings and we don't know what production impact it could have to me it's just as risky as a database configuration change is it a change or not I can see both sides of the argument after 30 minutes of arguing it's still not clear that we know the definition of what a change should be was rebooting a server a change yes because we don't want anyone rebooting servers willy-nilly especially if it's running a critical service how about turning off a server yes for the same reason how about turning on a server no we all thought that is until someone came up with the example of turning on a duplicate DHCP server which screwed up the entire Enterprise Network for 24 hours a half hour later we finally write on the Whiteboard A change is any activity that is physical logical or virtual to Applications databases operating systems networks or Hardware that could Impact Services being delivered I look at my watch alarmed that we've been in the room for nearly 90 minutes and we still haven't even approved our first change I push us to move faster but at the end of our two hour meeting we've only posted five changes on the Whiteboard surprisingly no one else seems frustrated except me everyone is vigorously engaged in the discussion even Paddy everyone is discussing risks of the proposed changes even discovering that one change wasn't necessary encouraged I say we'll pick this up on Monday get all your cards to Patty as soon as you can Patty what's the best way for us to process all the cards she says tersely I'll set up a basket later today in the meantime pile them up at the front of the table when we adjourn several people tell me on their way out great meeting and I wish we had more time to discuss changes and I'm looking forward to Monday only Patty has remained behind arms crossed we spent a lot of Blood Sweat and Tears creating our old change management policy and everyone still blew it off what makes you think this will be any different I shrug I don't know but we'll keep trying things until we have a system that works and I'm going to make sure everyone keeps helping us get there it's not just to satisfy the audit findings we need some way to plan communicate and make our changes safely I can guarantee you that if we don't change the way we work I'll be soon out of a job pointing at her old policy document she says we shouldn't just throw all this work out the window we spent weeks designing it and hundreds of thousands of dollars with Consultants changing our Tools around she tears up slightly I remind myself of how long she's been trying to get this process integrated into the organization I know that there was a lot of good work put into all this process I say sympathetically let's face it though no one was actually following it as the Auditors pointed out we also know that people were gaming the system just trying to get their work done I say sincerely we may be starting over but we need all your experience and skills to make this work it's still your process and I know this is absolutely critical to Our Success okay she says sighing and resignation I suppose I care more about our survival than whether we use our old process or not her expression changes how about I write up the outputs of the meeting and the new instructions for submitting requests for changes later that afternoon I'm back in the Phoenix war room when Patty calls I run out to the hallway what's up she sounds stressed we've got a problem I was expecting we'd have 50 changes for us to review next week but we're already up to 243 submitted changes I keep getting emails from people saying to expect more cards over the weekend I think we're looking at over 400 changes being made next week holy crap 400. how many of these 400 changes are high risk potentially affecting Phoenix the payroll application or worse I suddenly remember range Master Duty in the Marines as range master I was responsible for the safety of everyone on the firing range I have a horrifying vision of a mob of 400 unsupervised 18 year olds jumping out of trucks running to the firing range firing their rifles into the air hooting and hollering um at least people are following the process I say laughing nervously I hear her laugh with all the change requests coming in how are we going to get them all authorized by Monday should we put a temporary hold on changes until we get them all approved absolutely not I say immediately the best way to kill everyone's enthusiasm and support is to prevent them from doing what they need to do I doubt we'll get a second chance to get this right send out an email telling everyone to submit any change for next week by Monday Monday's changes will not need to be authorized but changes for the remainder of the week will no exceptions I can hear Patty typing over the phone got it I'll probably need to have some of my people help organize all of the change cards over the weekend frankly I'm stunned by how many changes there are so am I excellent I say leaving my concerns unvoiced