New Years Day Pager Call: "We Can't Get Into the Building..." Tech Support |
| New Years Day Pager Call: "We Can't Get Into the Building..." Posted: 03 Jan 2021 11:55 AM PST I do consulting tech support, usually systems admin type work, but in this business you will always be the "IT Guy" no matter what you put in your scope of service. One of my clients, despite having a revenue stream of over $5 million a year, and around 40 employees, still won't invest in even an MSP. The owner either spends his time fixing IT issues, has one of his family do it, or expects employees to troubleshoot these problems. If all those "support" options fail the overflow usually comes to my cell phone. That is what happened at 8AM on New Years Day. New Year's Day, around 9AM... Scene: The IT Guy (me) was up late at a New Years Eve Party (outdoors, socially distanced, small, and responsible before the haters start chiming in here) and didn't get to bed until about 3AM. Although I am technically not "on call" some of my clients will treat me as such so when I do not want to be bugged I put my cell phone on silent which is what happened before going to sleep. Around 9AM I roll over and see my cell phone ringing. Don't think it much, probably junk, until then I notice there are 10 missed calls and 3 voicemails. Surprised, and hoping some tragedy had not befallen a friend last night, I check the calls. They are all from the Business Owner ("Owner") and same with the voicemails. I go ahead and listen to one. Voicemail: "Hey IT Guy...I know it is New Years Day but my company is open (sure his employees love that fact) and we have lots of work to do. The first people to get to work though said the parking lot gate wouldn't open and the front door card reader isn't responding. No one can get into the lot or the building. Nothing seems to be working I went in through a window and the power appears to be on. Can't figure it out and really could use you if you could come in ASAP..." Three Years Ago... Now let me rewind to about three years ago when I started working for this client. My first job was a few server upgrades. Simple stuff, but before getting into this I always like to do a review of systems. The network closet was (surprisingly) organized and I was able to trace back everything except one older box on the rack. It had an IP but no documentation. Couldn't get into it any which way and the owner's kid who was doing the IT stuff had no idea what it did. After doing about a day's worth of digging, we came up with some old manuals that suggested it was running all the building security systems (card readers, parking lot gates, security cams, etc.) and we finally got into it by hooking up a monitor, keyboard, and mouse (no network access available). The login was still the default (which was in the old manual) and if the attached receipt was to be believed it was purchased in the late 90's. I did some hunting and it appeared to be connected to the internet and automatically updating. The last update ran a few years ago, but an internet search yielded very little on the brand name, model, etc. other than some random discussion board conversations. Not wanting to go down a black hole my inquiry ended there and I made the recommendation to the Owner to think about updating the system. It is a little foreshadowing here, but the email I got in response is paraphrased below: Owner: "Hey IT Guy....(all the scope of work was approved with this note)....I do not want to do anything with the security system. I had it installed when we moved into the building and as long as it is working spending any money on it is going to be a waste of money. (The solution I proposed was around $5,000)." Fine, then he decided to insert some unnecessary commentary... "I understand this is how you guys make money by selling unnecessary services, jacking up rates, and padding bills...I just want the services we talked about...Please restrict your work to the approve scope...If any work beyond that is performed I will NOT pay you for it..." OK, something in my head says to update the scope to include something in the notes: "Not responsible for (security system box with some identifying details). Agrees to hold harmless for any work that might impact this system. Any work performed on security system will be by separate agreement...Initial here." Owner signs updated scope of work and the job is booked. Back to present day... I am tired but figure might as well knock this out so I call the Owner. Owner: "Hey IT Guy...thanks for calling back so soon...I got everyone in but we have to leave the door hanging open and everyone is parked illegally on the street...My kid is out of town and I really need someone to look into this ASAP..." Me: (After going over some basic questions...Power on, check....Internet working, check...etc.) "OK give me a few hours to get ready and head over there. It might be that old security system you remember we talked about a few years ago. Sounds like if other things are working it probably failed overnight. Owner: "Umm I don't remember anything about that, but whatever is wrong we need it fixed ASAP...And oh how much is this going to cost me...." Me: "I'll do it on my usual emergency hourly rate according to our agreement (which is about twice my normal rate) or if you prefer I can come in on Monday and it will be the normal rate.." Owner: (grumbling audibly before answering) "That is fine just get here sometime this morning..." (hangs up) Noon, New Years Day... I'm in the network closet after finding a spare monitor, keyboard, and mouse. Hook up to the security system box and nothing. It does not even sound like it is running. Toggle the power switch on it and nadda. It is dead. I go back to my notes and look up the name of the company that manufactured the system finding an old news article about how they went bankrupt in 2010. Links to any support are all dead. Searches come up with a few older discussion threads on how similar devices have failed asking for help. Most useful response are the company is closed, support ended in 2016, and only way to keep it running was to find used components and perform a lot DIY. Owner: (red faced, obviously angry at dealing with angry employees, and appearing hungover) "So were you able to get it up and running..." Me: "No, the device is dead. It won't power on. (Go one to explain the specifics to a very impatient owner...) Owner: "That is unacceptable. I NEED this up and running now. Can you restore a back up or something???" Me: "It is some kind of proprietary hardware so even if I had a back up I would need a similar device to restore to it to and those are not available commercially since the company is not longer in business. I could look at getting one online via Ebay, but there is no guarantee that will even work..." Owner: (getting visibly annoying and increasingly angry at the situation) "What do you mean?!?!? What am I going to do??? I can't leave my front door open and the local cops are going to start ticketing my employees. My contracts require me to have these security features in place or I am supposed to close! And if one of their auditors hears I am open with this system down it will cost me big time!...." (The downside to being a jerk boss is your employees have little incentive not to be a whistleblower...) For Brevity... Turns out the kind of work the Owner does for his clients is highly sensitive and they take security seriously. He shouldn't be open right now with these systems down, but he is taking a risk because of the holiday. Chances are real if someone blows the whistle, if he stays open with them down, that he could lose his whole contract or pay big fines, so a solution is necessary and time sensitive. After a few hours I work up two solution (while continuing to charge my emergency rate). Keep in mind we are right up against a weekend and it is New Years Day in the late afternoon. Option #1: I track down a used device about a 6 hour drive away. The guy selling it actually responded to me and is available for in person pick up if I have cash. The box itself is several thousand dollars, drive with reasonable stops is going to be 14-16 hours, and then probably another 5-10 to see if it can even get up and running. My quote for this is $30,000, non-negotiable, 10 hour max on trying to new box, hourly rate after that. I'm doing the drive as part of the job (full rate) because I want to make sure it works before purchase (and see if that tech has any inside knowledge on the system). Yes I inflated my rate, but it is all weekend, emergency work and the Owner has annoyed me to no ends. Option #2: A friend, who does security systems, comes in to scope the job for a new system. At overtime rates he can rip out the old system and completely replace a new one that weekend. Means new everything. The old stuff is old and not compatible with any newer system. Also have to redo all the cabling, set up, migrate to new cards, and perform a bunch of admin work. It will take him and his crew all weekend, but he has the hardware in stock and can get it done. Bill about $100,000 for everything (including me helping at a nice rate and a little commission for the job). I present both options to the owner without recommendation (knowing he isn't going to pay attention to that anyhow...) The Decision To my surprise.... He is visibly agitated at the price tag for either option and after trying to negotiate to which he finds both prices are firm. If he wants normal rates his job goes into the same queue as the other work and gets done mid-month... Owner: "Let's do the second option. I can't risk this not working on Monday. I'll be on site all weekend in case you need anything and to make sure the job is done right...." TLDR... Jerk of a Business Owner ignores advice of IT consultant when it comes to old, discontinued hardware that runs essential systems for the building, thinking he is smart and saving money by avoiding being "fleeced" by a consultant who proposes unnecessary work. A few years later that system fails and the emergency fix costs him about 20 times more than the original solution. And for the curious, I'm going to make a lot of money off his incompetence so it will be a Happy New Year for at least me and my friend. [link] [comments]  |
| You are subscribed to email updates from Tales From Tech Support. To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
| Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States | |
No comments:
Post a Comment