But my web site is not working! Tech Support |
- But my web site is not working!
- of dodging
- What were you thinking Hubert?
- 3am & The Orange-lit Power Indicator
| But my web site is not working! Posted: 01 Jan 2020 11:45 AM PST A comical conversation that I had over the past 48 hours. I work in a hosting company that provides managed and un managed servers for customers. If your un managed then you have are given a virgin Linux or windows OS with admin access and a panel if you pay for it. All server side configuration is up to the customer. So over the Christmas period there and phone rings.. Me: "Server and cloud support" Customer "My website is not working.. must get it back on line asap" Now anyone in support knows that ASAP bristles.. i mean without saying that support engineers would just drag it out right?? After a while I find the right server and start to look at it.. Seems like a Linux server with a few ports are responding ( port 80 and 443 ) but no login on 443 and so it looks like his admin panel has an issue.. on top of that no port 80 which is weird. Looking at the servers console, its at a login prompt. Did a reboot and saw SQL errors on boot which would again point to a panel issue. Looking on his order I also see that we never supplied him with any admin panels.. Me: "It looks like there is a server side issue as some ports are working and others not.." Customer "Dont give me that, my website is not working!!" Me: "The networking looks ok, if it was a NIC issue than no ports would respond. Additionally I can give you KVM access so you can log in and troubleshoot the issue further.." Customer: "whats that?" Me: "You need to log in and check your server and I can give you a KVM to achieve that" Customer : "You don't understand, I know nothing about computers and my developer has moved back to India!" Me: "If there are backups, maybe I can reinstall the OS and then you restore.." Customer: "You have backups! Yes do that!" Me: "I have no backups, I assume that you do?" Customer: "If you reinstall the operating system, I get my websites back yes??" Me: "No" He then hangs up and proceeds to call back multiple times looking for a different answer from another engineer. I wanted to help this guy, I really did, but he is running a "critical" website with unmanaged server, a panel his dev installed by himself that has no backups on a virtual server that costs £7.99. Its hard to know exactly what he expected me to do. [link] [comments]  |
| Posted: 02 Jan 2020 12:41 AM PST TL;DR dodging work results in dodging bullets $me: obvious $2ndline_admins: dodging work $server_admins: dodging work $2ndline_mgr: clueless $teamlead: almost sprung a vein. absolutely has my back on this one. On this project, I'm hired as a desktop software packaging engineer (though capable of much much more). I noticed $server_admins have been hard to reach lately (whenever I need them for information on a backend for an application). I have some spare time so I pick up some tickets for software installs on servers in the Test and Acceptance environments. Makes my life easier, makes their life easier. Checking <Deployment Tool> logging. Why isn't this thing installing properly? Reading the logfile: ".NET version 4.<something> is not installed, please make sure it is updated and available." Hang on, that version was released 2 years ago. Remoting into the server, opening Add/Remove Software, Display installed updates ^%$#%\&#@ you have got to be kidding me!* Remoting into another server, opening Add/Remove Software, Display installed updates @&^%$#&\^%$#%*&#@ you have got to be kidding me!* Doing a baseline scan for the entire Test and Acceptance environment. Results are bad. As in mind-boggling bad. This needs to be fixed ff-ing NOW. $me: "$teamlead, I think we have a challenge." $teamlead: "What's up this time $me? Last time you said that you found a security risk in one of the applications that is used company-wide." $me:"I am one-upping you on that. Have a look at this." He has a glance, his eyes open wide once he realizes what exactly I found $teamlead: "I... @&#%&#^$%&^$#% how did this happen????!?!?" $me: "I have no idea. Who is responsible for keeping things patched and up to date in Test and Acceptance?" $teamlead: "As far as I know, $2line is handling patching in Production so I expect them to handle this too." RINNNGGG $2ndline_admin: "Hi $me, what's up?" $me: "I seem to be missing 4 ff-ing years of security patches on all test and acceptance servers." $2ndline_admin: "Oh, you need to call $server_admins for that. We don't do that anymore." RINNNGGG $server_admin: "Hi $me, what's up?" $me: "I seem to be missing 4 ff-ing years of security patches on all test and acceptance servers." $server_admin: "Oh, you need to call $2ndline_admins for that. We don't do that." Wait what? Oh, aha, I see where this is coming from RINNNGGG $2ndline_mgr: "Hi $me, what's up?" $me: "I seem to be missing 4 ff-ing years worth of security patches on all test and acceptance servers. Your guys say to call the $server_admins, $server_admins says to call your guys. What is going on?" $2ndline_mgr: "We used to do that but after the last internal reorganization it was assigned elsewhere, it's no longer in our tasklist." $me: "Who were there when this reorganization took place? And why has nobody checked if everything on that tasklist was assigned completely?" $2ndline_mgr: "yeah well, we have been busy already so I wasn't waiting on any extra work anyway, I left it at that after that project was signed off." I swear one of these days I will live up to my flair. Meeting is called, all people mentioned so far are present. $me: "Something needs to be done about this. I kind of know HOW it got to this situation, I don't know WHY nobody has even bothered to do a check until now. I just want this fixed." $2ndline_mgr: "I have done some reading on the project notes. Believe it or not, this task was no longer on the tasklist. So that is why it wasn't assigned." $me: "Like I said I don't $#&^&^$ care WHY. Either of you is responsible. One for not doing a proper transfer of the task, one for not monitoring the environments." $me: "I understand the rights of 2ndline have been largely revoked from doing any server installations." Because there have been a few mishaps, let's say the level of incompetence varies greatly between 2ndline people. $me: "I also understand you still have access to <Deployment Tool>. How about you 2ndline, you make sure the updates are available in <Deployment Tool> and $server_admins make the collections and schedules for the actual installation?" $teamlead: "Before you answer, do realize the only option is: YES WE WILL. And now get out, all of you!" Aftermath: it turns out there was monitoring software. That needed a minimum version of .Net installed, which wasn't available, so the automatic deployment of the agent failed, which wasn't monitored. Also, this software did not have AD integration, so adding servers was done by hand. So I checked the manual, exported a list of every server from AD, imported that list in <Deployment Tool> and made sure updates were scheduled for install. It took about a week to get everything to an acceptable level. [link] [comments]  |
| What were you thinking Hubert? Posted: 01 Jan 2020 07:47 AM PST Hello again! At the request of others in my last Hubert post, here is yet another Hubert tale from tech support. So no shit, there I was coming in for my scheduled shift at work. For those that don't know, my team provides technical support (of the networking variety) for job sites across the country. I'd go into further detail, but I don't want to give too much away and get figured out. Hubert was part of our internal IT support, supporting our main HQ. Originally he was hired on as an assistant, but at this point of the story his lead ended up resigning for another position. Such is life. The significance of this is, that meant Hubert was responsible for everything that needed to be done at HQ. Printer not working? Go get Hubert. Laptop having issues? Find Hubert. India support having issues with the phone software we use? Yep you guessed it, Hubert, Hubert, Hubert. On this particular day, I arrived and saw my boss working off of his laptop, which I found odd because we'll.. he's sitting at his work station with 3 monitors, why is he using his laptop? Turns out Hubert was trying to troubleshoot the issue with our phone software, by making sure the cat 5 runs we had going across our office we're good. His idea of troubleshooting was to start cutting cat 5 cables running through our area, and then run more cable. But he didn't do this at a time where my team would be out of the office or early morning/late evening. Oh no. He decided to do this when he waltzed in at 8am. And guess whose line was not only the first one to get cut, but the only one that would end up getting cut that day. If you guessed my boss, you've won your own "God damn it Hubert" T-shirt. Apparently the ass chewing my boss gave Hubert was legendary; Hubert swore up and down all he did was cut the Cat 5 cable so he can run a new cable, but somehow he didn't stop to think where the cable began or ended. It took the majority of the day to get fixed, and that was only because my boss ended up helping him so that he could get back to work that much sooner. Tldr; Hubert decided to troubleshoot a Phone software issue by cutting the cat 5 cable that feeds to my boss's work station. Edit: oh my God, over 500 upvotes guys thank you so much! Huge shout out to everyone suggesting titles for the story where Hubert lost his position, I'm currently at the bar laughing my head off at these titles. I may post one more tonight I'm not sure. You're also reigniting a long last passion for writing so thank you for that as well! [link] [comments]  |
| 3am & The Orange-lit Power Indicator Posted: 01 Jan 2020 01:19 PM PST Obligatory LTLFTP. Background on me: I was the only field IT person in a 200km+ diameter circle in a rural part of Canada. No helpdesk, no engineers, no escalations, sometimes no cell service, just me. Background on the story: Almost all the hotels in this area cater to heavy industry. Managers and consultants would travel in for multi-week shifts and the hiring companies would put them up in a hotel room instead of the work camp with all the blue-collar workers. Most of these people make $1000CAD/day, and stories involving them will get their own posts. Anyways, these hotels will bend over backwards for their customers as all it takes is one bad complaint and they can loose entire floors of bookings. So if something goes wrong that effects their customers, it's all hands on deck right-bloody-now. Me: me HSM: hotel shift manager HFD: hotel front desk Me: *wakes up at 3am to phone ringing* Hello this is $Me, how can I help you? HFD: Hi this is the front desk at $Hotel, our booking system is down and $HSF told me to call and ask you to come out. Me: *stalling for time as I wake up* Can you elaborate? Is something broken? It might be something I can fix over the phone... HFD: Well, the booking system is down on all of our computers, and one of our front desk computers is completely dead. Me: *finally starting to wake up* Oh...um...yah that kinda makes sense, your booking system depends on $Station1 for all it's information, so if it's $Station1 that's down, your booking system not working would make sense. Is it $Station1 that's down? HFD: Yup, that's the one...uh...hold on... *excited chattering in the background* ...$HSM wants to know if you can come right away, we need this fixed ASAP. Me: *still kinda sleeping* Can it wait 'till *checks clock*, I dunno, 8am? It's 3am right now, if I come it'll be an after hours call out with a 3 hour minimum not including travel time... HFD: *interrupts* HSM says we'll pay you whatever you want, shift change at $GiganticCorpCustomer is at 6am and we have to be able to do check-out for their people that are leaving or else it's gonna be... Me: *interrupting after finally registering the importance* Yup, no problem, I can be there in 45 minutes. But can we do a little bit of troubleshooting first? Just to make sure it's not something simple we can fix together right now? HFD: Sure, I can help with that. I walk them through trying the power buttons on the various elements of $Station1, and it seems nothing has power. Me: Okay, it sounds like it's a power issue. Can you check the surge protector? It looks like a fancy power bar with an orange light on it. We want to know if the orange light is on. HFD: *pauses* Yup, the orange light is on, and everything looks plugged in too. Me: Okay...can you flip the switch off and back on again? The orange light should turn off, then back on. HFD: *another pause* Done. Orange light is back on, but $Station1 is still down. Me: We tried the basics, I'll be there soon. I proceed to have the fastest shower ever so as to have time to brew coffee and grab a snack on the way out the door. 50km later, I'm at the hotel. Me: Hey $HFD...um...hey, where's $HSM? HFD: Oh, I think she's in the back...uh...either breathing or crying. I think I heard both. *hurries me to the other end of the front desk* This is a pretty big deal. $Station1 is this one here on the end... I look down at the space where all of $Station1's gear is stacked, and the orange light on the surge protector looks a little...off. Something just wasn't quite right. Like it was on, but way dimmer than it was supposed to be. I take a slurp of coffee, and blink the sleep out of my eyes. Yup, definitely something amiss here. Meanwhile, $HFD is wringing his ~17yo hands and staring with a blank expression, and I can hear heavy breathing and muttering coming from the back office, presumably from $HSF trying to calm herself. All of this is being processed with the back of my brain as I slowly take another slurp of my liquid awakeness. HFD: So, uh, do you think you can fix... Without really thinking about it, I reach down for the switch on the surge protector while $HFD was still finishing his sentence. $Station1 comes to life, and the other stations go from scary red-bordered error messages to the green checkmarks of self-tests. The sounds of the fans on $Station1 carry through the office, and $HSM comes running out with tears still drying on her face. Both her and $HFD stood silent with shock. Me: *still partly asleep, probably with some hardcore RBF going on* Surge protector was switched off. It's on now. Wanna test everything while I'm still here, just in case? Both of them immediately attack the stations with ravenous fervor. Everything worked fine. By the time they finished testing I was finished my coffee, and was much more pleasant and awake. HSM: *gushing with joy* Thank you thank you thank you! I thought we wouldn't be able to check anyone out! Charge us whatever you want! Me: *by this time the curious, detective part of my brain is awake and engaged* I'll make you a deal; if we can figure out what happened, I won't charge for travel, and we'll be able to avoid this in the future. It turned out that some well-meaning cleaning staff had tried to vacuum out the spot where $Station1 lived, and had bumped the switch on the surge protector without knowing it. $HFD had been too scared to touch anything while we were troubleshooting, so had instead told me everything was fine when I asked him to try the switch. And the lighting in the space was just such that the orange indicator looked like it might be on, even when it wasn't. TL:DR 3am call-out, client staff fibbed during troubleshooting, problem was power switch was off, I was the hero who got to bill 3 hours for 3 seconds of work and some travel. [link] [comments]  |
| You are subscribed to email updates from Tales From Tech Support. To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
| Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States | |
No comments:
Post a Comment