The Janitor is my Superhero Tech Support |
- The Janitor is my Superhero
- Do a backup on your own
- Of crashes, backups and the unexpected hero
- How Many Lights?
- The data only works when i’m in my hotel
- Internet Blackhole
- "Can we take our Monitors with us"
| Posted: 29 Oct 2019 02:02 PM PDT Hey all! I have another story to tell since my one about 'the thing in the box' and 'your server is where?' did so well a while back! This is another tale from my time as an MSP technician and actually occurred at my very first MSP job. This occurred back in September of 2013, for some of you this month might stand out because a certain something that rhymes with Ripto-Blocker started becoming big around this time. For story Purposes the Cast is as Follows! ME: Obvious, though at this point still a middle-range tech monkey.SDude: Sales 'professional'.TechDad: Senior technician at the company I worked at.Janitor: The hero of our story.I had been working at this MSP for about a year now, things were going relatively well. I was kind of on the fast-track because #TechDad liked me and saw potential in me as his padawan, which meant I basically just followed him around all day and did help-desk monkey things while learning by osmosis. The company had it's share of problems but my tech-side coworkers were alright... Sales on the other hand as anybody from an MSP background can tell you was a MESS. It was pretty routine for our Sales 'professionals' to sell things that they did not understand, which meant they would often sell the wrong things to customers without consulting us first or make promises that were impossible to keep all in the name of closing a 'deal' which they way their pay was structured netted them a 'bonus' for the number of clients that they signed. Enter #Sdude. This guy was your typical salesman, right down to wearing purple shirts with popped collars and gold chains, complete with spiked hair and fake tan. For some mysterious(tm) reason he was catnip to new customers and his sales numbers were insane... The mysterious reason is that he would promise them impossible things even MORE regularly than the other sales professionals. Anyway, #Sdude accosted #Me and #TechDad one day on our way into the building one day, accosted being the best word because the guy had clearly been waiting for #TechDad and basically sprung from the corner of the lobby like a goblin the moment he appeared. SDude: Hey where have you been!? Why are you late?!TechDad: I work 8-6, it's 7:30.SDude: Whatever, hey I have a hot prospect who needs support right now! I told them you would be leaving here an hour ago so you need to start driving!TechDad: What? What client? What's happening?SDude: My new prospect, their gonna sign next week but they need help NOW! You need to go now!A large argument ensues in the lobby with #TechDad demanding information and #Sdude demanding that he get in his car and drive to a so far non-contracted non-paying customer about forty-five minutes away. The owners of the company get involved. #SDude convinces them they smell money. #TechDad and I are in his car within the hour driving to an unknown location with unknown issues. We get onsite and without violating the rules of the sub a THING had happened and this very large, very old and very unprepared company had lost access to basically ALL of their data. Including, -gasp- the backups that they knew about. Now this was back in EARLY September so a lot of this was still new and we had not seen a lot of this particular kind of 'ware' before. We tell them the usual speil and then set to work with all the tricks and tools we can to try and recover this data which #SDude had helpfully told the customer was something we do all the time and could be completed in just an hour or so. Needless to say several hours... Then over a day goes by before valiant #Techdad is about to admit defeat. There is nothing he can do. They have no good backups. Their data is irretrievable. The owners of our company are furious with #Techdad for not being able to fix it so he can get back to paying clients rather than with #Sdude who with their help roped him into it. Then, as though sent from on high like an angel comes #Janitor. He pokes his head into the server closet where we've setup our shop because he see's we've propped the door open. He's done this a few times over the past day or so mostly because he seemed to have an interest in tech even if he dident understand most of it. Janitor: Hey fellas any luck?Me: Not so much.Janitor: Did anybody show you the server?Me -glances over at the server-Janitor: Naw! The one I take care of!Techdad -perks up at this and asks what server he means-So #Janitor takes us across the building to a small room that was clearly an old server closest... In which we found a working Tape Backup system... This #Janitor had been tasked YEARS ago with swapping the tapes in and out of this machine by some arcane I.T department of the past and had faithfully done his duty for years... Even figuring out how to order new tapes and label them. Needless to say some of the data was still corrupt but we managed to recover their network to a reasonable period from before the 'incident' and got a multi-million dollar company back on their feet... All thanks to #Janitor who is and always will be my hero. [link] [comments] |
| Posted: 30 Oct 2019 02:45 AM PDT This is my first post so I don't really know what I'm doing here. And it's not even a support thing either but I wanted to share this experience with somebody and I think this seems to be the right place. Cast: $Me $ITExpert (head of IT) I work in a company that has an IT department for it's own software and hardware. Most of our code is in php and we have a development-server where every developer has his own port and files where he can do whatever we want and test it. Only if we are finished with a project we comit our changes to a SVN that manages the versions of the program. $ITExpert is the brother of the boss so he gets away with nearly everything. He has limited knowlegde (His knowledge is from the 80ies) of IT security, backups and servers but manages all of it and nobody else knows what he is doing. Today he came to the conclusion that it is necesseary to move our development-server (its a VM) to another host. Something strange happens and the VM is now set back to the state of two weeks ago. That means two weeks of development is lost. I immediatly grabed the phone and talked to him. $Me: There is a problem with our server. The data is two weeks old. $ITExpert: Yeah there was a problem when I moved it to the other Host. It's no big deal right? You all have local backups of your development progress right? $Me: No? Why should we have local backups when there is a server that has a backup of it's own? $ITExpert: Well there is no backup of this server. I didn't know that I have to do a backup of this machine. I mean it's only the development-server and not the production server. $Me: There are only two servers you don't need to backup because they are restored from backups all other servers have to be backed up! $ITExpert: There is only one solution to this: Tell every developer that they haver to do a local backup of their files and everything should be fine in the future. If they need a bigger SSD they can get one. This is only the condensed version of this story. This entire event was hours long and the conclusion was that I wrote an email to everybody that they have to do a local backup of their progress. Maybe you can imagine the facepalms of the other developers [link] [comments] |
| Of crashes, backups and the unexpected hero Posted: 30 Oct 2019 12:40 AM PDT Some 20 years ago, one of our clients wanted to extend the space on their server. This server was delivered looong before I started working for this company. You know where this is going. $me = obvious I come prepared, I know what SCSI ID the current disk has, I checked what kind of controller we are dealing with and what kind of connector is used so I know for sure the additional disk will actually physically fit in the server. $me: $sa, backups have been made for the past week? We have a full backup and some additional ones? Shutdown,power off, add disk, connect, power on. So, I power down, disconnect, the new disk, thinking I did something wrong, power on. $me: it seems the original harddisk in their server died on me Right. So I dig up the install media, the license disk, the backup software, the information about this server (surprisingly pretty complete) and the backup tapes. I get to the point of actually inventory the tapes and starting to restore things. $me: eeeh $sa, I seem to be missing a tape, it is asking for #2 of yesterday. Lessons learned: [link] [comments] |
| Posted: 29 Oct 2019 06:22 PM PDT Background: I worked for a call center that had a contract with a large telecommunications company. My official title was "Level 3" tech support, but this was mostly a supervisor role to front line support. On one shift, I had an agent pass me a customer who was insisting on us sending out a tech. We serviced only business accounts, so we often attracted quite a few problem customers who loved to escalate easy fixes, some incurring a $300 fee for wasted tech dispatch. Conversation between Me and AC (Angry Customer): Me: After getting all the notes from the last tech, I know a few key facts. The modem has no lights, they have no connection to the internet, and they want a tech out immediately. I've checked and it will be two days for a dispatch, which I haven't told her I'm willing to do yet. "Ma'am, I've looked into the issue, after speaking with the last tech, and I believe this issue could be fixed over the phone with just a little troubleshooting." AC: "That's not my problem. I've done everything I can do, and your equipment isn't working. It is very important that we fix this as soon as possible!" She is shouting this, by this point. Me: "I understand your frustration, but dispatching a tech for fixes that can be done over the phone could result in a fee, as well as downtime for your service we won't reimburse." This line will typically agitate most people, but she's already upset and I'm trying to appeal to her rational brain, if such exists. AC: "I will not pay for you to fix your stuff that is broken!" Me: "There are a few things I want to check, and, if we can't fix it over the phone, I'll be happy to setup a tech to come out." The strategy works, and I get her on the idea that we're tackling the issue together, instead of her fighting against me. We check the modem: no lights. No inclination of a burnt out modem yet, so I'm not issuing a replacement. Check the power cable in the back of the modem, unplug and plug back in. No lights. In order to check where it's plugged in at, she has to get behind her desk, something she grumbles about the entire time. AC: "It's plugged into a surge protector." Me: "Does the surge protector have any lights on it, or any other devices plugged into it?" AC: "No, it doesn't." Me: "And, the surge protector is plugged into the wall?" AC: Annoyed tone."Yes, it is. No one gets back here, so why would that be unplugged.: Me: "If there is a switch on that surge protector, can you flip the switch and see if any lights come on?" She flips the surge protector power switch, and things start to improve. The modem gets lights on it. AC: "It's still broken, I can't get online." Me: "If the DSL light is flashing, it won't come on yet. I'm waiting to see if we can--" About this time, the modem comes in sync with our system and I can get replies from it. "Try your internet now." AC: "...it's working." Me: "I'm glad to hear it is working. In the future, if you have issues, we are always willing to help." TL;DR: Angry customer wanted a tech out to fix DSL. Modem was powered off at the surge protector. Saved her $300 and a two-day wait for a tech. [link] [comments] |
| The data only works when i’m in my hotel Posted: 29 Oct 2019 03:45 PM PDT So the company I work for provides sim cards to travelers so they can make calls and use data while abroad, just today I had to deal with this particularly funny costumer. The guy was probably in his 60s so as you'd expect he wasn't exactly the sharpest when it comes to technology. He called complaining about how the data only worked when he was inside his hotel, I immediately realized that the guy was connected to wifi rather than data. I still wanted to confirm so I asked him to send a screenshot which he obviously did not know how to do. I sent instructions on how to do it and he promptly replied. The response had me and my coworkers laughing for days. It was a picture of this man's feet. It was a pair, of wrinkly ass legs and feet in brown fcking sandals with yellow crusty nails. He even texted a couple of hours later asking if we had figured it out. This man literally looked at a picture of his feet, sent it, and went about his day thinking it was just what we needed, and judging by how this man's feet pic was the highlight of our week, i guess it really was what we needed. [link] [comments] |
| Posted: 29 Oct 2019 10:48 AM PDT Long time ago I worked for an origination that was running Microsoft Exchange 2003 for it's email. All our remote sites had a local Exchange server where the mailboxes for the office resided. One day we got notified that an Exchange server in one remote site was down. I traveled out to the site and determined it was a hardware failure. I contacted Dell and got replacement parts sent out. Before I left the office for the day I updated all the users on the status. I told them email will be down for a few days until we could get the server fixed. I also told them that they would not have any emails for the time period the server was down. We had a policy in place that if an email could not be delivered within 24 hours that it was dropped and the sender got a non deliverable response. Few days go by, the parts arrived and I was able to get the server back up and running. I sent out an email to the entire office notifying them it was all up and operational again. Also reminded them that they would have a gap in their email for the missing days as they were not able to receive any because the server was down. I did a walk around the office making sure all the users were back up and running and let them know there would be missing emails for the days the server was down. I get to the office manager. First thing she said was... "where are all my missing emails?" I told everyone why they would not have emails for the time period the servers was down. I told everyone multiple times, both verbally and written in an email. I was feeling like a smart ass. So I said... In the middle of the Internet is a blackhole. Every time your website wont load, or you are missing a file, or you don't get an email, it gets sucked into this blackhole and lost forever. She looked at me straight faced and said, "that makes sense." And she never asked about her missing emails again. [link] [comments] |
| "Can we take our Monitors with us" Posted: 29 Oct 2019 09:19 AM PDT So I work for a mid-size law firm, sometimes it feels like the wild west, because 80% of our laywers are partners and with that "C-level" and the diversity of software, hardware and shit goes wild among the divisions. Today I got a good surprise. 3 of our Lawyers heading to another country to for 1-2 weeks with their laptops (on monday morning) and called us they want to take not only they laptops (fine) but their fucking 24" Monitors with them. How the hell you know for weeks that you travel work related for a few weeks and want to take your monitors and Ask me 10 minutes before my work end and 2 days prior to deal with this shit. I am not a magician ( If I am please tell me) who can solve your really odd request on the spot. [link] [comments] |
| You are subscribed to email updates from Tales From Tech Support. To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
| Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States | |
No comments:
Post a Comment