"IT guy" ran off to Malaysia Tech Support

---

• "IT guy" ran off to Malaysia
• In which I am a patient in the hospital but also tech support for the staff.
• Story of a user who sat and waited for me for 2 hours
• Tales of an annoying employee part 3
• My Favorite Memory From Working Support
• Ice Bucket and Fan Verses Hottest Recorded Day in the State.

"IT guy" ran off to Malaysia Posted: 15 Jul 2021 04:19 PM PDT So I recently left an MSP that I worked for for ~2 years. When I first started, I learned that we had recently picked up a client with ~500 users/computers and 6 sites. None too crazy by itself, but...we did not have much documentation on them. Turns out, we picked them up as a client ~2 months prior, but only heard that the guy was leaving the company a week before he did. Note that he had worked there for 30 years before then. Originally, we were just going to be handling backups, AV, patching, and granting said "IT guy" RMM access. Now all of a sudden we were faced with covering all their IT needs, and knew very little about their setup. Here's a brief summary of what we discovered after he left to a land of no connectivity: He did not provide us with credentials to their switches/firewalls. Turns out they're owned by the company, but managed by the ISP, who also handles their phones. We were told about 3 VM fileservers, 3 terminal servers, and 2 DCs, all on one VM host, with one other VM host for redundancy. Turns out there were something like 11 physical servers, 9 DCs, 7 fileservers, 12 terminal servers, and a scattering of other servers that we didn't know shit about. Something like 65-80 server instances in total. Every site had various chunks of old, old, OLD equipment, like ancient scanners for proprietary film, and software running on domain-joined windows 2000 and XP machines for looking up indices of documents amongst their literal tons of physical paperwork. Faxing was handled by SPA devices living in extremely weird places, including: Above ceiling tiles, in kitchen cabinets, in users' desks, and in various closets by themselves. Various users ran XP VMs with DOS emulators which accessed domain fileshares, which were all SMB 1.0 with anonymous access to LITERALLY EVERYTHING. Everyday user work was typically done through 3 separate apps, all of which had inter-database communication. Those 3 apps operated off 7 different servers, and tied into two cloud services. No explanation was given as to how they tied together. I was only able to patch together the interconnects by observing network traffic. User passwords were all 6-digit, lowercase, with 1 number at the end, and saved in an unencrypted xls file accessible on the public SMB 1.0 fileshares. OH. And "domain users" was a member of "domain admins". The servers I listed above did not include the ~20 server 2003 instances that we shut down. The servers I listed above did not include the ~40 other VMs that did seemingly nothing. The conclusion we reached there was "he built a VM for every task, and if he messed it up along the way, he just built another one and left the originals running". Windows, office, and adobe licensing was all done through the cheapest routes possible (e.g. third party chinese websites). These license keys were stored in word docs alongside the installers, typically on the desktop of a user's sign-in on a workstation. About a week after I started, those licenses started expiring. The business owner was not happy to hear that he'd have to buy more licenses. We were told about 2 domains. Turns out there were ~15 - most just weren't actively used. Several of those other domains operated on servers that ran DHCP on the same VLAN as everything else. To get around this, he simply set every users' workstation to have a static IP with static DNS. Oh, and the ISP ran DHCP at those other sites too... After consolidating data, they ended up having ~10TB of unique data. Before this, there was a grand total of a little over 100TB living across the various servers. Apparently, in years past, he would drive between sites with a hard drive to update files between sites. Since the ISP managed the switches, they basically just handed off trunk ports to them. Some of the ports on the switches were configured as access to VLANs designated for other businesses - at one point a users' workstation got a weird IP after they moved it. Turns out he could see the neighboring business's computers. Later network scans I ran indicated that there were devices (including printers) not on-premise which could be seen. They vanished after we replaced the switches. The domain admin password was the name of the business, in all lowercase, followed by the year of its founding. Drive mappings were all done manually, often by IP. Users didn't know how to do this; he would do it in person when driving onsite. Cable runs in each building were from the switches to the workstations (male to male). No wall jacks or patch panels. Many unused cables were crimped as rollover or crossover. Several server racks weren't actually secured to anything. Found that out the hard way when pushing a cable into a switch port almost knocked the whole fucking thing over. The battery backups for the servers were all long dead. He somehow got them to not beep and not turn themselves off by doing some solder work within them. There were something like 20 other (undocumented) applications that specific users only had to use once every like...2 years or so, and we only learned of those over time. Sometimes it'd be a user saying that the workstation they pulled out of a basement cabinet drawer wouldn't boot, and it'd be a windows 95 machine running software that could only be found on floppy disks dug out of file cabinets. Other times it'd be something that used to run on servers that he'd boot up only when the users needed it - some of those he literally kept at his house. I was put in charge of figuring out and fixing all of it. Mind you, I was new to windows domain environments, so this amounted to taking pretty much the worst example of a network possible, reverse engineering it, reducing that picture down to what it needed to do, and improving it to modern standards - ones that I hadn't even learned yet. I was not allowed to impose downtime, even after hours. Finally, just as icing on the cake, I was also basically the main person supporting them, and across the entire company, there was only one user who could distinguish cable shapes by description, who left partway through the year. None of the users really knew what they needed on a day-to-day basis; they relied on the "IT guy" to do almost everything, whether it was plugging mice/keyboards in, changing the printer to print to (not the default printer; just the choice of it), or how to use the applications their job roles relied on. There wasn't a week I worked there where I didn't answer at least 2 calls from users asking how to do something in some app made just for their company. I gathered more information by periodically analyzing data pulled with Get-NetTCPConnection, Get-Process, and wireshark, than I did from talking to users. The only exceptions to this were those weird things that only got pulled out once every few years. Towards the end of my time there, the guy actually came back into town, and provided us his extended notes. He wrote them while he was on the plane to Malaysia, but didn't think to send them over on any of the days where he got internet access. Those notes covered everything that I had already found out. Needless to say, it was a point of great anxiety, and contributed to me leaving said MSP. (There were other reasons for me leaving, but that one moment contributed a lot) submitted by /u/Arbitrary_Pseudonym [link] [comments]

In which I am a patient in the hospital but also tech support for the staff. Posted: 15 Jul 2021 08:32 PM PDT I work for a university/hospital/med school. I'm a sysadmin, very little in the way of customer support but I've done it plenty in the past. In April I had a medical emergency and had surgery, and spent 10 days as an inpatient, with the strong pain meds and so on. Every interaction between a patient and anyone on staff is logged in to the EMR system. It was physically painful watching people navigate between boxes in the EMR system with the mouse when logging an interaction. It wasn't long at all before I absolutely had to start teaching everyone about the super-simple universal tab-to-jump to the next box. The nurses began to come in to talk to me to see if I had any other tips for them. One nurse estimated that the simple tab trick would save them 5-10 minutes per day each, which adds up to hours per year. I am in no way involved in training non-IT hospital staff, but now I want to work up some simple tips-and-tricks tutorials for nurses. And it all happened on the good drugs. I have no idea how many nurses I taught that one trick to. submitted by /u/weaver_of_cloth [link] [comments]

Story of a user who sat and waited for me for 2 hours Posted: 15 Jul 2021 03:43 PM PDT So to preface this: In my last role as a IT Engineer, I was the go-to person for everything. From Routine work, Installation and Upgrades to User support, I was the mug who was tasked with it. I did report to an IT Manager, but he was always busy doing work for Sales/Operations/Finance so would rarely work with me unless I was doing a project. Onto the story! So we have 3 other sites: 2 Planning/Work offices and 1 Servicing Yard. About 20+ people in each site at minimum. I get in at 9 to start my day. The plan is to start getting my furthest site ready for a Firewall Upgrade, as they are on a DrayTek and need moving to a Next Gen Firewall to help secure them. Instead I'm called on my mobile immediately by a Planning Manager (we'll call him Mike), who's been trying to get a hold of me for the past hour. Me: Hey Mike what's up? Mike: Hey Yucchie, can you give Emily (one of the work planners who has a reputation for not being so bright, and always asking us at IT for support) a call please? She says she can't use her laptop and she's been unable to work for about 2 hours now Me: Yeah sure thing, I'll give her a bell and see what's up My manager turns to me and says "Yucchie! I forgot to tell you. Emily needs a hand with her laptop and hasn't been able to work for 2 hours now" Me: Yeah, Mike just gave me a bell. I'll get a hold of her and see what's up So I call Emily, and the whole time I'm thinking of what could've happened to her machine. I was worried that her SSD was corrupted; as during that week, we had a string of machines all BSOD due to an SSD corruption I get through to her: Me: Hey Emily, it's Yucchie. What's up with your machine? Emily: I don't know. I can't do my work! (Nice. Real descriptive, love) Me: Can you open any programs or anything at all? Emily: I don't know. I can't do anything! Me: Ok, let me see what's going on your machine So I remote onto her machine and see that it's just on her desktop. I can't launch any applications. She right b. She can't do anything. Odd. Let's reboot it to see if it works… And would you look at that. Just like magic. You telling me this bi-atch decided to wait for 2 hours without doing anything?! And people let her?! Oh well, I have more important things to do instead of stressing over a users machine. So I decide to tell her how to at least be a smidge proactive, in case it happens again Me: So what we just did was restart the- Emily: I'm not listening to this (hangs up) She then Messages me on Teams, ranting about how "my kind" are always talking in a condescending manner or always talking in a "gangster" way and how she's not going to put up with my laid-back attitude that I have with people in the office whenever I visit. Huuuuuh?! Wtf? I'm not the idiot who decided to sit about and moan for 2 hours instead of trying to do SOMETHING. ANYTHING. Now I'm pissed. So I tell her how fucking stupid she must be to be sitting around for 2 hours and not ONCE think about rebooting her machine. Yes, I do have a laid back attitude, so that people feel more comfortable with asking me for help, as the last engineer used to just tell them to live with their problems ("it can't be fixed. You'll just have to use it like that!") and never gave a shit about them. I just bluntly messaged her that if she needed any more help next time, she'd need to go to her manager and have him call with her machine physically next to him, as I won't deal with her directly going forward. Just like Emily, I also don't have to listen to "this". I never did get any more requests or issues with her machine after that point. How odd. Thanks for reading submitted by /u/Yucchie [link] [comments]

Tales of an annoying employee part 3 Posted: 15 Jul 2021 05:58 PM PDT Sorry, I'm limited to one post every 24 hours but here's part 3. In my previous posts in this series, people were commenting about how we should have went to the user's manager. Hang on, that will be in the next part but for now, enjoy the continued saga about the most pain in the ass I have ever dealt with. If anyone can come up with a better name for this series, I'd love to hear it. It will likely be about 5 or 6 parts. Part 1 Part 2 So, about a month has passed since the last part. We havent heard anything from the user since my boss put him in his place. Life has been good. Until I refreshed my help desk and theres a ticket from... the user's boss! gasp I know, his boss is putting in a ticket for him? Well, let's dive into this ticket. UB: Hello, User has sent me an email to let me know that hes having some lag issues. He said you have already spoken to him but dismissed his issues. I am also having an issue using VNC to view his screen so I cant really help him with questions. Can you please give him a call and help him out? My Boss: oh, what the hell. We already told him there isnt a problem. HangmanMatt, can you call him and see what the problem is? Me: sigh and calls user Hello User, your boss put in a ticket for you. What seems to be the problem? User: The same problem as before. I cant work and that's cutting into my pay. Ive been running speed tests every day and the speed on your vpn is too slow. You need to upgrade it. When I'm off the vpn, I get 300 mbps download and 30 upload. I pay for the top internet plan in my area so it's not on my end. Also, my boss cant view my screen now so he has to help me with screenshots via email. Me: ok, one problem at a time, what are the speeds on the vpn? User: 50 down and 10 up. I have just started transferring documents to my home computer just so I can do my job and that takes a long time too. Me: ok, that's more than sufficient. We have over 100 other users working in the same manner you are. Some are on the west coast, none of them are having the lag issues you are describing. Second, you shouldn't be download documents to your home computer. They contain confidential customer information that needs to stay within our network. Third, you cant do your job from your home computer because our system can only be accessed internally, that's why you remote into a computer that's here on the east coast. User: well then fix my lag issue and I wont have to do that. Me: fine, walk me through your home network. Are you in wifi? User: No Me: ok... where is your router. User: in my basement Me: ok... so you have a cable running up through the floor to your office? User: no, my router connects to a 16 port switch that my friend who is a city network engineer gave me. Then those cables run throughout my house to the various rooms. I then have a router that goes to a switch on my desk for my desktop, my printer, and the computer you guys gave me. Me: I'm sorry, you have two switches? Do you have two independent ISP connections at your house? User: no, just the one. My router in my basement goes to the 16 port switch which goes to my office to my second router, which then goes to my 5 port switch, which goes to my computers. Me: who the hell told you to set your home network up like that? That switch is likely an old managed switch that the programming was never cleared out of which you're then running to a second router. Remove the managed switch and the second router and most of your lag should stop. User: No, you're wrong, I've had this set up for about 6 months and trying to work for you is the only issues I'm having. I took a networking class and was only a couple classes from getting my degree. I just didnt finish. I know what I'm talking about. Me: ok, well, I'm telling you now, that's your problem but since you dont want to remove it, live with your lag. Now, other problem. I'm gonna check your display settings to see if that's causing VNC to fail. checks display settings why is your scaling settings changed to 150%? that's what's causing VNC to fail. User: dont change the scaling settings, I cant see the screen properly if you change it back. Me: you know you can just zoom everything in right? User: no, I cant. It's still hard to see if I zoom. I need to use the scaling. Me: fine, so you understand your manager cant help you. User: I guess. I just wish you knew how to do your job so you could fix me but I guess I'll endure. At this point, I'm already done with this guy. I hang up the phone and had to take a walk. After a couple minutes, I return to my desk and update the ticket with everything I did, everything we discussed, and the users refusal to listen. Bare in mind, the user's manager put the ticket in so he saw my reply. UB: Why couldnt you get the issues fixed? Me: I'm sorry but he refuses to listen when I tell him what the problem is. He thinks he knows more than us and I cant help him. Maybe you can talk to him but he wont listen to us. UB: ok, I'll talk to him. Great now we might get somewhere... stay tuned for part 4. submitted by /u/Hangman_Matt [link] [comments]

My Favorite Memory From Working Support Posted: 15 Jul 2021 03:49 PM PDT I commented this elsewhere, but I thought people here might appreciate it. I worked in a support via email job and my favorite thing in the world was having an abusive customer raging about not being able to do something, using tons of inappropriate language, and just being an ass in general, when the reason they couldn't do X was that they are not an admin. Then it was pure bliss to CC their boss on the thread with a message of... This action requires administrative approval so I've gone ahead and CC'd (admin) onto this thread. Hi, (admin)! To catch you up I've included the previous thread discussing (employee's) request. Do we have your permission to go ahead with that? You could hear audible tire screeches from the deceleration in the attitude of the employee. submitted by /u/Zakal74 [link] [comments]

Ice Bucket and Fan Verses Hottest Recorded Day in the State. Posted: 15 Jul 2021 04:47 AM PDT TL:DR - Hottest Recorded day in my State, AC Fails in School's server room. I use ice buckets and an industrial fan to save Server Room. ​ 2012 Sophomore year of high school we got a new IT Technician who had enough certs to be legally allowed to teach classes as electives (Net+ and SEC+ I believe he has CASP but I don't remember if he had it before or after I started the year). I decided to take an elective with him as a SYSOP, that's what we called that class/elective/title thingy. Up to starting the year my only IT experience is my fixing my Gaming PC and once kicking a printer in 4th grade and fixing it (Probably how I knew I was destined to be in IT) That year my state had record temperatures in August/September right when school started. I was probably about 2-3 weeks into school at the time of this incident so I had learned the basic stuff such as reset passwords, printers, reimage. One day a few weeks into the school year the IT dude had to head out for the day for an Appt and he said as he left I'm in charge of everything for the day and said make sure you keep an eye on the Server room due to the heat. ( Turns out it was and still is the hottest recorded day in our state) I wasn't a problem child or anything during school and with my new status as the "Assistant Sysop" I kinda had free range to go in and out of class to assist with various issues throughout the day, especially when your IT dude was our or busy at the other school. So that day I would step out of class every hour or so and make sure the server room was good. About 1pm I go to check the server room and the AC cut out and temperatures were exceeding over 100F at this point. I started to freak out because I can't fix the AC myself and if it got any hotter I knew the servers would shut down and that would be a big no no. The School was still running Novel and just really old and outdated hardware and I'm sure if It was shut off and restarted unintentionally it probably wouldn't start up again type of old. After brainstorming on what I should do I got the biggest stroke of genius I ever received, probably cashed out for a few years). I propped open the door got one of the ridiculously big industrial fans and pointed to blow air into the server room. My high school had a large Ice maker in the hallway, I really don't know why we had it to this day because where I'm from its normally not that hot, even during the summer, and its literally just sitting in a hallway not even near the cafeteria. I went and found some of the biggest buckets I could find I filled them up with ice and placed it in front and behind the Fan. It ended up bringing the Temps from around 120F down to about 65-70 as long as I kept adding ice as it melted. For sure I saved the Server's from either shutting off and or completely failing. Kinda proud of it. submitted by /u/SixxTailsHD [link] [comments]

You are subscribed to email updates from Tales From Tech Support. To stop receiving these emails, you may unsubscribe now.	Email delivery powered by Google
Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States