End User deletes $5,000 and asks us to pay up Tech Support |
- End User deletes $5,000 and asks us to pay up
- Headphoning it in
- The yellow box that wouldn't go away
- The one where I'm the idiot...
- Slinky Time: Server is hang (Do Over)
- Monosylables and Their Expanded Meanings
| End User deletes $5,000 and asks us to pay up Posted: 02 Apr 2018 10:03 AM PDT Inciting Incident December, 20XX. Client is having an error message on their registers, caused by a corrupt file. This is solved by deleting a specific file, one that, by chance, stores offline credit card transactions! Since this is actual sales, it's important that a copy of this file be made in advance, just in case there are sales to replay. Due to timezone difference, time constraints, and the end-user not wanting to do a phone call, a brief back-and-forth discussion happens, ending thusly:
March, 20XX+1. Sitting in a small meeting room, $DatIzzy is leaning back. His internship was almost up, and he had to deliver the unpleasant news that $HospitalChain was canceling their support contract with us. $BigCompany was still going to offer other services, but they effectively used us to test out the system that they insisted on, that was never deployed before them and no one deployed after. The thing was undocumented, unique, and not even the $DeploymentGuys had a solid idea of how to troubleshoot. Which meant this was fun...
Context I work at $BigCompany, offering support for Point of Sales (POS) devices. We also end up supporting peripherals (scanners, scales, receipt printers, etc.) as well as related websites. However, I've managed to wiggle into credit card reconciliation, which makes me feel like a low-risk Sherlock Holmes. Back then, it's because there was a sort of a thrill to the chase, so-to-speak. Now... well, now it's because I can diagnose most issues at a glance, and be insufferable about it. This is a tale from my early career in IT, around the 6-month mark. I still wore dress shirts, though had dropped the tie. I had dress shoes. I worked hard to get here, and I was gunning to be brought on full-time! The Cast $Me: $DatIzzy. Yours truly. The Issue Missing funds are a serious matter, but my group isn't responsible for making sure the field properly reconciles. At least, not anymore. Back at the time of the story, it was my job to reconcile for them as part of the troubleshooting process. At the time it was only about three month's worth of information, for two properties. The information collection process is, well, slow. First, I need to go to $ReportingWebsite to find out what the registers actually recorded day-by-day. Then, I need to go into $Program to see what was actually sent out. Finally, I need to go to $SettleWebsite to see what actually got sent to their bank. Between those three, I can manage to find some trends. If $Program has more or less money than $ReportingWebsite, then there was an issue with the registers for instance. If $ReportingWebsite and $Program are the same but $SettleWebsite is off, then either transactions were rejected (if short) or there might be duplicated transactions (if over). $ReportingWebsite takes the longest, due to some clunky UI, website instability, and that, to make sure I don't need to pull this information again, I save a copy of every day for each location. Needless to say, it took me most of the workweek to get this done, because, of course, I still have other issues to work on... Lo and behold, they're missing... less than $5k, but close enough that I can excuse it being a shorthand. And I can pinpoint days as well. The problem with the high-level view, of course, is that it's high-level: I can pinpoint when the issues are, but not what the exact issue is for each day! So I dig in. This takes me about another day. It's a full week after I've looked up all this information. I've got hundreds of reports saved, some text documents and others PDF's. I've got my spreadsheet looking... well, passable. I call $Aurelia.
A call is scheduled during their slow time, which, coincidentally, is when my day is almost over due to the timezone difference. Still, I'm more than happy to provide this service. Some extra effort getting this missing money will be a boon! I wasn't even thinking of what would happen if the sales got declined or, worse, if transactions actually made it through...
Each property only had a couple of POS' each, so it was quick per site, with the major delay being the drive between the two. As it turns out, $Aurelia had indeed just been deleting the files. Now, that wasn't the end of the world, because instead of using compact flash memory and a modified version of an older windows like the rest of our POS', theirs had Windows 7 and an HDD. This meant that deleted items actually went to the Recycling Bin...
Knowing this was a situation of one person's word against another's, $DatIzzy considered himself to be up a well-known creek without a paddle. The only way to get in front of this potential storm was to fess up, throwing myself at $Manager's mercy.
The Aftermath $Aurelia and $HospitalChain did continue to pester us to write them a check. The latter does make sense -- it was a $BigCompany employee that was the cause of them not getting their sales, although, to be honest, I'm not sure if they were given the full rundown or not. From $Aurelia's standpoint, I like to imagine that she was on the hook for it all. She kept opening tickets up in an attempt to get something done, but we simply referenced $Ticket and closed it out, stating that a decision had been made. It's bad enough that $BigCompany wasn't going to get its cut of those sales, but to write out a check for the missing funds on top of that would've still been worse. I was ultimately brought on full-time as well. I don't know if this incident had anything to do with it. Honestly, I'd like to think that it was my hard work that brought me on. Though, this even did teach me a couple of important lessons at the end of my internship: TL;DR: Read the title. [link] [comments]  |
| Posted: 02 Apr 2018 05:30 PM PDT Some of my users work remotely. Some of my users are video editors. One of my users has need of a new iMac currently. A few weeks back, we had a snow storm on the east coast of the US. One of my editors (let's call her Julia) who lives several hours away and works remotely is working on a top priority trailer. With her computer on the fritz, Julia needs to remote in to one of our more powerful machines for the day to get it done. I make the executive decision around 7am that I should drive into the office to get Julia set up. I create a Logmein account and jump in the car. The roads aren't too bad yet, so it doesn't take long. I get into the office and get the Logmein set up and start the transfer of the video files onto the proper machine. I head back home (the roads are getting a bit icy, but not quite too bad). After a quick stop for some McD's, I make it home safe and sound. After a few hours of getting templates and fonts copied over and some other work, Julia starts up Logmein to get some video edits in, only...there's no sound. Go into System Settings to check audio output only to find that the headphones are plugged in. I spend a solid hour and a half looking up how to swap the output on High Sierra away from the headphones while they are plugged in, including looking up Soundflower and other digital audio routing software. No dice. We opted to work on other projects and finish the audio on the video edit first thing in the morning. Only about 3 hours wasted! LPT: If you are planning to work remotely on a High Sierra Mac and need the audio output, unplug your headphones. [link] [comments]  |
| The yellow box that wouldn't go away Posted: 02 Apr 2018 07:33 AM PDT I got to thinking over the weekend (which I generally don't advise) what other story from my past might make a good TFTS story. I've come up with one. It isn't as epic as my last and this lady was an angel compared to the last one, but I think you all will enjoy it. The Parties Involved Paralegal - A paralegal that worked in the legal dept. of News People. I've talked with her before. She isn't a techie, but she knows her way around a computer and can identify various parts with little help. LK - Me, the sometimes hero, as always. The Setting Nearly lunch time. Summer. Hot and humid outside. Just another glorious day in the corp.
When it comes to legal and accounting, the question of logging in was so they could start closing out confidential things while we started the process to remote in. They still have to approve it on their end and can kick us out anytime. Most lawyers didn't actually care, it was their assistants and paralegals who would freak if we started logging in while they had so much as their email on the screen. I've seen who we were suing, who was suing us and knew the salary and bonuses of 5 different execs. It's my ethics and honor that keep me from saying anything….but, you know. To each his own and all that jazz.
I remote in, there is no box. There's nothing but a few icons and the desktop.
I wait patiently for the picture to come in. It does and I immediately know exactly what that yellow box is. Some of you might know by now too.
We laughed a bit and hung up. Then I promptly told everyone what just happened. I left out her name so that I didn't shame her. Of course, anyone who wanted could have just done a ticket search. TL;DR - Post-It notes. They're not just for disabling mice. EDIT: I made up a fake name for my old company, and used the initials of that fake name as a fake domain for a fake email. Turns out that happens to be an actual domain for a very real law firm in SE Asia. What are the chances? [link] [comments]  |
| The one where I'm the idiot... Posted: 02 Apr 2018 01:00 PM PDT Sigh, here comes one of those stories that I can't blame on a user - I was the idiot. It happens... I'm a solo admin at a government agency - that thankfully outsources some of our support & infrastructure work to an external agency. Allows me to take days off and still have backup, and gives me somebody to call to help with the heavy lifts... Anyway, long story short - we were nearing the end of a physical to virtual transition on a server that was used to host video for our local PD. The previous physical server had two NIC's - one that connected to the internal network, and one that the squads used to offload video wirelessly. We'd managed to get everything swapped over & running on the new VM with new (upgraded) wireless hardware, and had been able to upload a couple test videos with no issues (other than the port list from the manufacturer being hilariously outdated & inadequate). Considered it a win, unplugged the old network and powered down the old server. Came back the next day - to find that videos failed to upload over wireless half the time. After a LONG day of troubleshooting, swearing and general asshattery - discovered my mistake. I'd unplugged the old wifi AP from the switch, but hadn't unplugged its POE injector. So it was still happily broadcasting away and offering addresses to the cars that just happened to grab it instead of the upgraded AP - where they then connected and talked to nothing. Whoops. [link] [comments]  |
| Slinky Time: Server is hang (Do Over) Posted: 02 Apr 2018 12:44 PM PDT This Post was Removed because I submitted 2 posts in one day. Here's the do over. On dark days, I remember a coworker's advice: "Users are like Slinkys. They serve no useful purpose, but it's fun to give them a little push off the top stair". tl;dr - File System 100% Full is not caused by a hung NFS Mount on a different file system
I grab the next incident in the queue. It's medium priority - 3 day turn around time. The entire text of the ticket was "Server is hang unable to do anything" and two hostnames. That's usually a clue that the user will not be a joy to work with. User is a "Service Owner", which means they are responsible for the box and application, and have minor mojo over the box. I ssh into the first server. I notice 1900+ processes. Check a few things, and discover it's half of an active-active pair and I cannot ssh into the other half. Being awesome (but not humble), I quickly find that there is a hung NFS mount. On the working box, processes are backing up trying to access it. For some reason, running a traceroute to the NAS clears the problem, and the processes start completing. Problems that go away by themselves come back by themselves, but for now, my work is done. I jump through the necessary hoops, and get the console for the other box. In our systems, this requires giving the secret handshake to get the console, and sacrificing a goat to get the actual root password for the box. The user had sudo privilege to reboot the box, and the shutdown process had hung trying to unmount the NAS. Power cycle the box and it's back. Both systems appear normal after reboot. IM the user and ask him to check things out. He says it looks good, and allows me to close the ticket. The next day, the user reopened the ticket. With a completely different problem - out of space on an application file system. The update to the ticket even included the error message, a df showing the problem, and the text "Problem happen again". User was advised to clean up his own file system. An separate email to the user advised him that "NFS server not responding" was not the same thing as "No Space Left on Device", and reminding him a new ticket is required for each incident. For good measure, I cc'd his boss and my boss. My boss responded that if we had further problems with the user, she'd reach out to his boss and suggest retraining. You may also enjoy Slinky Time: Password Reset [link] [comments]  |
| Monosylables and Their Expanded Meanings Posted: 02 Apr 2018 12:51 PM PDT But first, some context. I was in thrall to a fellow who loved the idea of refilled and remanufactured ink and toner cartridges. He loved the obvious cost savings, and could not be made receptive to the reports of clogged print heads and toner explosions. When I was the FNG I acquired the dusty mantle of The Printer Guy and had not been able to divest myself of it, despite having passed along most of the other trappings of the FNG to the next worthies that passed through my care. I can only imagine that this lead to some mistaken idea that I maintained this duty out of desire, rather than delegation. Manager and I had gone through a number of cycles of
I'm sure this is familiar. Expansion of monosylables is in parentheses.
SpecificallyGeneral (SG): Yup. (I am aware, and slightly suspicious)
SG: Yup. (This is true - it blurs and stains the cheque to bring it to the attention of bankers.)
SG: Huh. (You do love that $Beard brand, despite its ongoing efforts to destroy every medium-class printer we have)
SG: Yup. (I'm even reasonably certain that I pointed that out, in the misty past)
SG: Hmm...
SG: tsk (MICR toner)
SG: Good. (Good!) I realized, later, I watched an entire cycle happen in the space of a few minutes, and a man who was proud of finding a solution to a problem that they, themselves, caused. SG: Yup. (Time to update the ol' resume) [link] [comments]  |
| You are subscribed to email updates from Tales From Tech Support. To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
| Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States | |
No comments:
Post a Comment