From Russia With Love, Part 2 Tech Support |
- From Russia With Love, Part 2
- The day I learn humanity is doomed (and to CYA) Part 2
- Hotel Wi-Fi shenanigans part 4 "I should just blacklist everyone".
- Now I have internet, but no network!!!
- I have a PHD in Touchscreen using
- "Can I print yet?"
- Yay Internet Explorer supports it! kinda?
| Posted: 29 Oct 2018 07:55 PM PDT Hello Everyone. For those of you just joining in, part 1 can be read here. For anyone who would prefer a summary: I REALLY suggestreading part 1. This doesn't do it justice....anyway: We *accidentally* double encrypted most of our thousand-computers at the medical facility I worked at. Come Monday, we didn't even have enough working machines to properly see all the patients anymore. Our 5 man shop was collectively shitting ourselves. Ash, the turd responsible, would not stop crying. I would have preferred a network-wide ransomware outbreak. At least then we could have just paid the ransom.....but there is no ransom when it's an inside job. Just despair. Part 1 is still good even though you now know this, and I still suggest it before reading part 2. Sophie SafeYard: Our old full disk encryption software. Part Two's Prolouge: The Kubler-Ross model, more commonly referred to as the five stages of grief, states that someone faced with death goes through denial, anger, bargaining, depression and acceptance. The way things were shaping up to be, in hindsight, this model was fitting perfectly. Take for example where we are at in my 9th circle of hell. I would say denial arguably began over the weekend when I thought everything would be ok. It continued through the vomiting (I was not joking or embellishing about that part) up to when Ash tipped me off to what was happening. I was already angry, and seamlessly switched to this stage in full by the end of part 1. Act 3 - The Five Stages of Grief At this point, I'm toggling between combing the Sophie knowledge-base and the manual while waiting on hold. True to form, Casper's support department picks up within a minute. Hearing Sophie's hold music reminds me.....of just how much longer I could be stuck hearing it. I decide to swap roles with Tech 2.
A minute later and some shuffling of calls, and I hear a familiar voice. It's not the asshole! In fact, this is one of the tech's I like. Hail Mary! Oh, well there we go. we'll call her Mary. I've called in and talked to her enough during implementation that she knows me.
*No, no, no! I said Hail Mary, not Hara-Kiri!*My internal monologue screams at me....
We've now progressed to the bargaining stage. Over the next few minutes (I swear it was a half an hour, even though only 15 minutes had passed on my call time...) I started looking at the tool's support documentation. It's pretty straight forward. Piece of cake, really, if the data wasn't behind another FIPS complaint container that just had it's head chopped off across 700 some-odd computers.
We joke for another minute before we end. At least I've got that 6-foot broomstick tucked up my...ah, I digress again.
At this point, I know we're going to be doing massive re-imaging no matter what. Tech 2's phone is now on speaker. The office is now being slowly filled with laptops and crappy hold music. Back on my phone, it's time to call a local MSP we use for extra hands on projects. There techs all know our environment enough. Every single one of them has seen a Windows Deployment Services Server (WDS) before. Every one of them is trustable and dependable, at least with the simpler things in life. I wish I could say the same for their account manager....Dick.
Of course you will, you little leprechaun-shark-halfling. I bet you'd make your own mother sell you her house for a generic Asprin if her life depended on it....
Would my CIO have my back if I went for it? Yes. But I'm not one to put my balls in a sharks mouth, just because I'm in the water....
At this point, I had been watching our users for many years. There were certain departments that were smarter than average. You know the type. As long as they had good instructions, they would bake you a decent cake, even if it was the first cake they ever baked in their life. I've been wanting to do this for a long time. Every upgrade where we do need some occasional labor, I get denied. Well, it's now or quite possibly never. Not going to bug the CIO with this one. Just need HR's approval, really....time for another phone call.
Is that?...
The phone shows 53 minutes on hold (This part i remember exactly, and share with you so you can further feel my pain )....
\Explain everything you already know, not much different from how I explained it to Casper's Support**
It's at this point the guy starts going on about how Sophie isn't responsible for this type of incident.....you would think I threatened to sue them or something. Far from, as at this point I was begging, all but offering to fly to Europe and shine his shoes for any help he could provide. Shoe shining....now there's an idea, seeing as I'll never work in IT within a 2 hour radius ever again....
At this point, I want to hang
I pull open my junk drawer. "Hey Boss, I think I have a plan." Act 4 - When you're backed against the wall, break the goddamn thing down.
*dialing....ringing.....ringing.....*
CIO pretty much takes this as his Hail Mary and drops off the call to tell the other directors the good news. No pressure though, right? Tech 2 comes back with a working computer that has Sophie on it. We dock a non-working machine's SSD to it. Bingo. It knows. Encrypted partition is visible, and Windows isn't asking me to format it. Alright - if I login to the management server and assign the decryption key for that machine's volume to my user account on that laptop.....holy crap, I can open it. Ok......and now that it's open.....Oh my God, Casper's decryption tool recognizes the encrypted volume. It should be able to decrypt it. Sure, this won't work for all of our machines, but at least this buy us data recovery on important machines.
At this point, I have another idea. I can see all of the data. The users folders. Program Files. Windows. I'm focusing on exec machines with more specialized software and local files. What if i don't have to reimage them? What if....I could use Macrium to clone their data in its unencrypted state to the donor drives and do a few bootrec commands to make it boot again? Macrium says it'll take about 20 minutes to copy. Cool. That's enough time for me to go deal with the WDS bandwidth saturation. You see, we don't usually do this many computers at once, so the WDS is configured for unicast - this is where each computer downloads a separate image in its own, personal session. To solve the saturation issue and have more employees helping with the reimage process, I needed to change this to multicast - where a group of computers all watch the same "tv channel" until they each have a complete copy of the show. For Multicast, you specify how many computers need to be tuned in before the show starts. Once it starts, they all are in their own private session until they all have a copy of the show to continue on installing the new image. Then the session is released, and the bandwidth is available for other groups again. If you have a group of, say, 20 computers, this means they can all listen to the stream of data instead of 20 different streams. That's a 95% reduction in bandwidth. When you're trying to reimage hundreds of computers, it kind of matters. Once I had putzed around with WDS enough, Macrium was almost finished. This is it. The moment of truth.....clone completed successfully.
I restart the computer....at this point, Ash and Tech 2 are hovering like cartoon angels perched on each of my shoulders. I think we're all praying. About 10 seconds later....a familiar screen comes up. I can choose user.name or Switch User. Tech 2 and I are laughing.Ash is crying while he laughs. I have a means of decrypting a German full disk encryption program. ....I think these are good tears now. Stay tuned for the final part in our epic saga, where fate and aftermath come together. (Due Wednesday Evening) [link] [comments]  |
| The day I learn humanity is doomed (and to CYA) Part 2 Posted: 29 Oct 2018 08:20 AM PDT I work for a small MSP servicing around 100+ clients with 5-20 employees and our biggest client having 50+ employees. This is a story about one of the clients. Characters: $Me – Can't wait for Battlefield V to come out $POL – Pissed off lady (Who we never have to deal with again) $Brains - Cubicle mate who can retain any all information somehow. I don't know how he does it, but he puts all of us other techs to shame. 2 of us have College Degrees (not that this means anything really) $Bossman – Owner and Boss – great guy enough said For Obscurity purposes, this post will not be being going into any great detail for obvious reasons. The matter is public, and I don't want to cause any issues This is the follow up from Part 1. I'm still fairly new at MSP. Previously we had an issue with $POL and I found some things on her screen that was illegal. I was informed to get screen captures and submit to $Bossman. Did so and she was terminated, and court followed. Now, for what everyone is wanting. The client we serviced, was a wealth management company. They invest people's money and try to make them more money with getting a cut. Simple enough and the town they were located it, the residents DEFINITELY have expendable income. $POL lady had multiple things on her screen. I'm not going into great detail (for obvious reasons above), but after glancing at it for a few minutes (and if you've watched any CSI/Cop shows) you would realize what you were looking at. She essentially had a ponzi scheme. Now I did not get into it, but after what myself and $Brains saw, her computer was locked down after contact was made with client about what we found. We did some forensic work and found that a few million dollars was being siphoned off clients' accounts and numbers were being fudged to make it look like they didn't make as much on the investments as they really did. She was pocketing all that money. We terminated our contract with client (Mainly because they were a bad client and $owner of client had some choice words for $Bossman when he informed her of their findings). Court Followed. Remember, at this point in my career, I'm an FNG, not even 6 months into my career, and I'm in a court room being called to testify along with $Brains and $Bossman. I am petrified. We spent a few days going over with $LawyerBuddy (Will be/has been mentioned a few times in my stories) the types of questions and answer to give. $Brains and I essentially have the exact same things to say. So, I get called to the stand and get asked a series of question by $LawyerBuddy and $clientlawyer. The whole time, I'm sitting there in a pool of my own sweat, smelling like a lumberjack in the middle of summer chopping down pine trees (All in a suit I had to rent, don't know if that was a plus or not). My evidence is put on screen and all goes as it should. Court finishes and $POL is found guilty. Sentencing was at a later date and did not involve me, so I don't know what came of her. From this day forward, I document everything. CYA is the way to live in the IT world. Not only will it save you when an issue arises, and you need reference, but it cases like this and disgruntled people, it's always best. Fast Forward: $client is a client again. The owner sold the company after the court hearing as his/her reputation/name was being dragged through the mud. $newclient bought the practice and had a meeting with $Brains, $Bossman and me and we explain in detail what happened and how to help prevent it in the future. She really liked what we had to say and $newclient is now one of our best clients who always take our recommendation on upgrades/security changes. Edit: Grammar [link] [comments]  |
| Hotel Wi-Fi shenanigans part 4 "I should just blacklist everyone". Posted: 29 Oct 2018 11:37 AM PDT Link to the original story, also includes links to the followups. New residential customer with a "slow computer" wanted it cleaned and checked for viruses. Computer is running perfectly and is less than 6 months old, no viruses, no malware, and is fully up to date. Job took all of 20 minutes and we spent more time on small talk and paperwork than we did working. As we are finishing up the job and going over our generic advice we give all customers about malware protection and computer backups she says she was referred to us by an employee at the hotel in the link above. She claims she got hired on last month and wonders if I know the passwords for the the WiFi remote management and RDP connection. I inform her its against company policy to store passwords for customers unless we have a very specific contract set up and even then I'd need authorization from the owner to share any information I do have. She gives a vague deflecting answer about getting it at work later and doesn't push it further. The whole job felt super weird so I called the lady that runs the hotel now and told her what happened. She asked for the woman's name and when I gave it to her she got really upset and said that is her now ex-husbands new girlfriend. Why can't this guy leave my company out of their messy divorce? What possible outcome was this guy planning? Now the lady that runs the hotel is asking for me to get further involved in this nightmare and talk to her divorce lawyer. I know its not her fault that her ex is insane but I'm starting to regret taking them off the blacklist, the money we've made off this job is not worth the headaches. Of course it doesn't end there. I also received a negative review by the woman we did the work for, claiming our service is a rip off because we charged her the full amount for a 20 minute job, even after signing and initialing paperwork that specifically stated job price minimums. [link] [comments]  |
| Now I have internet, but no network!!! Posted: 29 Oct 2018 02:22 PM PDT $Me, working as a SysAdmin at an operating IT support for small and medium companies. Basically what we do is taking care of servers, network stuff and users at multiple companies. Also, we visit them on site every 2 or 3 weeks. We have a $Client (with 10-15 users) who has been really a pain in the ass lately, because of their shitty and slow ISP. Also, they like to run everything wirelessly (important detail). We did everything to improve their situation: installed a new local router, set up a WiFi repeater, even running a wireless nano router in client mode hooked up to their main printer (very important detail). Also, this $Client is... really not the sharpest knife in the drawer. On Fridays, I like to work from my home.. because why not, TGIF. 8:30AM Drinking my coffee, reading the local news when my phone starts ringing, caller is $Client. $Me: Hello Sir, how can I help you this morning? This leaves me no choice, it's too early for a Friday morning, I'm better to do it remotely and go on with my day. I change the WiFi password from his laptop, and as expected, I've got disconnected right away. $Client: I don't have internet. At this point, I'm assuming the laptop is trying to connect to the WiFi repeater, which didn't follow the changes on the main router. That's why prefer doing these kinds of changes on site, rather than remotely. $Me: Ok, we have to reset the WiFi repeater installed next to your printer. There is a little reset butto... No need to disconnect, but.. whatever. It's a fresh setup, they can live without the repeater for a couple of days, then I will reset it next week when I'm on site. $Client: The internet is back! Thank you for helping! \hangs up** 10:00AM Coffee finished, reading my e-mails, checking tickets, etc... Phone is ringing. $Me: Hello, how can I help you? At this point, the $Client is navigating me to the mapped network drives from their NAS, pointing at the red crosses in the explorer. $Me: Yes, because you've got disconnected from the LAN, the NAS is just asking for your username/password again. Enter it and you will be fine. $Client enters it, voilá. 1:00PM Sometimes miracles happen, don't they? No, they don't. I was wrong on so many levels. 3:00PM 3:15PM $Client: WE CAN'T PRINT!!! IT'S REALLY IMPORTANT, WE HAVE LIKE 5 MINUTES TO PRINT A VERY IMPORTANT CONTRACT! 3:45PMI'm dead inside, I give up. I'm staring at my fridge with the cold beer, when my phone rings again. $Client: Hello, this password reset did nothing to the internet speed and we can't print. Can you please reset the password to the old one? So I wasted my entire day on this nonsense issue, eventually doing nothing. Now I have to go on a site visit on the next workday and do it again from the beginning. Exactly, how I suggested 8 freaking hours ago. Before you ask: why didn't I change the password on the repeater and the nano router... well, this happens when the client tries to be smarter than the admin and keeps ignoring my advice every time. [link] [comments]  |
| I have a PHD in Touchscreen using Posted: 30 Oct 2018 03:14 AM PDT So this is a story all about how, I lost faith in humanity, optimism turned upside down, what a tragedy.It stars: %me%: Voodoo master of clicking a button; %user%: The true Goddess of chaos; Little pre-emptive warning, this is all internal so my team luckily doesn't have to be overly fake polite. As usual everything starts off normal, I get a call for one of our machines where customers can have themselves printed out tags because apparently the tag printer isn't working. I tell %user% to unplug and replug the tag Printers, %user% asks me which one and I tell her to do it to both of them, this is where my first red flag pops up as %user% tells me that "The cable can't be unplugged, oh wait they are off now" It's not a big thing but over time it's something I make mental notes of. So far so good now onto the brains of the machine, I tell %user% that in the right top corner above the printers is a black box, on the left top side of this black box is a green light and next to it is a button. %user%'s response is that there is no such black box, nor is there any lights. %user% then proceeds to say that theres the tag printers and the only thing else there is is behind the printers. At this point I get ready for a longer than average call. I once again tell %user% that the black box is above the printers and that if she opens the top of the box %user% should be able to see it. At that %user% first points out the WHITE box that has a green light with APC on it. At this point I tell %user% to not touch that white box at all. %user% then reafirms her position that a black box does not exist and suggest simply unplugging every single black cable, I politely decline this offer of pure anarchy and destruction. After respecifying the top right corner above the printers a few more times, finally out of the sheer power of pure belief, a black box manifests exactly in that position and it even has a green light, who would have guessed. But since no good can come without bad %user% complains that the box is on the right side and not the left side, really just wanting to get this over with I simply tell %user% once more that there's a small button right next tot he green lamp. %user% however insists a button does not exist, I ask %user% to please check again. Fast forward and %user% does finally press the button but remarks that %user% didn't think it was a button because it wasn't big. Alright final stretch the system is finally booting, I tell %user% to close the machine and tell me what's on the screen, surprisingly there's no argument over the existence of a display, instead %user% decribes to me the machine logging on and doing its normal startup. That's it right? Wrong, now as the machine is starting %user% tells me she has to tend to some customers for a second, I don't mind this as it's a usual thing, however once %user% picks the phone back up she quickly says "Hello?" before removing the phone from her ear at super sonic speeds and putting it down to tell her coworker "He hung up" despite me instantly replying. I call %user% back and she (of course) claims we got disconnected (no big deal). Now the machine is finally booted and we can tes- nope another customer, sure why not what's the wor- "Hello?" Yes! "He hung up again" \)click\). I take a deep breath and restrain myself from faceplanting into my keyboard before calling back a second time where %user% of course once more claims we got disconnected (It's a rare thing to happen but usually when Users hang up on us on purpose we know it because they are so SLOW at doing it that we can hear them utter their last sentences away from the phone before they do so), I carefully consider whether it's worth telling %user% that she should wait at the very least 2 milliseconds after questioning whether someone is still on the line but decide not to, on the basis that I do not assume she has the capacity to even process such a request (At this point I'm quite internally annoyed as you might be able to tell) I walk %user% through the touchscreen gestures needed to activate the Maintenance menu so we can perform a print test and %user% fails to do this, more than 10 times, I have been personally infront that type of machine and know how accurate and fast you have to be to access the maint menu and let me tell you, it's VERY lenient. In between those 10 touchscreen inputs %user% voices the complaint of complaints, the unholy grail of abominations. "I'm not a technician". After holding back the urge to say "You use a touch screeen every single day for more than the last year and you want to tell me using one is a foreign concept to you?" I simply told %user% to make sure she was accurately pressing the areas in a fast enough pace. After this failed a few more times we finally settled on me calling her in 30 minutes after she dealt with customers. Once I called her back after the 30 Minutes it took a few more failed inputs before %user% finally found the mojo allowing her to accurately perform 5 touchscreen presses in succession. And the crowd goes wild! After finally performing the print test I wished %user% a nice day and went on to more relaxing things, like juggling the act of writing Support tickets, performing support and writing a Reddit post. In closing:I wish every device would just allow support to remote connect, would make a simple print test not dependent on the user being able to perform a 5 touch combo. [link] [comments]  |
| Posted: 30 Oct 2018 03:57 AM PDT This happened just now. I'm assisting in the installation of a new network copier. We've disconnected the old one, and wheeled it outside. The new one is being lowered from the wagon outside right now, and in full view of he staff in this office. There's currently an obvious gap where the copier used to be. I'm stood next to this gap, repairing the Cat5 cable that got accidentally squashed when the old one moved out. A middle aged lady appears next to me. User - "Can I print yet?" Me - "When the new copier is installed, then yes, but as you can see, there's no device here right now" User - "Oh, OK, so where's my printout gone? Is it here?" Me - "Did you send it after the old copier was taken away?" User - "Yes" I fucking give up. [link] [comments]  |
| Yay Internet Explorer supports it! kinda? Posted: 30 Oct 2018 03:44 AM PDT Working in support I always strive to optimize things, be it automization for Ticket input, lookup scripts for internal data, tools to make mass pinging PCs (Complete with a ding noise to tell you its done) etc. With that I of course after getting clearance distribute these tools to my fellow support members. And by that virtue end up also having to support my self written tools as well as our callers. My most recent "brilliant" invention is a simple tool that grabs a list, splits it into arrays and then checks user input against the array contents telling you whether input exists or does not exist in array. Right in the phase of testing if it works for others (everyones setup is the same in our team) for some people it works perfectly, for others it doesn't work at all. Looking into the issue I check the settings of IE to see if there is any differences between 2 people and surprisingly at first glance it seems like there isn't. Narrowing ti down a bit IE seems to block the execution of the script portion for some reason, confused I recheck the IE settings. I compare the person who's having issues to my own and then to the person where it works. Until It finally dawns on me. IE arbitrarily decides that for some people it's rated as a restricted zone while on others it sees it like a normal Internet zone. The exact same file, on copies of the exact same windows image. Another Issue I found was that if it is opened through something such as an Email, IE will always see the temporary file as a danger and run it as restricted zone. [link] [comments]  |
| You are subscribed to email updates from Tales From Tech Support. To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
| Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States | |
No comments:
Post a Comment