That time a vendor underestimated how insane I am. Tech Support |
- That time a vendor underestimated how insane I am.
- Swapping users to fix the issue
- Customer who won't buy a £20 graphics card doesn't understand why he won't be able to use his new monitor.
- Text messages can't time travel.
- I didn't touch anything!
| That time a vendor underestimated how insane I am. Posted: 23 Jun 2020 12:32 PM PDT setup( I work for an IT MSP and one of our clients is this big industrial metal fab company in the middle of nowhere. They're so far out that if I'm going to be there for more than a day at a time it's cheaper to put me up in a motel overnight than pay mileage reimbursement there and back. The site consists largely of a main building with the server and network stuff in it and then a couple of smaller buildings about a quarter mile away across a dirt field where they store metal stock. This story happened around October after some heavy rains, that'll be important later. ); loop( The client had been reporting increasing issues where one of the machines in the shed across the field would take hours to pull down just a few kilobytes of gcode from the server when loading jobs. Nothing else in that shed had issues like that just the one machine. Whenever we talked with the vendor they'd insist it was some network or configuration issue on the Ubiquiti point to point solution that beamed the network out to the shed. We'd show them wireshark captures and ping graphs and all kinds of things proving that it was some issue in their software and that none of the other machines with the same software in the same shed had the problem. We replace the cat5 line to the machine with shielded cat6e, showed it would do it even with everything else in the shed turned off, showed the machine itself could access the rest of the network at full speed even while the software was struggling. Every time the vendor would just insist it was our point to point gear. After several months of this the client had had enough and wanted something done about it, now. The vendor asked us to try moving the machine to the main shop but the thing weighed more than a school bus and moving it would have been in the mid 5 figures. We didn't have a quarter mile worth of fiber in stock let alone the SFPs and the client didn't want to pay that much for a temporary test to prove the vendor needed to fix their shit. We were stumped until I had a brain wave. We had a pair of old desktop UPSes that would still hold a charge, some boxes of Cat5e that another client had found in a storage room and didn't know what to do with, and some little desktop switches, you can probably see where this is going. The next week I go onsite to enact my devious plan and discover the entire field is a mud pit now. Undeterred I find a couple of cardboard boxes and duck tape some trash bags to the bottom. I go to the server room and tape and tie the cable to the bolted down rack, climb atop some filing cabinets, pull some slack cable, and toss the entire 1000 foot box of cat5e out a second story window onto a snow bank. Anger in my heart I retrieve my box and carry it back to my car, and then idle my way into the field feeding cable out of the box all the way to about 95 meters into the field, got one of the bankers boxes out of my back seat and put a UPS and a switch in there, crimped the connections and plugged stuff in, and then continued this process all the way to the shed dropping another box about every 95 meters until I got to the machine. I get everyone ready for the test and have all of our network diagnostics running and a vendor on the line and drive my way back into mudhell to turn on the UPSes. The unhappy beeping of a choir of APCc and Cyberpowers singing in my ears we start the tests. Surprise surprise the issue is still present and our link to the office is, all things considered, astonishingly good. In the little over hour and a half it takes before the UPSes finally die the vendor finally agreed to sending out a warranty covered rep the next day who had the issue fixed in about 30 minutes. Victorious I took apart the whole mess, hosed down the cable, and went into town to buy and expense some new socks. ) TLDR: Vendor tried to insist a problem with their equipment was our network, I called their bluff and burned a box of cat5e and a pair of socks to prove they were wrong. [link] [comments]  |
| Swapping users to fix the issue Posted: 23 Jun 2020 01:03 PM PDT Many moons ago I worked for a studio that created computer graphics for films. This was a place where a lot of CG techniques were invented and it grew from a small "computer division" to nearly 1K cg artists. In that process the company went from the front half of an industrial building to 13 buildings in an industrial park. Only 2 of those buildings were built by them, the rest were general purpose industrial buildings and had original occupants. It made for great building names, my favorite was Cookie Bay because it had been a cookie factory. We also had "Far Away Bay" and "Far Far Away Bay". Because of the addition of many computers and workstations and people the AC systems in these buildings were kind of bodged together and added to randomly. This also led to power distribution being weird since much of the original infrastructure was removed, it was just modified. That meant things like the breaker panel that was just across the room had nothing to do with the outlet next to you, that was fed from halfway down the building. A few of you elderly folks will remember these things called CRT monitors. We had a very large collection of 19" CRTs for the CG artists to use. There was also a roving team of MonCal people to keep the colors proper on those. You gotta make sure your green alien was the same green on every screen. I was a paladin of a maintenance droid, I got called in when all other techs got stumped. This time I was called because of an issue where the image on some of the monitors was vibrating. It was a small shift that in modern terms would have been less than a pixel. But I did notice it. They had tried swapping monitors, then cables, then whole machines. Still some screens had a case of the wiggles. Now mind you this was not an issue for all the users in that area, just a few. When we went in a night to really get into it and suss out what was going on and not interrupt production we could not find any issues. The screens were perfectly stable. The next day there would be complaints from a few users. Then they tried swapping users, this worked! We had noticed that there were users who did not notice the slight wiggle in their screens. There were others who were very sensitive. By relocating the users and selecting for ones who did not complain about the twitchies the issue was sort of resolved. I was still bugged about the issue. I then heard from my older bother (he was the telecom guy) that the PBX was having issues with the T1 lines feeding it. They would lose sync from time to time. When they tried to troubleshoot it at night all would work perfectly. Then the next day it would fail. (My brain twitched) I went to see what was up and learned that they had moved the T1 handoff from the fiber to a new server room in the building with the twitchy monitors. Those went via coax cable from the new location to the old building where the PBX was. I asked if he had used the isolation transformers for that connection. He looked at me sideways and went Hunh? A year before this we had some video lines between the buildings and discovered a ground differential between the buildings. The quick fix were some gnarly 75 ohm transformers to isolate the 2 systems. Those did their job, but had been retired and put on the shelf. The telecom and IT teams did not communicate much with the video people so there was a knowledge gap. I am the gap filler here. Transformers were installed and the PBX was a happy camper. I still was perplexed by the issue. Why only during the day and not at night. Whey would it work sometimes and not others. Hmmm. Then it hit me... AC! (In several connotations of the term) They had all these various HVAC systems bodged on the roof with power coming from whatever panel they found extra power in. Because these units were taking power from one end of the building, but getting grounded locally there were some weird ground currents going on when all the HVAC units were running, like on a warm day. At night when testing was done the AC units were off so there was no problem. To test it we setup one night and watched a monitor that we knew would wiggle. We then went and overrode the timers and turned on all the HVAC systems in the building. As they came on the wiggles would start. We surmised that we were sitting inside a huge magnetic field because of all the wiring running all over the place. There was nothing to do about it short of gutting the whole place. They were in that building for a few more years and LCD monitors started coming out. Those are not bothered by magnetism. In the long run the whole company moved to a new campus. I stayed behind with the small practical unit until the place got sold. I got sold too and rode with that company until they filed chapter 7. The evil building was eventually gutted and turned into a county medical center. Teal Deer: There were twitchy monitors in one area and users were swapped out to find ones that did not notice. [link] [comments]  |
| Posted: 23 Jun 2020 06:30 AM PDT Cast: Me: Me User Boss. Tldr at bottom So I worked in this small computer shop 2016-2018 as an apprentice and this is one of the stories I can tell. So I was at the shop alone When a customer comes up to the shop. Hello sir how can I help? I ask Yeah i need a new monitor preferably something HD. said the User. So I show him the monitors. I can't remember the exact model but the one he liked was a AOC or ACER with A 27" ips display which was 1080p. Honestly not a bad call. He brought his computer with him and I noticed a major problem straight away. His pc was VGA only. The monitor in question had HDMI and Display port no vga. instantly I deduced he will need either a new graphics card or a different monitor. We had a low profile gpu (think it was a gt 710 or something similer) which just came out of a pc we upgraded for a customer, it was never used outside of the shop and was going used for cheap. Unfortunatly sir that particular model of monitor will not work with your computer in it's current state. I said. As you can see here you pc can only output a vga output. The monitor only has HDMI and Display port. To remedy this we will have to insert a graphics card with a HDMI port. We have one on sale here for £20. How much will it cost? User asks. £20 I replied. I don't accept that answer. Said the user. Is there a way to do it without the graphics card? No way unfortunatly. I replied. I can do it for £15 with the monitor if you want. Both come with 2 year warranty. The guy pulls out his phone and starts googling. You are lucky you have it cheaper than pc world. I'll take the monitor but not the graphics card. My mate knows more than you he will get it setup for me. So I sell him the monitor and he leaves. 3 days later he comes in and buys the graphics card. Tldr user buys a monitor that is incompatible with his pc and called me an idiot. [link] [comments]  |
| Text messages can't time travel. Posted: 23 Jun 2020 06:29 AM PDT I work in the medical side of things and my day to day is managing systems that track patient information for multiple uses such as a patient's inpatient history, upcoming clinic appointments, etc. One of our systems is used to book patients into our outpatient clinics, manage their appointments and update their outcomes. It also plugs in to an external text messaging service that will automatically text the patient if their appointment time or status has changed, for example if it's cancelled. Our team (Me) gets a support ticket from a clinic secretary (CS) reporting an issue with the text message service. Paraphrased: CS: Hi, I have a patient who has arrived into the clinic here today but their appointment was cancelled two weeks ago. They say they never got the cancellation text but they should've been texted two weeks ago when the clinic was cancelled. (Patient is understandably very annoyed, but this is the only report of this issue so I do a quick audit of the activity recorded against the patient's record...) Me: Okay, I'm just going through the patient's account history. I can see today's appointment was booked a few months ago by you. When did you say today's clinic was cancelled? CS: Two weeks ago, the consultant due to run the clinic was going on leave so we were told the clinic wasn't going ahead and we were to reschedule the appointments. Me: Okay. And when (I ask, knowing the answer already as I can see the audit) did you actually update the patient's appointment information on the system? CS: Yesterday evening. Me: So not two weeks ago? CS: No. Me: So the reason the patient didn't get a cancellation message two weeks ago is because you didn't actually cancel the appointment on the system two weeks ago. (Our text messages go in batches at different times during the day, so the message generated by the record update late the day prior hadn't yet gone out.) Me: The good news is the text service is fine and the patient will get the cancellation message today, but if you don't update the appointments in a timely fashion on the system then patients won't get the texts in time. CS: Okay thanks. It goes without saying that if you or someone you know does manage to invent time travelling text messaging, please let me know. [link] [comments]  |
| Posted: 23 Jun 2020 07:46 AM PDT So for the last few days I've had to go back and forth with my friend over his computer. He bought a new ssd and wanted to move windows to it, easy enough. So I talked him through it, linked him to guides and youtube tutorials and eventually got the ssd installed. The next day he asked me to come over to his house to fix it, I looked inside and the power cables werent connected to the drive, an easy enough mistake/fix. The next part is where things got messy. A day later, I get a call from him saying the PC either won't boot or goes straight to bluescreen. So I troubleshoot with him, getting him to tell me exactly what was happening, retracing anything he might've done etc. He insists he did nothing between me being there and the PC refusing to boot. I try and get him to run disk repair and when that failed just getting him to reinstall windows. He claims he can't do either so I'm stumped. He claims literally nothing has changed so I start thinking it's an issue with the new drive itself. The next day I agree to go over and look at it again and... The case is a complete mess. There are cables everywhere (I'd tidied up the cables the day before). The GPU is unplugged, the network card is laying on the bottom of the case, the ssd is still plugged in properly but it's suspended in a tangle of cables and the hdd was halfway out of the drive cage. He starts going on about how moving the hdmi cable up to the motherboard stopped it not booting so he thinks its an issue with the gpu... He was still adamant he hadn't touched anything. After about 30 mins of cable management and reinstalling windows, the PC was running fine and he finally admitted he did touch something. Turns out he tried to split the windows files between the two drives so he could fit a game onto the ssd. Then when windows bluescreened and wouldnt boot he tried what I did the day before, plugging the drives in properly, but with both the hdd and ssd. In the process he somehow managed to pull up all the cables that were tucked away, and unplugged the GPU. "I didn't touch anything" seems to be something I hear all the time when going over to friends or relatives houses to troubleshoot/fix their systems. 9 times out of 10 they always have. [link] [comments]  |
| You are subscribed to email updates from Tales From Tech Support. To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
| Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States | |
No comments:
Post a Comment