We're a multi-billion dollar enterprise. We know better than you do. (Hint: no. you don't) Tech Support |
- We're a multi-billion dollar enterprise. We know better than you do. (Hint: no. you don't)
- Power cables don't go in your mouth
- There is a reason people start by asking if it's plugged in.
- Ticket: Networking Scientist's network isn't working. (I'm new to networking.)
- Definitely a developer job!
- The one where it's up to us to fix even though we didn't break it
- Some Like It Hot
- Wetware Blockchain for screamtesting a printer
| We're a multi-billion dollar enterprise. We know better than you do. (Hint: no. you don't) Posted: 03 Aug 2021 12:54 PM PDT Posting here for the first time in a while, mostly because I want to tell this story. Background: A certain large healthcare company ($vendor) offered to migrate us from our on-prem deployment of their software to their environment for a reasonable rate. We agreed because the on-prem server was getting close to retirement, and we were going to migrate them to either a VM or an AWS server soon anyway. There's been no issues with the software or the database itself, and I'm actually quite happy with how it turned out. Main Story Where the problem was, was with our interface. See, we have a single practice management software ($PM) that ties together our large, multi-specialty practice and the various EHRs our doctors use to keep their records. The idea is that the $PM is the primary patient database and the EHRs pull data and schedules from it via standard HL7 interfaces. For those not familiar with them, an HL7 interface's messages usually take one of two forms: a text file created by one system and read by another or a message sent by one system to an IP address and port, where another system reads it. Pretty simple, and its what connects the world of healthcare tech. While $vendor was hosted on-prem, a file based interface was pretty easy to accomplish. $PM sends files to a folder on a file server, and $vendor reads them from there. With a server that's not hosted on site, $vendor's people decided that the best thing to do would be to build a VPN tunnel and push the HL7 data to their server via that. Now, we've built out interfaces with outside partners before, primarily to accomplish getting data to and from the labs that test our specimens for us. I've run these projects not once, but half a dozen times in the last 2 years. We tend to do this by sending the message files to our SFTP server and having the other party read them from there via SFTP. It's efficient, HIPPA compliant, and I haven't had a single complaint about it in the 7 years we've been doing this. But no. $vendor INSISTED. They fought tooth and nail to get it set up via VPN because to them, that's the easiest way get an interface up and running. So the $PM rep and I decided to try it despite our misgivings. We got a network engineer to setup a site-to-site VPN tunnel on our firewall, tested it to make sure we could connect to $vendor's server, a process that took almost a week due to difficulties getting the $vendor to communicate with the network engineer and went ahead with the migration as planned. We then spent the next two days troubleshooting the HL7 interface because despite messages going out from our end, the $vendor wasn't recieving them. After two days of back and forth, with $vendor's people insisting everything was good on their end (despite us being able to ping their server's IP, telnet their port, and even packet tracing to make sure the HL7 messages were actually leaving our server), I ended up reciveing a message from their lead that basically ammounted to "actually, we don't know what's wrong and want to do this via SFTP." Well la-di-frickin-da. 20 minutes later, I have the SFTP configured for them and our $PM has set about reconfiguring the interface to actually do its job. Tl;dr: vendor wants things done their way. Clients knows a way that works for them. Turns out, the client's right. [link] [comments]  |
| Power cables don't go in your mouth Posted: 04 Aug 2021 12:31 AM PDT I was working for a large cable tv/high speed internet provider and got a call from a residential user that had internet issues and a few too many drinks. After a little bit of chatting and verifying information I was having the guy power cycle his cable modem. He had to lay under his desk to get to the modem for some weird reason ?? Well, he also took my instructions of power cycling his modem to the extreme to unplug all 3 cables from the back of it, coax, rj45, and power. Not in that order. I guess he needed a free hand and out of bad decision making he decided to hold the power cable with his teeth/mouth. After hearing a scream and a thump he started cursing a lot. Well, he was under his desk so when his wife came into the room she angrily asked who he was talking to, to which he said tech support. I guess she didn't see the phone and didn't know what was going on so assumed she was being cussed at. It turned into a verbal domestic situation and I disconnected the call shortly after. I watched the modem for a few minutes through the headend and the power cycle brought it back online after I reprovisioned it, never heard from him again and none of the other reps had them that night either. This call stands out in my memories from 15 years ago. [link] [comments]  |
| There is a reason people start by asking if it's plugged in. Posted: 03 Aug 2021 09:24 AM PDT My mentor always said to start with the obvious first, because about twenty common things will makes up 50% of all the calls. So we went through the routine and nothing worked. The database was there, the install sounded clean. What should have worked just didn't. As an hour dragged to two, I finally asked him to uninstall and reinstall, which took a half an hour. Still nothing. We tried pasting the account and password, and spell check didn't bark. Finally I heard him say "Are there two 'a's in Manager? I'm in! Thanks!" <Click><Dial tone> I stared at my phone in mute shock. He had misspelled the same word the same way for three hours. I went back to the queue because I was behind, muttering "Two A's in manager." [link] [comments]  |
| Ticket: Networking Scientist's network isn't working. (I'm new to networking.) Posted: 03 Aug 2021 07:32 PM PDT At [hugest software corporation], one of my very first issues came-in as a networking problem. I always (pro tip) look-up who I'm about to meet, to see who they are. This researcher was a networking security scientist. Their network wasn't working and they couldn't figure-out why. Having had setup a few consumer-grade switches and hubs, I had a rudimentary knowledge of basic networking. You know, make sure all the wires are attached at each end, the little springy clips have to be engaged, LEDs light-up when things are working, that sort of stuff. So, when I get assigned THIS ticket, I'm thinking; this is going to be embarrassing. What the Hell am I going to be able to do for a NETWORKING F'n SCIENTIST?!?!? I go to their office, sweating, and THANK GOD, they weren't there. I see wires running all around the edge of the office, all kinds of networking gear pulled-out, ports lit-up but not flashing, and one 5-port hub had none of the ports lit-up at all. Huh. Looking a bit closer, I see the hub has a power cord pulled-out from the back by several inches. Probably troubleshooting and removed power for some reason... I plug it back-in. EVERYTHING lights-up! I verify that there's connectivity to the corporate network, and my work is... done? Riiight, CAN'T. BE. THIS. EASY. She walks in, looks around, and hesitantly asked; "You... fixed the problem?" "Well, looking around, I saw that the hub was un-powered and the cord was pulled-out from the back. Plugging it back-in did the trick. I was really concerned that it was going to be something to difficult to troubleshoot!" I smiled bigly. She replied "Well,... I can't... believe that I didn't... well, uh, thanks!" I kid you not, later that week, I catch her eye, coming across a large cafeteria, and she sheepishly ducked her head down and looked away. I learned something very valuable from that experience. [link] [comments]  |
| Posted: 03 Aug 2021 04:35 PM PDT First story from me, might be a bit long but hopefully not too dull. So, doing a short stint heading up 'digital' at a small but renowned local company. We do a lot of work with a huge international broadcaster, and they distribute programs to satellite TV. They got in touch for a favour. We normally deal with technical briefs and design work but this is an odd one - head of design(HOD) says yep no problem. So the issue is - when our client sends a new TV show over to a satellite TV company, they sent over an accompanying graphic/show logo which is used for 'coming up next' style overlays and I think in the EPG. It's a pretty archaic format, its a low Res gif format with a 32 colour palette that must have certain colours in it. Now normally they use an ancient Mac with some 90s software on to create this file but, disaster, the machine been retired! So armed with this minimal knowledge and a couple of examples I said, well, ok - we just need to configure Photoshop and they can generate this no problem. I ping it back to the HOD and say, look it's a photoshop job, give it your best guy for a day to look at. He's not happy but gives it to the guy, who does, well, nothing. I come in early to an arsey email from HOD cc'ing in the MD etc etc - "it's a Dev job", not possible etc etc Classic FFS right FINE - boot up Photoshop at 8am (god knows why it was on my machine, years since I used it) - some googling and testing and 1 hour later I sent over 12 point instructions with screenshots. Well, that was a mistake. HOD refused to try it and claimed it didn't work etc etc. I'm pretty busy as I have the next day off so no chance to smooth things over. I come in after my day off, out of a meeting first thing and the HOD and designer are both over the shoulder of my junior dev (JDEV), muttering complaining etc. Poor JDEV is sweating away! Turns out they had taken the opportunity all day yesterday to try to get my JDEV to build some abortion of script to combine images into the format and bundle the correct pallette of 32 colours etc. He was barely anywhere struggling with installing image libraries, and just couldn't make any progress. HOD was furious and stormed over "Your devs are rubbish, this is a simple job! etc etc". I respond: "I've sent instructions that are simple enough for client to follow, and they have the software already. You went behind my back and wasted development time." "And finally... JDEV is extremely colourblind; that's why he doesn't know what the hell you're talking about." [link] [comments]  |
| The one where it's up to us to fix even though we didn't break it Posted: 03 Aug 2021 03:08 PM PDT I support healthcare software that has interfaces with third parties, e.g. labs, scheduling software, other electronic medical records (EMR) programs, etc. I got a case saying that data is no longer transferring to a third party after the customer updated the software of the third party. I figured that maybe the customer built a new server for the new version of the third party's software and didn't know how to change the IP we're sending to. Simple enough, right? The case was put in by Laura (names changed to protect the innocent) and in the case description, she requested that I call Debbie. I figured Laura is some admin telling me to call someone in IT. So, I called Debbie and the phone menu answered. I hung up immediately because it's the third party and I figured that they won't be able to get me into our software to change the IP, or get me on the server that has our software on it, etc. I called Laura and she asks that I call Debbie anyway. It seems Debbie is going to be able to explain the issue better. But Laura does tell me that they upgraded the third party's software a couple days ago and appointments haven't been reaching the third party since then. She said "a couple days" and I thought "Today is Monday, so it was probably done over the weekend." About this time, the customer's account rep for our company starts sending out emails. Apparently the customer is going crazy (note: I've already talked to them, but okay) and we need to escalate this issue! I put off calling Debbie for a couple minutes and email everyone on the thread that I'm working the issue but just haven't updated the case yet because I'm still working on it. It's NOT being ignored. To my knowledge, the account rep is just doing their job. So, don't think she was angry at me. In fact, I sent her a quick message on Teams to tell her that I'm calling the third party as requested. While we chatted, she mentioned that the customer updated the third party software on Thursday and only now do they consider it an emergency and have submitted a case. So, it's not just one day of missing data. It's three. I then called Debbie. She explained to me that the issue is not with the transmission of data. They're getting the data from our software. The problem is that the new version of their software only allows 15 characters for a phone number where they used to have a maximum of 20 characters. The phone numbers we are sending are 18 characters and that is keeping their software from processing the data files. (Note: Yes, this is the US and yes, phone numbers are 10 digits (including area code). But we send "(111)222-3333CHome" for instance. That's 18 characters. The interface engine hasn't been updated in a long long time, so yeah, it could be better but I didn't write the code.) So, they changed their software to accept fewer characters. Didn't tell the customer this before go-live. And now, for some reason, it's up to us to fix it. As of now, I've made the only change I can to shorten the phone numbers. The 'CHome' part is no longer appended to the number, so that's within their limit. Our software doesn't have a lot of customization of the data, so it's lucky I could do that. New appointments are being accepted. And I resent the appointments created since Thursday, so all the missing data is there on the third party's system. [link] [comments]  |
| Posted: 03 Aug 2021 05:45 AM PDT This story is short, but sweet. My IT department benefits from having both a front-facing customer desk as well as a back office for hands and feet. I'm manning the front desk one crisp winter day when an accountant brings in her laptop for support. I greet her as she comes in and she has a noticeable Latin accent (important for later) But her English is excellent so it's not like I'm having trouble understanding her. She explains that she's having issues with her PC shutting off randomly while she's working. I ask how often and she describes it happening every half hour or so. Hmm, unusual but not unheard of. So assuming it's some kind of hardware issue I start troubleshooting by running diagnostics, but they come back fine. Whatever, I have her login and try to replicate what she was doing when it last crashed. Time passes and... nothing. I'm stumped, everything seems to be working as intended. So we chalk it up to the ghost in the machine being on a smoke break. I send her off with the instructions to give me a call should it start acting up again. sure enough, bout a half an hour later She sends out the SOS. At this point I'm fairly certain there's something wrong with her setup. I get her desk location and let her know not to touch anything because I'm on my way. I get to her office and when she lets me I noticed the air seems considerably warmer even though the heat is on for the entire building. I just write it off as the cubicles being close to a vent or something and move to put a steading hand on her desk so I can lean under to take a look at the outlet. MISTAKE! The cubicle desk is surprisingly hot! I sort of recoil in surprise (which also startled the customer) and bend my knees a bit to try and get another look under the hood. Yep sure enough she's got a personal heater on full blast right underneath her setup! I asked her about it, turns out this lady is from Brazil and our Midwestern Winters are just driving her insane! I explained to her the amount of heat was likely too much for the CPU and was probably causing The computer to shut down because of it. Trying to be sympathetic I suggested moving the heater out from underneath the desk for now and maybe getting some cheap risers to keep the computer above the desktop since she was working with a docking station anyway. She swings by later to let us know that moving the heater seemed to resolve the problem. No word on whether or not she went with the elevated PC suggestion though. [link] [comments]  |
| Wetware Blockchain for screamtesting a printer Posted: 04 Aug 2021 04:14 AM PDT Two weeks ago, we had an external technician (ET) in our office to start to redo all of our patch panels. Last friday, we got the last missing printer from the printer swap started last year. It didn't want to work. Yesterday, I hears my teamleads boss (TB) talking to another colleague just outside my office, asking him, if there is any way to wire a network port so it only not supports printer. I am in no way anything it related in this company except user, but this was my cue. I opened my office door and explained wiring of an Ethernet port as simple as I could. Further, I made an educated guess, why the new printer wouldn't get connected to the authentication and print server. At the time, I was talking about VLANs, TB was looking at me, as if I was talking Mandarin. I suggested that he needed to contact our corporate Helpdesk that they adjust the VLANs on the ports of the switch. He didn't contact the corporate helpdesk, because ET and his assistant EA where in the building to finish the last patch panels. After some blind guessing, we somehow figured out that one port in the office across the hallway seemed to have the right VLAN assigned. Now we just needed to redo that one patch... After some discussion someone suggested to pull plugs out of the patch panel one after another, until the link light on the printer goes dark. We all agreed to it and I was about to go upstairs to look at the printer, as TB ment9that he has no signal on his cellphone. Tje patch panel is in the basement of this quite old building with walls and ceilings full of steel. The Printer was two stories above. In the end ET was at the panel. EA was in the stairwell at the bottom. TB was in the stairwell on my level and I was sitting next to the printer. ET would yell "now?" EA would yell "now?" TB would yell "now?" I would yell "still lit" TB would yell "still lit" EA would yell "still lit" ET would plug the plug back into the panel, would pull the next plug and would yell "now?" After 20 something tries we found the right port and were able to bring the printer online. tl;dr: we build a wetware Blockchain to transport screamtest requests and results. [link] [comments]  |
| You are subscribed to email updates from Tales From Tech Support. To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
| Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States | |
No comments:
Post a Comment