Killing them (not so) softly, part 2 Tech Support

Killing them (not so) softly, part 2
Ah Why Thank You So Much Sir
Your hardware is tripping the circuit breaker, we need a new one!
Save your work.
I'm not sure that is the right UPS
The Enemies Within: Exposure gets you... problems. Episode 126
of overtaking
A gift from a friend

Posted: 11 Dec 2019 08:52 PM PST

While I'm working for the Earl Scheib of consulting firms (we'll do anything for $200/hr) I'm assessing the cybersecurity risks of the hundreds of vendors that touch my client's big pile of health care data. I spouted off and now I have to pick five vendors and vote them off the island.

Of course, this is going to be more complicated than I thought.

I can't use this as an opportunity to punish vendors who have annoyed me. Instead, I need to select a few truly bad actors. I contact the other consultants at my firm who have worked on this project and asked for their reports. I'm going to put all this in a master spreadsheet and find the five worst.

Sounds simple, right? Wrong.

I'm grinding through everyone else's reports when I get a meeting request for tomorrow afternoon from a handful of people I don't recognize. It's my boss and a few unknowns from $BigHealth, my health insurer client.

Oh, shit. I'm going to get some vague 'guidance' from the client that will make this harder. Great.

I'd like to prevent this. I have to make my list of five before the call so I can seem like I'm competent enough to handle this task on my own.

I don't want to seem biased against the vendors that I've reviewed, so I go through the reports from other consultants. I'm not just looking for occasional bad practice, I'm looking for repeated ass-pucker.

I find a few, including:

A 'healthcare outcome metrics' firm that queries patients on their surgeries. They know all about chemotherapy side effects, but not encryption. When pressed, their answer was "we don't need to do that".
A pharmacy qualifying vendor- they go through prescriptions and bills to determine what is and what isn't covered after the fact. Their reviewers are contractors using personal laptops. They've lost three laptops (which might hold sensitive data) and they hid that from us for a year.
An insurance broker who has two sales employees with felony convictions for Medicare fraud, which exposes our client to some kind of liability I haven't looked up yet.
A company that "Does big data things with healthcare data to improve outcomes" but doesn't think security matters. I think I'm going to have a conversation with the responsible consultant that may end in yelling, since that's all the detail I get in the report.
A vendor who wouldn't fill out our questionnaire, answer any questions or allow the consultant to enter their property, yet re-signs contracts year over year.

I also have a few vendors who I'd like to fire for no reasons better than "you were complete schmucks and tried to lie to me". However, I must be a fair executioner. I have to make sure the service they provide for $BigHealth isn't unique so they can just name a competitor before they cut the incompetent vendor.

I put together a compact deck- one slide for the review process and two slides per problem vendor- who they are, what they do for $BigHealth, what they're doing wrong and how we could replace their service with existing or new vendors.

The next day rolls around and I've got my slides ready to go for the $BigHealth 'alignment' call. I email them to Shi for comment because he's something of a micromanager.

Radio silence. I occasionally send myself an email to make sure everything's working.

I spend my time writing other deliverables, laundry and writing stories here.

My phone starts ringing.

me:"Hello?"

Shi:"Where are you?"

me:"In the physical plane, metaphysically or career wise?"

Shi:"The call?"

me:"With $BigHealth? That's not for another hour"

Shi:"With me to prepare for that call"

me:"I didn't know about your call"

Shi:"The meeting is in my Drafts folder. I didn't send it"

me:"I see. I wish to apologize for not attending a meeting I wasn't invited to"

Shi:"No reason to be sarcastic about this. You should have known I wanted to talk to you about this"

me:"I'm sorry. Next time I'll be proactive about this and reach out when you want to talk to me"

Shi:"That's better"

Shi sends me the meeting link and I click on it immediately.

Shi and a more senior member (who I can only call Mr. Bland) of the Health Care team are on the call already.

Shi:"I finally rousted LawTechie. Now we can talk about $BigHealth"

We have a 25 minute call that seems to repeat the following:

$BigHealth is an important client to our consulting firm.
Our contract with $BigHealth is up for renewal and things are 'sensitive' right now, but Bland's turning it around.
I should consider the "bigger picture", which can't be revealed to me because of the first two points.

We don't actually discuss which firms we're going to cut or our methodology.

We all say "great meeting" and end the call.

The 'big' call with $BigHealth people, including Client Director goes smoothly. Mr Bland talks about the "twenty thousand foot view" and "Provide Aircover" and I wonder if I should be climbing into a Lancaster soon. I present my methodology and reasons for cutting firms that present risk but can be replaced. I'll be informed when I'm to tell the firms they're cancelled and any other details, then the call ends.

I start to think about other things when my phone rings.

Client Director:"I'm approving three of the five to be terminated right now. Contact them, make sure they return or delete our data and tell me when it's done"

me:"That was quick. I was expecting it to be more complex"

Client Director:"Why?"

me:"I always assume there's a bigger picture"

Client Director:"You're talking like Bland. Don't. He's an idiot."

to be continued.

submitted by /u/lawtechie
[link] [comments]

Ah Why Thank You So Much Sir

Posted: 11 Dec 2019 06:00 AM PST

So this one was a bit more of a recent interaction between the IT security team I work with and the ISO.

$ISO walks in: Guy I would just like to congratulate you all for doing such a great job. All the other Regional ISOs are talking about how quick and proactive you guys handled those 2 major threats to our systems.

$team entertains $ISO for a few minutes

$Tech1: We'll sir thanks for the praise but we need to get back to work real quick

$ISO leaves

$Tech1: $Hakaari do you know what he was talking about?

$Me: nope, $tech2?

$Tech 2: Nope, but I'm gunna go through the incident tickets for the past week to see if I can find out

This was 2 days ago.

We still don't know what $ISO was talking about.

submitted by /u/Clay_Hakaari
[link] [comments]

Your hardware is tripping the circuit breaker, we need a new one!

Posted: 11 Dec 2019 06:29 AM PST

Hi TFTS,

Obligatory long time lurker, first time poster, etc etc.

First some background:

I currently work as a biomedical field engineer, which means I install OEM medical equipment in hospitals, clinics, medical environments.

I also perform repairs in the field and test to make sure everything is OK, according to OEM and medical standards. I am technical support, not clinical - we have other employees for those questions, usually brand reps with a clinical background.

Anyway, onto the story. First the characters.

$me - raptorboi, me - biomed.

$rep - OEM brand rep, very friendly guy. We get on well.

$BME - biomedical engineer at customer site.

$BME_Boss: $BME's manager, a senior Biomedical Engineer. I've met him multiple times, and he seems like a nice guy.

The story:

I get an email from $rep.

$rep: Hey, raptorboi, customer has an issue with a device, biomed says it is tripping the RCDs on-site.

$Me: Hmm, ok. Have they tried to plug the device into another circuit to confirm that the device is actually tripping the RCDs, and isn't just a RCD?

Now, and RCD is a current monitoring device that constantly monitors the electric current flowing through one or more circuits it is used to protect. If it detects electricity flowing down an unintended path, such as through a person who has touched a live part, the RCD will switch the circuit off very quickly, significantly reducing the risk of death or serious injury.

$rep: OK I'll let the biomeds on-site know and see what happens.

I go back to my work and a few hours later, get a call from $rep.

$rep: Hey, the customer wants you to attend the site and fix the unit.

$Me: Have they tried plugging the device into another circuit?

$rep: The biomed that reported the fault says that we should attend the site and perform a field repair, as the unit is under a service contract.

$Me: Ok, but that customer is only on an annual preventative maintenance (PM) contract - 2 site visits a year. They get software upgrades and PM parts only, but a field repair will cost them money. It's also a 4 hour drive, which is also chargeable. Are you sure that they haven't just tried another circuit to confirm the fault? I performed Preventative Maintenance on the unit only last month, and it was perfectly fine.

$rep: Yes they know, but you're technical right? Did you want to call them? You can talk on their level. Here is $BMEs number.

$Me: Ok, no worries.

I call $BME.

$Me: Hi, this is raptorboi from OEM. I hear you're having trouble with one of our devices?

$BME: Yeah, it's tripping our RCDs, can you come out and fix it?

$Me: Sure, but I just want to confirm that our device is actually faulty. Have you tried plugging it into another circuit? Or maybe the circuit it is on has too much stuff on it?

$BME: I'm busy, can't you just come out and fix it?

$Me: Your site is on a Preventative Maintenance contract, all field repairs are chargeable. I just want to confirm that our device is faulty before we charge you-

$BME: Look, I know what I'm doing. I've worked with stuff way more advanced than your devices. Your stuff is under a contract, and it's faulty - now come out and fix it! I have more important equipment to deal with so just do your job!

$BME then hung up. He also sent an impolite email to $rep that the local field engineer was belittling him and trying to get out of work. He also CC'd the Theatre Manager, as well as my boss.

I get a call from $rep shortly after.

$rep: What happened? Our boss is asking why we have a customer complaining about you.

$Me: I literally called to confirm the fault, and $BME wasn't helpful at all. I couldn't even confirm if he'd tried plugging the device into another circuit.

$rep: Ok, let me see what the boss wants to do. He's on the phone with the customer now. Now you're definitely sure the unit was OK when you performed Preventative Maintenance last month?

$Me: Yes - I'm sure. You know I'm thorough.

$rep then hangs up to talk to our boss, explain everything and asks what he'd like us to do.

$rep: Well, the boss has given the OK to go to site. He talked with the Theatre Manager, and this BME has kinda thrown you under the bus. Boss said if our unit is faulty, the field repair will be free like under a comprehensive contract. If there is no fault, charge them for a field repair. But be thorough. The customer says they will have the unit in the Biomedical Engineering workshop.

Next day I drive the 4 hours to site, and head to the Biomedical Engineering workshop. I find the unit and see $BME_Boss nearby. He's the only one here.

$Me: Hey $BME_Boss.

$BME_Boss: Hey, raptorboi. I just heard about this thing. $BME was going on about it this morning over the phone, and something about you not wanting to come fix it? That can't be right. Also, weren't you just here last month for the 6-monthly service on this thing?

$Me: $BME said it was tripping the RCDs, but didn't confirm if he'd tried multiple RCDs. Also, yeah I was here last month - no problems with this device.

$BME_Boss: What? Well, $BME is on his day off today. Ok, since it's tripping RCDs, run an Electrical Safety Test to make sure it's electrically safe.

I setup my test gear and test the device - result is good. Unit is electrically safe and shouldn't trip any RCDs.

$BME_Boss: OK... Let's just plug it in here and see what happens.

We plug the device in. Power it on. Unit boots just fine, no RCDs are tripped. No problems. Try another two circuits. No problem.

We then take the unit to Theatres, where the unit usually is when it isn't being used. Plug it in. Power it on. Unit boots OK, no tripped RCDs. Try another circuit - no problems.

We then take the unit into the operating theatre, where the unit is used in a surgical procedure. Plug it in. Power it on. Unit boots OK, no tripped RCDs. Try another two circuits - no problems.

$BME_Boss: Hmm OK, run your tests and let's see if it trips anything through normal use.

I setup my gear, run all my tests and all good - no worries.

$BME_Boss: So the unit is OK?

$Me: Yeah, nothing wrong. Can't fault the device.

$BME_Boss: Hmm OK - This visit is chargeable isn't it?

$Me: Yes, your Theatre Manager wanted us to come out ASAP.

$BME_Boss: Ugh, what a waste of time. If someone told me about this I would have confirmed the fault myself. Anyway, let's head back to the workshop and I can sign off on your paperwork. It's been a bit slow this week so I'll even shout you a coffee.

As we are heading back, we run into the Theatre Manager.

He tell us an RCD is tripping in another theatre where they're setting up for a procedure.

submitted by /u/raptorboi
[link] [comments]

Save your work.

Posted: 11 Dec 2019 01:23 PM PST

Not the craziest story, so I'll keep this one short. I work in a library/lab kinda situation, so people come use the computers when they need them. We have 3 actors in this performance: me, the customer, and the off-stage idiot from support. I don't know the customer's name, so we'll call her [Janet]. Janet walks up to me from one of the machines.

[me] - How can I help you?

[Janet] - uh, the screen went blank

[me] - uh, what?

[Janet] - it just... died

I check the machine and it is, in fact, dead as hell. I check under the desk and (thanks to [idiot in support]'s terrible cable management) can be so easily unplugged that I don't know how it stayed plugged in for so long.

[me] - sorry, but it looks like this computer got unplugged. Did you save your work?

[Janet] - no, I was just gonna type it and print it... ugh.

[me] - well... I'm really sorry. I'll work on this computer so it doesn't happen again later. In the mean time, use this machine. It'll stay plugged in.

[Janet, in the most passive of aggressive tones I've ever heard] - No, it's fine.

I thought that was the end of it. Embarrassing, but fairly harmless, and totally the fault of [idiot].

Plot twist: She's still here, typing up what she lost. It's been 2 hours. This means she didn't save AT ALL during 2 hours previously, and was just planning on printing it and being done. Also she just went up to go to the bathroom, and the computers sign out after 5 minutes of being idle. She'll lose all her unsaved stuff. I hope to god she learned her lesson about saving, but I have a terrible feeling that she didn't. I'll keep you posted.

Moral: Save your goddamn work.

EDIT - Update: She stayed for about 3.5 hours. I think she just printed what she made and left. No idea if it's saved or not.

submitted by /u/dougmantis
[link] [comments]

I'm not sure that is the right UPS

Posted: 11 Dec 2019 08:03 AM PST

This was when I was a very new engineer, just finished training no real field experience.

New installation of a huge server and its 5kVA UPS. All I had to do was check the wiring and turn it on. I'd checked the manual the previous night so I was sure of all the technical details.

I look at the setup and say to the customer "I'm not 100% sure that they have specified the right UPS, this is a variant B and I think we need variant A". Customer "Have you ever worked on one of these before?" Me "No, let me phone in and check". Customer "We haven't got time for this, I trust the Sales guy, I'm sure it's fine." "I'm goonna turn it on, you don't know what you're doing." Me "I wouldn't do that"

KABOOOM!!!! This thing weighed several 100 pounds and I swear it moved.

Me, with ears still ringing, "You'd better talk to the sales guy and let us know when he's sorted it."

I can't remember what the customer said but he didn't have as big a smile as me.

submitted by /u/crapengineer
[link] [comments]

The Enemies Within: Exposure gets you... problems. Episode 126

Posted: 11 Dec 2019 09:28 AM PST

Today's tale is short.

My boss had a meeting with our marketing director. The marketing director wants to demonstrate our core product to people while away from the office.

So here's what mister marketing requested: "Guys, can we setup https://ourcoreproduct.domain.com to NAT to our private configuration website but block all public requests unless it's an IP we allow?"

While.. that's kinda the job of a firewall. But having our core products configuration site facing any public IP scares me. If it were an ideal world, it would be on a non-routable IP to begin with, with NAT only from our private ip range. But to have it public facing is just a non-starter in my book.

Sadly, this guy usually gets his way. Hilarity to follow.

I have a few more stories to share. EMC doesn't document well, and VMWare hilarity.

submitted by /u/nerobro
[link] [comments]

of overtaking

Posted: 11 Dec 2019 03:44 AM PST

TL;DR when system boot overtakes storage device, bad things happen

20+ years ago

$me: obvious

$sysadmin: system administrator

$sysadmin: "I am going crazy!!!!"

$me: "What's up?"

$sysadmin: "Ever since upgrading to $NewOS we have some unexplained crashes. Whenever we go there, all is fine. Tickets are soaring about this one."

$me: "Any similarities in hardware? Software?"

$sysadmin: "Haven't really checked yet."

After making a list of systems pull up the hardware configs, we have:

All of them have a $Portion $Sasquatch 5,25" half-height harddisk.

Let's build one like that and test the hell out of it.

Install $NewOS. Reboot

$system: *whirrr* starts

$me: "What the..."

Fiddle some more, try reinstalls, updates whatever, system keeps running. At 5:30PM I decide I am done for the day. Power off, close door, go home.

Next day, power on system

$system: *BSOD CRASH\*

Wait, wut?

Reset system, let $NewOS grab itself together (which takes time with this POS disk)

$system: *whirrr* starts

What the actual...

Reset system, let $NewOS grab itself together (which takes time with this POS disk)

$system: *whirrr* starts

Aha