Posted: 29 Apr 2020 05:38 PM PDT

I'm at $BigClient, which is taking a Citroen like approach to infrastructure and operations. "We recognize that the McPherson strut is simple, efficient, good enough for most use cases and accepted by everyone in the industry, but we shall do it with hydraulic fluid at high pressure. What could go wrong?"

Except $BigClient's far away from a competent Citroen shop. $BigClient's Citroen has gone through a few years of 'just keep it running on the cheap' upkeep without access to factory parts.

I've got an odd patching problem on a handful of servers. Systems are rolling back to insecure versions (2.0.2 ->1.4.6) and nobody knows why.

Or at least, nobody's talking.

I don't know what to do yet, so I decide to go and get lunch. I work out the possibilities.

There's something wrong with our validation procedure- they're actually patched and we're reading the wrong thing.
There's something or someone else downgrading these systems.

Number 1 requires more documentation, which $BC doesn't seem to want to show me. Number two might be hiding in logs, which are emailed to me on a regular basis.

I walk back to my cubicle, grab my laptop and a notebook and find a quiet corner to figure things out. I find one in a tiny conference room.

I read through my emails and search for any of the logs from the api servers.

I spend about ten minutes on Stack Exchange for the appropriate sed, awk, tee and cat munging to pare them down to what I want. Eventually I dump them all to Excel, because I am a bad person.

Some filtering and I can see what's going on. The system orchestration updates each server every other midnight. I see about three quarters of them download the 2.0.2 version as a part of the night's update.

Every two nights a (seemingly) random selection of servers updates. I scribble the order on the conference room whiteboard and stare at them for a few minutes.

Nothing in the orchestration system logs shows another process loading the older 1.4.6. version. But something is.

Nothing in the logs emailed to me obviously points to another process.

I take a walk to get a coffee and think. Nothing comes to me and I have to scour the kitchen for unflavored coffee. I walk back to my conference room to find an intern-like person.

me:"Hey, I apologize. I didn't know the room was reserved. I'll take my stuff."

Other person:"That's ok. Are you Rob?"

me:"Nope, sorry"

I take my stuff and make my way back to my cubicle.

A few minutes searching leads me to a shared root password for the servers stored in the password vault.

I login to one of the remaining servers running 2.0.2 and look at the running processes. Nothing obvious like "random updater".

I'm stumped.

I lean back and stare at nothing in particular trying to come up with some ideas.

Unfortunately, it's fairly packed and I'm next to a bullpen.

Voice 1:"So the Sky Caps put blotter in the vat without telling anyone"

Voice 2:"Hilton Honors kicks' Marriott Bonvoy's ass any day."

Voice 3:"No, I'll pick her up at 4"

The voices wash over me in some clip reel workplace sitcom haze. I'm not going to get anything done. I take a walk around the offices to get the lay of the land. It's a Hanna-Barbera cartoon of grey cubefarms, tan breakrooms, free coffee but no snacks. The only attempts at color are people's cubicles. Family pictures, shirtless men with fish, desk toys and action figures. It's like a mall- everything's pleasant, non threatening and in identically-sized stalls, with colorful (but bounded) individuality, all for commerce.

Then I find the Hot Topic meets Successories manifesting in a cubicle. There are two dorm-room sized posters of the gold Bitcoin-coin, along with framed inspirational quotes about success and perserverance set against pictures of Game Of Thrones characters and muscle-bound men in insignia-less camo. A new leather jacket with an embroidered skull is on the back of the chair. This person is either a hoot or insufferable.

I keep walking. I have a breakthrough.

Where are the API servers getting the older version to install? Maybe that'll lead me into the library. I'm not yet Adso, but perhaps I'm one of the other ,lesser scribes copying my book and scribbling fanciful drawings of the things I miss, like decent coffee and a cell-mate that doesn't snore.

I walk back to my cubicle. A different intern-shaped person is in the conference room, all alone.

I can't save them. Eventually they'll be standing in the corner of their cubicle looking away while the middle manager cleans out the rest of their team.

I'm in my seat. Some searching results in a few possible repositories. Some more searching finds me the one repo that still has v1.4.6 of this application.

Just to make sure, I compare a downloaded copy of v1.4.6 and the installed version of v 1.4.6 on one of the servers.

I search all the folders and files for the URL of the repo server and find it.

In the application itself. The server waits every two days and looks to the repo. If the installed version is not equal to v 1.4.6, it downloads v 1.4.6 from the server and installs it, then forces a restart.

This code is commented out (made non-executable) along with an actual comment:

/REMOVE BEFORE PRODUCTION

I quickly scan through the API servers to find one of the ones still running 2.0.2. I search for the term "REMOVE BEFORE PRODUCTION"

And there it is, in the application code.

Except it's not commented out.

In a text editor, I write up my findings, conclusion and a recommended fix- delete the upgrade code snippet, increment to 2.0.3, push it out using the orchestration tool and call it a day.

LC Chat won't let me attach my text file, so I breathlessly LC Chat my document, line by line at Vincent, the poor bastard tasked with closing audit finding 162, the mystery of the random rollback.

Vincent:...

Clearly, Vincent is choosing his congratulatory language carefully.

Vincent:"Can't apply the fix. The application is owned by Development. They're behind on other things, so they won't update the software until next quarter."

me:"It's about thirty lines of code we can comment out"

Vincent:"Can we say it's fixed for the audit since we know what the problem is?"

me:"No. We can patch it, or we could write up a remediation plan and get it on some schedule."

me:"But that's more paperwork than the actual fix."

Vincent:"But Ops isn't on good terms with Development."

me:"So they're not going to touch it any time soon."

Vincent:"Probably not"

me:You guys own that repo server, too"

Vincent:"I don't see how that's good for anything"

me:"We cut out the update code in 2.0.2 and call it 2.0.3. We name the file 1.4.6 and replace the existing 1.4.6 on the repo server. Either the app gets updated via your orchestration server or it updates itself. We're fixed in two days either way.

Vincent:"But policy requires that we get approval"

me:"There's an exception, if you have a superior in Operations to sign off, you can call it an emergency fix. Ask Trevor. He just needs to not tell anyone else. You submit the ticket and eventually the devs will get to it and fix the problem for good. Until then, you pass that part of the audit."

Vincent tells me he's going to talk to Trevor. I'm going to take a walk. Out of curiosity, I go back to the Hot Topic cubicle to get a look at its occupant.

The jacket is gone and the monitors are off. Mystery person has left for the day, I assume. I look at the large jars of nutritional supplements with macho names- Gorilla Rage, LumberJacked, Psycho Focus".

I notice the name-plate on the outside of the cubicle.

Oh, no.

Ian.

To Be Continued...

edit- made modifications to satisfy Internal Audit 8-)

submitted by /u/lawtechie
[link] [comments]

Did they send me expired grease?

Posted: 30 Apr 2020 07:06 AM PDT

So, I'm not a tech support pro, but I am to go-to guy among my friends, since I know enough to google the issue and have built my own PCs before.

Dramatis Personae:

OldFriend (OF): Wonderful woman - and wife to AG - I have been friends with since our university days. Avid player of JRPGs and Jump&Runs since the PlayStation 2. Currently in love with her Nintendo Switch.

ArchitectGuy(AG): Husband of OF. Convinced of his own technical abilities, since he fixes everything around the house and is an architect. Nice guy, but overly-concerned with his manliness. Has been known to gently mock his wife for her gaming ways.

Me (ME): Me

It is lockdown season. Gamers all around the globe have time for their hobby and OF spends hers playing through the Nintendo eShop, using frequent skype conversations with ME to stay in touch and challenge me to play through her latest Mario Maker 2 levels. But about three weeks ago something new happened:

OF: So, my husband is really bored and wants to try gaming. But he doesn't like the kiddy games I play.

I can hear her eyes rolling through the voicecall

OF: He wants a gaming PC, but I'm really out of the loop with the latest hardware. Since you bought a gaming PC recently, could you recommend one to us? I don't think he'll use it a lot, but I'd love to play some RPGs that aren't on console anyway, so I said I'd ask you for recommendations.

ME: Sure, but if you just want to try some RPGs, your Dell Laptop should be fine for some older ones...

OF: I know, but he wants to do it properly. A real gaming machine... Could you find a tower for 700-800€?

ME: Of course, that's no problem.

After the call I shop around for about 20 minutes and send her a link to a good offer for 699€. Then I make a crucial mistake. I recommend using the storefront's configurator to pay 30€ extra for a gold-rated PSU, just to be safe and maybe consider upping the RAM from 8 GB 2666 to 16 GB 3000. Apparently this triggers an unprecedented avalanche of tech-elitism and penny-saving from AG. Who hops on our next Skype call.

AG: So I played around a bit with that configurator on the storefront. It looks like these components are pretty interchangeable.

ME (innocently): Sure, the components are from all kinds of companies. The store I recommended buys the stuff and assembles it for you.

AG: That can't be free.

ME (still not seeing where this is going): I guess not. It's a bit more expensive than buying the stuff yourself and assembling it. But they make their money from you buying all components through them, so the markup for assembly isn't a lot in the end.

AG: I've compared prices on [MythicalFemaleWarrior]. And I could get the exact same components quite a bit cheaper from there. Or better ones for the same price.

ME (with dawning dread): I wouldn't recommend that. Building a PC is a hassle and if you break something during assembly chances are you're not getting warranty for it.

OF (sounding slightly exasperated): See, I told you. Let the pros assemble the machine. They even have a 36-month warranty, so if anything arrives broken we can just bring it in to their store here in town.

AG: I can handle assembling a few parts. I'll just order them from [MythicalFemaleWarrior]. Thanks for recommending the parts, I looked around some and they look really solid.

AG leaves the call

OF: This will not end well. This will be the doorbell again, I know it.

AG and OF have a doorbell with an intercom, installed by AG. The intercom has never worked.*

ME: This will not end well.

Things are quiet. OF continues to do unspeakable things with Bullet Bills, taunting me to use their projectiles as a deadly staircase only to drop a Goomba on my head, when I finally do.

Until yesterday, when I receive a call from AG and OF. OF is on voice, but AG is broadcasting video from their garage, an open case lying on his workbench.

AG (without any introduction): What kind of strange grease do these computer people use? This stuff is useless and kind of grimy. Did they send me expired stuff?

ME (confused): Hello AG. What?

AG: I'm building the PC and had trouble putting the RAM in.

ME (my brain choosing to ignore the grease comment): Did you open the clamps before trying to push them in?

Ten ominous seconds of silence

AG: I think so, yes.

ME: Push them down on both ends, then slide the RAM in. It should take a bit of pressure, but not too much. And they will click in place, when the RAM is sitting right.

AG: But what about the grease? Did they send me old gunky stuff?

ME (brain slowly remembering that this was the start of the call): What grease?

AG: The grease they sent me with the parts. I used it to lube up the RAM, but it didn't help at all.

ME (harbouring a terrible suspicion): What does the tube of the grease say?

rustling sounds, a tube comes into view.

AG: Super Thermal Grease.

To my shame I must confess that I started laughing at that. As an excuse I can only say that OF laughed harder than me. As my penance OF will drop off the poor components and a bottle of Scotch with me, so I can try to rescue the DIMM slots and RAM by applying alcohol internally and externally. But AG still wants to finish building the PC himself.

submitted by /u/Cachar
[link] [comments]

Go figure.

Posted: 29 Apr 2020 10:50 AM PDT

Got this one close to a year ago.

A user called in saying that their laptop kept overheating. It never turned off or told them that it was overheating, but they said it was too hot.

I ran OBD and didn't see a significant spike in temps and everything passed it's test. I did notice that it felt pretty warm, but after a three hour intense OBD, I figured it may be expected. So I assumed "Eh, the user just thinks it's hotter than it should be. Maybe they're blocking the fan intake/exhaust making it extra hot."

I gave it back to the user and told her that I didn't see any issues, but to be sure that the intake and exhaust are never blocked so that air can move freely through it. She's happy with the suggestion and I don't hear back from her.

Until about a month later.

The user calls again saying that now it's shutting off and saying that the temperature is too high. So of course, now I need to take a closer look.

When I got the laptop to my desk, I run a primary OBD and within 2 minutes, it cuts off. The thing is SEARING hot. I noticed something was smeared on the keyboard as well, so not wanting her dinner from last night on my hands, I grabbed some wipes to sanitize it.

However when I wiped over the keyboard, I noticed what looked like a light colored pet hair come out of the keyboard. So I wiped over it again to get it, and more came out. I wiped for a solid 2 minutes before I finally decided it was time to figure out what the hell I was dealing with.

So I unscrew the laptop and all manner of pet hair comes out of this thing in a cloud. It coated the motherboard, it was in all of the cracks, it was in the fan, it was packed into the keyboard, it was inside the trackpad, I mean everywhere. It took me 2 full hours to clean it out in gloves and mask, wiping it all down with germicidal wipes and alcohol. I didn't even bother with the keyboard and trackpad, I just grabbed one from a parts laptop and tossed the originals in the garbage.

Lo and behold, when I reassembled it, there were no more heating issues. Go figure. Why did it take so long to get to this point? My guess is there was just enough clear space around the CPU and fan to get by a month prior.

I took it back to the user and told her about the pet hair and asked why there was any in there, to which she responds, "Well my cat really likes to get on it when I'm working, so I let her sleep on it every night."

I promptly told her to never let the cat on the laptop again. Never heard back.

submitted by /u/Music4lity
[link] [comments]

Context, who needs it?

Posted: 29 Apr 2020 12:58 PM PDT

A large part of my job is being an escalation resource for tough issues my team runs into. Another tech is also an escalation resource. It's more difficult than it needs to be...

IM exchange I had with a coworker earlier today:

Coworker: Hey! On $user's machine, do you remember what version of chrome you updated it to?

Me: No. Chrome updates itself and we don't manage them. It updated to whatever the current version is.

Coworker: Ok, thanks.

Roughly half an hour later.

Coworker: Even after updating chrome to the newest version user is still getting the "400 bad request" error.

To be clear, there was no previous conversations related to anything about this. Whatever Coworker is talking about is the first I'm hearing of it. But sure, let's roll with it.

Me: …To what site?

Coworker: site.client.com

I've disabled popups for that site. Still bad request.

Let's not question how popups being blocked or not would give "bad request" for a website.

Me: Well I can't hit that page from my machine or from an external device, it's probably an issue on the client side.

Coworker: She's connecting to the client via VPN before trying to access that site.

Me: Ah, well that's important information you should mention.

But that doesn't change the answer, currently. Their vpn, their site, their issue.

Coworker: Ok thanks.

Coworker then decided they didn't like my answer, then proceeds to talk to the other escalation tech, who gives them the same answer I did. Because I still have a desire to be helpful, I went searching the ticketing system for a ticket for the user's issue. All I could find was a recent ticket complaining about issues with their PC, and in the notes Coworker says the PC was replaced. And I recalled something that needs to be done for that specific client. Back to chatting:

Me: Taking a guess based on the only ticket the user has, did you replace their PC?

Coworker: I did.

Me: Did you get the hosts file copied over? Client needs a few entries for their stuff to work correctly.

Coworker: No, but I didn't do that when I replaced another user's PC recently. They access the same client. That user had no issues.

Me: I can't speak to whatever magical routings that PC managed to acquire, but that's one of the first thing Client's support will ask about.

Coworker: Ok, I will try it and let you know.

Another half hour later…

Coworker: Ok! User got into the site, but now she's trying to view something and she's getting a different page not found error when she goes to view. It comes up in a pop up.

Me: Check the hosts file for that server name in the popup's address bar and replace it with the IP, just to confirm it's not an issue with that. But the user is fully in the client system, you'll have to contact the client most likely.

Coworker: Ok thanks…

Me: Next time, if you remember to be more forthcoming with what you need help with, we could have gotten to this conclusion in maybe 30 minutes, instead of the two hours this took.

At this point most of my team has been told, multiple times, to list what you've done when you need help and escalate. Almost all of my first replies to requests for help are "What have you tried already?" And more than a few times they've tried nothing. One tech now basically stream of conscious documents his troubleshooting steps into a word document but also manages to include random screenshots with no context to them. It's kind of impressive, actually.

Bonus, also happened today:

Boss: Can client console push the client software to a PC, even if it doesn't have the client installed?

Me: It can. You just need the IP address and to make sure the service account for the software is in the local admin group, which is done via GPO and should already be there.

Boss: Ok, I'm talking to Tech about $issue. He says he's tried it a few different ways and nothing has worked.

Me: Ah, $issue. Yes, OtherEscalationTech and I gave him a few different things to try. But tech never got back to us if those didn't work. Hard to help when we don't know they need it. Without information can't determine if it's an issue with Tech or the process.

Boss: Tech claims he's reached out to you and you haven't been helpful.

Me: Well here's a screenshot of the entirety of his messages to us about $issue. As you can see that was a week ago, and we've heard nothing since.

Boss: Ah… Well he should be reaching out to you soon. Thanks.

I also know that the reason Boss was talking to Tech is his continued installing of licensed software without making any attempt to find out if we actually have licenses available for that software. I can only imagine that conversation was not a fun one for Tech.

submitted by /u/TheRubiksDude
[link] [comments]

I need a new laptop - my manager said it is ok.

Posted: 29 Apr 2020 08:12 AM PDT

This short tale is not a particluar individual experience but a breakdown of the exact conversation I have sadly had many times with many different users.

When users are assigned a laptop the policy is that they keep this until it is no longer economical to repair or burst into flames. The IT dept are the owners of the equipment and have purchased them from their own budget. On other different fiscal years each department would have purchased them, but IT still has the final say on what, when, if to buy.

User: Hi OP, I have a problem can assist/help?

ME: Sure, what's up

User: The battery on my laptop is only lasting for ...(insert - 30mins, 60mins, less than an hour, less than 2 hours). I cannot work this way.

ME: Hmmm ok, well....

User: I've already asked my manager and they say I can have a new laptop

(which = wahhh, wahhh, boss can I pleae have a new laptop, this one is broken!)

ME: Well sorry I'm not giving you a new laptop

User: Why not my manager said I can!

ME: I'm not goping to spend £1,000 when you current laptop is only 2 years old and a new battery only costs £80

Now this normally is the end of the subject but I do get some persistent users then try to rattle of extra problems with the machine and 99% are resolved with the offer (notice I say I offer :> ) to re-image the machine. The other 1% is:

ME: ok, well if you USB port (or whatever) is broken for over a year why did you not report it earlier if such a bad issue?

User: I just coped with it, until now.

ME: Mentally thinking "sigh, ohh f**k off!"

For the 1% that a reimage will not work or laptop is out of warranty I always have a healthy stack of more or less used identical spares to hand out and in most cases they are a little bit more dinged up. Only new hires get a new laptop and new a laptop assigned to exsiting users if no good spares available.

Epoc of the story is, try to get a new laptop out of me by whinging and stretching the truth to your boss, get declined and then take the gamble on what you have to work on in future.

Also, why and how in hell did you think a bad battery means a new laptop!

submitted by /u/FlashPan73
[link] [comments]

Breaking News

Tech Support

Thursday, April 30, 2020

Bad Architecture, part 3, digging deeper... Tech Support

Bad Architecture, part 3, digging deeper... Tech Support

No comments:

Post a Comment

Popular

Blog Archive

Fashion

Beauty

Travel