• Breaking News

    [Android][timeline][#f39c12]

    Monday, February 10, 2020

    When an Unstoppable Addiction Meets an Immovable Web Filter or A Cautionary Tale on AD Privileges Part 2 Tech Support

    When an Unstoppable Addiction Meets an Immovable Web Filter or A Cautionary Tale on AD Privileges Part 2 Tech Support


    When an Unstoppable Addiction Meets an Immovable Web Filter or A Cautionary Tale on AD Privileges Part 2

    Posted: 09 Feb 2020 11:13 PM PST

    Part 1 here

     

    Greetings, and welcome back to another tale of tech failure support. Sit back, relax, kill Mr. Poofers (google it), and please do the needful. To set the background $Me works as an L2 tech, which is to say the end of the line. My team gets tickets for $Org that were not able to be resolved by the helpdesk. If we aren't able to resolve the issue, then we will generally engage the engineers at the relevant vendor. That, or we tell the $user they are out of luck. We handle everything from diagnostics to AD administrative tasks. The way our system works is that tickets get assigned to our queue, and we have dispatchers who assign tickets to individual technicians on our team.

     

    Let's set the stage:

     

    $Me - The protagonist of this story, runs on coffee and lo-brau brand beer. He also has a cape that flutters in the breeze of a "hero-wind" branded fan.

    $User - Fateful ticket generator. The source of the story

    $L1 - Level 1 Helpdesk

    $TM - Technical Manger, our resident IT Dr. House who makes final decisions on process.

    $M - Manager of $User

     

    My office is right next to the area the L1 phone jockeys are in, and I'm the unofficial L2 point of contact for the helpdesk. If they need help with a ticket and it's quicker for them to ask me as opposed to just following the escalation process, I will generally jump in and help out with their callers. Before I begin, I should explain that we basically have two types of AD accounts. The first kind is the standard user account that most employees have. They get a generic set of access to various applications, and any additional access they need requires them to submit a request to be added to a security group in AD.The second type is a special kind of account that has certain privileges that are usually reserved for special use cases. These accounts have unrestricted web access and that's where this story begins.

     

    Where we last left off, I explained to $User that we would review the web filters and send a temporary access request. $TM had me fire off an email to $User's supervisor for review. A review of their repeated attempts to visit multiple NSFW websites. I should also note here that nobody in the organization should need access to these websites. While I was fully expecting the user to be insta-fired given the evidence against him, it didn't happen quite the way you might expect. The email chain went something like this:

     

    $M: Yes, I spoke with $User. They told me the helpdesk was rude and refused to give him the access he needed. I don't care what you have to do, just make it happen. Maybe you should hire competent people who can do their jobs without all this nonsense about forms and such. Just do what you need to do.

    $M obviously didn't read the site list we sent. Before I could reply $TM did for me.

    $TM: Let's be clear. Those forms exist for a reason. In this case, $User wanted to access several NSFW websites. Do they have a legitimate business reason to access those sites? Or did you not read the logs $Me sent? As for the request, $L1 clearly stated that there are processes we need to follow. I've attached an MP3 file with the phone conversation $User had with $L1.

    A few hours go by

    $M: I'll have a conversation with them.

     

    A few more hours go by and I can only assume a sizable ass-chewing ensued. A little while later we get a sheepish email from $M that reads:

    $M: Go ahead and disable $User's account. We won't be needing his services anymore.

    I know it's a little anti-climatic but I found it entertaining. Have a good evening everyone and wear your seatbelts!

    EDIT: This story has been simplified for easier reading but if anyone has questions, comments, compliments, death threats, etc. I'll be happy to respond to them. Aside from disabling an AD account, my day should be pretty light work-wise.

    EDIT 2: Bonus points for those of you who catch my (not so) subtle references in part 1/2. There probably won't be a part 3 because I live on Valve time, but you're likely to get a card game or a VR prequel that literally nobody asked for!

    submitted by /u/blueblood724
    [link] [comments]

    When they just sit there... and make you watch... that's physical torture

    Posted: 09 Feb 2020 12:55 PM PST

    I'm not by any mean a tecnician, nor have any part in the pc system.

    I'm just a 30 yo guy who has had a pc for most of his life, and make pizzas... But today my friends i felt your pain.

     

    So, this happened yesterday, Saturday evening, we work in a family owned restaurant, and I'm on the pizza side of the business in front of the cashier.

     

    I noticed we started having troubles half through the service, orders were slow but didn't know any better.

    After couple hours i had a time break, and while cleaning i overhear my boss ranting, so i asked and he started complaining that the main pc screen (a touchscreen monitor where we send orders from) was "half" .

    what?

     

    I peaked from behind and noticed the aspect ratio was 4:3 instead the usual widescreen 16:9.

    I was just about to touch the monitor panel to set that back to normal, he grabbed my hand, blocked me and started mentioning my EVERY. SINGLE.THING.THAT.HAPPENED.UP.UNTIL.THAT.MOMENT.

     

    Me: "W-well just let me touc-"

    Boss: "No, you don't understand, i touched the program, and it closed, so i touched the menu bar and it opened the Google, so i turned all off, unplugged every cable, now it is still like this and the phone support will answer in 2 hours etc...." (11 minutes of this)

     

    Me: "Well touchscreen commands won't work as you think because the aspect ratio of the image isn't now the same as the touchscreen area, it's not like your smartpho-"

    Boss: " no you don't understand, my daughters also tried it all, we clicked the "ctrl alt canc" and it opened but it is still like that, so i had to use the wireless keyboard and the trackpad to close the orders and use the smartphones..." (6 minutes of this, for real, I was watching the pc clock)

     

    Me: "yeah well, your daughters really suck at this (cool environment in this way), if you just let me-"

    Boss: "no you dont understa-"

    Me: " just f*ck back to the side-

    i pushed his chair back, bolt in front, press the 2 stupid touch sensitive buttons on the screen for the menu while he's screaming: "I ALREADY DID IT, IT DOESN'T CLICK ANYTHING" (screaming like an innocent convicted dragged into prison for life).

     

    I proceeded to stick using the display buttons to navigate the monitor menu instead of pretending to click it with the mouse like they all tried to do, and it goes back to normal as our lord and savior the spaghetti monster always wanted.

     

    It was tough, i pushed through, and came out broken inside, remembering all the times stuff like this happened even here and knew it won't ever stop.

    submitted by /u/dluca22
    [link] [comments]

    Telecom Chronicles Part 1: Stop! What are you dOiNg?

    Posted: 09 Feb 2020 11:37 AM PST

    Recently, I and a few of my colleagues worked on scripting, testing, rolling out, and training our department on a new phone system. The testing schedule was a bit cramped and gigantic red flags or straight up broken systems were rammed through.

    The usual characters in these capers are:

    $b: $me, a sentient barnacle

    $wd: work dad, one of our most senior members, solid dude, wanted on every project for being well-informed, resourceful, and level-headed

    $dev: an abrasive, hard-working, and often mistaken dev who supports our endeavors

    This particular day, we had encountered a number of odd issues that excluded entire portions of our intended routing, or resulted in significant failures that dumped callers into an infinite loop of hold music with no further options.
    We created tickets to our $dev for each unexpected behavior.

    $b: ok, let's test our IVR options, messaging, and on-call system for after hours.

    $wd: sure thing. You're assigned as the on-call agent, I'll call in as the customer.

    $ivr playing on wd's phone: Thank you for calling $HEALTHCORP. Our offices are currently closed. If you are experiencing a medical emergency, please hang up and dial 911. If you would like to speak with an on-call technician, please press 1.

    $wd presses 1

    [HORRIBLE ROYALTY-FREE HOLD MUSIC] for 45 seconds

    $ivr: Good bye.

    $wd and I stare at each other.
    My eyebrow quirks.
    He shrugs.

    Our phones are sitting on the conference room table, untouched. My phone starts ringing. The number shows the outbound test phone number.

    I tentatively pick up and immediately switch to speaker.

    $ivr: This is an emergency, afterhours call from $HEALTHCORP. Press 1 to accept.

    I figure I might as well make the best of it, and press 9 to make sure our invalid response prompts work.

    $ivr: Good bye.

    $b: Well that's wrong. A: there's nobody calling anymore so why did I get a call B: why does the system keep hanging up on us?

    We try it all again just to make sure the behavior can be replicated and to run a trace of the phone traffic, but instead we have a surprise cameo from a rising star.

    I dial out. Almost immediately, we get a response. $Indian dude: hello, who is this?

    $wd: who are you and what are you doing in our VoIP sandbox?

    $id: my name is [OH MY GOD THATS A LOT OF SYLLABLES]. Who are you?

    $wd: I am on of only 4 people who are supposed to be working on our telecom test environment. So bud, what are YOU doing in our sandbox?

    $id: I'm testing $proprietarytool. (Proprietary tool is not supposed to enter testing until we are done with our work and the rollout is done, AKA, 3+ weeks).

    Turns out this dude was actively turning on and off features in a our testing environment when he wasn't even scheduled to be in there, effectively causing all the unexpected behavior.

    $wd and I asked $id, a nice enough gentleman in the end, to buggar off and not come back in the kindest way we could.

    At this point, $wd and I check in with our telecom dev lead to see why this nice gentleman is starting his work.

    $b: hey $dev, it's $b. We had a nice chat with $id. Why's he testing $proprietarytool right now?

    $dev: we told him to go ahead as it will speed up our deployment schedule.

    $b: but he's activating and deactivating features and routing that we are testing for the rest of the week. You know, the system that we send you tickets about when we encounter an issue? The system that you adjust parameters for as we test different scenarios?

    $dev: oh, I guess we'll ask him to hold off.

    $b: alright, no worries.

    REEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE

    EDIT: Hunting typos

    submitted by /u/Breowleh
    [link] [comments]

    Last Sale

    Posted: 09 Feb 2020 08:04 AM PST

    This happened around 1996.

    I was the field service tech for a cash register company. One of our biggest customers was a large corporate owned chain of supermarkets.

    We didn't service the entire chain, just the eight stores in our service area.

    I was on call for this one.

    About 3 AM my pager goes off. I call the service to see whats what.

    A drunk had driven into a power poll directly behind one of these markets and knocked out the transformer.

    The local power utility had done a fantastic job and got power back online with a temporary repair but with the power on only one of this stores 12 cash registers came back online.

    These dinosaurs had been installed originally back in the '70s. They had been upgraded over time but they were old. We were the only ones still servicing them and our parts supply consisted of old boards that had been repaired by us. There were no "new-old stock" or any other kinds of replacements for these things.

    Now keep in mind this store was open! Corporate decided that if they had one working register and the lights were on, they were going to open.

    So I load up my truck with all of the spares I can find and head out.

    Out of 11 machines, I had to replace 10 power boards, several CPU boards, and more than one memory board.

    After 7 hours or so, being stopped every 5 minutes or so to answer the age old question "Well why can't you open another register?" I get them all up and running, but not a single one of them had its program intact. I needed to download it from the one working machine. Explaining to the LONG line of shoppers that I would have to shut down the one working register for the 15 minutes or so it would take to record the program (on audio cassette! Did I mention these things were old?) well, that did not go down well. It was like a chorus of "Why don't you just open another register?"

    But shut down we did.

    15 minutes later I had my tape. One by one, 15 minutes at a time I loaded the program to the remaining registers. Still had to tweak the program a bit. Machine names and such. These machines were programmed in hexidecimal so that was no easy task either.

    12 hours later I have the store up and running.

    Heres the stupid part.

    In the back room they had 12 BRAND NEW Point of Sale terminals all programmed and running! They didn't even power off during the blackout because they were all on UPSs. They had been training back there. The only reason they hadn't been installed was because they were waiting for new pedestals to put them on and for a trench to be cut for the new network wiring.

    The bill for this adventure was rubbing right up against 5 figures for parts and OT/DT labor and Corporate tried to cheap out and talk us down from the full amount. The best part was when they suggested that it would be good customer relations. We pointed out that they did not buy their new system from us and therefore they were no longer a customer! They eventually paid the full amount.

    Their new system went in a week later.

    I did the math and the repair bill would have just about covered the cost of the new hardware.

    submitted by /u/bradley547
    [link] [comments]

    When you tech support the tech support with a quick look at the manufacturer’s site

    Posted: 10 Feb 2020 03:50 AM PST

    Hhi all, first time posting here. I'm not even completely sure I'm in the right place. As the title suggests, I had to find a solution to a problem for the tech support department, but I'm not tech support, unless you count being the closest tech guy in my department and also helping out with small issues. Maybe it's easier to just share the story?

    I worked in my college's library at the laptop circulation desk. I was hired to maintain and check-out the laptops that we offered. Being the egotistical kid I was, I quickly asserted myself as the first line of help for all computer-related issues. I worked hard to streamline a lot of the processes we used to make my life easier. You know, like I'm sure most of us do! I should note tgat I actually was a student at the time as well, so I didn't really get to handle fixing the more campus-wide issues.

    My main tech support communication was the man who was in charge of maintaining the computer lab. I never really had to deal too much with him. Mostly it was only broken laptops needing replaced or complaining about how weak the batteries were until they finally would order new ones. He'd take them at the end of the term and image them. Past that, it was my realm.

    As many colleges do now, we had a system to pay for prints with the swipe of your student ID. I want to say ours was called Uniprint, but it's been eight years since I moved on and my memory is as strong as a rusted bear trap.

    Now, I never had trouble with the system for the most part. I even was the one to push for them to install it on the laptops too. Of course, that all changed one day.

    A patron came up, and told me their print just released a blank page. I apologized, and printed it on my own system. But then another patron had the issue. And another. And more, until "there's some issue with printing sometime, let me take care of that for you" was repeated several times an hour. It was irritating to everyone: me, the patrons, the other library workers. I complained about it to the computer lab guy, but he said he wasn't sure what the issue was, and that it was just our system.

    So I started troubleshooting on my own. It happened on both the wired and wireless computers. It seemed to only affect PDF documents. I just couldn't find a correlation to solve the problem. I did also learn that it actually was happening in the computer lab as well.

    This went on for several months. I wouldn't hesitate to say 6-12 months. Every time I brought it up, the actually tech support people would just tell me that they had no idea and they had looked online for a solution but found nothing.

    It was at that last month mark that my limit was reached. I didn't like having to get up to print people's things off. Call me lazy, because I am, but I just couldn't believe that there were no reports of this issue from anywhere else. So I finally took it upon myself to check out Uniprint's website.

    Take a guess what I saw the moment their page loaded.

    The first post on their main page at the time was about EXACTLY that issue. It explained exactly why the issue was happening (something about Acrobat sending the file to print, and deleting the item in the queue when it was done, then sending it to the Uniprint station). It linked to a patch for it. It had been dated several months prior to when I finally decided to look.

    I was livid, because again, IT WAS RIGHT THERE! I went to my boss, the librarian that did most of the management for the public area. She had been barraged by people about this problem too. I told her that I found a fix. I explained to her the why, and she easily understood it. She had me write up an email for her to forward all of this info I found to the head of IT. He seemed a bit surprised about how easy it was to fix, but they had to get it approved by whatever team had to approve that stuff. Cue a few more months of waiting until it was finally fixed.

    And I, the one person who was apparantly smart enough to go to the manufacturer's website, got no recognition from anyone other than the other library workers, who were quite happy that the annoyance was finally over.

    submitted by /u/chibiachika
    [link] [comments]

    Government Work, a Contractors Tale

    Posted: 08 Feb 2020 03:57 PM PST

    Hey All.
    Disclaimer: no fancy IT techy stuff in this story, just Gov't shananigans.

    BACKGROUND Electrician here working with a commercial/industrial company that is mainly contracted to Gov't. Rural Australia, we cover a sizeable area and deal with all the different agencies.

    Been here about 15yrs, but over the past 8yrs have grown and manage the "tech" side to now have me and 5 techs. Fire, security, CCTV, door access, cabling, installs, servicing. And pretty much anything that needs to talk to something else.
    Small town, we're pretty much the only ones who cover this range of services, so we get around and know everyone. A lot of the gear we install and service is networked, so we work closely with the different Gov't IT departments.

    ACTORS $me - me, contractor
    $SPoC - site point of contact for the Law Enforcement site (admin and site maintenance co-ord)
    $LEHIT - Statewide Head of IT for Law Enforcement (they were new at the time, there is a different person in this role now, so I think I'm ok posting)
    $LocHIT - local Head of IT (the office that does sysadmin and on-site tech support for all the different agencies, except help desk. Known him for years)
    $LocSysAdmin - local Gov't sysadmin (also friend)
    $LocTech - local Gov't tech (good guy)
    $LEOIC - Officer in Charge of the Law Enforcement site (high rank, nice guy, known him for a few years)

    STORY Government work as a contractor can be difficult at times. Different agencies and departments (Local, State, Federal, Commonwealth) overlap in their scope so apart from working to various Industry Codes, you have the inter-agency/department and political minefields as well.

    Got a call one day after lunch from my office just saying "Urgent job, there's a problem with the data outlets you put in at <law enforcement> site".
    Two months prior, I'd done a handover of our part of a renovation at this site. Power and data for a new multi-purpose room. It's primary function was as a regional emergency response/co-ordination center, so it had all the bells and whistles.

    Arrive at site, grab testing gear, sign in, sign out a site access card, head up to $SPoC office to see what's going on. $me - "Hey $SPoC, what's this about my data outlets?"

    $SPoC - "None of the PCs can access the network and there's a group using the room in an hour."

    $me - "All 12 PCs? Anything else in the room not working?"

    $SPoC - "I called law enforcement IT help desk, they said they could see the switches and everything else is ok, just not the PCs. They said it was a cabling issue, so I called your office."

    $me - (weird, but there's been weirder) "Ok, I'll check it out and report back."

    Head down to the room, see the PCs are indeed on, lift the lid on the first of six floor boxes we put in (2 data outlets each for the 12 PCs plus power outlets). See the two PCs using this box are unplugged from the data outlets. Lift the lid of the next to see the same. All 12 have been unplugged. That's strange.
    Head to the server room and see..
    Sweet Mary Brown! Someone has finally been in to tidy up the racks! With colour coding for different services! When I was there 2 months ago the racks looked like they'd been collateral damage of a cable factory explosion.
    Look at the patch panel and switch where the patch leads for my outlets should be to find just holes. Look around the room to see boxes of new patch leads. Ok, easy fix then, but there are obvious questions before I just start plugging in.

    Call my friend at local IT on his personal mobile, because I know he hates it.
    $LocSysAdmin - "What did you do?" (I accidentally shut something off 2yrs ago. He wont let me forget)

    $me - "Nothing yet, but if you like I can start randomly turning power off. Hey, are you guys involved in the server room tidy-up at <law enforcement site>?"

    $LocSysAdmin - "Yeah, the new $LEHIT must have weird fetish for tidy server rooms and asked us to do it. We have $LocTech doing that. Why?"

    $me - "I'm on-site because <law enforcement IT> have impugned my cabling and terminations for the PCs in the new fancy room, but all I'm seeing here are holes. Holes in the room outlets, holes in the patch panel and holes in the switches. I mean, I'm just a sparky, but aren't these COMputers supposed to be, you know, plugged in to stuff?"

    $LocSysAdmin - "Ha. Ha. We had to send $LocTech to (reboot the thing) at one of the schools, he should be back in a bit. Wait, what do you mean holes in the room outlets?"

    $me - "All the PCs are unplugged from the data outlets. I know $LocTech wouldn't have needed do it, so I'm going with my usual answer for weird shit - Cryptids. Hey, they want to use this room pretty soon so I may as well put the patch leads in. Can you check what colour they're supposed to be and if the ports have been changed from what they were?"

    $LocSysAdmin - "Says here red, and no changes to the ports (reads off ports)."

    $me - "Thankyou wizard-man."

    Go and plug in the PCs, put the patch leads in, start seeing all the pretty lights. Call $LocSysAdmin.

    $LocSysAdmin - "I can see them. What would we do without you? Why are you even there? The site knew we were doing work on the racks and law enforcement IT gave us the job. $SPoC should have just gone and seen $LocTech."

    $me - "You've been here too long to ask silly questions like that. Hang on, $SPoC is walking toward me, holding a phone, looking concerned. Don't hang up."

    He hung up. (This is where it gets all government-y)

    $me - "Hey $SPoC, you're PCs are back up!"

    $SPoC - (holding phone out to me) "I know, thank you. I have $LEHIT on the phone, they want to speak with you"

    Polite introductions.
    During this I'm wondering why the statewide head of law enforcement IT is wanting to speak with the contractor. Also, that was pretty fast for $SPoC to know. Also, $LEHIT or their team must have been monitoring those ports to know the same time $LocSysAdmn did. Also, what have I walked into.

    $LEHIT - "So I can see the PCs are back, may I ask what the issue was?" (Said very pleasantly. I'm now fully alert)

    $me - "Someone had unplugged the PCs at the floor box outlets."

    $LEHIT - "Hmm. I see. Where are you at the moment?"

    This is where I say the stupid thing. $me - "In the server room checking the traffic lights on the switch."

    $LEHIT - "I see. Was there anyone from local IT or local law enforcement there with you?"

    $me - "No, just me and the racks. They're looking very nice now by the way."

    $LEHIT - "Thank you for your help today, could you please wait outside the server room and pass me back to $SPoC please?"

    $SPoC - (speaking to $LEHIT) "Yes. No. Yes. He always has in the past. No. Ok."

    $SPoC - (after hanging up) "I've been asked for us both to wait here" (closes server room door)

    15 minutes later the $LEOIC is there, in person. During this time $LocTech is back and at work. $LEOIC - (serious face and no handshake) "$me, what did you do? I've had calls from $LEHIT and the (his boss, the head of law enforcement for the state). $LEHIT has issued an alert of potential network security breach."

    $me - "All I did was look at the lights on a network switch to see if the PCs were ok."

    $LEOIC - (smiling now) - "Relax it's ok. I think the new $LEHIT is making an example out of you about our local procedures. It does mean you can't just go in there by yourself from now on. Thanks for getting the PCs back up. See the shift sergeant to give a statement before you go please. $SPoC, you're with me."

    $me - (relieved there may only now be a small bit of paperwork) "Ok, thanks."

    Give statement, sign out, sign in access card, head to car, light a smoke.
    Text message from $LocSysAdmin - "You out yet?"
    Text back - "Yeah."
    Get a call from $LocHIT

    $LocHIT - (I can tell I'm on speaker) "Hey $me. First, thanks for not bringing $LocTech into this. Second, $LEHIT has their team checking the networks for your nefarious activity. And finally.. (audio from a what can only be a submarine movie - sirens and someone yelling BREACH BREACH, with $LocHIT and $LocSysAdmin laughing in the background)

    $me - (laughing) "Dickheads."

    I hang up. No one has asked about the cryptids.

    TL;DR: If you give me a key to your server room, don't act all surprised when I go in there.

    NB: I found out later how/why the PCs got unplugged. One of the staff wanted to hide from a co-worker to write a report in peace. When the PCs couldn't access the network, they pulled out their work laptop and tried it in each floor outlet.

    submitted by /u/OlderSparky
    [link] [comments]

    No comments:

    Post a Comment

    Fashion

    Beauty

    Travel