• Breaking News

    [Android][timeline][#f39c12]

    Thursday, August 2, 2018

    Phone Error Tech Support

    Phone Error Tech Support


    Phone Error

    Posted: 02 Aug 2018 01:11 AM PDT

    Just a quick one, whilst i'm waiting for some software to load. I work in a fairly large government organisation, and some of the users are a little clueless.

    One of our users comes into the office. They've got some problems with their email, and we're there fixing it. It's no suprise that a few users will come in this week, as we've just migrated from a local email server, to a cloud based one. We're getting a lot of things done these few weeks, as for us, it's the best time to get upgrades done.

    Clueless user: "Who's that over there?"
    HIT: "That's the guy here to upgrade our phone system"
    Clueless User: "Awesome, maybe that will solve the error my phone has"
    HIT: "Error, what error? How long have you had issues?"
    Clueless User: "About 1 1/2 years now"
    HIT: "OK... What's the problem?"
    Clueless User: "Well, the phone works fine, but the big red light to the right of it keeps flashing. That's got to mean an error, right?
    HIT sighs. "That light notifies you of voicemails"

    After deleting 150-odd voicemails, the error is resolved.

    submitted by /u/the123king-reddit
    [link] [comments]

    ESPN (or Extra Sensory Perception of Networking)

    Posted: 01 Aug 2018 06:39 PM PDT

    The following story just happened.

    My morning break rolled around and I'm standing at the coffee machine in the break room waiting for it to dispense the warm brown diarrhoea that constitutes as coffee at my workplace. It's free and my use of it is purely for effect this morning.

    A user appears in the corner of my eye.

    "Oh IT!" he says. I do have an actual name but I don't think he has bothered to learn it yet. "The printer out the back is not working."

    Ah the old "not working" fault description. My absolute favourite. Never mind the ticketing system I've shown him how to use and asked him to log tickets with multiple times. Bailing me up on my break or in the hallways is much easier to do I guess.

    "Sure I'll take a look at it after my coffee." I manage to say through gritted teeth, ignoring my top two pet hates. No real description, no ticket. Not calling me by my name was the icing on the cake. I've already internally decided not to ask any more details on the problem. I'm off the clock and I'm yet to ingest any caffeine at this point.

    Hoping that this interaction is over, my attention is turned back to the coffee machine.

    "Wireless"

    "I'm sorry?" I stutter confused and caffineless.

    "Wireless."

    "I'm sorry I'm not following..." I trail off, brow furrowed in bewilderment.

    "Do you think it could be the wireless?"comes the response

    "For fuck sake... I don't fucking know! You just literally told me about the issue. The printer is on the other side of the factory. I can't fucking diagnose the issue from here. Not withstanding you told me nothing about the actual problem itself. How about some symptoms? Be more descriptive man! Besides, can you see I'm on my break. Please kindly fuck off and leave me to my 15 minutes of peace"

    ...is what flashed through my mind in a second.

    What I actually said was, "I'm not sure. I'll have to check it out."

    He left unaware of the storm boiling in my brain and the lack of caffeine in my blood.

    As a post script to the story, the coffee was weak and watery and upon inspection the printer was working fine.

    submitted by /u/fruntside
    [link] [comments]

    Spectral interference

    Posted: 02 Aug 2018 12:38 AM PDT

    I do tech support for a cell phone carrier via chat. I got a chat from a woman who was complaining about poor service on her phone, but was also chatting FROM that phone, so I couldn't actually do any troubleshooting with her, because most of it involves turning off the phone and/or other things that would end the chat. I eventually convinced her to go to her computer and chat back in, but not before this happened. Also, I cleaned this up heavily, as her actual messages to me were in texting shorthand and about 25% emoji.

    Woman: So I'm getting bad service when I'm traveling, but also when I'm at home as well. I'm only getting about 1 to 2 bars. But I guess that can't be helped, because of the death.

    Me: I'm sorry for your loss, but could you please clarify?

    Woman: Oh, it's not my loss. It was in the 40's. Actually, it's a really old building, so I'm sure lots of people have died here.

    I'm so glad this was chat so she couldn't hear me laughing my ass off at this point. I wasn't sure if she was joking at first but the longer we talked the clearer it was she actually thought ghosts were interfering with her cell service.

    submitted by /u/gwynforred
    [link] [comments]

    Transferring data and the importance of reading

    Posted: 01 Aug 2018 05:36 PM PDT

    I work for a large phone manufacturer and a while ago I spoke to a lady who was trying to transfer data from a very old phone to a very new one. Doing this involves transferring data from the old phone to a computer program that will then connect to another program that can download it into the new phone, which isn't as complicated as it sounds if you follow the instructions we can provide you with.

    This woman has however been struggling with it for quite a while and has called us several times before. It becomes obvious why only a few minutes into the call when it turns out that she asks unrelated "why"-questions about everything, doesn't really listen when you answer her questions and wants to read everything that she sees on the screen out loud to you. If you interrupt her she'll tell you to "please let her finish" and starts over. So, as you can imagine this becomes a rather long winded call, but after struggling for a little while we manage to transfer her data to the first program without any major incidents.

    By then I've asked her perhaps two or three times if she's installed both programs and has both cables close at hand since her phones use different cables, which she says she has. She goes to plug the new phone into her computer.

    C: "It's broken! The cable doesn't fit anymore."

    Me: "Are you using the cable you received with the new phone?"

    C: "No, why do I have to do that?"

    Me: "As I said earlier you have to use the cable you received with the new phone because the old one doesn't fit."

    C: "I've plugged it into the computer now and nothing has changed."

    Me: "OK, so the program looks exactly the same as it did before you plugged it in? Is there a notification on your phone?"

    C: "No, it only says…"

    She starts reading the program's generic welcome text, which pretty much only tells you how to use it and what it can do, out loud to me. After she's finished we do quite a bit of trouble shooting, which includes restarting everything, checking the cable and connections and asking the customer to reinstall the computer program, but despite everything it's still stuck on the same screen. So nothing has changed, and the customer is getting a bit upset. We're about 30-40 minutes into the call by then, and about 20 minutes into my lunch break.

    C: "I've done everything now, why can't you just fix this? I never had this issue with my [phone from other brand]! If you can't solve this within the next ten minutes I'll just throw away this phone and never by anything from you again."

    Me: "I'm sorry that it's taking a bit longer than expected to fix this issue, but I am trying to solve it as soon as possible. Just to check, are you entirely sure that you can't see anything else in the program that wasn't there earlier?"

    C: "Well, it says "Download for PC" here, but that's really not important right now!"

    submitted by /u/ABreakLikeThis
    [link] [comments]

    More from Aviation Maintenance: Language Barriers

    Posted: 01 Aug 2018 09:52 AM PDT

    While in Germany, as I've mentioned on occasion, I found myself in charge of the POL (Petroleum, Oils and Lubricants—aka, Hazardous Materials) program and cabinets for my Company as well as the Tool Room. Between the two jobs as well as any engine work that came up, I was rather busy and was in dire need of an assistant. Towards the last eight months or so of my time there, we had a new Private First Class (PFC) show up and he was given to me by the Platoon Sergeant to answer my prayers.

    Unfortunately, PFC Delmonte (Like the fruit brand…reminds me of his name) just added to my workload instead of lightening it.

    PFC Delmonte was a Power Train (transmission) mechanic from Puerto Rico by way of New York City and really didn't care for the Army. This was his first post-training assignment, but he'd already figured out how to game the system. When he arrived he put on a show of not being able to understand much English (How on earth did you enlist, then?!) and therefore, was required to be accommodated with ESL (English as a Second Language) classes. After work in the barracks? Could hang out and BS with us all day. Get him in the shop? No hablo ingles.

    I learned quickly he seldom cared about much to do with work and was mostly disengaged. I was able to prove this theory once while we were painting a grill my squadleader, KG, had built from aircraft stands, a wall locker and a 55 galllon drum. The plan had been to paint the whole thing black using some spray paint I had in excess, and seeing how I had too much to do that I didn't want to putz around with this all day, I grabbed Delmonte and had him help me out.

    Watching him handle the spraycan, something piqued my familiarity. Having grown up in some unsavory areas, I realized he wasn't actually spraying the grill—he was tagging it. Smiling evily, I waited as he became engrossed in what he was doing before suddenly shouting.

    ZeeWulf "Oh, %$&@, it's the cops!"

    He threw that can one direction and started sprinting away across the airfield in the opposite direction.


    Needless to say, I realized from his behavior that he wasn't always the sharpest tool in addition to his lack of care for his duties. The best place for him would be checking in and out tools from the tool room. He wouldn't get into too much trouble in there, and would "expose him to more English."

    Around this time, in order to combat people using inappropriate secondary containment for solvents (and provide tighter inventory control in general) I had implemented a check-out policy from the POL lockers. Mechanics would have to go to the tool room to check out spray bottles of the various solvents. It would be the responsibility of the person running the tool room to check those items out as well, since the lockers were right next to the tool room.

    One afternoon, while I was working on an engine swap on one of our Blackhawks, there was a commotion down by the 'hell hole' in the tail boom. A mechanic was sliding out of the underside access panel, obviously dizzy and disoriented, holding onto a bottle of solvent. I climbed down to help him and determine what chemical he'd just obviously been unintentionally huffing—you see, the "hell hole" is a very tight space right where the tail boom of the helicopter joins with the main body of the fuselage. The ventilation in there is poor, so one needs to exercise caution in any solvents they might be using since the wrong stuff could create a very dangerous atmosphere in short order.

    I looked at the bottle and did a doubletake.

    ZeeWulf "Snuffy, just what the hell were you doing spraying this in there?!"

    PFC Snuffy "I was trying to clean it up…"

    ZeeWulf "You're supposed to use Isopropyl for that!"

    PFC Snuffy "I did!"

    ZeeWulf "No, this is Trichloroethylene! You're lucky you didn't pass out and die in there!"

    PFC Snuffy "Well that's what Delmonte gave me…"

    My head whipped over to the tool room, where I saw him casually not caring about what was transpiring mere feet away.

    ZeeWulf "Delmonte. WTF. He asked for Isopropyl, and you gave him trike?!"

    He shrugged.

    Delmonte "So?"

    ZeeWulf "Snuffy could have died. Did you even look at the bottle?"

    Delmonte "No."

    He didn't look like he cared too much. After that, I took away his POL locker key and made everyone come to me, instead. He got a little piece of paper telling him to be careful, but due to his 'language barrier' they couldn't actually punish him for negligence.

    TL;DR: Soldier in Germany attempted to gas another.


    Enjoy these? There are more!

    submitted by /u/Zeewulfeh
    [link] [comments]

    Devices Run On Magic

    Posted: 01 Aug 2018 11:05 AM PDT

    First post here, been lurking for a few days.

    So some backstory; I work at a small WISP (Wireless ISP) as the Sysadmin/Tech Support Supervisor, so on top of managing the servers and VMs I also take general support calls from the end users (We're a small company with about 5 people in the office so most of us wear a few hats).

    I will be [Me], Old Lady will be [OL]

    [Me]: Thank you for calling $WISP, this is Bukimari how can I help?

    [OL]: My internet's not working and I've already tried rebooting the modem, fix it!

    [Me]: (Understanding that she is referring to the wireless router as we don't use modems) Alright ma'am, I'd be glad to help you with this issue, can you verify if your 'modem' is powered on at the moment and has any lights on it?

    [OL]: It doesn't have anything on, it usually has a bunch of flashing lights but right now nothing.

    [Me]: Alright, it sounds like it's not receiving power at the moment. Let's double check that everything is plugged in properly to the wall outlet or power strip and also plugged in properly on the back of the device.

    [OL]: Well I'll have to get a flashlight to see, the power's out at the moment.

    [Me]: ...

    Needless to say, I had to explain to the poor woman that the equipment would have to have electricity to function and once the power was back on it should all come back up as well.

    submitted by /u/Bukimari
    [link] [comments]

    Off By Zero Error

    Posted: 02 Aug 2018 03:42 AM PDT

    Not sure if providing tech support to myself qualifies, but I think it should?

    I work in financial planning, but one of my roles these days is creating various technical tools that my team uses to create financial plans. A couple days ago, I was working on one of these tools - basically, it's an Excel sheet designed to take a cashflow table as input, and clean it up into a format we prefer(change a few names, delete empty rows, sum up the totals, etc.). I've been using an English-language version of this for months with no issues, but my team covers all of Canada, so now we need a French-language version as well.

    I translated the text, and expected to see the same nicely-formatted output we get in English and...it's trash. Half the rows are deleted, nothing is formatted right, and it's a mess. So, time to figure out what's going on. I look at the formulas, and most of them seem to be working fine, but there's a couple oddities. All the text formatting is right, but anything based on numbers(e.g., "are any of these greater than zero?") is failing.

    Quebec uses number styles differ from English-speaking Canada. We would write a thousand as "1,000", while they write it as "1 000". Simple enough for most purposes, but these numbers are being put into an Excel sheet, and that means Excel needs to treat them as numbers. And it seems that to Excel, "1 000" is a text string - it can use commas to separate digit groups, but not spaces.

    Okay, let's fix that. A bit of text substitution to clean out the spaces, and it's coming through as "1000" - that should be a number, right? Weirdly enough, no, it's not - IF(1000>0) is still coming out false. So I try another text-cleaning approach, and then I check it by adding something to the result. I figure that if addition works, it's a number, right?

    Sure enough, addition works fine. 1000+1 = 1001, so we're good. Except that, when I look at the final output, we're not good. Something is clearly wrong, and when I dig through it turns out to be a "find the largest of these" function - the largest one of a group of five positive numbers is zero. However, the one that I tested by adding one to it, that one(and only that one) works fine. It seems that a string of digits from a text function isn't really a number in Excel's mind, but the result of an addition is.

    I ponder this for a minute, and then realize what I need to do. In the cells where I'm doing the text cleaning, I go in and add "+0" to all of the cells I'm doing the text cleaning in. And sure enough, everything is a number now, and it all works perfectly.

    Sometimes the fix is even stupider than the problem.

    submitted by /u/Alsadius
    [link] [comments]

    Thought the whole system was going down, turns out people just need a reminder of the basics sometimes

    Posted: 01 Aug 2018 12:36 PM PDT

    Can't believe I can actually post on here now. LTR;FTP

    I work at a med-sized organization as a web dude, so technically not IT, but am required to help if I can, if not, then ask other IT members and ask for help. We use a Content Management System (CMS) where staff can write their own content for users, this way the web people aren't having to write everything and can focus on development.

    $M-Me $L- Lady $I - $L's intern

    $M working alone because others come in later. $I knocks on the door and says she needs permissions to access the CMS, I tell her to email us so someone else on the team can take care of it. I'm still relatively new so can't do everything yet. She says OK and promptly sends an email. Good. I go back to work.

    A few mins later, $L comes and starts talking about not being able to change something on the website, and $I is standing behind her. I look over to $I and ask "is this related to the thing you just came here for a few minutes ago?" $I replies "no we can't change a link for some reason, it won't change in the website". I get worried because this shouldn't be happening. All the systems are working fine and if there is a problem with trying to change links, it could be a big thing that requires contacting the people who've been here for much longer. Nobody here is bad with computers, plus $I is a student worker, so she should know about basics at least. This could be bad.

    I lock my computer and walk over to $L's room with them, and they've got the CMS open in editing mode on one monitor, and the website open on the other where they're trying to make edits and change a link. $L shows me the problem where the link in the CMS is changed to a different link, but when you click it on the website, it still redirects to the old link. I hover over the link in the CMS and it seems to be linked properly, so nothing should be wrong. There shouldn't be an issue here, everything is working fine. It's a relatively simple CMS to use, and pretty straight forward. I decide to reset and redo the whole process and see if that helps. So refresh the CMS, copy the link, link it to some text, press save, and refresh the website. When I click on the link, it redirects to what they wanted!

    $I screams, literally screams, "HOW??? WE JUST DID THE EXACT SAME THING!!!!". $L thinks and says "you know, the only thing we didn't do that he did was press save". $I with her high pitched voice replies something like "no wayyyy". I say something like "haha yeah happens sometimes, I've probably done this as well" and leave feeling relieved.

    Moral of the story, don't panic, the system is probably not dying. User's just need to be reminded of the save button.

    submitted by /u/amature_writer
    [link] [comments]

    Server Broke Computer FIX ASAP

    Posted: 01 Aug 2018 04:51 AM PDT

    Alt account due to colleagues i'd rather not have know of this recognising my main account

    This tale takes place back in '09 when i was but a pimple-faced young girl fresh out of uni, with a head full of big ideas and booklearning but shy of any kind of hands-on experience.

    I had taken up work as a part-time IT fixer at a local fix-it shop straight out of uni. The hours were bad and the culture was severely leaning towards the "sleazy old pervert" side of the spectrum but it paid rent at least. To be fair I did dress somewhat provocative, as I like having a touch of femininity in such a male dominated area. Besides, I had gone through a bad breakup a few months before and some primal part of me enjoyed the attention I got from my male colleagues. But I digress, so on with the story.

    I was enjoying a quiet Friday afternoon, lazily browsing some goth clothing webstore or another when my boss poked his head in and told me he had a customer in need of on-site support ASAP and since everyone else was either busy with other things or on vacation it was my lucky day. He tossed me a crumpled piece of paper with an adress on it and the words "Server broke computer FIX ASAP" on it.

    A server that broke a computer? I had heard about the reverse happening with bad setups but this was something new for me. I decided to go all in on this to show my boss I actually knew my way around IT so I grabbed a toolbox and rammed everything i could think of needing in it. Recovery discs, 'ware killer USBs, extra cables, screwdrivers, pliers, anything that could be remotely useful in bringing a downed computer back to life went into the toolbox. Once I had packed everything but the kitchen sink I got in my car and headed out to the adress on the paper.

    The adress ended up being a strip club in the bad part of town. Not really what I had envisioned when I headed out but a customer is a customer, right? The bouncer let me in through the back door and I began my search for the downed computer. A quick chat with the bartender and he showed me a small office in the back. On the desk stood a Dell that looked like it had been dunked in a swimmingpool, complete with a small puddle spreading out on the desk around it and a big wet spot on the floor. When I asked the bartender what happened and how a faulty server could cause a minor flooding he nearly bent over double with laughter. It wasn't a case of a IT server choking and dragging a computer down with it, but a case of one of the serving girls watering a potted plant near on the desk and fumbled the pitcher of water all over the computer on the floor next to the desk.

    Well, the poor computer was busted alright. Nothing I could do to save it but bring it back to the shop and fetch a loaner for them while we attempted to recover what we could. When I returned with the loaner I had to take a shortcut through the area the showgirls used to prep for the show and damn was that place bustling. Some guy (I'm guessing the manager) even tossed me a skimpy shirt a few sizes too small for my bust and told me to get ready for stage when I was heading back out after plugging in their loaner. I tossed it back, told him about the computer and headed back to the shop to clock out. To be frank I kind of took it as a compliment that some guy thought me good looking enough to be on stage so I never told my boss about that part when he asked how the job went on Monday.

    submitted by /u/Tech_Witch
    [link] [comments]

    Access Denied

    Posted: 01 Aug 2018 08:06 AM PDT

    Looking for a complete list of $404Comp tales? Here's the index!

     

    It's been a while...
    Sorry for the long gap since my last post! Got married, honeymoon'd, and shortly after my wife and I were putting together the next sizable chapter of my DnD campaign that we finally got to play this past Saturday!

     

    The Cast
    $LawnGnome: Contrary to his name he's actually quite tall and lengthy.
    $Savy: Yours truly, also avid fan of lemonade.

     

    $LawnGnome has been with us a couple months now and is eager to take on larger scale projects so we've been throwing him more regularly into disaster-level situations with plenty of folks on standby for support in case he gets walled. This particular support case was "mission-critical" level so naturally we threw him into the fire assigned the ticket to him.

     

    $LawnGnome: Ugh, I can't keep up with this anymore.
    $Savy: Hmm?
    $LawnGnome: Oh, sorry. It's just that the database over at $HearingAgency has gone offline for the third time this week. Staff over there gets an error immediately after they try to open it.
    $Savy: Umm, that's not good. What caused it?
    $LawnGnome: Don't know. It's been working fine, but all of a sudden it just refuses to open this week. The error is about having to repair the database.
    $Savy: eye twitch Let me take a look...

     

    Now, $HearingAgency used to be pretty quiet when we first took them on as a client, but they recently had one of their admin folks retire. Why is this relevant? Well, it turns out that said admin person had took it upon herself to compact the database on a weekly basis while she worked for the Agency as a form of maintenance. So after months of her being gone from the company, the size of the database had grown to 2GB? Not too shaby for a database, eh? Well, of course I wouldn't be typing up this Tale if that was the case.

     

    $Savy: ...this...this isn't a database; it's an abomination!!!
    $LawnGnome: What??
    $Savy: The whole thing is built on Microsoft Access!
    $LawnGnome: What do we do?!
    $Savy: We're gonna need some candles and possibly an intern to sacrifice to the IT gods.
    $LawnGnome: ....for real?
    $Savy: What? No, but just we need to call their director and let her know what's going on.

     

    [insert montage of everyone running around and screaming]

     

    To add to the complexity of the situation, the "database" had multiple versions of itself on the shared drive, and no one knew which was the original database. Further the issue with custom FrontEnds being developed for each individual staff member that was pointing them to all different copies of the database. How do we go about fixing this?

    • 1. Cry
    • 2. Salvage the database
    • 3. Identify the master copy and obliterate the impostors
    • 4. Do the same for all the FrontEnd databases

     

    We managed to barely squeeze some life out of the database by compacting it from its hard cap of 2.00GB to a walloping 1.96GB which was enough to open the database again. We normalized everyone's FrontEnd shortcuts so that everyone would point and work out of the same copy of the database.

     

    So...No Interns were harmed in the making of this Tale?
    After a thumbs up check we reported back to the Director and let her know that the database is stable but is no longer something we could support and that the database would fail again if no action was taken. She was happy that we brought this issue to her attention and assured us that she had contacted a developer to migrate the database to a more suitable platform. The kicker? That very same director ended up retiring a month after she made that promise to us.

     

    TL:DR: Someone had the brilliant idea to turn Access into their org's mission-critical database and let it reach the 2GB hard cap. Hilarity and Panic ensues.

     

    Edit: Spelling, round 1
    Edit 2: Typo

    submitted by /u/savacli
    [link] [comments]

    Tales from Desktop Support - Printer Edition

    Posted: 01 Aug 2018 10:16 AM PDT

    Hello fellow system supporters, tech teams, and printer punchers!
    I have a few short printer tales because The Cutover Part 3 is still a work in progress.

    The one where a camera company made printers.
    It's not Canon, but another one that was popular in the 35mm film era.

    To start, the service tech does not believe in DHCP, says it's too unreliable.
    So this company buys 20 of them anyways, tech pulled an IP and then set the printers to hold that IP static, the PC team didn't know the glaring problem with the method used until it was too late. We like to make note about printers if people will tell us about them.

    IT comes along later and re-maps the IP schema for various reasons (new hardware, getting rid of outdated technology/servers/etc.) It happens, deal with the fallout of not following protocol.

    The screams of offline printers may still haunt the dreams of those poor technicians...

    The only way to resolve it?
    Renew lease on printerOS?
    Ha!
    You had to shut down the printer, power down the subsystem, and then flip the main switch to drain all residual charge.
    Then and only then would it refresh it's IP address.

    The one that took 30 minutes to install-

    HP's Laserjet 400 pro series is by far one of the most annoying designs ever as it like to switch to Russian, it's heavy, and the driver installer is weird. I followed protocol, tried the simple driver first, it failed and made the mistake of using the full-installer.

    The installer did nothing for 30 minutes, but managed to load the print driver in less than 10 seconds!

    I cancelled the install hoping it would just stop at that point and let me load the proper HP software after the Sysadmins re-packaged HP's crap.

    Nope! Rookie mistake!
    Driver terminated
    *Windows Chime*

    New device detected - Discovering drivers now.
    *Standard Windows Error*
    Tell me what it is! I know nothing!

    I sigh and resign to watching the screen for 30 more minutes.
    It finally worked, but it took so long to do so much nothing.

    The Russian printer:
    In a new office suite, there are two identical printers located in adjacent offices and one of them decided to switch to Russian, we're not sure why or how, but it was odd. Even though I have a fantastic on-demand Russian accent for being an American, it was not enough to read or understand Cyrillic.

    I tried the konami code on it by stepping through the second printer's English menu to change the language.

    No dice, something wasn't quite right with the order in the menu.

    Aha! Try Google Translate!

    That was really hard because of the constant up/down at the desk/matching my English keyboard to the Russian characters.

    I finally reach enlightenment and remember now Google has a camera translator! After some very oddly spelled words, I got close enough to find the proper language menu and set it english.

    Day saved, user happy.​

    submitted by /u/techtornado
    [link] [comments]

    Band and his merry cluster of 365 initiates

    Posted: 01 Aug 2018 09:38 AM PDT

    Before we begin, allow me to introduce the players in this round:

    Band: There was a time that Quantum was a professor at a local community college. Band was one of his better students, which helped to get him into $themed. Band was a clever guy, started as an intern a little after I did. Wasn't afraid to get his hands dirty, able to keep up the pace with more complicated tasks, all around good guy. In the interim between the time I left $themed from being an intern to finishing my Bachelor's degree and coming back, Band had become a real boy, as some other tech had left and he'd moved into their position, and if I recall correctly, he'd actually gotten a promotion to an analyst at the point this story takes place. If not, he was the highest level of technician that $themed had to offer.

    Buchji: Buchji was the IT guy for the Pharmacy. Started as a pharmacy tech (actually dealing with drugs and such) long before I got my first taste of technology. He eventually fell into the IT role for the pharmacy and through a long and complicated tunnel he dug his way through to the IT side of the house as opposed to the Pharmacy side of the house. Being the pharmacy tech, he had his own office with all kinds of stuff that normal techs didn't have access to for various reasons. I first met Buchji when I was an intern and thought he was just an angry older dude. What I could have never told you is that he and I would become the absolute best friends, literally doing magic and successfully doing the impossible. He and I became the dream team, if someone else couldn't figure it out, they fielded it to Buchji and me, and we would figure it out. There was literally no problem that we were given that we didn't at one point or another find a proper resolution to. At this point in the overarching story though, we'd worked together a lot and were very good friends, but we hadn't figured out the capabilities we had when we worked together just yet.

    Turtle: Turtle was my direct boss. He and my father had been friends for years, and by extension he knew me. He and $Heart (his boss) worked with my dad for years and helped me get my foot in the door. Turtle loved to give us hell about all kinds of things, but would always come through in a pinch or when you needed him. 10/10 fantastic boss, would work for again.

    Tinker: Tinker was a higher level technician, on the same level as Band. He's known for being super good at tinkering. He was incredible with electronics and soldering and fixing broken… pretty much anything. He was also very good at hardware (computer and otherwise) modification. He's also *the* guy that $CEO calls when anything is wrong with anything technological she has.

    Wykydtron: The AVP of Information Technology at $themed.

    $CEO: The CEO of $themed.</p>

    *It is also important to note if you've not worked in a Hospital/environment before, there are software suites all over the place that depend on ancient software to run properly. Some of them still exist because the Hospital doesn't want to buy something else if the old system works (even if it is the worst kind of security vulnerability), or the vendor is no longer in business or supporting the software in question, and no one could find something that works better.*

    ------------------------------------------------------------------------------------

    Our story begins on day like any other, I had just walked in the office for the morning and was browsing through ticket ques and finishing a rebuild for a user whose hard drive had died. I'm tired, its 7AM, and I hadn't gotten my morning can of $energydrink (green/original, for those wondering). I skim through the ticket que and determine that nothing major is going on at the moment, and head to go grab a can of $energydrink and breakfast from the on campus café.

    On my way, I knock on Buchji's office door and check in on him for the morning, as he was on the way, and offered to get him breakfast if he was interested. He politely obliged and we headed off. We made it to the café and got our breakfast/$energydrink and proceeded back to my office. Not a minute after we'd sat down (it's around 8:30AM at this point), Turtle walks in and is literally freaking out about a call he'd just gotten from Wykydtron that $CEO doesn't have office on her computer anymore, but it wasn't just her, it was all of $Administration. Turtle proceeds to dispatch Tinker to go check on $Administration and $CEO and figure out what is going on.</p>

    <p>It was about that time I turned back to my breakfast/ticket que and I realize that the flood gates have opened. It wasn't just $Administration that had been hit by whatever it was that took out office, it was the entire environment. I quickly proceeded to scarf down my breakfast and I believe it was the fastest I'd ever chugged $energydrink, but I knew it was going to be an incredibly long day and the sooner I got started, the better off I was going to be.

    Not having been informed on exactly what happened, I start to ask questions. As a group we often talked about the goings on and changes that were going to be made to the environment, as my team was the one that the desktop images/updates/patches/changes lived with. Band was actually the guy using $softwaresuite to allow us to mass deploy updates and patches and deal with asset management. It was planned that there was a clearly defined list of 80 devices on 4 floors of $themed that were going to have $OldOfficesuites forcibly removed and have $OnlineOfficesuite pushed to them without technician intervention. The initial tests in our office had gone off without a hitch and Band was prepared to test on a larger group of users, but not something so big that, if it went wrong, the team would suffer brutally. We, as a collective, tried to automate everything we could automate to make life a little easier for the techs. Quantum and The Sailor had already moved on from $themed to their next big jobs at $Fortune100. Between Chewy, Band, Honey Badger, Macguy and $me, there was plenty of custom software to go around. Those guys, as well as the rest of the crew will come in other stories though.

    What was intended and what actually happened were quite different though, if you haven't caught on just yet. What actually happened was that $softwaresuite lost the file that clearly defined the 80 devices that were supposed to be our test environment, and as opposed to just stopping and asking questions like a good software should, it fired off to every device in the environment, and there was nothing we could do to stop it by the time we realized it. All we could do at that point is man our machines and try to reinstall everything we could as quickly as we could.

    It was impossible to get through all of the tickets (as there are 10,000 devices and 15 of us techs, only 11 of which were allocated to deal with tickets) that came in that day, but we did as well as we could. The next morning we came in, we found that a huge portion of those tickets we'd closed the day before were reopened. After further investigation, we found that $softwaresuite was still pushing the delete/uninstall command and had been all night.

    Awesome.

    We dug in and found the way to stop the delete scripts from being pushed out to more devices, but the damage had been done, and we actually ended up spending almost 2 months to get $OnlineOfficesuite redeployed by hand. And while you're wondering why we didn't just use $softwaresuite to push out $OnlineOfficesuite, we tried that but ran into catastrophic failure on devices that had the original scripts pushed to them.

    We finally did get everything back to where it was intended, it was just not exactly how we intended to happen the way that it did. It was that incident though that made us shape up our Incident Response plans and make sure we were well prepared and had a plan of attack for if anything like th… *snickers* who am I kidding? Management didn't learn any lessons from that incident, and it was actually just in the last couple of weeks that $themed got hit by an *actual* cyber-attack. From what I could tell, resolution tactics were the same and took an extended period of time with all of the techs available to issue proper resolution to that issue.

    Tl;Dr: Had a software suite powerful enough to push changes to save techs a lot of time upgrading to a new suite of products. Had a clearly defined parameter of "wild" machines to test on. Parameters lost, software suite shits bed, techs end up spending 2 months doing the job that should have been paced for around 6 months as emergency countermeasures. Management learns nothing.

    As a last note, during that two months there was a lot of day drinking ($soonMhine, for those curious) and a lot of frustration, but also a lot of pizza from $bestlocalpizzajoint, because morale was super low. We got through it and Band kept his job, and is actually still working for $themed. He hasn't caused an incident like that since.

    Edit: If you missed my last tale about Quantum and the Sailor, you can read through that Here

    submitted by /u/CorporateGandalf
    [link] [comments]

    No comments:

    Post a Comment

    Fashion

    Beauty

    Travel