• Breaking News

    [Android][timeline][#f39c12]

    Wednesday, January 13, 2021

    A failed application 'one-size-fits-all', 250 broken telephony reports, and the sudden realisiation that no money has been saved as IT never paid for that system in the first place. Tech Support

    A failed application 'one-size-fits-all', 250 broken telephony reports, and the sudden realisiation that no money has been saved as IT never paid for that system in the first place. Tech Support


    A failed application 'one-size-fits-all', 250 broken telephony reports, and the sudden realisiation that no money has been saved as IT never paid for that system in the first place.

    Posted: 12 Jan 2021 11:12 AM PST

    Some context for you: the company in question has around 3500 staff, grouped into about 25 business units, spread across about 20 sites. There are three service desk teams, each supporting some of the business units. There are seven telephony systems across the company, five of them with call recording systems hooked in. Five of these telephony systems and four of these call recording systems are supported by head office. The final two telephony systems are supported by the team I'm on, one of which has a call recording system linked to it, which we also support.

    One problem with this call recording software was that elements of the UI used Flash. The system had been around for a loooong time and undergone a few upgrades, but the last one was about 5 or 6 years prior to the events in I'm about to recall for your entertainment. Another fun fact: it only worked in Internet Explorer, not Chrome, Firefox or Edge. Our local business unit, who relied on the call recording system quite heavily, had been pushing for an upgrade for some time, but group IT kept saying no. That was until the CIO ordered a review of all systems and services used across the company, with the intention of eliminating any traces of Flash from our infrastructure. Finally, we were getting an upgrade.

    It's also worth mentioning that the call recording system was originally implemented by our local business unit, back when they had their own internal IT team, before their internal IT was merged with the group IT. Even though we still spent 90% of their time supporting just them, had contracts that differed from group IT, had bonuses based on local business unit performance rather than group-wide performance, etc., we still officially reported to group IT.

    Perhaps a decade after the system had been purchased, other business units decided they wanted to start recording phone calls, so group IT opted to go with the same system we used. Albeit a newer version, with fewer features licensed. Four new systems were setup, linked into four of the other phone systems across the group. These were all supported by the service desk team in head office.

    Around this time, a new service delivery manager came in and wasn't happy that there were three service desk teams supporting different business units. He wanted all the teams supporting all the users across the business. He wanted duplicate systems removed or merged and maintained centrally. He wanted device and systems configurations unified across the group. He was chasing a one-size-fits-all approach to IT.

    So when the decision to upgrade the systems was made, the task of coordinating it was given to the central projects team, and my team leader was assigned to the project to advise them of our requirements, risks, etc. For background, she was useless. And even more so when it came to projects. In one instance several months earlier, I had been taken off a project when it was moved to the central team and she was put on in my place. After a few months of the same issues cropping up over and over, I asked to see the issues register for the project. She didn't know what that was. It turned out she hadn't reported any of the issues we'd encountered, or reported any of the remedial works we'd been performing to the project manager over the course of the project. But whatever. I've got no involvement in the project, so don't really give it any thought.

    The week of the upgrade finally comes around and the new systems are run alongside the existing ones for a week. We receive confirmation from the project manager that user acceptance testing has been a resounding success and that they're ready to deactivate the existing servers and switch over to the new ones. This switch-over happened on a Tuesday evening. On the Wendesday morning, I get in at 8am to find that all the wallboards are down.

    I report this this to my team leader, who has decided to work from home as she was apparently up until 2am copying the config over from the old system. I'm a little confused by this, as surely this was done while the old and new systems were running side by side. I mean ... how do you do UAT on a system that has no config? But whatever. She says the wallboards just need logging into and pointing to the new server, which confuses me further because the wallboards are provided by our business unit's CRM, it is the CRM that links into the call recording software, and I don't have access to the CRM config. She tells me to raise it with the developers. So I do.

    edit: I later discovered that projects had only tested two of the four systems; the phone systems the recording system were linked into were running two different versions, so they tested the recording software on one system for each phone system version. Our site's phone system was on the same version as another site's, and the other site was chosen for testing rather than ours.

    Me: Hey [lead developer]. Have you had any reports about the wallboards not showing call stats?
    Lead Developer: Yeah. The call recording server is down. I'll ask [infrastructure engineer] to have a look when he's in.
    Me: The server isn't exactly down. It's been taken offline. The call recording system was upgraded last week and switched over last night. Here's the new server address.
    Lead Developer: Give me a few minutes.

    A few minutes go by and I start getting phone calls from managers about call stat reports not being received. I try logging into the new call recording system and find I don't have permissions to the reporting section, so I drop a message to my team leader, who tells me to ask the head office team for access. I send them a message on Teams and log into Barracuda with my auditor credentials to see if I could find out if any reports have been sent. The last report had been emailed at 6:30pm the previous day, from the old system. None had been sent from the new one. And then I get a reply from the head office team leader.

    What reporting section?

    Now a little information on the reports that this system generated. Most other business units had reports generated by the BI team, but our business unit used the call recording system. Team leaders and managers had permissions in the old system to generate scheduled reports, to be sent hourly, daily, weekly, or monthly. The could be run on extensions or groups that they had access to view stats on. This meant that a team leader could only generate a report for individuals in their team, or their team collectively. Meanwhile, managers could create reports for anyone in the teams they managed. Finally, the operations manager and a couple of analysts had access to create reports on any group or extension that existed within the system.

    There were about 250 reports in all, some used purely to give managers visibility of what their teams were doing, others used to gauge performance for bonuses (which were awarded to some teams every other week), there were reports for which numbers were calling us the most, some targeting new employees, some targeting employees that were under review.

    While the business probably didn't need all 250 reports, a few were critical to the business at an operational level, so getting them restored was a big deal. So I call the head office team leader ($HOTL).

    Me: Hey $HOTL. There should be a reporting section in the phone system. The permission group is called 'Reporting Admin'.
    $HOTL: I've never seen that section before. And you have global admin on your system. That will give you all the admin permissions.
    Me: I don't think you had the reporting module on your systems. But we had it on ours. And you're right. If I have global admin, I should be able to see the module.
    $HOTL: What was it for?
    Me: It tracked call stats. It let you run reports for calls and talk time and stuff for extensions and groups.
    $HOTL: Why did you need that? The BI team generates call reports.
    Me: It's just how it always was. Our CRM linked in to the reporting module for wallboards too.
    $HOTL: Why does your team always have to do things differently to everyone else?
    Me: Because our team started using this system before everyone else?
    $HOTL: Well now all our systems are the same. If we don't have the reporting system, then you probably don't either.
    Me: So what do I tell the business now that all their reports are gone and the wallboards are broken.
    $HOTL: Talk to [Service Desk Manager].

    FFS.

    As I get off the call, the lead developer tells me the same thing.

    Lead Developer: I don't think the new system has the call reporting system.
    Me: No. I just found out it doesn't.
    Lead Developer: So what now?
    Me: Can you replace the call stats section of the wallboard to say that it's broken and IT is looking into it?
    Lead Developer: How is "Group IT f**ked it all up again. Call stats will be back shortly."
    Me: I'm not sure about the 'shortly' bit.
    Lead Developer: I was feeling optimistic.

    He put a clean message up with an open timescale for the fix, and I called my team leader again, who also said that I should talk to the service desk manager. So I spoke to the service desk manager, who said that all call report requests should go to the BI team and that is how it has always been. I told him that our business unit had previously run reports from the call recording system. He reiterated that requests need to go to the BI team and that no such requirement was given during the project meetings. Thanks, team leader. So I called the BI team and asked if they were able to generate call reports from our phone system. They said they could, but they'd need details of the reports needing generating.

    Well this will be fun.

    I hopped into vSphere and booted the old recording server up, exported a list of all the reports, and did a bulk 'run now' on all the reports. While this was happening, I got a call from a Senior Systems Engineer the central infrastructure team. I'll call him $Bro because he was pretty cool.

    $Bro: Hey Escaped2theMountains. Did you just online the old call recording server?
    Me: Yeah, sorry about that. I need to export some config information that wasn't copied over to the new system.
    $Bro: Wasn't [team leader] up until 2am copying all that over?
    Me: She was, but she skipped all the call reports.
    $Bro: Why would she forget to copy over the call reports?
    Me: Because she forgot to tell projects our system had a call reporting module, so projects never bought the call reporting module.
    $Bro: How many reports were there?
    Me: Around 250.
    $Bro: Oh.
    Me: Our wallboards also rely on the call reporting module, so they're all down too.
    $Bro: Oh.
    Me: Yeah.
    $Bro: So why are you exporting the details for the reports if you can't set them up in the new system?
    Me: [Service Desk Manager] told me to send the details over to BI team to recreate.
    $Bro: You expect them to create 250 reports for you?
    Me: No. I expect them to buy the call reporting license.
    $Bro: Ah. But, next time, can you warn us before you turn a server online?
    Me: If I had asked, would you have said yes?
    $Bro: That's not the point.
    Me: If I had asked, would you have asked why?
    $Bro: Probably not.
    Me: So if I had asked, you wouldn't have found out how my morning is going.
    $Bro: Just warn me next time.

    Once I had a list of all the report exported and and examples of each report generated, I saved them to the shared drive, shut the server back down, and popped upstairs to see to one of the business analysts. I bumped into the business unit's Managing Director on the way, who said she was on the way to the lead developer to ask him about the wallboards and missing reports. I told her that there was a problem with the call recording server upgrade that had taken place overnight and that she'd get an email from one of her business analysts that she will need to pass on to the BI team.

    MD: Why do I need to pass it on to the BI team?
    Me: Group IT or Projects or whoever was responsible for the upgrade never bought the call reporting module. I've been advised by [service desk manager] that requests for call reports now need to go to the BI team.
    MD: And how long do you expect that to take? We need those reports now.
    Me: It would probably take the BI team months, if they can do it at all. But I'm counting on them not doing it at all, which is why the email requesting the reports needs to come from you.
    MD: How does that ... ah. You want group to buy the reporting module?
    Me: No. The BI team will want group to buy the reporting module.
    MD: Because I'm going to demand the reports be created today, which they cannot do.
    Me: [smiles]
    MD: Did [lead developer] tell you to do this?
    Me: No, but I would like to say I've been inspired by him.
    MD: [smiles]

    I eventually get to the Business Analyst and we start working through the list of reports. We decide to separate them all into groups, based on how essential they are. The ones used to inform staff bonuses went into the highest priority group, which we marked as needing to be completed by midday on Thursday. That was about 27 hours away. We also threw some of the global reports (i.e. reporting on all incoming and outgoing calls) and the repeat incoming call ones into this group. The group totalled around 70 reports. We then had about 60 into a second group, that we marked for completion within a week. About 40 marked for completion within two weeks. And the remaining ~80 for completion within a month.

    The business analyst writes this up into an email, attaching the .csv with the report configs to the email, and linking to a folder containing the 250 example reports that he's saved in his section of the shared drive. We call up the MD and he forwards it over to her. The MD then forwarded it over to the Head of BI, with the business analyst CC'd in. A reply comes back about 5 minutes later saying they'll need to arrange a call as such a request could take months. The MD replied to say that was unacceptable, with the CEO, CIO, Service Delivery Manager, Service Desk Manager, Head of Projects, the Project Manager involved in the project, the business analyst, and my team leader CC'd in, stressing the importance of these reports to the business. I was still sat with the business analyst, so I got to see the loooong CC field in all its glory. And, thankfully, my name didn't come up once.

    Though, unfortunately, it did make it's way back to me. The Service Desk Manager found out that I'd raised the issue with my team leader and the head office team leader, and I ended up getting conferenced into a call about an hour later. Oddly enough, my team leader was absent from the call ...

    Service Delivery Manager: Hi Escaped2theMountains. We need to understand what is going on with the call reports and you seem to know more than anyone else in IT about it.
    Service Desk Manager: Yes. Apparently you've been maintaining 250 reports for your business unit. Who asked you to do that? We aren't budgeted to provide that support.
    Me: Hi all. I'm sorry you only found out about this requirement for the recording system now. I can confirm that the business unit had around 250 scheduled reports configured in the old recording system. But we don't really support it.
    Service Desk Manager: What do you mean, you don't really support it? It's an IT system. Who creates the reports? [Team leader] said she doesn't.
    Me: Did she not explain that the business unit create the reports themselves?
    Service Desk Manager: No?!? Why are they doing that? They shouldn't have admin access to the system. It's an IT system. Who authorised you to give users admin access to the recording system?
    Me: They didn't have admin access. They had reporting access. There were three levels of permissions for reports. Admin, creator, and viewer. Team leaders and managers had creator permissions, so they are able to create scheduled reports. And then...
    Service Delivery Manager: This sounds like a security risk. What is stopping a team leader from listening to their manager's calls?
    Me: Recording and reporting have separate permissions. And, as I was going to say...
    Service Desk Manager: How much time are you spending setting these permissions?
    CIO: Can everyone please let Escaped2theMountains say what he needs to say. And can someone get projects on this call.
    Me: Thank you, [CIO]. So we enter each phone extension into the system, and then put each extension into a group, and link the extension to a user. We flag each extension in each team as an agent, team leader, or manager. We then set recording and reporting permissions in each group. For example, in Team A, Team A Agents have access to their own recordings and no reporting, Team A Team Leaders have access to all Team A Agent recordings and Team A Agent and Team Leader reporting, and Team A Managers have access to all Team A recordings and reporting. So if an agent in Team A gets promoted to a Team Leader, we go into the group and flag them as a Team Leader and their access is updated automatically. This is part of our 'joiner, leaver and transfer' process, which you, [Service Desk Manager], recently signed off on when we updated it a few months ago.
    CIO: What about access for people that aren't in Team A.
    Me: We can also grant individual users permissions to recordings and reports for each group. I would show you, but that functionality isn't included in the new system, and the old system is offline. But there are very few people with access to recordings outside their own teams. Just the senior managers and directors. And the business analysts, but they only have reporting access.
    CIO: And you said there are also creator and viewer permissions for reports?
    Me: Yes. These are set at a user level. So agents have no reporting permissions, most team leaders have report viewer permissions, and some team leaders and all managers have report creator permissions. But they can only view and create reports for groups they have reporting access to. If the Team B Team Leader tries to run a report on Team A, it will show no results as they don't have access to query Team A's call stats. So even if we accidentally gave an agent user reporting permissions, all reports would be blank as they don't have any reporting permissions to each group.
    CIO: And you don't spend time supporting this at all. Just creating extensions, users and setting permissions.
    Me: If the disk space on the server fills up, the reports will fail to run, so we delete some old reports off the server. And when the SMTP relay address changed last year, the reports stopped working as well, so we had to fix that. But for individual reports? No. We don't touch them. That's all done by the business.
    CIO: Thank you. So, [Service Delivery Manager]. Why didn't we purchase a call reporting license for [business unit's] instance?
    Service Delivery Manager: The requirement was never raised with us and we had never paid for it in the past.
    CIO: So [business unit] was getting the module for free?
    Service Delivery Manager: I would presume so...
    Project Manager: Sorry to interrupt. The module wasn't free. [Business unit] were paying for it.
    Service Delivery Manager: Why were [business unit] paying for it?
    Project Manager: Because they had always paid for their instance. Both the core license and the reporting module license. (I found out that the PM only found this out 15 minutes before getting onto the call)
    [silence ...]
    Me: [Business unit] have had this system since at least 2004, back when they had their own IT department. The rest of the group only adopted the system 6 or 7 years ago, after their IT was absorbed into the group. I guess administration of the system moved over to IT, but the bills still went to them. But that's why we were on an older version to the rest of the group. There was a newer version out when the other systems were setup and ours was never upgraded.
    Service Delivery Manager: But now we pay for it.
    Project Manager: Yes I believe so. All four systems are under one contract.
    Service Delivery Manager: But we paid for four systems before. And we pay for four systems now. So that doesn't add up.
    Service Desk Manager: The system at Site E wasn't upgraded. It was decommissioned.
    [silence ...]
    Me: Am I still needed on this call? Shall I go and tell [business unit] that they can cancel their contract for the recording system?
    CIO: We will handle that. [Service Delivery Manager], can we order a reporting license for [business unit]?
    Service Delivery Manager: I'll contact the vendor now.
    CIO: Thank you. Escaped2theMountains, I think we have everything we need. Thanks for your support to the business this morning.
    Me: No problem. Let me know if you need anything else.

    The reporting module was installed later that day and my team leader was tasked with creating the 250 reports. I didn't hear from her at all the next day, so I guess she was up all night and slept through the day. It was quite a peaceful day, actually. And the wallboards were working when I got in. Turns out the lead developer put a piece of code into wallboard thingy so the message would only appear when the reporting endpoints returned an error, otherwise it would display the call stats as intended. I loved that guy ...

    submitted by /u/Escaped2theMountains
    [link] [comments]

    Travis the POP3 champion

    Posted: 13 Jan 2021 01:23 AM PST

    This is from years ago when I was a Systems Consultant for a fairly big MSP. I've always wanted to tell it properly, so I think this the right place for this lovely yarn.

    *Be warned, this is a long read!*

    At the MSP, I was a System Consultant for multiple companies. The MSP hired me out, I travelled like a hired gun, fixing things here, getting projects done there. I was part of an operational server group, and we were pretty good at what we did.

    It's 2014, near the end of the day and I'm finishing up a few things. My Operations Lead, comes up and asks me if I'm available to meet a new client, for a 'major issue'. No job code or anything, a brand new client that heard about our company as a MSP in the city. Client knew a guy that knew a guy, and eventually hit my Operations Lead. I figured, a lot of my operations work is done, my big projects are on hold anyway. So yeah, let's see what I can do to help.

    We hoofed it since their office was on the outskirts of downtown, and we were in the core. I ask a few details; what's the issue, how much time/effort we talking about? My Ops lead was sparse, didn't know the exact details, but knew that there was some email issues.

    When we arrive at the office, say our hello's to the CFO. She ushers us into a meeting room, and says she'll be back in a moment with the IT Manager.

    Now, this is where it gets weird. There's about four other guys in the same room. I thought they were employees of the company, so to be polite I start handing out my business card and introducing myself. I greet two people, then the CFO steps back in.

    I'm paraphrasing here, but this is the jist of what the CFO says to everyone in the room:

    "Thank you all for coming here, I'm hoping you can help us at Company X. Our email system has finally crashed, and I'm looking to each of YOU to provide a solution. I've brought our IT Manager to help me make the decision and choose one of you".

    Turns out, the four guys in the room were all representatives for different MSP's in the City. The CFO called a bunch of companies to come in, and present different ideas. I got past feeling embarrassed pretty quick, because now we're down to brass tacks with this company's issue.

    IT Manager lays it out the best he can. He's also the IT Manager because he's good with CAD licensing, but not so great at infrastructure. He doesn't know much about the server gear, so Company X hired a third party contractor to run the VMware/Storage/AD/Exchange environment. We'll call this third party guy Travis.

    IT Manager says the entire company of 200+ all have trouble logging in, keeping mapped drives, can't synchronize email, calendars, network resources constantly drops, and everyone has finally reached their breaking point. He asks us all desperately for our plans on fixing this, and will award the winning company the work. I ask what the order of priority is. Top of his list, is getting email functional again.

    We present our plan; remediate, rebuild the exchange server, then start looking at the infrastructure. IT Manager hears the rest of the plans, and based on our plan of attack he picks our company. The other 4 guys clear out, and I make my way to the server room to see what exactly I'm dealing with.

    The server room wasn't pretty. Network Cables running across the ceiling tiles from the ISP drop to switches on the other side of the room. Network was spaghetti. Zero cable management, all plugged into whatever port they could reach. Physical licensing servers strewn across the floor. Each one with their own keyboard/mouse and teetering monitor. UPS devices were desktop models at full capacity, other UPS batteries were dead, and the alarms silenced. Paperwork was everywhere, stacks of random project paperwork was anywhere space was available.

    Once I logged in, the underlying infrastructure wasn't in much better condition.

    The third party contract, Travis, in his infinite wisdom, migrated all the users OFF Exchange server into some third party POP3/SMTP service.

    The entire office was using outlook 2013 with Goddamn POP3 email. No IMAPI protocol either! We're talking about 200+ users dealing with the worst email system on desktops and smartphones. No functionality, no synchronization of sent items, no calendars, just a terrible setup.

    The backups were useless, all Veeam backups between 4 different USB connected drives. Only project data, some SQL databases, and their CAD licensing servers. No AD information backed up I could find anywhere.

    At least the VMware hosts were running. I noticed at least 2 of their drives blinking red. Uh-Oh.

    The RAID wasn't busted, but it needed attention. I ran the Serial#'s of the Dell through the website. Out of warranty since 2 years ago. Okay, this is bad.

    I called up our warranty boys, give them the serial# and FRU of the Dell disk. About 15 min later, they tell me the device is registered to some company in the United States (We're in Canada), shipping address, company name are completely different. This is a local company, no branch offices anywhere else.

    Finally, I meet Travis. Travis tells me the servers were purchased online from Ebay. Gulp.

    My Ops lead works with our procurement department on pricing out a replacement disk. I focus on the VMware infrastructure.

    Luckily, was all licensed properly, HA is enabled. The Vmware ESXi hosts were all willy-nilly configured with different NTP servers. The PDC was no better, it was still picking up time from the host. It was something like 5 minutes off, no wonder everyone's network drive was disappearing!

    AD was not structured. No separation or hierarchy of departments. Some OU's were completely empty. Computer OU was full of objects, half of which were dead or old computers that haven't logged in for years.

    Travis tells me, it was a joint decision to move away from Exchange server (and all the wonderful synchronization abilities, like contacts/calendars/tasks, ability to tie into AD) because the backups were too large. Also, the POP3/SMTP service is owned by the company he works for.

    All users also have local .PST files they're keeping their email since the move to SMTP. Which means, there's no email backup. Their computer dies, all their client correspondence dies with them. Each user had about ~1-5GB of PST data. Also, Travis kept outlook because it could read the .NK2 autocomplete files.

    My Ops Lead calls up for some warm bodies. Desktop resources to help out while I get an Exchange server spun up on their VMWare infrastructure.

    It was an all-nighter. Among some of the laundry list items: Building a new Exchange server, fixing NTP issues, including AD into backups, finding replacement disks for their servers, helpng desktop team to manually import each user's PST into their newly created outlook profiles, and fixing just obvious issues like enabling AD recycle bin (instead of re-creating an AD account when someone gets re-hired). Those are just some of the issues I remember.

    I couldn't believe that Travis the consultant, actually talked the IT Manager OUT of exchange into POP3/SMTP. Shame on this dude for putting the company into this situation. If he spent a little more time looking at infrastructure, just looking at the list of requirements, and maybe getting help from one of his peers… they could have saved themselves a lot of trouble.

    In the end, we weren't awarded a contract. We got their email up and running, configured on their smart devices, and fixed what we could. The CFO was thankful, but still held onto Travis. We just went in, and did what we did.

    TL;DR - if you have a third party contractor, make sure he doesn't try to sell you on moving to a POP3 service for the entire company!

    submitted by /u/dstar_888
    [link] [comments]

    DONT TOUCH THAT!

    Posted: 12 Jan 2021 06:27 PM PST

    Obligatory double post to spread my pain across the internet.

    This was back when I worked for a cloud filing cabinet, storing and sorting files for hospitals and other large corporations.

    In this roll, within the first year everyone had quit above me, leaving me the senior lead for the large corporations side of the business.

    The healthcare professionals had me by about 5 years experience.

    Our stars:

    Me

    Boss: Real cool dude that is one of the best bosses I worked under.

    The Engineer(E): SQL Wizard

    The Nurse: Ratchet

    Needy: Special guest

    You might have noticed that the client I am working with is a nurse....and not some company CEO or accountant. You might think that this shouldn't have been my problem and you would be right.

    Scene: There I was, happy go lucky, getting crushed under tickets, project deadlines and general pressure, thinking I definitely had the day ahead of me to do my actual job.

    Then the phones started ringing.

    Act 1: What?

    I knew that it was a hospital because the Caller ID started with Saint and I let it ring because it's not my problem.

    Then the 2 HC professionals both said "NOT IT" simultaneously and started bickering while the phone rang, unanswered.

    My trainees had their own calls to make or were already preoccupied.

    Stupidly, I picked it up before it went to voicemail and I heard both of the specialists laugh followed by a few jeers of "Sucker!"

    Me: Helldesk, this is Absinthe.

    Ratchet: ITS GONE! WHERE IS IT? WHY DONT I HAVE IT? THE REFERRALS AREN'T GOING THROUGH! DO YOU KNOW HOW IMPORTANT THIS IS? WHY ARENT YOU TALKING?!

    Me: .... Do you have a ticket open with us already?

    Overwhelmed by the sheer force and vehemence I am immediately put on the defensive and she continues to attack.

    Ratchet: WHAT DO YOU MEAN DO I HAVE A TICKET?! IM CALLING YOU ARENT I!? DO YOU KNOW HOW IMPORTANT THIS IS? PEOPLE ARE GOING TO DIE!

    Me: I have no idea what you're talking about.

    Ratchet: THE SQL! IT'S GONE! I NEED THE SQL OTHERWISE HOW AM I GOING TO GET THESE REFERRALS STARTED!?

    Me: -turns my phone volume down to minimum- Okay, The SQL is down... I should be able to help with that.

    Ratchet: -unintelligible quiet muttering-

    -i turn up the phone a little bit.-

    Me: Can you repeat that?

    Ratchet: I SAID YOU CANT CONNECT TO THE MACHINE, THERE'S PATIENT DATA ON IT!

    Me: Uh... We store your patient's data... It's what you pay us to do.

    Ratchet: I NEED TO TALK TO SOMEONE WHO ISN'T A @#$! IDIOT.

    -Click-

    I sit there for a minute, stunned and HIPAA certified.

    Me: Boss, there's a nurse that thinks we don't store their data?

    Act 2: Homework

    Boss: was it from Saint Jerks?

    I look through my call history and confirm since the user never identified herself.

    Boss: That has to have been Nurse Ratchet. She's... Well you work with the billionaires I'm sure you're used to it by now. But she's #&$+ Nutzo.

    My boss then spends the next hour helping me go through their documentation, teaching me all the buzz words I need to get her to shut up and get me connected.

    On cue she calls again and I hear a chorus of, "Not it!" And " Finish what you started!"

    I pick up the phone.

    Act 3: Striking back

    Me: Alright. Listen here Ratchet, your system runs on {technical buzzword} and needs {admin access} and if I can't get both of those I can't help you.

    Ratchet: Well okay, it seems I got someone worth my time this time around.

    Me: Open up Teamviewer and let me connect so I can see what's going on with this SQL Server.

    Ratchet: YOURE NOT ALLOWE-

    Me: Ratchet, I meant through your computer. Which is designated as the access point via your own documentation.

    Ratchet: Fine.

    I get connected to her computer and start looking around through the documentation and on her computer. Everything looks.... Wrong.

    Me: Ratchet, why can't I interact with this session?

    Ratchet: that's because I'm working from home, and it's remoted to my work laptop that is remoted into the SQL server.

    Me: Why?

    Ratchet: because my dog is sick.

    Me: that doesn't explain why everything is wrong.

    Ratchet: nothing full screens, just use the scroll bars to get around the remote connections.

    Me: So... I have a 4 inch window, to scroll around your work computer... To find the scroll bar to work on the SQL window... And hope I can find it?

    Ratchet: Yes.

    2 hours of soul searching and window shopping later I find that the SQL server is perfectly fine and has everything it needs.

    Me: Ratchet I dont see any problem with the SQL server.

    Ratchet: why are you looking at the SQL server? That's confidential!

    Me: You said the SQL was missing! What am I supposed to look at?

    Ratchet: my computer!

    Me: why didn't you say that then?!

    Ratchet: am I talking to that same idiot as before?

    -click-

    I might have broken my headset and stolen a newer one from a trainee.

    More research: it turns out that she used to have SQL manager on her computer and it held a couple lines of simple SQL code to pull data from the server which was then turned into the documents we stored and sorted.

    Act 4: You knew?!

    I talk with my boss again who suggests looking into Ratchets old tickets and compare notes with mine.

    Way way back in the archives I find a dusty One Note that is dated 4 years previous and authored by one of the two Techs that refused the call. (That one note is a mess, don't get me started.)

    I notice that the 4 year old ticket shows Windows XP but the documentation on the SQL code shows Windows 7. Except... I was working on a windows 10 machine earlier.

    Then I find the cause of my issues.

    Her IT department got her a new laptop and didn't back up anything before trashing the old one. All her data is on their servers so they figured nothing important was on the laptop itself.

    This is the 3rd time it has happened and when I said it out loud with my sanity cracking the healthcare pros said "yep. So far as I know it happens every 4 years."

    I had been working on this, blind, for more than half the day and skipped lunch.

    I notice that the second iteration of code was implemented by E when he was the HC lead.

    Act 5: Finally

    I conscript E to help me put this ticket to rest and call up Ratchet again.

    This time I change up my tactic and just distract her while he works.

    -Call start -

    Me: Hey Ratchet, I found that someone smarter who's going to get this all fixed up.

    Ratchet: Finally.

    Thankfully she didn't fight about getting everything set up for connection this time and I explain the weird ass system that is needed to move around on her work computer.

    E: So, what the heck am I doing here Absinthe?

    -mute-

    Me: you're going to help me get this crazy &$-+ fixed up so she stops calling us. You're the writer of the most current code and just because I can read it doesn't mean I can write it out.

    E installs SQL on Ratchets machine and starts transplanting, updating and getting the code into working order. Ratchet starts to notice.

    Ratchet: HEY! WHAT'S HE DOING I SEE SQL! HE'S NOT SUPPOSED TO BE TOUCHING THE SQL SE-

    -unmute-

    Me: How's your dog doing? That must be awful to have him sick.

    Ratchet: Oh, the poor little darling he ate my husband's boots and he's just been sick all day!

    I basically talked her back and forth while he worked, double checking the tables on the server and making sure they're calling to the right etc. (Been awhile since I've had to use SQL it's all slipping away)

    Ratchet: Why is he touching the server!?

    Me: Wanna hear my recipe for chicken parm? I bet it's better than yours.

    Ratchet: oh no it's not! I've won the county fair cook off with my chicken parm!

    Ratchet: Why is Patient data on screen!?

    Me: What's your favorite color?

    Ratchet: Purple

    On and on this goes through the building, testing and implementation of the code.

    An aside:

    I had to put Ratchet on hold to take a call from one of my other needy clients that I hadn't been able to work on that day.

    Needy: Absinthe! You were supposed to call me at 10, why haven't you answered my 27 calls?

    Me: Sorry Needy, I've been stuck with a customer for... 7 hours now.

    Needy: but what about me?!

    Me: This customer is the healthcare side's version of you, except the Healthcare Absinthe threw me under the bus and didn't pick up the phone.

    Needy: oh no!

    Me: Yeah, I'll call you in the morning.

    Needy: okay

    Switch lines:

    Ratchet: Why aren't you responding he's not supposed to be touching that. What are these documents!?

    Me: So what's the weather like?

    Ratchet: Oh, it's just beautiful right now, perfect walking weather if poor little Buckley hadn't eaten that boot.

    Eventually everything is tested and working properly. The documentation is given another layer and an essay worthy ticket is on the books, closed.

    Ratchet: Hey! It's working now, amazing! Me and the girls have been wondering where those documents have been for a month! Thanks!

    -click-

    A month? @#$!&$ month?

    I guess people dying might not have been an exaggeration.

    Fin

    Edit: a couple minor mistakes that were bugging me

    submitted by /u/absinthangler
    [link] [comments]

    How I fixed a fax machine with Velcro

    Posted: 12 Jan 2021 01:19 PM PST

    User calls and says that they haven't received a fax in a while and when they try to send one it tells them it failed to send. So I call it and get a busy signal, then I ask if they have sent out or are expecting a large fax and they answer no to both. Then I plug my handset into the phone line and find that there is no dial tone. I look under the desk to make sure it is plugged in and guess what I find.

    She is using the cord as a foot rest. Once she took her foot off faxes just started poring in.

    So the cord is now tied up with Velcro and she has requested an actual foot rest.

    And that is how for the 76,239,672nd time in history, Velcro has saved the day.

    submitted by /u/waldcha
    [link] [comments]

    He Sure Fixed It Alright!

    Posted: 12 Jan 2021 05:29 AM PST

    In the early days of my IT career, I worked as the only phone support person for a company that provides construction labor and project management. We have make-shift office networks setup across the country on our job sites. Because of the industry the company is in, it was much more laid back than your typical corporate environment.

    I get a call in my first year there from a construction foreman with a very thick Southern accent, CF from here on. This was almost a decade ago now, so I'll remember the dialog as best I can.

    Me: Company Name IT Support. How can I help you?

    CF: Hey buddy. I just wanted to let you know that I just can't get my printer to work. I've tried everything.

    Me: OK. Let me see what I can do. Can I please have your username, so that I may connect to your computer?

    CF: Alright, it's <spells out username>.

    CF: Wait a second, I gotta go. I'll call back later.

    He abruptly ends the call. This is not unheard of, as these guys have duties on the site that come first and things can come up that require attention right away. I think nothing of it and I go back to my other duties.

    Roughly an hour later, CF calls back.

    CF: Hey buddy. I just wanted to let you know that I fixed it!

    Me: Oh yeah? What was the issue?

    CF: What I did was... I picked the printer up 3 inches... and then I dropped it!

    At this point, I'm trying my best to not laugh in the guy's ear.

    Me: OK. Well when the issue happens again, please give me a call back.

    I wrap up the call and have a good laugh with some of the other IT folks. It turns out the guy did call in the next day and did have printer issues to no one's surprise.

    submitted by /u/trippSC2
    [link] [comments]

    Not our Department

    Posted: 12 Jan 2021 04:00 PM PST

    An unfinished tale with an infuriating runaround.

    I've mentioned before that I work with traveling sales people and sometimes they retire and take on a new contract to keep selling with low sales quota pressure.

    This user retired in July and started his retirement contract in August.

    His re-implementation was a total clustertruck and I fielded about 12 tickets in one call for him, sending tickets to various departments basically asking them to do their jobs and get him selling again.

    One of my tickets went to Department A.

    Department A took the ticket.

    I forgot about it.

    Cue last week.

    Me: Helldesk. This is Absinthe.

    Retiree: Absinthe, I know I bug ya a lot, but I still can't get into {sales tracking program}.

    Me: Weird... I remember that being a ticket...

    -one ticket search later-

    Department A lead: We don't deal with this.

    This was dated one day after the ticket went out.

    The ticket was still in department A's ticket queue. Unassigned.

    Second ticket update: December 20th, my coworker had sent a message to Department A stating: "If this isn't for your department, please route ticket accordingly as our records show this to be your department's program. "

    No further updates, still unassigned.

    -send a scathing email to both my boss and Department A lead about how a sales person hasn't been able to track their darn sales for 6 months and the ticket was left in the wind.-

    -send out about 20 Teams messages across the board to be told. Sales Training deals with the tracking program.-

    Me: Thanks for holding there Retiree and sorry about all the incessant clacking, if I put you on hold for more than 2 minutes I lost points on my call reviews. I'm going to reach out to the proper authorities and get this fixed for you.

    -leave a message on the training line as per company practice.-

    Forget about the ticket.

    Monday: Ticket from {Sales Trainer} Subject: We don't control this system and Kate (manager with name change) said to make a ticket for HellDesk to get it to the proper authorities.

    Call back Sales Training and leave a message essentially saying WTF.

    Get back into the thick of it with my boss and the entire Helldesk looking for some scrap of what to do to fix this.

    Eventually we get an email chain going and hear from Boss'Boss'Boss'Boss.

    BBBB: I'm pretty sure this falls under Kate's jurisdiction.

    For a moment everyone is happy and the mystery is solved.

    Until I hit reply all and point out that Kate put in a ticket saying "Not my Department."

    And the mystery goes on... With no one knowing how all these users get added to the Sales Tracking Device.

    Only had like 15 new hires this month.

    Documentation is key.

    Document your stuff.

    I guess I have my projects set for the next year or two.

    submitted by /u/absinthangler
    [link] [comments]

    My baby face

    Posted: 12 Jan 2021 10:48 AM PST

    Ok so this JUST happened. I had been corresponding to a student parent who was given a hotspot since they do not have wifi at home. Hotspot wasn't charging past 30% and had to be kept on the charger all day.

    When I first saw her she looked like a meth head with dreads tied up on her head, sunken cheeks, face tats, and to top it all off was causing the who lobby to smell of weed.

    The way we give out the hotspots is in a case with the serial number and a charger. It's a nice ass casse too like clamshell nintendo DS case nice.

    She brings in the device and hands it to me.

    I ask were the rest of the kit is and she responds why would I need to bring the rest of it it's just an issue with the charging.

    I take it back to my team lead and ask him what to do. Was told that when she brings the kit back she can get a new device.

    I then proceeded to drag myself back down to her preparing to get yelled at.

    When I finally reached the witch and gave her the news she exploded on me.

    Exclaiming that how dare I make her come down here for support in a PANDEMIC! Now she has to go and sanitize so that her kids don't get sick. (she only talked to me and the security guard and we were both 10 feet + from her). She storms out and I turn around and shrug it off.

    Sit back down at my desk and see that her ticket has an update. OH BOY did I not expect the laughs that it would cause.

    Little bit of background I just turned 20 6 foot 3 145lbs I get mistaken for a high school kid a lot but do have a full beard atm.

    This was her response word for word " I really don't appreciate you having me come up here during a pandemic and risk the exposure not to get anything handled and for you to send a child to handle your business."

    I have now been dubbed by my fellow techs as baby man.

    Told her she is free to come get a swap if she can bring back the full kit :)

    submitted by /u/blaze8n
    [link] [comments]

    Private hospital mess

    Posted: 12 Jan 2021 07:49 AM PST

    I was tasked with setting up a 2nd seperate network in a private hospital. I had to replicate a hospital network that consisted of an air conditioned server room and full wifi coverage, but on a tiny tiny budget and in a broom closet.
    This was all for 1 single doctor who wanted to subcontract to the hospital and not pay for any of the hospitals excellent infrastructure or support staff.
    The network had to go from the doctors office, down the hall to where there was a fax machine and they needed wifi as far as the operating theatre.
    There was all kinds of drama over a week or so, many basic things they didn't understand about computers. But bigger issues too, including the company selling them software said a NAS was all that was required when they actually needed a server. While changing the doctors Windows account on the hospital network to a local one her Yahoo mail wouldn't log in.. was an outdated browser issue, but seriously Yahoo mail for important doctor/surgeon emails?! WTH
    Everything I fixed effortlessly and got them up and running within minutes, the unexpected server was up and running the very next day.
    The hospital was constantly ringing work, complaining and begging them to sack me. My boss was onto me, he even sent a coworker with me to verify that I was telling the truth.
    One day I get a call, the fax machine wasn't working and the entire network was down.... As I arrive I notice my friend from another tech company, he was there to replace me. We shake hands, he apologises and said "if I had known it was you I wouldn't have accepted the job". I was fine, I had never felt so relieved to be free of a problem.
    I walked down the hall, UNPLUGGED THE FAX MACHINE FROM THE BLEEPING NETWORK PORT and left my friend to it, it was his mess now.
    After being forced by my boss to write a multiple page explanation of every single thing that happened, I didn't get fired and we never spoke of it again.

    submitted by /u/FixTechStuff
    [link] [comments]

    No comments:

    Post a Comment

    Fashion

    Beauty

    Travel