• Breaking News

    [Android][timeline][#f39c12]

    Saturday, September 28, 2019

    Tech support can be really helpful if you aren't rude to us Tech Support

    Tech support can be really helpful if you aren't rude to us Tech Support


    Tech support can be really helpful if you aren't rude to us

    Posted: 27 Sep 2019 05:28 PM PDT

    Today I got a call from a user that needed the latest version of iTunes on his computer. But it wasn't available from our software store. He asked me if, since he had an admin PC, he could uninstall the iTunes from us and downloaded it on his own.

    Here's the crux, we don't have support on admin PC's. But I figured a simple yes or no would be fine. I guessed the answer was yes, but to verify I told the user I would be right back and put him on hold. I walked over to our coaches and asked if my guess was correct. They said yes, but the lack of support to admin PC's goes so far that I can't tell him that.

    I return to the user and inform him that sadly our support doesn't cover his PC, and it apparently goes so far that I can't answer this question.

    The user sighs and asks if at least could help him uninstall the iTunes from our store, because he can't find the uninstall option in Software center (where all our software is found). I inform him that I am afraid I can't do that either.

    User: "So you can't help me get rid of a program you installed on my PC?"
    Me: "Sadly, no."

    The user was quiet for a few seconds and then this conversation followed:
    User: "... Say I was using another PC. How would I go about uninstalling a program?"
    Me: "... Well... If you are using another PC I would tell you to open Control panel and go to uninstall programs. But this is only for other PC's of course. PC's that aren't admin PC's."
    User: "Of course. Of course. I would never do this on an admin PC. I am using this other PC next to it. I promise."
    Me: "Good. We wouldn't want to break the rules for the support."
    User: "Heavens, no. I would never ask you to do something like that."

    I then ended the call and split the call in to two cases. One for the iTunes thing where he was denied support. And a separate one where he asked my about how to uninstall a program on a nondescript computer.

    I have been in similar situations before. Situations where I could have used some loophole to help a user. But the user had been so rude and unpleasant that I chose not to. So it was nice to talk to someone who understood that I didn't make the rules.

    submitted by /u/NerdyGuyRanting
    [link] [comments]

    Payroll Troubles - The Tale of Edith

    Posted: 27 Sep 2019 02:02 PM PDT

    Timeframe: Around March of 2019

    The Client: A car dealership that sells new cars from their affiliated manufacturer

    CB = Me, Codebooker, Private Tech Guy Who Does IT Work

    OL = Edith, a 75-year-old lady who works in the accounting office within one of my clients

    ---------------------------------------

    Phone: RINNNNNG

    CB: Hey, thanks for calling -business name here- this is Codebooker, how can I help you?

    OL: Oh, Hi Codebooker, it's Edith, from -client name- in the accounting office, I'm having trouble doing payroll

    CB: Okay, Edith, can you please describe the problem you are having

    OL: I am trying to pull up the employees but I can't find anything when I look for their names

    CB: Okay, do you mind if I connect to your computer so I can see what the issue is?

    OL: No, go ahead. I just want to get payroll done, this is driving me crazy.

    (I remoted into her workstation)

    CB: Okay I can see your screen now, can you please show me what you were trying to do

    OL: Sure

    (OL proceeds to go to Google and search for the employee's name, then scrolls down the page and gets confused when she can't find him)

    CB: Hold on for a second, Edith

    (I minimized her Chrome window and maximized the accounting software that was running in the background)

    CB: Okay, try it again

    (OL then begins doing payroll and I guess it was muscle memory for her because she was off to the races. I never tried explaining the issue to her, because I've met this woman in person and it would be like explaining nuclear physics to a newborn child)

    OL: Thank you so much Codebooker

    CB: Is there anything else I can help you with?

    OL: No, that's it

    CB: Alright, Edith, have a nice weekend.

    Phone: Hangs Up

    ----------------------------------

    tl;dr - Old Lady (user) tried to do payroll from Google Search instead of her company's payroll software

    submitted by /u/codebooker
    [link] [comments]

    We Can't Do That, I Don't Know Why

    Posted: 27 Sep 2019 06:13 PM PDT

    We'll call $TeamMember1 the target of my frustration.

    I recently joined a company on the team of process/product owners. This is a fairly new team so there are no documents or guides for how we should all work together outside of standard project documentation. Recognizing this, our team leader advised us to work together on getting to know how each-other works and finding ways to standardize things like request forms, etc.

    Now, I am the new guy being that I came from outside of the company. I am also the most technical since I came from a system administrator role. Plus, I love tech and feel it is grossly under-considered in process/business structuring. Thus, I have been informally recognized as "the tech guy" on the team.

    To kick off our project we scheduled an internal demo of PowerApps by one of the AVPs who works closely with our leader. I had a week or two of time before the meeting. So, decided to dig in myself and begin learning how it works with flow and Sharepoint to prepare for any dev questions I was struggling with.

    The meeting starts and $TeamMember1 has not shown up because of an emergency she's dealing with. No biggie. I set the scope of the meeting and he begins walking through some templates and generally having an open forum discussion with us. I bring up that our Sharepoint is an on-prem copy and begin asking how he's managed to develop in the on-prem site without offloading to our App-Dev team who is offshore and sucks balls.

    About 15 minutes into the meeting $TeamMember1 comes barreling into the room apologizing profusely and blatantly frazzled. We assure her it's fine and I continue my discourse with the AVP. He mentions that there is a gateway already set up and we agree a follow up for him to walk me through how he set it up for his team would be ideal. Immediately after, $TeamMember1 jumps in and says something along these lines:

    "Hey, sorry guys, I know I was late and don't really know what we are talking about. This tech stuff sounds like you're talking Chinese to me. But, $Voxmanns, I just want to remind you that the point of this meeting was for us to meet as a team and learn about PowerApps and how we can use it for this project. $AVP we are..." and goes for an additional 10 minute explanation of the ENTIRE scope of the project after calling me out in the meeting as if I was off topic and had not already done this.

    That's fine. I can give a pass on this and just say I would have done it differently for sure. I bite my tongue through the explanation and let her have some discourse with the AVP without interrupting to let her catch up. This is when I learned she's one of those people who just doesn't have an understanding of how flexible modern technology is - asking very simple functionality questions like "Can we log notes" and things along those lines.

    However, we got to a point where we were discussing documentation and adding attachments. From my short time with PowerApps and Sharepoint I had read that attachments aren't always great since Sharepoint items aren't structured to function as a document library. So, seeking a solution I asked the AVP "Hey, do you think it's better for us to keep separate documents and update PowerApps separately (double work but separation) or do we go directly through PowerApps and use screens as separate documents?"

    Before he can answer, $TeamMember1 butts in and says "No, they need to be separate." I can do one, I can do plenty from seniors and superiors. I have little patience for my teammate interrupting me twice in the same meeting because they don't get the picture. The following exchange occurred between her and I:

    $Voxmanns: "Why do they need to be separate?" (with a firm but polite tone)

    $TeamMate1: "We need some documents signed off by SVPs, present them as PDFs, etc. We need to have the document separate from the app."

    $Voxmanns: "So, PowerApps can't do that?"

    $TeamMate1: "Well, it's an application to be used for forms and organizing. And I know what you do is a little bit different than what me and (other teammates) do but at least for our process we need to have the documents separate." (She likes playing the 'you're different' card on me)

    $Voxmanns: "Okay, so we need document versions of the form. Are you saying PowerApps can't do that?"

    $TeamMate1: "Well, I haven't used it very much so I-"

    $Voxmanns: "Okay," turning to the AVP "Can PowerApps export to PDF or other documents?"

    $AVP: "Yeah, PowerApps can export" and goes on to say the importance of mulling ideas over and over to find the best way forward (I am guessing he saw my frustration).

    Admittedly, I need to work with a thicker skin, or at least hide it better. Asking the "Do you know that" question seemed to work and will hopefully work in the future with her. It is just incredibly difficult to work with a teammate who has to assert their control and does not consider anything they don't understand as a reasonable solution to explore. Hey, totally open to any stories or tactics you all use when dealing with these types of people - it's definitely one of my weak points. Felt nice to shut down the document rebuttal, though! I'll call it a draw for now haha.

    submitted by /u/Voxmanns
    [link] [comments]

    I found a bug

    Posted: 27 Sep 2019 06:32 AM PDT

    Obligatory first post, mobile user, yadda yadda. This was an interaction I've just had with a fellow developer at work.

    I get back from seeing a user and one of the new devs calls me over, saying he's just found a bug, and he sounds really excited. He's only been here a few weeks but is integrating really well, so I'm a little surprised that he's so zealous about this.

    I wander over to his station to see what it is, and he starts pointing at his screen. I can't see anything that looks out of place in our system until a literal bug walks across his screen.

    A very funny little joke, but nothing I haven't seen before so I prepare to get on with my day. Then I realise the bug is in his monitor. He takes a screenshot to prove it isn't some kind of overlay, and wipes his finger over it.

    That's a new one on me.

    TL:DR; Newbie dev calls me over to look at a bug, turns out he has a friend living in his monitor.

    submitted by /u/SuicidalTurnip
    [link] [comments]

    Tech Support to Tech Support

    Posted: 27 Sep 2019 07:08 PM PDT

    SETTING:
    Mid 1990s at a large, multi-national cutting-edge high-tech company

    BACKSTORY:
    We had introduced networked PCs for all engineers and most technicians in our division. We were provisioning them and supporting them ourselves because corporate IT didn't exist yet. Because of the primitive nature of networking for MS Windows, we were using DEC Pathworks and our DEC VAX 11/780 mainframe as a server for file shares. And DEC's version of email.

    This DEC VAX was the computer that supervised the entire, very sensitive, very high-tech research and development factory. This factory operates 7x24. Now, we operated this computer to be ultra-reliable because factory downtime cost 1.5 million USD per hour. We had only one scheduled annual warm-down for 12 hours between Christmas and New Years when the factory was idling.

    Then corporate started putting together our first IT department made up of PC enthusiasts with that wild-west frontier spirit of that era in computing. Along with the development of corporate IT, they rolled out cc:Mail as the corporate email solution.

    Now, IT didn't have any presence at our campus as we had been self-sufficient up to that point. So there were no IT servers on our campus. But they needed some file shares for the cc:Mail databases that made the application operate. So, we created several file-shares on our VAX to support their cc:Mail services. (Each database/fileshare could support only a few hundred users). That worked pretty well except that cc:Mail was rather flaky and had a rather high failure rate.

    Now, IT's solution to everything was to simply reboot. Which is fine for an end-user client machine. But you can't go around willy-nilly rebooting servers where users have open documents, etc. But the IT of that era hadn't yet developed that kind of customer-centric mindset. So that is the back-story that led up to...

    THE PLAYERS:
    Me and Corporate Help Desk (CHD)

    THE INCIDENT:
    One hot August afternoon I was building new client systems and answering our Help Desk. The fateful call comes in.

    Me: "Hello, {department} PC Help Desk, how can we help you?"

    CHD: "Hello, this is the corporate help desk. We are having problems with one of the cc:Mail databases on your campus. To clear the problem, can you re-boot the server where that database is provisioned?"

    Now, I had never had the CHD call us. Usually, it was the other way around, I had to call THEM for something.

    Me: "Well, that database is served from our factory mainframe and we can't just reboot it whenever we please."

    CHD: "But what will it take to reboot that server?"

    Me: "The next scheduled re-boot of that server is Christmas day. Would you like to hold?"

    Yes, I admit that was rather snarky of me and I wanted to see what was their reaction. We had VERY different concepts of operating network services.

    CHD: Long silent pause. I thought maybe the call had dropped. But finally they replied: "Well, I guess we will have to find some other way to clear the problem. Thank you."

    I like to think the I gave them a little encouragement to develop more user-friendly ways of handling problems like this.

    AFTERMATH
    A few years later, management decided to revert to IT-supported office computing services rather than rolling our own. But on OUR terms. We had a critical list of 20-30 Best Known Methods (BKMs) and SOP that we had developed to keep computing and networking in their place, as aids to doing business, rather than distractions and hindrance to getting the work done.

    We met with the corporate IT people and presented our list. Their reaction to the items on the list ranged from mystified to horrified. They proceeded to cross off almost every point and argued that they couldn't possibly do business like that. But we insisted, and it took almost 2 years to transition from our office computing environment to the corporate IT provided solution.

    A few months after the transition period, our division had moved into a new campus, we were operating with IT supported office networking and computing. And then corporate IT proudly announced their new office computing initiative. Oddly enough, it contained almost exactly the list that we had originally presented to them.

    There are some sub-plots in this scenario that I will post as their own stand-alone stories that should amuse many readers here.

    submitted by /u/RC531976
    [link] [comments]

    Every once in a while you get reminded how it's possible for some people to screw computers up so badly.

    Posted: 27 Sep 2019 05:24 PM PDT

    • LTL, FTP, I think I finally have a story worth posting here.
    • CAST:
    • GF: Grandfather, has a computer that he mostly uses to read email and check facebook, but I've been attempting to teach him some more things to reduce my tech calls to his computer. Mixed results so far, but he is picking up a few things.
    • GM: Grandmother, has never used anything more complicated than a graphing calculator, often says "computer" like it's a dirty word.

    On to the story:

    • ME: (trying to explain to GF why I need an external service to send emails from my homelab while he can send emails from his webmail just fine) "Residential IP addresses are usually blocked by other email servers because of spammers, so I need to use somebody else's address to send emails. Your webmail isn't actually in your computer, it's on another server outside of your network"
    • GF: "So my emails aren't blocked?"
    • ME: "No"
    • GM: "Why can't you rent a PO box from the post office instead of using somebody else's address? How do people know it's from you?"
    • (GF attempts to explain the difference between IP address and mailing address to GM while I leave the room to avoid laughing in GM's face.)
    submitted by /u/NathaninThailand
    [link] [comments]

    Desktop support story from the 90s

    Posted: 28 Sep 2019 03:39 AM PDT

    So like the title says, this story was taking place during the mid 90s. I was working as a desktop support tech for this company. There was no remote access so in most cases, I would just walk to the customer's office and solve the issue there.

    This particular ticket was from somebody in payroll who was complaining that their desktop couldn't print. When I arrived at the person's office, He explained that he's trying to send this document to the department printer which was down the hall but it won't come out. He then explained that this is a sensitive document so he is going to leave me in the office and wait by the printer so he could grab the print out before someone would see it. I sat down at his chair and looked at the document. I couldn't believe what I was seeing. It was an excel file that contained a table with columns that looked somewhat like this:   First Name   Last Name   Salary  

    followed by hundreds of records of ALL COMPANY EMPLOYEES.
    I took a few seconds to look through the data. I was particularly interested in the people who were working in my department, but as I was scrolling through, I saw a lot of information. I was tempted but decided not to save the document to a network share. I fixed the printer issue, checked that the guy got it and left.

    From that day forward, as I was roaming around the building, when I saw people walking in front of me, I was also seeing this imaginary bubble with their salary figure right above them...

    submitted by /u/silverbluenote
    [link] [comments]

    So... can we just keep clicking ok?

    Posted: 27 Sep 2019 06:41 AM PDT

    A few weeks ago we received a ticket about a department having trouble connecting a medical device to the software on their desktop. I can't troubleshoot the medical device (biomed department does that), but I can troubleshoot the software and hardware on the computer, so I head down to take a look.

    When I walked in there were 2 users staring at the screen, just waiting. I introduce myself and they thank me for coming and start explaining how they need this to work and it's not working. They're stressed out and need this working now for a patient that will be coming soon.

    I sit down and have them show me the software and the error they're getting. They show me and explain what it's supposed to do that it's not doing, then they continue to stand right behind me hovering and watching over my shoulder.

    I try to ignore their hovering and see that it is a COM error. The software can't see the device at the assigned port. They have a serial cable that connects the device to a serial to USB adapter that plugs into the computer.

    So I check the settings in the software, find the port number it needs to be, and launch device management. I don't see the device under the COM port at all, so I begin checking the connections. While I'm unplugging and reconnecting each connection they're telling me how they've tried that already and it didn't help. I just nod and say I'm just going through my checklist. Everything is secure and it's not showing up, so I try moving the USB to another port as they tell me they've already tried too.

    It's still not showing up, so I try removing the serial cable and plugging the medical device directly into the serial to USB adapter. Bingo. The COM port is now showing up, but is the wrong number. No problem. I change the port number, launch the software, and it works.

    They cheer (softly, but I'm not exaggerating. This department is easily excitable) and begin praising me and ask what was wrong. I explain that it looks like the serial cable is bad and recommend they get a new one. We don't keep those in our stock room, but I tell them we can order one on their behalf if they need us to. They thank me, say they can take care of it and send me on my way.

    I close out the ticket and move on thinking this is the end of it. But, of course it's not.

    Yesterday morning I get a page for a high priority ticket from the department. I'm reading through the page and see my name specifically mentioned and one of the users from before. I open the ticket on my phone and make a note that I'm on my way. It was only 10 minutes before I normally leave, and I was already finished getting ready so I just left early.

    I get there and head straight to the department fully expecting to show up and see them trying the same thing as before. I walk in and no one is around to meet me for this "high priority" issue. I look at the setup and see that they are trying to use the serial cable again, but this time they're using it with a different medical device. As I'm debating calling the user she walks in and sees me.

    She's explaining what it's doing and what should be happening as she logs in to the computer to show me. She hops up to let me sit down and I snag a screenshot of the error to attach to the ticket. It was a long error so I'm reading through it as she hovers and is trying to tell me what she wanted to do but she didn't want to break it so she just waited for me to get there.

    The error was saying it would take over 8 hours to transfer the logs, because there were over 1800 communication errors from the data transfer. It identifies it as an issue with the serial cable as well. I explain, again, that the serial cable is bad and that's what caused the error. I also explain that these transfer errors most likely mean that pieces of the information on the device are being lost during the transfer.

    "Well, can we just click 'ok' at the bottom there and continue?" She asks, pointing to the button at the bottom of the error.

    "Sure, we can try it and see what happens." I tell her as I click "ok" and see it go on to the next step instead of closing the program.

    Naturally, she gets very excited and wants to keep going. I slowly follow her instructions, taking time to read each step for any errors or warnings. We get to the data transfer screen where she can select which of the recorded sessions she wants to transfer and print. We try it, and she says the graph looks right and prints it out.

    "Great! Well it looks like it's working again. So.. can we just continue to click 'ok' and ignore that error?"

    I stare at her for a moment processing the fact that she either forgot what I just told her about the cable or thinks it's not a big deal. "No, I wouldn't recommend doing that. This cable is going bad and there is a chance that parts of those tests you recorded aren't getting transferred. You could be missing seconds from several parts of these tests and I don't know what effect that has on these results. I highly recommend getting in touch with biomed today to see if he can help you get a new cable for this."

    "Oh, okay then." She sounds like she still doesn't understand why this is an issue, but asks me about the biomed guy to confirm she knows who I am talking about so she can get in touch with him.

    Hopefully she actually gets it replaced this time.

    Edit: There are several people mentioning cutting the cable or taking it from the department once it was found bad. In hindsight I could have taken it to biomed for them and let him know the situation, but he makes rounds through the hospital and is easy for the department to get ahold of so I didn't think to do that at the time. My department does replace other common cables/peripherals (ethernet, USB, mice, keyboards, etc) that we find to be broken or causing issues so that we do not get repeat tickets for it. Serial cables are one of the few things that we do not carry since the only thing they are used for in our environment is biomed equipment, so my go-to was to let biomed handle it since they declined letting me order a new one for them.

    submitted by /u/clover-kitsune
    [link] [comments]

    So close, yet so far

    Posted: 27 Sep 2019 06:18 AM PDT

    A user called in stating that "email was totally gone", so I connected in to see that the Office upgrade we're pushing just went sideways on their machine. I pull up Software Center and see it sitting in the Installation Status queue as failed, so I pull it up and retry it. The usual progress wrapper we put on these kinda things pops up and I see it start churning, so now it's time to sit and wait.

    After a few minutes of silence between us on the phone where I get some much needed sips of coffee in, I decide I want to get back to listening to music.

    me: Hey, so this is probably going to take a little while, so I'll leave my remote window up and keep an eye on it in case it tanks again. The first bit takes some time while it runs some cleanup bits to make sure all the old stuff is gone, but once that finishes up the rest is usually pretty quick.

    her: Oh okay, sounds good. Do you know how long it'll take?

    me: Eh, usually about 30 or 40 minutes. You can still work on stuff while this runs in the background, just not any Office stuff. But yeah, I'll keep an eye on it and if I see it fail again we can just install it manually.

    her: Sounds good!

    So we hang up, I punch up my ticket, and I go back to doing whatever I was doing. I look over and see there's an error message up on her main screen regarding Excel not starting properly. Weird. I hit Okay on it and wait for a few seconds. I watch her mouse over to a different Excel document and try to open up that one. Lo and behold, another application failed to start error comes up. Okay, that's on me. I didn't specify that Excel and Word are part of the Office suite. Fair enough, so I open up my preferred method of remote, non-verbal communication in times like this, Notepad.

    me: Hey, sorry, Excel and Word are both going to be broken until the installer finishes up.

    her: is it okay if i go on the internet?

    me: Yeah, that's fine. Just keep in mind you'll have to restart at some point in the near future.

    me: ok thanks!

    I watch her pull up our homepage, navigate over to OneDrive, and get in to her files. She digs through her stuff for a bit and then eventually finds the same sheet she was going to open previously. I watch her click on it, mouse over to the Open button, and then immediately shatter my hope by clicking Open with Excel rather than Open with Excel Online. She gets the same application failure she was getting previously and her mouse settles and doesn't move throughout the rest of the the install.

    A for Effort, I guess.

    submitted by /u/uvclay_
    [link] [comments]

    The (Friday) night I bricked a server (The Beehive - Part 1)

    Posted: 27 Sep 2019 05:28 PM PDT

    Per popular choice on the previous tale, today's tale will be the beginning of The Beehive arc, which should be a three-parter.

    This tale takes place in the end of my career at $agency. I was now one of the oldest there, with same $boss and $pm from before but now supported with $kohai1 (and $kohai2 later) for dev work, and many other marketing/design colleagues.


    The first time I met $queen was on a minor project during training, I found odd that the Head of Digital Marketing was there, but she made relevant SEO remarks on our delivered product. But I realized this was a cover to meet with $boss to discuss a big upcoming project, despite us being miles away from home and the clock being 7pm.

    Said project is another story I wasn't involved much at first since we were not chosen for the dev part. But when it got FUBARed we took over but that doesn't deserve a tale of its own.

    I compare organizations with shareholders as beehives. Like every company, they need to output honey and keep the hive healthy and workers busy. But inevitably, the beekeepers take their share and split hives as they please, they might have feelings for their bug slaves but the honey must flow. So they buy queen bees and just drop them in the hive to be accepted by the workers.

    Since the previous six figure failure, $queen was moved to a new company but weirdly enough in the same market and recently acquired by a trust fund, still as Head of Digital Marketing. Maybe somebody had a friend high up ?

    The other moving part of the Beehive tale is $hostingProvider which is also a beehive of sort. Despite being a local company, they had been acquired by a much larger US based group. $hostingProvider was mostly telco but with a solid sales team of all-flavours-of-EU speaking bees, so they also provided hosting services.

    We already worked with them for the same minor project I mentioned previously, so the pattern was quite clear when $queen asked us to switch from their current hosting provider to $hostingProvider.

    We were knee-deep in our audit of their current ecommerce infrastructure and multiple brands they managed, which in itself was "funny" since the first meeting we had was about the revamp of their corporate website.
    $boss and $queen at it again.
    A deal was struck to employ us for 2 days a week starting now, mostly to fix all their issues with $bigEcommerceCms.
    So it was a another fun surprise when we were asked to handle the switch to $hostingProvider, but come on, "you already know them and how they work, and they know how you work !".

    It was true we had another unrelated client with a $bigEcommerceCms instance hosted by $hostingProvider, so $boss and $queen must have rejoiced. But this is the tale of "The 10 pages requirement you regret you billed us 1 hour", we had total control of the sales and dev environment, not here.


    Now the crux of today's tale is that $previousProvider was also the agency responsible for everything technical up to now, so they were pretty pissed at $queen. This was shown by giving them and thus us a hard deadline to move everything before they shut their machine.

    And we were talking about at least 4 instances of $bigEcommerceCms in various conditions and versions, multiple static website from years old HTML pages to Flash based one, all around a dozen of domains. The only solace in this is that they already had moved their email to Office 365, so we didn't have to think about that.

    So the marketing bees were harassing us with tickets about their needs of custom stats export, years old bug they thought could be finally be fixed and various questions and sales operations, and at some point I told $pm we had to focus on the switch and nothing else.

    Given the deadline, this was already eating up their 2 allocated days. Despite the years, management was not much more efficient so the deadline was rapidly visible even behind the mountain that was our backlog. $Kohai1 was still in training for $bigEcommerceCms so I assigned him the small stuff and handled everything else including most of the project management work.

    Up to now $queen had been mostly bugging us with technical questions about $bigEcommerceCms performance and modules since she wasn't up to speed with their brands and catalog. And having a good chat with her $hostingProvider sales team friend. Despite our now paused infra audit, she took the decision to order a load-balanced set of machines : 2 prod machines behind a load balancer, 1 for global db and assets storage and 1 for preprod needs.

    Now mind you this was not our other client setup, the first time we had a load-balanced infra, and the only $hostingProvider technician familiar with both $bigEcommerceCms and the foul language we speak was US-based and half a day away.

    Oh and they used the worst ticketing system I've ever seen and we were second-grade users since not directly client of the mother-ship company.

    Now 5000 signs in, you have the full context for this tale and future parts. I wont leave you without what the title promised :

    The (Friday) night I bricked a server


    Since non switch-related issues were still actively pushed all week, I arrived Friday with merely a plan, an energy drink, and the assurance everybody had been briefed on how the ball was supposed to roll.

    The morning was spent smoking the bees and reminding them to end their business by noon before downtime. Afternoon was spent copying $bigEcommerceCms instances and other websites from $pissedPreviousHostingProvider.
    Of course $queen wasn't happy we couldn't use the downtime to slip a patch or two, but the custom CRM integrations was full of hard coded paths merely not yet detected by our audit and she forgot some website were hosted elsewhere than on $pissedPreviousHostingProvider. Luckily they had the credentials, so on the list it goes.

    So it was already 4pm when everything was OK on the preprod machine despite the config being far from what our other combo of $bigEcommerceCms@$hostingProvider

    I asked $queen on the phone to verify all preprod versions :

    $me : Is everything OK to you ? Instances are perfect copies before downtime, so all orders of this morning are accounted for.
    $queen : yeah but $shittyStaticSite is an old version, and our shareholders are reviewing all of our domains so pls fix it ASAP

    I had 4 websites full of honey not live at 4.30pm on a friday when I called $pissedPreviousHostingProvider who gave us the previous SFTP access :

    $me : Hi $pissedTechnician, $queen is telling me $domain is invalid, could you look at your config ?
    $weekendAwaitingTechnician : (After a few minutes) Yeah your access was linked to a previous vhost folder, our bad.
    $me : Thanks I got it, I'm sorry to ask but when are you leaving ?
    $cleaningDeskTechnician : in about 30 min, why ?
    $me: Just in case anything happens...
    $pissedTechnician : You know our deadline is set in stone right ? And is today. We won't host anything on Monday even if you ask nicely or with honey. (Thank $queen for that you fool)
    $faunaHatingFool : Yeah I know, just 30min and after that I'm on my own.

    Luckily during the week I had time to tweak my previous bash script responsible for copying instances of $bigEcommerceCms and automatically adjust some server/db/cms settings to handle Prod VS Preprod.

    It was a beauty of cobbled together variables representing the multiple machines (Prod being 2 IPs + 1 shared storage and Preprod 1 IP + same storage) and databases, and I was very proud that it could be launched from any machine to another and rsync/ssh would gently do its job.

    Each website was a parameter of the script, along with a keyword for source and one for destination. I didn't test machine to machine copy, but I was confident my gigantic switch to generate machine related variables would work.

    Now that everything was OK on Preprod at 5pm, I used the script to copy $shittyButNowCorrectStaticWebsite to please $queen, and it worked flawlessly, from Preprod to both Prod machines. All according to keikaku all I had to do was copy everything to Prod machines and ask for DNS modification.

    As I was preparing to copy $bigEcommerceCms instances, $boss and $pm comes by and asks me if everything is alright. I rant about $queen and $currentlyAtThePubTechnican warning and tells them this should be the yearly instance of Friday deployment they'll pay overtime. They laugh and leave, everything is OK and I hit Enter on my third monitor since $kohai2 had not yet stole it.

    Now all commands were encapsulated in SSH commands for them to work on any machine, and verbose so that I could have an ETA on operation. And I didn't like the currently rolling folders of my "delete before copy" operation so I canceled it quickly. I realized they didn't mention my vhosts at all, but much more close to root folders.

    I open my 100% correct script that previously worked for $staticWebsite to check the operation :

    rm -rfv /$Folder cp -Rv /$Folder 

    This is of course a dramatization of the script, but the keenest among will see that /$Folderis a very dangerous construct.

    Because sure enough, I launched my script without specifying the final parameter of destination since I was distracted. Therefore $Folder is empty, and the SSH connecting to localhost.

    I've just rm -rf /ed my preprod server I spent the day crafting.

    Thanks $boss, thanks $queen, thanks $hostingProvider and their custom folder config which required me to change how folders were concatenated.


    I was alone, I could have shouted. But the cold sweat of my greener days never came. The $senpai of my previous tale was long gone, I had long since took its place.

    Bricking a machine was also not my first rodeo of this kind, so I didn't chastise much of $Kohai1 when he rm -rfed his preprod vhost and all associated backups (we grabbed from offsite), or later $kohai2 when he chmod 7XXed his cloud machine (we reinstalled in 5 min).

    $Kohai1 chose anger and spent the week ruminating his mistake. $Kohai2 was green and spent 10 min frozen in fear hidden behind his wall of monitors before calling for help.

    I therefore transcended my exhausted mortal body to grab the title of Sensei with my bare hands, attaining peace. I floated among lotuses and no bees were in sight.

    Not a single thought of rage or regret passed upon me, no doubt in my skills and what the future was holding.

    I must say this was a profound personal experience since I was on a spiritual path for some time to cope with my environment. I'm not ashamed of my blunder, and I'm proud of how I handled the consequences despite the madness around me. The mark of a truly desperate professional.


    On the mortal plane, I just sighed :

    $ root@preprod : ls -al $ root@preprod : ls command not found 

    Yep, I'm fucked. Machine is bricked.

    I call $queen :

    $me : Hey $queen, don't worry if $staticWebsite is no longer online, I've hit a pothole and need time to fix it up.
    $queen : Nothing major I hope ? I'm heading home so here's my personal number.
    $me : I'll keep you posted and will call you to handle DNS on your side.

    I texted $boss and $pm as well.

    All I had to do was re-copy everything manually from their source. Some stuff could be salvaged from Preprod via scp, and the autoconfig part of my workflow was a sub-script I could trigger manually. $pissedHosting didn't cut our access yet, maybe they felt guilty for that vhost.

    It was 10.30pm when I called back $queen :

    $me : Everything is copied on both machines, here's the load balancer IP again. You do your DNS magic.
    $queen : Thanks, but I can't leave without giving you my utmost gratitude. No company has gone this kinda of length for us before, I know I was making the right choice picking you. Don't worry, all the shareholders will know your effort, and I won't forget it.
    $me : Thanks, but right now I need to eat something. See you on Monday for postmortem.

    The keenest among you will sure have picked that $queen was in charge of DNS, not the IT department. And during the whole story, was much more tech-savy than your run-of-the-mill marketing bee. This will of course bode well for the future.


    If you want more, ask for future tale (1 choice per user, upvote other people's choice) :

    • Beehive Part 2 : The Great Friday Hack
    • Interesting clients : "are we an accomplice in money laundering ?" (~10 votes already)
    • More pressure : "no budget this year, but we're still the most visited website of the state for 3 weeks" (~5 votes already)
    • Less greenery and more bullshitting : "I'm 100% affirmative it can be done (but I haven't done it myself and you will do it here)"
    • $hostingProvider learn who they're dealing with : "The 10 pages requirement you regret you billed us 1 hour"
    • Good project management : "the marketplace of ideas"
    submitted by /u/PM_ME_FRENCH_CHEESE
    [link] [comments]

    Back in the day, part of my job was to liaison with government reps when receiving warranty claims for military electronics returns.

    Posted: 27 Sep 2019 05:38 PM PDT

    So here's how it worked: These incredibly complex "black boxes" for military aircraft would be returned from the field with tags affixed by the (fucking awesome) local repair crews explaining the deficiency. These guy knew their shit I tell you!

    The shipping containers were always opened with a company rep (me) and a gov't rep (we'll call him Tito) present.

    Tito was kind of a dick. He would always look to find a way to ensure the defect was covered under warranty. I get it, I'm a taxpayer too.

    The first one I recall, we open the shipping container and remove the shoebox size "black box" and there is a clear and present footprint on the top of the device. Someone had actually used it as a foot stool.
    Me: "Warranty exclusion."
    Tito: "Are you kidding me?"
    Me: "Tito, show me where in the documentation is foot stool mentioned?"
    Tito relented. ;)

    Another one. Remember these are highly complex electronic devices:
    Me: "The field report tag reads, 'water observed leaking from device', exclusion."
    Tito: "Are you kidding me?
    Me: ...<eye roll>

    It was always fun dealing with Tito. He would try and shut down our operation just to flex his muscle. We were not a shitbag outfit. Far from it. Almost everyone I worked with understood that what we were building was protecting our boys and girls in the air. And we fucking took it seriously. (We had times where one of our 'things' went to shit and everyone worked around the clock until it was right. We understood the job, we understood the risks to our Flyboys/girls and dammit, we fucking took it seriously!)

    One of the last times I saw Tito, he was in the middle of trying to shut down the entire production line for what amounted to a very minor deficiency. (The kind of thing that would normally be caught and corrected using our typical procedures, which were pretty fucking impressive!) His boss, uncharacteristically, in the middle of the entire workcenter, with everyone in that workcenter in earshot berated the shit out of him in his heavy Italian accent, "Tito, you don havea the fuckin authority, to shuta down this whole a fuckina line." Apparently, boss man had had enough.
    It was a good day. :)

    And also,
    FUCK YOU TITO. :)

    submitted by /u/accidental-poet
    [link] [comments]

    No comments:

    Post a Comment

    Fashion

    Beauty

    Travel