The tale of moving a server which CANNOT be powered down ... 5 miles to a new office Tech Support |
- The tale of moving a server which CANNOT be powered down ... 5 miles to a new office
- The server on wheels....the companies next big idea
- Screaming fire in a crowded theater
- I need my printer next to me in the new office
- "Why do I need to call them? Just fix it"!
- Working with whitelists
- Chrome won't install!!!
- I hate sales monkeys
- Percussive maintenance works...sometimes
| The tale of moving a server which CANNOT be powered down ... 5 miles to a new office Posted: 23 Jun 2020 03:06 AM PDT I just remembered this one, and at the time it felt like just another day in cowboy IT. Now in hindsight I see it for the mad and brilliant plan it was. The scenario: SMB who outgrew their office, and in-office datacentre, migrated to a new office building and most things out to a datacentre. There was however, still, a few things in the old office building. And a rapidly looming 'end of lease' date by which point the office HAD to be empty. So, we come to the problem. I don't recall the name of the server. This was around 2009 or so, and the server was a ML350. Which wasn't particularly new even at the time. This server had some crucial software that we needed for six more months. Now, the old office only had a 2Mbps leased line, and the server had a lot of data with the software. Of course, the office was closing in three months. So, a review was done. Do we attempt to migrate it the data? Well it's not needed for more than six months, at which point it can all go. It would've taken a lot of time and effort to build a replacement server and migrate the data. Do we power it down and move it? Nope can't do that either - server is so flaky it might not ever come back up. In comes the hero of our story. We'll call him Phil, because it seems to fit. Phil suggests we simply move the server ... without powering it down. "The server has two power supplies. If we take this spare rackmount UPS, and put it in the car, run a long power cord to the server, we can carry it down the stairs while it's running, and then get it in the car. We can move it while it's powered on, and get it into the new rack! Simple!" Sounded horrendous. So of course, we got the "let's do it!" mandate. This is the way this beautiful horrendous piece of IT went down.
We didn't lose any disks, but a few people lost some hair. [link] [comments]  |
| The server on wheels....the companies next big idea Posted: 23 Jun 2020 01:13 AM PDT So I work for a large multi national tech company. I am in no way involved with this new product. The company at the highest level recently released a new flagship product. Without ousting my company...think central server for an entire office that also prints, copies, scans, etc. More or less, they got rid of a paper tray and put a server in there instead. Revolutionary. Anyways they are testing this new product in a remote office that I sometimes work in. One day my group is slacking off at lunch and one of the testers of this new flagship product comes over to tell us all about it and how we can put our software on this server in the machine. Think all your companies transactions. This is the ensuing conversation between one of my engineers and this tester. E - engineer , T -tester. T: so now instead of buying an Azure cloud environment they can buy this machine and host the software (ERP) on prem and have everything centralized in one location. E: does the machine have wheels T: of course, how else are you supposed to move it around easily E: so god forbid they ever get robbed and someone wheels out the 10k dollar machine and they lose their database server as well T: (dumbfounded) excuses himself Not really tech support. But always gives me a chuckle. [link] [comments]  |
| Screaming fire in a crowded theater Posted: 22 Jun 2020 01:47 PM PDT This story happened literally moments ago. I've mentioned in the past that I'm a developer and part of my job is troubleshooting my software when someone complains or files a ticket. Two Saturdays ago, one of our contractors deleted a VM. This VM housed our only version of a real-time tool for our call queues, one for which no backup existed. This project took me multiple hundreds of hours to build out the first time using a whole host of technologies. Like a miracle worker, I fixed it by early Monday, spending 26 hours of my weekend to do it. The only difference here being that the URL changed (for various reasons that don't really matter). Flash forward to this afternoon. I get an email from the CTO labeled high importance "Real-time not working in Site 2". All it says is "CEO informed me that Real-time is not working in Site 2 and hasn't been for days". Like the good dev I am, I made the assumption that it wasn't working and no one had filed a ticket or said anything (which can and has happened before). I ask CTO if he had any additional details his response back was, and I quote, "No". I end up sounding the alarm and engaging the networking, server, and operations support folks to start probing and find the problem. Ping tests from different vlans, remote accessing multiple boxes, double checking AD and the box itself... and nothing... it works just fine in every test. After spending an hour engaging a bunch of different folks, I get a call from one of our ops support folks "Oh yeah, I was walking around with CEO since he's at Site 2 today. I told him the real-time tool wasn't set up on our monitors yet. The URL changed and we're still setting the links back up". I spent an hour panic hunting for a problem because CEO misunderstood something an Ops person said and complained to CTO about a problem that didn't actually exist. The worst part of this is that I now have to tell CTO that CEO was wrong. That's not a conversation that's going to be easy or fun. !@#$ing C-levels. [link] [comments]  |
| I need my printer next to me in the new office Posted: 23 Jun 2020 03:12 AM PDT So this happened about 19 years ago. We had an office with a number of laser printers (think big brand with two letters one up each from GO). We were moving the entire office of 150 staff from one place in London to a new office about 1 mile away. It started at noon on a Friday and the office was up and running by noon on Sunday with nary a problem (apart from a dead token Ring switch) So the printers were big, floor standing with 3 paper trays and their own "on the floor" paper storage cupboard on casters. About 4' tall. The new office was all shiny and sparkly, fresh paint, carpet tiles on the floors; EXCEPT in the reception area and the strip to the C-LEVEL offices where it was very expensive laid wood strips. So the morning of the move and there are maps everywhere showing where departments are in the new office; and copious signage telling you that if you have IT problems to call the new IT Support Desk number and a tech would be with you in a few minutes. Printers were automatically rolled out configured to each user on login so you'd get the printers all the printers. Each person had a guide sheet that just told them to find the nearest printer and then print to that one. Each printer had a big A3 sign on the wall with its name "Printer XX" and they featured on the site-map prominently. About 10am and i'm fixing a problem in Reception. Pretty minor issue; new PABX and the receptionist wasn't used to a particular call parking feature. Suddenly I hear a particularly nasty squeaky scratching noise. Like a heavily loaded item being pushed over wood. I look round the corner and there are two members of staff diligently pushing one of the printers across the office through the C-Level area, and digging nice deep runnels into the wood as the printer's caster breaks haven't been let free and the printer is too heavy. I run over and stop them before they do any more damage and ask them what they're doing. User: We were trying to print and our printouts were not coming out. So we went to find our old printer. So to save time we decided to move this printer back to near us. Me: Why didn't you follow the guide sheet about printing? User: There wasn't any guide sheet. me: Every desk has a "IT Services at the new office - read this first" packet along with the "Facilites Services one also". User: Well we didn't see those so we decided to do this. Me: why didn't you call the IT Support number? User2: We rang "old extension number" but no-one answered. Me: We're in a new office; all the phones are new and you know there are new numbers. Every desk has a note pinned to the divider that lists key contacts including the IT Service Desk. User1; look are you going to help us with this printer or not? Me: Please just leave the printer here and go back to your desk. Call "new service desk extension" and someone will come and set you up on the printers. User1/2: Ok well you could have made this a lot easier -- they walk off. At this moment the CEO decides to walk through reception and sees me standing next to a printer and and big runnels in the wooden floor. I managed to explain things and we had in the end to get the moving company to come in with a skate and floor protectors and move the printer; plus get the flooring company back in to sort out the damage. Not what you call a good day. TLDR: Users move big printer across expensive wooden flooring because they can't be bothered to read documentation in new office. Cause $$$$ damage and leave IT Manager to explain things to CEO. [link] [comments]  |
| "Why do I need to call them? Just fix it"! Posted: 22 Jun 2020 01:18 PM PDT Greetings, and welcome back to another tale of tech
Here's the cast:
$Me - The protagonist of this story, runs on coffee and lo-brau brand beer. He also has a cape that flutters in the breeze of a "hero-wind" branded fan. $Users - The sort of people who the internet usually associates with a name that starts with a "K" who usually want to speak with your manager. $SDL - Service Delivery Lead. Nice guy, advocate for IT staff to the suits and c-suite manglers.
As I've mentioned before in previous posts, my team is the L2 team for the $org L1 helpdesk. We are a much smaller team as we only take escalations for issues that cannot be resolved by the L1 team on chat/phone. We have an internal DL that is designed for the IT staff to email when something is happening that requires the attention of my team (outages, planned maintenance, etc)
Scene: It's a typical Friday morning. Things are pretty slow and I'm reading TFTS while watching my queue to see if anything new has been assigned to me. Suddenly my Outlook notifications are going crazy. "Must be a major outage" I think to myself. Some prior knowledge, the L1 team is not to share our internal DL address under any circumstances. If a user calls in requesting a ticket update, the agent should note the caller's comments in the existing ticket and the ticket owner gets a notification. A few days ago, an agent had such a call, but instead of following process, they give the caller the email address to our internal DL. Annoying, but whatever it's fine. The agent was given a talking to and we got the issue resolved.
Back to the story. I check Outlook and I see a ton of emails like these:
All annoying, and all emails about things that should be handled by the helpdesk. Lots of emails back from $TM (our technical manager) asking for ticket numbers. Surprise, none of these people had existing tickets. As it turns out, the one person who got the internal DL, shared it out as a way to "get help right away without needing to call in". This lead to a parade of Users who suddenly decided their issues couldn't be fixed by L1 and that they needed an instant escalation. To top it off, while some of them understood they needed to follow process, others got indignant and made comments that they didn't have time for a ticket and needed someone to call them right away.
Long story short, everyone who needed a ticket finally got a ticket and an email was sent out as a general notification from IT from $SDL explaining that we have a helpdesk for a reason and reiterated the policy for reporting an IT issue. This all went down a couple months ago and we still get occasional emails from random users who want help for a ticket that they don't have. A little post-script to the story is that a week later, one of the other guys on my team gave his personal cell phone number to a user which lead to him getting tons of calls over the weekend after that user shared his number with her department. Nobody had any sympathy for him... Hope you enjoyed! I'll try to upload some more stories later on from my time at $GovOrg. As always if you have any questions, comments, or death threats, feel free to post them in the comments. Later! EDIT: Re-uploaded to remove proper names and commentary on the reason for the re-upload. [link] [comments]  |
| Posted: 22 Jun 2020 04:17 PM PDT A colleaugue of mine was involved in this story way back when. The scene: It were the days of the wild west internet. Outlaw javascript popups roamed the internet without any popup blockers to play sherrif. A certain car $company produced slightly more luxurious cars than a golf caddy could afford. (The actual name is important later) $company asked theIT business $HavePatience to develop a whitelist solution, so $company employees couldn't surf to none-approved websites. (Be it by typo or malicious intent) It would also autocorrect and point the user to the nearest match of the whitelist. Due $company wanting a solution fast, someone at $HavePatience made a typo. They typed the wrong ammounts of L's in $companyName. The name with that typo was a womans name with a profession that would break rule2 if I'd mention it here. During the demo with $companies bigshots, they ended up on a site with a ton of popups. My colleaugue managed to fix the issue pretty fast. (When telling the story, he mentioned some jokes arround that 10 minute break, but again rule 2) Funniest part: $company decided to buy the solution, so their employees wouldn't also make the typo. [link] [comments]  |
| Posted: 22 Jun 2020 07:01 AM PDT I work Tech Support at a mine. Users are not known for their computer literacy. The company is switching over to a new voicemail system that users have to register and create a PIN on the webpage. Catch is, you can't use IE. Any other browser will work and the instructions were very clear and easy to follow. Queue user stomping into my office saying that they can't get Chrome to install on their PC so they can't complete the registration, and that I have to come over right now and fix it! So I walk over and there is Chrome browser window open, with 10 or so downloads of chrome at the bottom, and an error box saying chrome is already installed on the PC.... Yeah, User was opening chrome, to download and install chrome, then complaining when the installation failed. TL;DR - User is using the Chrome browser to download and install the Chrome browser and getting an error that it was already installed, then told me it was broken and she couldn't install Chrome. [link] [comments]  |
| Posted: 22 Jun 2020 10:24 AM PDT I guess I am not the only tech support guy that has a special relationship with his lazy ass sales monkey colleagues. Just one of the examples; Me = me SM = sales monkey Ping - direct email of SM. SM; Do you know if our system backup db works across OEMs? So if we do a DB export from OEM-A and then upload to OEM-B would the settings work as I assume the db structure would be the same. Me; I do know. And our system backup does not work across various OEMs SM; All good. A little over a week passed. Ping - Support ticket of SM. SM; Is there any way to transfer a configuration of a OEM-A server to an OEM-B server? We have a customer with 800 iot devices moving from OEM-A to OEM-B so they can continue getting updates and expanding their system. Short of manually noting all the config and re-entering it all, is there an easy way to achieve this? Me; On June 9 you emailed me this question. On June 9 I told you couldn't do this. Why would the answer be different on June 18? SM; Stutters a bit. Me; No worries. I will close the ticket. (And informed my team to avoid a reputation of this annoying 'game'). [link] [comments]  |
| Percussive maintenance works...sometimes Posted: 22 Jun 2020 09:08 AM PDT Many years ago one of my first jobs was working as PC tech for a college. I had a professor come down to my office saying his laptop was making funny noises and he was unable to get it started. He was hoping that I could get it working again. I took the laptop; plugged it in; powered it up and heard...click-click...click-click...click-click as the hard drive attempted to spin up. Knowing this wasn't good, I informed the professor that the chances of his computer booting up were slim to none and we would more than likely need to replace the hard drive. Of course...that's when he drops the "Everything I have is on that computer." I knew the answer already; but still asked the question, "Do you backup your work to your network share per college policy?" This was before USB flash drives, folder redirection,VPN's, and other such features were common so everything was saved to the local drive. Of course the answer is,"No, never have." I ask him to leave the laptop with me to see what I can do. After he leaves and as I start taking apart the laptop to pull the hard drive a thought strikes me….just maybe….I take the screwdriver I have in my hand; turn it around so I am holding the pointy end and firmly strike the case where the hard drive is located with the back end of the screwdriver. Miracles beyond miracle...the click-click stops and the hard drive spins up. The OS starts and I am greeted with the login screen. I log in with our technician account and run a few disk tools(Norton Disk Doctor or something similar) we had which "correct" the errors. I called the professor and explained the good news(leaving out the part about some percussive maintenance) and asked him to stop by my office. When he arrives I explain in no uncertain terms that he is extremely lucky and that he needs to back up his files as soon as he is back in his office and plugged back into the network; as I could not guarantee how much longer that hard drive would continue to run. He takes his laptop, expresses his gratitude and promises he will back up his important files as soon as he is back in his office. Well this is TFTS….so you know the story can't end there…. Guess who was back in my office about two weeks later with a computer that would not boot...guess who had not backed up their important files..and one guess to who was not so lucky the second time around? Edit-This was before backup/cloning of hard drives was common to do in these scenarios. [link] [comments]  |
| You are subscribed to email updates from Tales From Tech Support. To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
| Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States | |
No comments:
Post a Comment