Hard drives are HARD Tech Support |
- Hard drives are HARD
- There's no way this system will still be running in a decade, we don't need offboarding
- Thanks for the heart attack, I guess
- Hard drives are hard
- Do you have a time machine?
- You just type the XML into Word, right?
- "Oh I forgot to tell you?"
- I used Google before calling
- RV tech support
| Posted: 30 Jun 2021 12:14 AM PDT A little short story. Working in a public school isn't the easiest IT gig. Budgets are tight, and there's quite a "make do and mend" and "Don't fix what isn't broken" attitude to IT procurement. That "make do and mend" philosophy had me retrofitting SSD's into machines that were otherwise acceptably specced (3rd gen i5's, 4GB RAM, enough to work with). This left us with a stack of 3.5" HDD's of various capacities, mainly 320GB and 500GB. Well, time comes to clear out some space. The 320GB's are just a bit too small and slow to be of use, but the 500GB's probably still have some life on them. So we pull the several 320GB's off the shelf and go ham with the "Data protection officer", also known as a ball peen hammer.
Our site manager comes rolling along with his 6lb sledgehammer. despite the progress we're making beating the living crap out of these drives, site manager wants to show off his might. WHACK One drive completely obliterated.
WHACK Two drives obliterated in one stroke. He stacks 3 up. WHACK
As he hit the stack, one of the drive flung up, bounced and got him right in the gentleman's vegetables. After recovering from the plumshot and hit on his pride, he decided to leave data destruction to the IT department. [link] [comments]  |
| There's no way this system will still be running in a decade, we don't need offboarding Posted: 30 Jun 2021 02:41 AM PDT When you're dealing with systems that haven't had proper support for years, you're inevitably going to run into landmines. My boss has been away on sick leave for a while, and it was just extended further. As such, I delegated his approval tasks to myself so we could get the low-impact stuff moving. Part of that was approving people for access to this ancient ordering system. I approve the 30 pending requests and go on with my day. Today I wake up to a few messages from the new people. No license available! Wait, what? Oh right. We're limited to 4000 licenses. There is no way we have 4000 active users. After digging up the database info, it turns out that we have licensed users that last logged on in 2011. Is there any automated method of cleaning out licenses for offboarded users? No. Is there any manual way that I can trigger? No. Did anyone at any point create some way of handling user licenses? Adding/removing licenses in bulk? Anything at all?! No. Oh, and is there a test or dev environment for the system? Also no. After double checking that I still have the email from my boss giving me full authority to do whatever I deem necessary with this outdated system, and absolving me from any consequences if it breaks due to the lack of training in the system, I get to work. SQL Update. All users with a license that have not started a session since 1/1/2020, remove license. 2912 records updated. 1088 users still have a license. Every user I know has logged on recently still has a license. Servicedesk informed. No issues should arise, but if they do I'm to blame. Post-mortem sent to my boss and scrum master. Sweating gradually decreasing. The system didn't explode. Another day saved. TL;DR: Another fire put out by smothering it with napalm. [link] [comments]  |
| Thanks for the heart attack, I guess Posted: 30 Jun 2021 01:01 AM PDT I work at a large car dealership. It's 5:40PM, 30th June in Australia as I write this. It's the end of the financial year, a very important time in the car industry. Earlier I was trying to wrap my head around a weird problem with our wireless, while our admin clerks were furiously tapping away at their keyboards, trying to process the deals the branches have finally sent through after 5PM before end of financial year rollover of our system can be performed. Outside, the wind and rain beat against the windows and roof of our building. My boss had just left for some holiday time, my other colleague had left for the day. I was the only sysadmin left in the building. My train of thought was suddenly derailed as everything went black and I heard the contact breakers for the generator power panel thumping closed. We'd lost power. I was already hearing the plaintive cries of the admin team crying over the cubicle walls. Thankfully, I've been in this situation before. Due to a major power incident years before, the server room, switches and some of the IT area are connected to a generator. Even though it was just after 5PM, a few of the staff with laptops around the group were still in the office. A bit of quick rewiring, and we had enough laptops going to keep the admin team busy while I tried to find the reason for the blackout. A quick call to the local power utility gave me no clues as they said nothing was happening in our area, so I decided to walk over to the other side of the warehouse we call an office. The other side of the warehouse is rented to a different car dealership, and the main transformer and plant room is located on this side. There's no internal access between us any more since they moved in, so I had to walk outside and around the front of the building. I wandered over in the driving rain to see the other dealership manager hanging out of the door and having a smoke. I asked him if he had lost power, he affirmed he had. I told him the local power utility said there wasn't a problem in our area. He says to me "Could those guys in the plant room be doing something?" We are in the process of getting solar panels installed. My boss was the project manager for this installation and kept me in the loop about the project, but he hadn't scheduled anyone working tonight. I walked into the plant room with the dealership manager in tow to find two of the electricians elbow deep in the main switch cabinet supplying the building. I asked them what the (&% they were doing. They tell me they're preparing for the solar panels they are installing. I asked them why the %#(& they cut the power without telling us to receive a shrug and a verbal guarantee that the power would be back up in 20 minutes. The manager asked them why the $*@) they cut the power without telling him, to receive a "oh.......... sorry". Turns out he was finishing end of financial year paperwork as well. I was thinking there was no way my boss would be that stupid to authorise a power cut on the most important night of the year, and called him to confirm. He wasn't told about any sort of work being performed tonight, and said he was going to call the person in charge at their end and tear them a new @$$%(. Anyway, 30 mins later and the powers back up. I'm sitting here typing this story backup and waiting to hear from the admin team about any potential database record corruption before they roll the system over. TLDR: Electricians cut power to the building on the most important night of the year for our company without scheduling it or telling anyone. [link] [comments]  |
| Posted: 29 Jun 2021 09:46 AM PDT I've supported NAS products for many years, and occasionally you get hard drive failures. This is 2 short stories. First one takes place with a university. They had a NAS at every site that stored a lot of geological research data. One site seemed to be cursed. The storage would often be sluggish, and it had a much higher rate of disk failure than all their other sites. We even replaced the whole unit, but they kept on getting frequent failures. We finally had them swap the unit at the remote site with one from their main office, and the failures continued to be at the remote site. Something was going on at that site. We got on a conference call with the remote site and the main office to discuss the issue further. System was in a secure rack, this is their only storage, and they have always had a history of high disk failure at that site on all equipment. Strange. I discussed the issue with a friend that worked at IBM (this is back when IBM made hard drives). He asked me "at what elevation is the site, because most hard drives are rated for a max of 10,000 feet."? Turns out the site is at an observatory at the top of a volcano, and the elevation was well over 12,000 feet. Hard drives need to "breathe", or at least the head needs to float on a cushion of air. The thin air at the high elevation caused frequent head crashes. Their only solution was to switch to very expensive (at the time) solid state storage. Second story, different storage company, different customer, similar symptoms (also geological info). A site had frequent disk failures, and the failures were only at that site - shipping the unit to a different site made the problems go away. I got pulled into the case and I said "ah ha, is the site at a high elevation?" I was told "no, it is at sea level". hmm, interesting. We had them ship the unit to us and sent them a different, well burned-in unit. We couldn't get theirs to fail, but our tested unit started crashing with disk errors within a week. I was truly stumped. The issue didn't happen all the time, but it was rather frequent. I got on a long conference call with the customer home office and the site to fully discuss the issue. A few hours into the call, the site said that they will have to disconnect because a storm was coming. I started asking questions such as "is this on a UPS, is auto-shutdown enabled, etc". They let me know that consistent power isn't an issue because the site runs off a generator that is rated for sensitive electronics. They have to go and "seal hatches and strap things down". Hatches?!?! The NAS in question was on an oil exploration rig. I learned that oil rigs float, and they rock with the sea when it gets choppy. When they told me "sea level", they were 100% accurate. I've learned that sometimes, it's best to get on the phone with the customer and not assume anything. Even the smallest detail might be a clue into what the issue is. [link] [comments]  |
| Posted: 29 Jun 2021 02:41 PM PDT (Sorry I've been out of the group for so long, life happens, work happens and you realize it's been weeks since your last post) Backstory: I worked for a beer distributor for 9 years (not anymore). Here is a conversation with one of the on premise (bars, restaurants, taverns) supervisors. The players: Me (hey that's me!) Sup (Supervisor) Ring!! Me: IT, this is bambam67 Sup: Hey B!! (He would never call me by my fist name, just my first initial. I think he thought that made us close friends. It did not) Me: Hey Sup, how can I help you? Sup: Okay, you know how we are supposed to do a survey of beer signs and merchandise every time we go into an account? Me: Yes, you can't start doing inventory or orders until you fill that out…or skip it leaving it blank. Are you having issues with the survey? Sup: Well, kind of… Me: Are the surveys coming up correctly? Sup: Yes Me: Does it move you over to inventory after? Sup: Yes… Me: Then what's going on? Sup: I took 'salesman name here' route this month…and I was also trying to do my job as well… Me: And… (I knew this wasn't going to be good) Sup: I didn't fill out any of the surveys for the month. Me: You skipped over the surveys? Sup: Is there a way I can go back and fill in those surveys? Me: Unfortunately, I don't think so. Sup: But there's gotta be a way! Me: Look, every time you send in the EOD (end of day) that survey, along with the inventory and orders go to the server and is processed. The survey is sent to the Beer Company. They determine if we need more signs, more merch. GM gets a report each month. Sup: But you're in IT, you have to know a way around this! I could get written up for not doing the surveys. I ride my team to do it every time. Me: Sup, the only way I know how to fix this is to build a time machine, go back a month and tell you directly to do the surveys and don't cut corners. Sup: (quiet on the other end) Me: And no, I'm not building a time machine. Sup: Alright, I better go face the music. Are you sure we can't do anything? Me: Positive. You better go talk to GM now and tell her what happened. It will go better if you tell her first instead of her finding out on her own. Sup: Okay, thanks for the advice. Click Sup didn't get written up. He did NOT take my advice and simply let fate take a crack at it. GM never noticed the empty reports for those accounts and Sup got away with not doing the job. This was not the first nor the last for him. Cheers! [link] [comments]  |
| You just type the XML into Word, right? Posted: 29 Jun 2021 09:42 AM PDT I used to work for a company that aggregated and distributed mobile content. I managed the support team for the content submission interfaces. We had a manual web interface, which almost no one used because it was one title at a time. And for a provider with hundreds or thousands of titles it simply wasn't an option. Most people used the XML interface. But this meant a lot of otherwise non-technical people attempting to write XML. A typical support request was to get some XML, debug a big of tag or structure stuff, and help the provider resubmit. One day we got a frantic screaming phone call from a submission specialist at a large company. His XML was "perfect" and he's been getting rejections for hours. So we calm him down and get him to email his XML. 10 minutes later we get an email with an MS Word document full of XML. To his credit his coding was perfect. He had folllowed our schema down to the last detail. Which was especially impressive given that he was using Word and not an editor. So of course we asked how he was submitting it, as the interface allowed either a direct file submission or a URL to the XML feed. "You just put the XML in the Word document and upload it right?" Fortunately this was a quick fix and their submission was in an hour later. [link] [comments]  |
| Posted: 29 Jun 2021 01:42 PM PDT Some back story. I'm the single SysAdmin supporting a ~150 user law firm. My boss is the CTO, I have no one reporting to me. We have 3 offices in our firm, 1 main office and 2 satellite offices. Let's call them Main SatA SatB For some additional context, we had an ENS(Layer 2 connection site-to-site basically) line through Comcast from both SatA and SatB to our Main office so that our case management software can properly communicate with the SQL server in our main office. Well SatB is in BFE and had basically 35/10 internet for the office (there is <10 people working there but still) that was pitiful and ungodly slow. 2 months ago I went down to SatB to switch over their ISP and put in 500/500 fiber. Part of that entailed setting up site to site VPNs between our firewalls from each of our satellite offices to our main office, and them to each other. Then rebuilding the routing tables to pump everything through that VPN and off the ENS lines. Everything except our phone system... Fast forward two months until yesterday. I get calls from multiple people telling me that no one in SatA can call the managing partner of our firm who just so happens to be working out of SatB this week. I do various things to troubleshoot it in our phone system, none of it works. I show 0 connection between our phone switches in SatA to SatB. I can't ping SatB's phone switch from either Main or SatA. Thankfully I can ping it internally from SatB's network. I added all the phone system subnets to the firewall and made sure they were correctly configured. Everyone from the main office has been able to dial to SatB just fine this whole time. SatA however is still unable to make connection. I've been conversing back and forth with my boss this whole time who is coming and going from meetings. Come to find out that my boss put in the disconnect order on the ENS lines about 45 days ago because we were no longer going to be using them. (We're in progress of switching to a managed phone system). Comcast told us it would be 90 days until the disconnect. It wasn't, boss didn't think to mention it to me. Whoops. tl;dr Comcast is always late, except when you expect them to be, then they're early. And, as always, good communication is key. So now I get the lovely task of going into our switches and redoing all our routing tables to remove any reference to the ENS line and then reconfiguring the site to site VPN between firewalls to allow the new subnets. Wish me luck. [link] [comments]  |
| Posted: 29 Jun 2021 07:26 AM PDT I do phone support and most callers don't bother with any sort of troubleshooting before calling in - until today. I do my greeting and get: "I hate to call you for something this simple, I used Google to find the fix but it's not working." Insert surprised Pikachu face here I verified that the user did in fact try the correct fix for Outlook not showing new message notification by the clock AND that all the settings were correct. Using my Google Fu I found a more detailed fix for the issue (put Outlook in cache mode) and everything is back to normal. Now if we can just get them to reboot before calling.... [link] [comments]  |
| Posted: 29 Jun 2021 10:24 AM PDT TL;DR at bottom. After working in IT for about a dozen years, I would occasionally do a little tech support. Mostly easy, but this one time... So a relative of mine had moved into my area with her RV. She was staying in an RV park that was also a trailer park. The spots came with cable TV and we went to the cable company to get internet from them. They hand over a cable modem and away we go. In the RV, there's an alcove with a TV. The alcove is wider than the TV and at the back is the wallplate. I unplug the TV from the wallplate and plug the modem in. Does not connect to the interwebs. Get on the phone and got a great technician. Since this was a new RV and it had only been in 2 places that provided wi-fi for everyone before, I didn't know if the cables and connections worked, so I took the laptop outside and connected there. Did not work, so the tech did some stuff on his end and I fiddled with cables and detached and reattached. Then I had to ask him to wait because a train was coming by. Okay, he says. Then hears how very very loud it is and is impressed. This RV park was pretty much on the train tracks. I have no idea how it's legal to be that close to the tracks. Looking at the map, there's 50 ft between the actual track and the first row of RVs. We were in the 2nd row. So after we can hear ourselves think again, we manage to get the connection working. I say, okay, I'll see if it works inside again, but if not, it's not your problem, since it does work outside. He's pleased with me, I'm pleased with him, it was cool. Go inside and of course it does not work. The RV did have a manual, but it wasn't actually for the right model. Looked online, nothing helpful, just some info that didn't make sense. I was stuck. Lady goes to the local RV store the next day and asks them. The guy behind the counter says the same b.s. the manual says. I can't remember all the details, but it was nonsense. I mean, "that's not how any of this works" level nonsense about wi-fi not working unless you pay for additional services and stuff. Thankfully, another guy at the store has the real knowledge: there's a button on the wallplate that switches the connection between cable and the antenna. I'd looked at the wallplate, but not with a strong flashlight. It was dark in that alcove and I did not see the black on black tiny button at the bottom of the wallplate. The button was pressed and the internet flowed throughout the RV. Trains were still really loud, though. If there's a moral to the story, I guess it's that when dealing with unfamiliar equipment, use bright enough lights to make sure there's not something wacky and/or tiny going on. I just now googled "rv cable antenna switch" and almost all the ones I found were white with a black button or black with a red button. I got the special one! I thought there was another connection from the tv to the antenna, but if I'd pulled the tv out and looked at the back, I would have seen that there was only one connection, which might have helped. TL;DR: Trains are loud, RVs are weird, flashlights are helpful, documentation could be even more helpful if it were even in the vicinity of correct. [link] [comments]  |
| You are subscribed to email updates from Tales From Tech Support. To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
| Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States | |
No comments:
Post a Comment