Law enforcement. Disk mirror. You already know how it ends, don’t you? Tech Support |
- Law enforcement. Disk mirror. You already know how it ends, don’t you?
- Replacing a failed RAID drive
- That's not my job...
- First lesson in IT
| Law enforcement. Disk mirror. You already know how it ends, don’t you? Posted: 01 Mar 2020 06:46 AM PST The Tennessee Sheriffs Office debacle (https://www.reddit.com/r/sysadmin/comments/fbtedb/sheriffs_office_accidentally_deletes_dashcam/) reminded me of my own work with a local PD. A city's police department did not have a support contract with us, but the town hall did, so the PD would intermittently call us. Because they were on accessible VLANs to the town hall, troubleshooting was easy. I took the call. PD point of contact was a thoroughly unpleasant and technically inept officer who apparently was our contact because she could turn on a computer without assistance. "The server had a blinking orange light." Yes, good afternoon to you, too. After going through the usual speech (you don't have a contract, time is billed in 15 minute increments at HIGHVALUES, etc.) I dive in. "Yes, I see the primary disk has failed. I've checked and this server [note:not purchased through us] is still under its three-year warranty. You'll need to contact the manufacturer at [website] with this disk serial number of [long string of letters and numbers from Seagate spinning rust]. When the warranty drive comes in, do you want up to come out, swap the disks, and rebuild the RAID array?" "No! You charge too much, I can do it!" Not even a blink. That's kind of how she is. "Okay. Please make sure you have a good backup before doing anything with the disks in case something goes wrong; it's rare something does do sideways, but it is a possibility." "I know what I'm doing." "Fair enough," and I ring off with the usual pleasantries, write my notes and close the ticket. Four days later, my phone rings with samesaid officer. "All the data is gone! What did you do??!" [insert speil about billing costs, etc.] I remote into the iDRAC and take a look. Yup, data's gone. Check some more. Yup, definitely gone and it's time to tread carefully. "I see what's happened. The dead disk was replaced, but the RAID rebuild wrote the blank disk over its mirrored disk rather than the other way. The data disk had been overwritten and will be unrecoverable." Cue her losing her shit - it's all our fault, we need to get this up and running, etc. "Sergeant," I calmly reply, "I've pulled up the original call, and we did offer to come to perform the work but were waved off. The data is not recoverable from that disk. We should be able to restore from the last known good backup. May I remote into your system to check?" Grudging assent. I look through their backups, all red, all failed. For months. "M'am… when was the last time you looked at the backups? I'm should them all redlined with failures going back for as far as it maintains records, about ninety days." "The previous admin said that was normal." She's been in this position of tech support for at least three years. I lag a moment, and almost BSOD'd at that comment, before recovering. "Sergeant, the data on the disks is gone, and the backups are no good. What does this server do, precisely?" "It holds all the arrest records and mugshots." "I… see. There is a recovery company we use; while I cannot guarantee the data is recoverable, I can put a call into him for you. He charges EVENHIGHERVALUES and all time is payable whether successful or not. I have yet to see him recover nothing." "Forget it," and she rings off. I have no doubt it's entirely our fault in her eyes, too. [link] [comments]  |
| Posted: 01 Mar 2020 07:32 AM PST First post on this sub. TL:DR at bottom. Years ago, back when I was a desktop tech for a fortune 500 company, I was trying to break into server side support... So I hung out with the server guys as much as I could to learn from them. One day, I was with one of the senior server techs (SST), who just received a replacement drive for a failed one (simple stuff... But I wanted to learn everything). We walk into the server room, and he says something about needing to put the new drive "at the end" of the DAE. At this point I'm still under the assumption that he's smarter than I am, and ask him to clarify what he means. SST - "All new drives need to go into the last slot of the DAE, so I need to remove the bad disk from slot 5 (16 disk DAE) and move each drive down one until the last slot is open" Me - isn't it really important to keep the disk in exactly the same place for parity? Wouldn't changing the drive order screw up the data? SST (irritated that a lowly desktop tech is questioning him) - no, the system knows which disk is which and needs the new drive at the end. Me - I'm not sure about that... Everything I've read says just to replace the drive. SST - I know what I'm doing Me (not wanting to be there when he pulls drives, and knowing I'm about to be very busy) - alright, I'll leave you to it. I've got some desktop stuff to do. 15 minutes later, I've got quite a few angry calls and emails about home and department folders being down, and all I can say is that the server team is aware and working on it. Took them until the next morning to recover the data from backups, and I learned that just because someone is in the field longer than me, doesn't mean they know more than me. TL:DR - Server tech re-orders RAID5 DAE against my recommendation, loses all data. [link] [comments]  |
| Posted: 01 Mar 2020 08:21 AM PST A story from a few years ago when I was a trainee tech working solo on a third party job. I get called into the bosses office so they can explain an issue to me, apparently 3 rooms worth of computers (about 20 ISH) suddenly won't turn on anymore. Perfect timing also as as there was apparently a very important meeting on this day. I explain straight away that doesn't sound like it would be an IT issue but I can certainly go and look. I go and visit the rooms and discover that not only do the computers not turn on but every source of power in each room also seems to have failed (lights, fans etc.). I report this back to the boss and explain that there is a bigger problem and that the power itself has failed in all 3 rooms, the boss looks bewildered and i says "Yes, that's why I have just asked you to go and fix it". I explain that is not an IT issue and I will work with an electrician to identify an issue if needs be. Their reply? "Why would we need another person to so the same job as you, it's powered so it's your responsibility". So anyway a back and forth of sorts occurs for a while and I end up going away to speak to my actual company who tell me to just appease the customer and figure it out. An electrician finally shows up after more back and forth, it turns out that a circuit breaker panel had malfunctioned and required a repair (I think it was replaced in the end but I cannot remember that part). So yeah that's the story of people not understanding what my role actually is. TL;DR: Was instructed to figure out a large electrical fault whilst being a trainee technician. [link] [comments]  |
| Posted: 01 Mar 2020 10:06 AM PST Obligatory "Long time lurker, first-time poster" TL;DR at bottom. Here's a quick little one I thought of off the top of my head whilst browsing Reddit at work. Back in High School I decided to take a student-aide/student-IT period where I was the only Freshman working with a group of other students learning to do basic support whilst also setting up and maintaining every electronic at our school. We mostly imaged machines, did inventory, set up the computer labs and answered the summons of any teacher that needed help with any of their technology. This exchange happened early on in the year, while the more experienced other tech (OT) and I were finishing setting up a computer lab that, admittedly, had enough power-strip daisy-chains to probably constitute a very real fire hazard. OT: Hey alanmew can you help me with this row of computers? None of the monitors or machines will power up, and none of the power-indicating lights are on either. Me: Did you check to make sure they were all plugged in? OT: Yeah of course I checked, I'm not an Idiot. Me: Ok, lets go see. While I look through the machines, checking their power supplies, connections, and anything else an inexperienced 14-year-old could come up with. After an embarrassing amount of time I found the issue: The computers and monitors were plugged in: into a power-strip that was connected to itself. It was a quick fix and there were no real consequences other than me not letting him forget about this incident for the rest of the year. TL;DR: I write a bunch of exposition for a story about forgetting to plug in a computer. [link] [comments]  |
| You are subscribed to email updates from Tales From Tech Support. To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
| Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States | |
No comments:
Post a Comment