Only $25k for a server? Let's get the $100k version - it's bound to be more reliable! If only we knew how to use it... Tech Support |
- Only $25k for a server? Let's get the $100k version - it's bound to be more reliable! If only we knew how to use it...
- In which we discover a locked account has more than one meaning...
- HR Strikes Again
- Tales from Aircraft Maintenance: Code Brown
- Symbolic links challenges
- A repair that I'm really proud of!
- Uptime correlates exclusively with the cost of your hardware
- Sometimes it's not just the clients you need to watch out for.
- The Tech Support Aura works on everything!
| Posted: 01 May 2018 08:29 AM PDT Scene: I was working in IT in the early 2000s. I had cleaned up the company's core network (getting rid of the 10Mb hubs) and now the company realized it was time to tackle the big enchilada: the ERP system. The setup, in which stupid decisions are made This system was critical. Through it ran all the company's financials, HR, shipping, parts processing, etc. And I promise I am not making this up: it quite literally ran on a system that was the size of a washer/dryer set and was so old it didn't even support ASCII. A steady parade of tech (well, ONE tech that still understood it) came by to service the line printer, the tape drive, etc. A committee for evaluating replacements was formed. After sitting through presentations by various sleazy ERP salespeople, a product was duly chosen. As "luck" would have it, the consultants that sold and installed $PRODUCT also sold and installed hardware. And the company's hardware they sold: IBM. $CONSULTANTS duly produced a quote for a $25,000 server. It was nothing special, a regular rackmount job with RAID. The sort of thing that one would have gotten for $10,000 or less from Dell or HP. It was going to run Linux. I knew Linux, but thought we were getting ripped off. Cue conversation at a committee meeting which somehow I managed to attend:
Consultant sees his boat going down the toilet and gives me the evil eye.
Consultant starts to look excited.
Consultant starts to look VERY excited.
Consultants duly provide a quote. The clearly put everything they possibly could into it, plus a generous profit margin. $80,000. Managers are STILL concerned that it's too cheap. So they add a separate "disk expansion shelf" (read: proprietary JBOD) to bring it up to $100,000.
Which would have been fine, if $Consultants actually had anyone that knew AIX. The plot thickens The AIX equipment duly arrives, followed by a consultant to do the install. This isn't the salesguy consultant, this is the nerdy consultant. I'll call him FreshPrince because he was always talking about his favorite TV show, the Fresh Prince of Bel Air, only FreshPrince was the opposite of Will Smith. In EVERY way. Of course, by this point, management is nowhere to be seen so it's FreshPrince and me.
I busy myself with the tasks of the day. FreshPrince makes grunting noises as he struggles with the latch on the rack door, the rails, the server, the power, the install CD-ROM... After much effort, and a few calls to tech support, he seems to have managed to configure something or other.
I give him some information. Correct, even. Muttering ensues. For hours.
Hilarious... The amber lamp of doom Eventually I -- not FreshPrince -- set up the printers, the application is duly installed, migrations occur, rejoicing happens, consultants leave, more rejoicing happens, etc. Then one day, the amber lamp of doom shows up on the system. WTF is this? Through various phone calls and twisty little menus, all alike (shoutout to smitty for any AIXers out there), we arrive at the conclusion: a disk in the RAID has failed. (For about the 20th time that week, I was again thinking "this would have been so much simpler on a $10k Linux box") IBM duly dispatches an engineer with a hard drive. Engineer arrives.
Engineer calls support. An hour-long conversation involving many more twisty little menus ensued, while I eavesdropped and took notes for the future. Eventually the defective drive was identified and replaced.
I HATE IT when they say that.
Multiple escalations occur at this point, since nobody in AIX support has any idea how to repair a degraded RAID. Eventually, some hours later, the magic smitty incantation is produced, the RAID is rebuilt, and the Amber Lamp of Doom extinguishes. Epilogue 2 years later, I managed to convince management to let me build a $10k Linux box. User complaints were way down after that because it was much faster and -- yes -- more reliable.
[link] [comments]  |
| In which we discover a locked account has more than one meaning... Posted: 01 May 2018 05:24 PM PDT I'm not a true tech support person, but being the security person in charge of the logs, I spend a lot of time working with tech support to troubleshoot issues. One of those issues, one that plagued our organization for a time, was account lockouts. The powers that be decided to reduce our failed login threshold, which was good, in that it was more secure, but was bad, in that our wireless ISE system was proficient in generating login failures any time a user moved from access point to access point. This caused people to get locked out when they had effectively done nothing wrong. People would even get locked out after logging into their computers, leading into all sorts of trouble with email and other things. Eventually they backed off on the threshold while we tried to find a way to make ISE stop generating the failures, but we still had a lot more lockouts than normal to deal with. Usually, it was the user having the wrong wifi password in their phone, or running a script with their old credentials after changing their password, but every once in a while we had something... or someone... special. To facilitate troubleshooting persistent lockouts, I had created a couple of queries in our logging system that tech support could use for troubleshooting, but inevitably i was called to assist pretty much every time. TS will be tech support, Me is yours truly and this is everything going through my head.
Excellent. Now we wait. About a half hour later, another techie rolls up to my desk, grinning from ear to ear.
[link] [comments]  |
| Posted: 01 May 2018 12:58 PM PDT Maybe some of you recall my previous work with the HR department, and how they filter all IT emails. Today, one HR user called in for support. Submitted for for approval for TalesFromTechSupport, "The Tale of the Skype Profile Photo" Tech: This is the XYZ Help Desk, proceed with gathering initial information Tech: OK, so we're logged in. Do you already have a photo in mind that you want to use for your Skype/Outlook picture? User: Oh yes. I just came back from a cruise with my boyfriend and there's a picture I want to use...continues talking about the cruise and unrelated information Tech: OK, great. Clicked the Upload Photo button. Please browse the photo you want to use. User: That doesn't look right though. Tech: Is that the photo you selected? User: Yes, but the top of my head is cut off. Tech: Can you show me the photo? User: It's right there. Can't you see it? Tech: I mean, where is it on your computer? Downloads? User: No, it's in Pictures. sigh Here, let me show you. Tech: OK, the photo you're showing me ends at about the middle of your forehead. Is this the original photo or a cropped version. User: No, this is the photo. Can you scroll up a bit? Tech: As I mentioned, this is the top of the photo, there's nothing more to scroll up. User: No, there is. You know like how on Google Maps, you can view all around. How can we do that here? Tech: You can't, this is just a photo. Google... User: What do you mean, you can't? Of course you can. Here, let me show you. Tech: Google uses multiple cameras to generate those 360 degree views. You can't do that with the photo you took. User: I took it with my iPhone, they have those video pictures. Tech: Was this photo taken as one of those video pictures? User: No Tech: Then, this is the picture you have, and we can't scroll up on this picture. User: You obviously don't know what you're talking about. You're not a very good tech. Speaking in a way where both of these sentences sounded like questions. Tech: If you have another picture, we can use that. User: No, I want to use this picture because it shows the line of chairs down the deck from me. Tech: We can use the photo, it's currently set as your Skype photo. User: But it's cutting off the top of my forehead. Tech: Yes, the photo clearly stops there. There's no more photo, that's it. User: You should be able to scroll around it. Tech: It's just a photo, it can't do that. User: whining Yeeeeeeees, it caaaaaaaaaan. I want this photo as my Skype photo framed properly. Tech: Your photo has been set. If you can find another photo to use, we'll be happy to set that for you. User: I'm going to call back and get someone who knows what they're doing. [link] [comments]  |
| Tales from Aircraft Maintenance: Code Brown Posted: 01 May 2018 07:24 AM PDT Many years ago, I was an Air Force avionics technician. I had some very interesting experiences while in uniform, some of them I can even share with all of you. It was a beautiful warm day, the moose calls of C-17's refueling could be heard all across post. I was relaxing on an especially long refuel when a call on the radio interrupted my wandering mind, thinking of all the places that I could be that aren't here. An In-Flight-Emergency (IFE) was called. No details, just that the emergency was called. All refuel operations were stopped along with all non-essential activities. With this call, everyone on shift and several of the guys coming on shift that were at work scramble an emergency tow crew and response team, orders are being shouted on the radio directing people for recovery. With the bird on approach, we get another call. Medical emergency. This is bad. Potentially really bad. The jet lands and taxies normally. As it parks on the spot we swarm the jet ready to respond to any emergency. I walk onto the jet and catch a horrible smell. Like fermented death and curry. As I climb up to the flight deck (don't ever call it a COCKpit), the smell gets worse. Once I was standing behind the pilot's seat, I think I am going to be sick. The pilot, copilot, the relief pilot and one of the loadmasters have all "soiled" themselves uncontrollably and repeatedly on a 6 hour flight. They also had a widespread electrical issue that was the cause of the IFE. Two twelve hour shifts later in an unimaginable stench, we discovered a failing circuit breaker was the cause of intermittent partial power loss. Edit: I forgot to mention that we had to change the Pilot, Copilot and 2 auxiliary seats. Months later when the jet passed through, it still had that funk. [link] [comments]  |
| Posted: 02 May 2018 02:34 AM PDT I think my brain just melted... Quick Setup: We support an HPC system for a research facility. There's a natural size limit on the home directory, so storages mounted at <several-complex-system-paths> for high I/O stuff, archival purposes, security,... are provided. Symlinks (symbolic links) help the users accessing these folders without interfering with each other and not having to mess with complex system paths. All is well documented and explained in the manual everyone has to read before using the cluster. Additionally every user get's a quick rundown where this is explicitly explained (basically "just use /home/$user/$symlink-to-secure for secure stuff, /home/$user/$symlink-to-highio for stuff where you expect high I/O to occur, ..."). You can't get it wrong... was what I thought. Today this conversation with the frigging head of the HPC users ensued (slightly changed for privacy reasons), who uses this system for years now. I'll call him $record. You'll see why. $record: So if I save my report into /home/$user/$symlink, will my data be stored in my home directory or in $symlink-target? $me encounters internal compute errors $me: No. Your report will only be stored once, at $symlink-target. It's just easier to list /home/$user/$symlink instead of $complex-system-path. We created these symlinks for all users for this exact reason. $me: oh god why... $me: ....yes. $me: internal screaming and twitching eyes $me: You decide where to store your data. $me goes into internal mental breakdown mode while maintaining a straight face I eventually got him sorted. After a solid hour of discussing with this record of a person. TL;DR: Symbolic links synchronize time-space with users, clouds and alcohol. [link] [comments]  |
| A repair that I'm really proud of! Posted: 01 May 2018 02:30 PM PDT TL;DR I repaired a motherboard and am really proud of it! So I'm making this post because I don't have a lot of techie friends but I'm just super psyched and had to tell someone. I work at a small, locally owned computer repair shop (we have three employees), and I've been here for almost three years. I received all of my technical training here, so I have no formal training what-so-ever. Over the last year, I've been slowly teaching myself about board-level repairs, mostly on MacBooks. I've managed to pull off a few repairs, but after completing them I still felt like I had a lot of questions and wasn't sure if I had just been lucky enough to stumble across the solution. This time, however, I can confidently say that I figured it out entirely due to my knowledge of board-level repair!! Here are some pictures for reference. https://imgur.com/a/pBO7Q2M Now, here's the story :) A customer brought in a MacBook Air that seemed to charge fine (the light on the adapter would go from green to orange), but it wouldn't turn on. The tech checking it in poked around a bit, but couldn't really find anything, so he left it for me to look at later. When I got my hands on it, I tried resetting the SMC and that worked fine, but it still wouldn't turn on. Next up I went to test each power rail to see if they were all reading what they should, so I disassembled it to have the bare motherboard on my desk. I pulled up the schematic pdf and the boardview on my computer to find where to test each rail, then hooked it up to the adapter. The first one I went to check was PPBUS_G3H, one of the more important rails that is always on whenever the computer is on power, and noticed that the area where I was touching was hot! I felt around a little bit and was almost burned by the problem component! It was a tantalum capacitor that was part of the circuitry providing the 3.3V and 5V rails power, and it had the word CRITICAL right next to it on the schematic. I immediately disconnected the adapter. When I looked closer, I noticed that the thing was even cracked! I then went into our parts closet to see if we had any donor boards, and we didn't. So I just got another MacBook Air board that seemed close in age to the one I was working on and carried it back to my desk. I pulled up the schematics for this board as well and found the page describing the same circuitry as my problem component, and there was one exactly the same! I identified it on the board, double checked the polarity of each of them, then went to the rework station and swapped them out. The swap went without a hitch, which I think is pretty cool considering I did it with just a heat gun (not even a rework station, we don't have a lot of money around here), tweezers, and a syringe of flux! I took it back to my bench, cleaned off the flux and, lo and behold, it worked perfectly! All in all, very customer, very happy tech :) Thanks for reading! [link] [comments]  |
| Uptime correlates exclusively with the cost of your hardware Posted: 01 May 2018 10:22 AM PDT The story of management who wanted to buy expensive hardware reminded me of another story from the same company I mentioned previously. It was the 90s, and I was a software developer building a website. Although I wasn't tech support, the role still gave me ample opportunity to observe many of the foolish things that happen around tech. In this company, there was an IT department that was responsible for maintaining systems. Other departments paid them for setup, support, and hosting servers, but had no authority to insist on timely resolutions. The prices were high and the service was bad, but other departments had no choice but to go through internal IT. My manager ordered hardware to run the website. We got a shiny new Sun, despite my recommendation for Linux. Uptime was important, and you only get it with tried and true big vendors! So we got another Sun which did nothing but mirror the first, and a switch to flip over to the second if the first went down. Both the machines had RAID. Uptime was really, really critical! I suggested that if they were serious about uptime, they might want to consider getting a UPS. However, we didn't have any budget left, due to the expensive decisions that had already been made. It took several weeks to get the system set up, because of the IT department's inherent laziness. During the time I worked there, we never had a disk failure, and we never switched over to the backup machine. But we also never had a power outage. We did, however, have lousy uptime. The reason? Our internet connection was not especially reliable, and would drop a couple of times a month. Although the IT department theoretically had someone on call 24/7, nobody would pick up the phone after hours. If the internet went down at 4:00 in the afternoon, it wouldn't be back until midday the following day. Unless it was a Friday, in which case it would be back on Monday. Unless Monday was a holiday... We easily had 3+ days of downtime every month. TL;DR: Spending lots of money on hardware is pointless if you can't enforce an SLA for your internet connection. [link] [comments]  |
| Sometimes it's not just the clients you need to watch out for. Posted: 01 May 2018 06:27 AM PDT So this happened all of 5 minutes ago and I currently waiting on the repercussions (thankfully I work with some understanding people - or at least, I hope I do). Last week I was sent to complete my MTCNA (Mikrotik router accreditation), which I passed. Today has been a quiet day in the office and with not much to do, I thought I would reinforce my newly learned skills and play around with one of our old 751G's. I connected it up to my laptop eth0 port and up the device popped in winbox. Logged in via MAC connection and noticed there was an old config on there. Never to mind, I thought. Lets wipe the config... It resets, I connect back in and go to view the wlan0 interface to configure it to our office WiFi so I can have internet while I work on the router, but there is no wlan0 interface. Odd, I thought, but whatever. I keep poking around, maybe it is just not showing up since I set it not to get default config. I check the full interface list. For some reason there are 24 eth interfaces and 2 SFP interfaces. wtf? so weird. Why does this little 5 port router think that it's a cloud router. Then it hits me: This is the office router, I just wiped it completely clean. No config export. No backup. Nothing. Fuck. I have since messaged my manager and boss and they are not super happy about it, but I think they will not blow up too much at me. They are level headed people. Looking at it, I can see where I went wrong. I had my eth0 interface on my laptop disabled and thereofre never saw the connected 751G and defaulted through the wireless network to our cloud router. I just happened to guess the right password for it and didn't check the MAC address was matching before wiping it. [link] [comments]  |
| The Tech Support Aura works on everything! Posted: 01 May 2018 02:08 PM PDT LTL, FTP, HTTPS (Helping To Transport Poo Scoops) I'm a school tech, so most of my support calls are along the lines of "I didn't do anything [other than try to circumvent the filter] and it broke!" Outside work, I do a little bit of everything. Today, I got a call that a user's peripherals weren't powering up. These peripherals are large and agricultural, but they follow the same rules as everything else. There's two, but only one port, so I asked him if he made sure the correct cable was plugged in on both ends and it was turned on. He said he had tried that FIVE TIMES and nothing worked. I plugged in the baler, and it worked, then the planter, and it worked. I rebooted a hundred horsepower of $Kumquat to test. He blames "magic" but may have skipped some troubleshooting steps when the new tractor didn't do what he wanted. The user is my father. People assume that I can fix anything with wires, and I let a few of them get away with that. Right now, I'm going to fix a track light that isn't lighting. Six credits of Electrical Engineering should come in handy :) [link] [comments]  |
| You are subscribed to email updates from Tales From Tech Support. To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
| Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States | |
No comments:
Post a Comment