• Breaking News

    [Android][timeline][#f39c12]

    Wednesday, May 2, 2018

    Only $25k for a server? Let's get the $100k version - it's bound to be more reliable! If only we knew how to use it... Tech Support

    Only $25k for a server? Let's get the $100k version - it's bound to be more reliable! If only we knew how to use it... Tech Support


    Only $25k for a server? Let's get the $100k version - it's bound to be more reliable! If only we knew how to use it...

    Posted: 01 May 2018 08:29 AM PDT

    Scene: I was working in IT in the early 2000s. I had cleaned up the company's core network (getting rid of the 10Mb hubs) and now the company realized it was time to tackle the big enchilada: the ERP system.

    The setup, in which stupid decisions are made

    This system was critical. Through it ran all the company's financials, HR, shipping, parts processing, etc. And I promise I am not making this up: it quite literally ran on a system that was the size of a washer/dryer set and was so old it didn't even support ASCII. A steady parade of tech (well, ONE tech that still understood it) came by to service the line printer, the tape drive, etc.

    A committee for evaluating replacements was formed. After sitting through presentations by various sleazy ERP salespeople, a product was duly chosen. As "luck" would have it, the consultants that sold and installed $PRODUCT also sold and installed hardware. And the company's hardware they sold: IBM.

    $CONSULTANTS duly produced a quote for a $25,000 server. It was nothing special, a regular rackmount job with RAID. The sort of thing that one would have gotten for $10,000 or less from Dell or HP. It was going to run Linux. I knew Linux, but thought we were getting ripped off. Cue conversation at a committee meeting which somehow I managed to attend:

    Me: I'm concerned about the price.

    $Manager: I know. Can you really get quality for only $25,000?

    $VP: Right! Our last server cost over $200,000!

    Me: We could have the same for $10,000.

    Consultant sees his boat going down the toilet and gives me the evil eye.

    $Manager: What if we got two? Then we'd have a backup.

    Consultant starts to look excited.

    $VP: That's still only $50,000. Are we sure we're not buying something so cheap it's dodgy?

    Consultant starts to look VERY excited.

    $Consultant: You know, we're an IBM reseller. We could sell you an AIX box. It's like what you're used to.

    Consultants duly provide a quote. The clearly put everything they possibly could into it, plus a generous profit margin. $80,000. Managers are STILL concerned that it's too cheap. So they add a separate "disk expansion shelf" (read: proprietary JBOD) to bring it up to $100,000.

    Me: Nobody here knows AIX.

    $Consultant: You're good with Linux. We'll train you. For free even!

    $VP: That's a great deal! Where do we sign?

    Which would have been fine, if $Consultants actually had anyone that knew AIX.

    The plot thickens

    The AIX equipment duly arrives, followed by a consultant to do the install. This isn't the salesguy consultant, this is the nerdy consultant. I'll call him FreshPrince because he was always talking about his favorite TV show, the Fresh Prince of Bel Air, only FreshPrince was the opposite of Will Smith. In EVERY way.

    Of course, by this point, management is nowhere to be seen so it's FreshPrince and me.

    FreshPrince: So I'll just install this then.

    Me: Great.

    I busy myself with the tasks of the day. FreshPrince makes grunting noises as he struggles with the latch on the rack door, the rails, the server, the power, the install CD-ROM... After much effort, and a few calls to tech support, he seems to have managed to configure something or other.

    FreshPrince: I'm all done.

    Me: You haven't asked me about the printer setup. $Application needs printer access.

    FreshPrince: Oh! Uh. Right. So, uh, tell me about these printers?

    I give him some information. Correct, even.

    Muttering ensues. For hours.

    Me: So how's it coming?

    FreshPrince: Would you believe it, they redid how they do printers in AIX 5! On AIX 4, it was, uh, somewhere else, and you have 5. I can't even find where it is in the menu! Hahaha!

    Me: You've spent all afternoon not configuring the printers?

    FreshPrince: Oh don't worry, I've got a hotel room, I'll be back tomorrow!

    Hilarious...

    The amber lamp of doom

    Eventually I -- not FreshPrince -- set up the printers, the application is duly installed, migrations occur, rejoicing happens, consultants leave, more rejoicing happens, etc.

    Then one day, the amber lamp of doom shows up on the system. WTF is this? Through various phone calls and twisty little menus, all alike (shoutout to smitty for any AIXers out there), we arrive at the conclusion: a disk in the RAID has failed. (For about the 20th time that week, I was again thinking "this would have been so much simpler on a $10k Linux box") IBM duly dispatches an engineer with a hard drive.

    Engineer arrives.

    Engineer: Which drive is bad?

    Me: Heck if I know. It took me two hours just to find out that the problem is a bad drive.

    Engineer: I can't replace the drive unless I know which one.

    Me: I was sorta hoping you could figure that out since you're from, you know, the people that made this thing.

    Engineer: That's not my job, but as a favor to you, I'll see what I can do.

    Engineer calls support. An hour-long conversation involving many more twisty little menus ensued, while I eavesdropped and took notes for the future. Eventually the defective drive was identified and replaced.

    Engineer: All done. Sign here.

    Me: The amber light's still on.

    Engineer: It's supposed to do that.

    I HATE IT when they say that.

    Me: That light was never on until the drive failed.

    Engineer: I know. You have to reset the lamp.

    Me: How?

    Engineer: I don't know. Sign here.

    Me: Also, there's no activity on the new drive.

    Engineer: You have to trigger a RAID rebuild.

    Me: I hesitate to ask, but... how?

    Engineer: I don't know. Sign here.

    Multiple escalations occur at this point, since nobody in AIX support has any idea how to repair a degraded RAID. Eventually, some hours later, the magic smitty incantation is produced, the RAID is rebuilt, and the Amber Lamp of Doom extinguishes.

    Epilogue

    2 years later, I managed to convince management to let me build a $10k Linux box. User complaints were way down after that because it was much faster and -- yes -- more reliable.

    $Manager: Wow, technology is so amazing! Two years ago, we had to pay 10 times as much for this!

    Me: ...

    submitted by /u/Universal_Binary
    [link] [comments]

    In which we discover a locked account has more than one meaning...

    Posted: 01 May 2018 05:24 PM PDT

    I'm not a true tech support person, but being the security person in charge of the logs, I spend a lot of time working with tech support to troubleshoot issues.

    One of those issues, one that plagued our organization for a time, was account lockouts. The powers that be decided to reduce our failed login threshold, which was good, in that it was more secure, but was bad, in that our wireless ISE system was proficient in generating login failures any time a user moved from access point to access point. This caused people to get locked out when they had effectively done nothing wrong. People would even get locked out after logging into their computers, leading into all sorts of trouble with email and other things. Eventually they backed off on the threshold while we tried to find a way to make ISE stop generating the failures, but we still had a lot more lockouts than normal to deal with. Usually, it was the user having the wrong wifi password in their phone, or running a script with their old credentials after changing their password, but every once in a while we had something... or someone... special.

    To facilitate troubleshooting persistent lockouts, I had created a couple of queries in our logging system that tech support could use for troubleshooting, but inevitably i was called to assist pretty much every time. TS will be tech support, Me is yours truly and this is everything going through my head.

    TS: "This user keeps getting locked out, Joe Tech unlocked her account several times this morning but it keeps happening. We can't figure out why."

    Okay, I'll play your game. I pull up my query and filter for her username. No lockouts.

    Me: "Sorry bud, her account isn't getting locked out. Why is Joe Tech unlocking her account when it's not locked? You can see that in AD when you look at the account, it tells you right there if it's locked out."

    If a tree falls in an active directory forest but no one's around to unlock it, does the user still make a sound?

    TS: "But she keeps saying it's locked and we don't know what else to do."

    Of course. Keep unlocking it even though it's clear from the logs and from AD that her account is not in fact locked out. Isn't that the definition of insanity...

    Me: "You do realize that's the definition of insanity, doing the same thing over and over and expecting a different result?"

    TS: "Yeah, but can you fix it?"

    if i could fix stupid i wouldn't be working here...

    Me: "This is beyond my ability to troubleshoot remotely and it's not ISE. She's not getting locked out. Next time she calls saying it's locked, send someone over to look at the error message, since she obviously can't take a screenshot and email it at that point."

    TS: "Will do."

    Excellent. Now we wait.

    About a half hour later, another techie rolls up to my desk, grinning from ear to ear.

    TS2: "You're not going to believe this."

    Me: "Try me."

    god i love the smell of of vindication early in the morning

    TS2: "So she calls us after she locks her account out again. I stopped by her desk, and she showed me this." He reaches over and locks my computer. "See? She was letting her system idle until the screensaver locked it, and when you hit a key to turn off the screensaver..."

    Me: "It shows a lock, and says the account is locked." It hits me. "She thought that that meant her account was locked out?!? She never even tried to put in her credentials?"

    TS2: "Not so loud, she made me swear to not tell anyone."

    Me: "Her secret is safe with me."

    Not.

    submitted by /u/starla79
    [link] [comments]

    HR Strikes Again

    Posted: 01 May 2018 12:58 PM PDT

    Maybe some of you recall my previous work with the HR department, and how they filter all IT emails. Today, one HR user called in for support. Submitted for for approval for TalesFromTechSupport, "The Tale of the Skype Profile Photo"

    Tech: This is the XYZ Help Desk, proceed with gathering initial information
    User wants to update their profile photo for Skype for Business and Outlook
    User is in HR - our ticketing system lists what department they're in, and after the lunacy of that last post, you know what kind of people work in my company's HR department
    After verbally attempting to instruct the user with changing her profile photo and being unsuccessful, we get a remote session. It's on the Online Web Access for Outlook login page

    Tech: OK, so we're logged in. Do you already have a photo in mind that you want to use for your Skype/Outlook picture?

    User: Oh yes. I just came back from a cruise with my boyfriend and there's a picture I want to use...continues talking about the cruise and unrelated information

    Tech: OK, great. Clicked the Upload Photo button. Please browse the photo you want to use.
    User complies and the photo is uploaded

    User: That doesn't look right though.

    Tech: Is that the photo you selected?

    User: Yes, but the top of my head is cut off.

    Tech: Can you show me the photo?

    User: It's right there. Can't you see it?

    Tech: I mean, where is it on your computer? Downloads?

    User: No, it's in Pictures. sigh Here, let me show you.

    Tech: OK, the photo you're showing me ends at about the middle of your forehead. Is this the original photo or a cropped version.

    User: No, this is the photo. Can you scroll up a bit?

    Tech: As I mentioned, this is the top of the photo, there's nothing more to scroll up.

    User: No, there is. You know like how on Google Maps, you can view all around. How can we do that here?

    Tech: You can't, this is just a photo. Google...

    User: What do you mean, you can't? Of course you can. Here, let me show you.
    User goes to Google Maps, drops the little man on the street, and scrolls around the 360 street view.
    See? Now, how can we do that with the photo so the top of my head isn't cut off?

    Tech: Google uses multiple cameras to generate those 360 degree views. You can't do that with the photo you took.

    User: I took it with my iPhone, they have those video pictures.

    Tech: Was this photo taken as one of those video pictures?
    Considering making a drop by the user's desk to capture the screen if it's a video and use that as the picture not cutting off her forehead

    User: No

    Tech: Then, this is the picture you have, and we can't scroll up on this picture.

    User: You obviously don't know what you're talking about. You're not a very good tech. Speaking in a way where both of these sentences sounded like questions.

    Tech: If you have another picture, we can use that.

    User: No, I want to use this picture because it shows the line of chairs down the deck from me.

    Tech: We can use the photo, it's currently set as your Skype photo.

    User: But it's cutting off the top of my forehead.

    Tech: Yes, the photo clearly stops there. There's no more photo, that's it.

    User: You should be able to scroll around it.

    Tech: It's just a photo, it can't do that.

    User: whining Yeeeeeeees, it caaaaaaaaaan. I want this photo as my Skype photo framed properly.

    Tech: Your photo has been set. If you can find another photo to use, we'll be happy to set that for you.

    User: I'm going to call back and get someone who knows what they're doing.
    click

    submitted by /u/megamanxtc
    [link] [comments]

    Tales from Aircraft Maintenance: Code Brown

    Posted: 01 May 2018 07:24 AM PDT

    Many years ago, I was an Air Force avionics technician. I had some very interesting experiences while in uniform, some of them I can even share with all of you.

    It was a beautiful warm day, the moose calls of C-17's refueling could be heard all across post. I was relaxing on an especially long refuel when a call on the radio interrupted my wandering mind, thinking of all the places that I could be that aren't here. An In-Flight-Emergency (IFE) was called. No details, just that the emergency was called. All refuel operations were stopped along with all non-essential activities. With this call, everyone on shift and several of the guys coming on shift that were at work scramble an emergency tow crew and response team, orders are being shouted on the radio directing people for recovery.

    With the bird on approach, we get another call. Medical emergency. This is bad. Potentially really bad.

    The jet lands and taxies normally. As it parks on the spot we swarm the jet ready to respond to any emergency.

    I walk onto the jet and catch a horrible smell. Like fermented death and curry. As I climb up to the flight deck (don't ever call it a COCKpit), the smell gets worse. Once I was standing behind the pilot's seat, I think I am going to be sick. The pilot, copilot, the relief pilot and one of the loadmasters have all "soiled" themselves uncontrollably and repeatedly on a 6 hour flight.

    They also had a widespread electrical issue that was the cause of the IFE. Two twelve hour shifts later in an unimaginable stench, we discovered a failing circuit breaker was the cause of intermittent partial power loss.

    Edit: I forgot to mention that we had to change the Pilot, Copilot and 2 auxiliary seats. Months later when the jet passed through, it still had that funk.

    submitted by /u/ITSupportZombie
    [link] [comments]

    Symbolic links challenges

    Posted: 02 May 2018 02:34 AM PDT

    I think my brain just melted... Quick Setup:

    We support an HPC system for a research facility. There's a natural size limit on the home directory, so storages mounted at <several-complex-system-paths> for high I/O stuff, archival purposes, security,... are provided. Symlinks (symbolic links) help the users accessing these folders without interfering with each other and not having to mess with complex system paths. All is well documented and explained in the manual everyone has to read before using the cluster. Additionally every user get's a quick rundown where this is explicitly explained (basically "just use /home/$user/$symlink-to-secure for secure stuff, /home/$user/$symlink-to-highio for stuff where you expect high I/O to occur, ..."). You can't get it wrong... was what I thought.

    Today this conversation with the frigging head of the HPC users ensued (slightly changed for privacy reasons), who uses this system for years now. I'll call him $record. You'll see why.

    $record: So if I save my report into /home/$user/$symlink, will my data be stored in my home directory or in $symlink-target?
    $me: It will be stored at $symlink-target, but accessible via the symlink in your home directory.
    $record: So it will be physically stored in my home directory.
    $me: No, it will be stored at $symlink-target.
    $record: So it is NOT stored in my home directory?
    $me: Correct.
    $record: Then why can I see my report file /home/$user/$symlink/?
    $me: Because when you access a symbolic link, it shows you the content of the directory or file the symlink is pointing to.
    $record: So my report will be stored in my home directory, but also in $symlink-target. How does the system synchronise these files then? Isn't that quite a waste of space?

    $me encounters internal compute errors

    $me: No. Your report will only be stored once, at $symlink-target. It's just easier to list /home/$user/$symlink instead of $complex-system-path. We created these symlinks for all users for this exact reason.
    $record: I understand. So if I save my report into /home/$user/$symlink, my data will be stored in $symlink-target and in /home/$user/$symlink.
    $me: Almost. The report file is physically stored in $complex-sytem-path and $symlink points to that directory so you don't need to worry about complex system paths. You only have to access $symlink to see your report file.
    $record: Ahh. So my report is stored somewhere and also in my home directory. $symlink just internally synchronizes it with $complex-system-path.

    $me: oh god why...

    $me: ....yes.
    $record: I think this is a waste of storage, though. Why can't I store everything in my home directory directly?
    $me: Because as documented the home directory has a size limit. $complex-system-path has not. The symlink is just there to access it easily.
    $record: Hmm I understand. It's a Cloud storage and I just store everything in my home directory and the system stores it where it's supposed to be.
    $me: That's one way to view it.
    $record: Then how does the system decide where to store my report correctly?

    $me: internal screaming and twitching eyes

    $me: You decide where to store your data.
    $report: Why would I decide where to store my data?
    $me: So you can decide whether you store your report in a secure place or in a high-I/O as clearly documented in the manual for $system.
    $report: I read that manual.
    $me: Then you should be able to decide where you store your report files.
    $report: But I don't want to waste space, because the symlink has to synchronize the files somehow in the background. Ain't that a bad decision performance wise AND a waste of space?

    $me goes into internal mental breakdown mode while maintaining a straight face

    I eventually got him sorted. After a solid hour of discussing with this record of a person.

    TL;DR: Symbolic links synchronize time-space with users, clouds and alcohol.

    submitted by /u/waka42
    [link] [comments]

    A repair that I'm really proud of!

    Posted: 01 May 2018 02:30 PM PDT

    TL;DR I repaired a motherboard and am really proud of it!

    So I'm making this post because I don't have a lot of techie friends but I'm just super psyched and had to tell someone. I work at a small, locally owned computer repair shop (we have three employees), and I've been here for almost three years. I received all of my technical training here, so I have no formal training what-so-ever. Over the last year, I've been slowly teaching myself about board-level repairs, mostly on MacBooks. I've managed to pull off a few repairs, but after completing them I still felt like I had a lot of questions and wasn't sure if I had just been lucky enough to stumble across the solution. This time, however, I can confidently say that I figured it out entirely due to my knowledge of board-level repair!! Here are some pictures for reference. https://imgur.com/a/pBO7Q2M Now, here's the story :) A customer brought in a MacBook Air that seemed to charge fine (the light on the adapter would go from green to orange), but it wouldn't turn on. The tech checking it in poked around a bit, but couldn't really find anything, so he left it for me to look at later. When I got my hands on it, I tried resetting the SMC and that worked fine, but it still wouldn't turn on. Next up I went to test each power rail to see if they were all reading what they should, so I disassembled it to have the bare motherboard on my desk. I pulled up the schematic pdf and the boardview on my computer to find where to test each rail, then hooked it up to the adapter. The first one I went to check was PPBUS_G3H, one of the more important rails that is always on whenever the computer is on power, and noticed that the area where I was touching was hot! I felt around a little bit and was almost burned by the problem component! It was a tantalum capacitor that was part of the circuitry providing the 3.3V and 5V rails power, and it had the word CRITICAL right next to it on the schematic. I immediately disconnected the adapter. When I looked closer, I noticed that the thing was even cracked! I then went into our parts closet to see if we had any donor boards, and we didn't. So I just got another MacBook Air board that seemed close in age to the one I was working on and carried it back to my desk. I pulled up the schematics for this board as well and found the page describing the same circuitry as my problem component, and there was one exactly the same! I identified it on the board, double checked the polarity of each of them, then went to the rework station and swapped them out. The swap went without a hitch, which I think is pretty cool considering I did it with just a heat gun (not even a rework station, we don't have a lot of money around here), tweezers, and a syringe of flux! I took it back to my bench, cleaned off the flux and, lo and behold, it worked perfectly! All in all, very customer, very happy tech :) Thanks for reading!

    submitted by /u/WillyWonkaTheFearful
    [link] [comments]

    Uptime correlates exclusively with the cost of your hardware

    Posted: 01 May 2018 10:22 AM PDT

    The story of management who wanted to buy expensive hardware reminded me of another story from the same company I mentioned previously. It was the 90s, and I was a software developer building a website. Although I wasn't tech support, the role still gave me ample opportunity to observe many of the foolish things that happen around tech.

    In this company, there was an IT department that was responsible for maintaining systems. Other departments paid them for setup, support, and hosting servers, but had no authority to insist on timely resolutions. The prices were high and the service was bad, but other departments had no choice but to go through internal IT.

    My manager ordered hardware to run the website. We got a shiny new Sun, despite my recommendation for Linux. Uptime was important, and you only get it with tried and true big vendors! So we got another Sun which did nothing but mirror the first, and a switch to flip over to the second if the first went down. Both the machines had RAID. Uptime was really, really critical! I suggested that if they were serious about uptime, they might want to consider getting a UPS. However, we didn't have any budget left, due to the expensive decisions that had already been made. It took several weeks to get the system set up, because of the IT department's inherent laziness.

    During the time I worked there, we never had a disk failure, and we never switched over to the backup machine. But we also never had a power outage. We did, however, have lousy uptime.

    The reason? Our internet connection was not especially reliable, and would drop a couple of times a month. Although the IT department theoretically had someone on call 24/7, nobody would pick up the phone after hours. If the internet went down at 4:00 in the afternoon, it wouldn't be back until midday the following day. Unless it was a Friday, in which case it would be back on Monday. Unless Monday was a holiday... We easily had 3+ days of downtime every month.

    TL;DR: Spending lots of money on hardware is pointless if you can't enforce an SLA for your internet connection.

    submitted by /u/while-1-sleep
    [link] [comments]

    Sometimes it's not just the clients you need to watch out for.

    Posted: 01 May 2018 06:27 AM PDT

    So this happened all of 5 minutes ago and I currently waiting on the repercussions (thankfully I work with some understanding people - or at least, I hope I do).

    Last week I was sent to complete my MTCNA (Mikrotik router accreditation), which I passed. Today has been a quiet day in the office and with not much to do, I thought I would reinforce my newly learned skills and play around with one of our old 751G's.

    I connected it up to my laptop eth0 port and up the device popped in winbox. Logged in via MAC connection and noticed there was an old config on there. Never to mind, I thought. Lets wipe the config...

    It resets, I connect back in and go to view the wlan0 interface to configure it to our office WiFi so I can have internet while I work on the router, but there is no wlan0 interface. Odd, I thought, but whatever. I keep poking around, maybe it is just not showing up since I set it not to get default config. I check the full interface list. For some reason there are 24 eth interfaces and 2 SFP interfaces. wtf? so weird. Why does this little 5 port router think that it's a cloud router. Then it hits me: This is the office router, I just wiped it completely clean. No config export. No backup. Nothing. Fuck.

    I have since messaged my manager and boss and they are not super happy about it, but I think they will not blow up too much at me. They are level headed people.

    Looking at it, I can see where I went wrong. I had my eth0 interface on my laptop disabled and thereofre never saw the connected 751G and defaulted through the wireless network to our cloud router. I just happened to guess the right password for it and didn't check the MAC address was matching before wiping it.

    submitted by /u/Stoned420Man
    [link] [comments]

    The Tech Support Aura works on everything!

    Posted: 01 May 2018 02:08 PM PDT

    LTL, FTP, HTTPS (Helping To Transport Poo Scoops)

    I'm a school tech, so most of my support calls are along the lines of "I didn't do anything [other than try to circumvent the filter] and it broke!" Outside work, I do a little bit of everything.

    Today, I got a call that a user's peripherals weren't powering up. These peripherals are large and agricultural, but they follow the same rules as everything else. There's two, but only one port, so I asked him if he made sure the correct cable was plugged in on both ends and it was turned on. He said he had tried that FIVE TIMES and nothing worked. I plugged in the baler, and it worked, then the planter, and it worked. I rebooted a hundred horsepower of $Kumquat to test.

    He blames "magic" but may have skipped some troubleshooting steps when the new tractor didn't do what he wanted.

    The user is my father. People assume that I can fix anything with wires, and I let a few of them get away with that. Right now, I'm going to fix a track light that isn't lighting. Six credits of Electrical Engineering should come in handy :)

    submitted by /u/trainbrain27
    [link] [comments]

    No comments:

    Post a Comment

    Fashion

    Beauty

    Travel