• Breaking News

    [Android][timeline][#f39c12]

    Tuesday, April 17, 2018

    So our server was hacked by the mailman. Tech Support

    So our server was hacked by the mailman. Tech Support


    So our server was hacked by the mailman.

    Posted: 16 Apr 2018 12:13 PM PDT

    This just happened about an hour ago and is also my first time posting here.

    I own a small MSP in Georgia. At one point in my life I was a pretty decent technician but these days my job is mostly shaking hands. I try to work a ticket or two every day though just to keep in shape so I can talk intelligently. Anyway to get to the story.....

    Today one of our system monitors alerted us to excessive login failures at one of our largest customers. This is an alert that is set up to let us know if someone has failed to log in successfully several times and is designed to gives us a heads up if there is a brute force attack happening. We have the threshold set pretty low and we get one alert a week just on the shared computers usually. But this alert was on a faxserver at one of their smaller remote locations. No users typically are at the fax servers so I decided to go ahead and investigate. I fired up screenconnect and was greeted by the windows login welcome screen just spinning. After a few seconds it hit the password authentication window but almost instantly blinked out of it and was trying to log in again. RED FLAGS immediately! I watched for another 30 seconds or so and saw it hit the login screen again and fail password check 3 more times again almost instantly! Clearly this was some sort of bot trying to brute force its way into the system. This is a pretty secure system as things go and we take things like this incredibly seriously. I am trying to rack my brain and figure out where an attack like this would even come from and why it would be hitting this server which is much less exposed than a lot of other things on the network.

    I grabbed two of my senior techs real quick and put them on the case to try and figure out what was happening and where this was coming from. We didn't want to log into the system because it might have a keylogger going and we didn't know what the situation was so we were pushing out commands on the backend through Labtech. Everything kept getting weirder and weirder. We chased down some suspicious processes with open connections, found something talking to amazon ec2..... something talking to azure......but we were able to determine with some effort that those were benign. We couldn't find an outside source hitting this machine in the firewall or through the switch. So one of my techs said, "Maybe it has something already on it trying to brute force itself that will phone home once it gets a domain login???"

    So we decided to isolate the machine on the network to test this theory. Sure enough the attack continued even with no communication from the outside. It didn't make a lot of sense though..... if the machine was already compromised there are better ways to get passwords? Maybe this is an amateur attempt? So we start looking for rogue processes. Not much is really running on it and everything looks pretty standard. Regardless though something is causing this so we start terminating whatever looks like the most likely offenders. No luck, every 30 seconds 3 failed login attempts about as fast as you can blink. Eventually we are digging deep and killing svchosts. Nothing is working. So we deploy a tech to go pick up the server and bring it back to the shop and get it off their network. In the meantime I call management and let them know we are seeing an attack on their network and we are investigating.

    This place is only a few minutes away, but as the tech is driving over the attacks suddenly stop. One of the processes we had killed had stopped it. My tech thinks ESET was the last thing he killed. Maybe we have a compromised ESET process???? How would that even happen??? <panic sets in> Maybe we have a compromised ESET server??? I play through in my head the thousand machines we have running ESET and start calling my deployment tech (who was sick in the hospital today god love him) and start asking him if he had changed anything with deployment and when the last time we rebooted the ESET instance was. I am pretty close to a full on freak out at this point. My tech goes ahead and reboots the server to see if the assault continues. After the reboot though it was quiet. We pushed out a temporary admin account and new password and went ahead and logged into the box to start poking around. We dug into the event viewer security logs to see what was going on and started to see all of the audit failures. Weird thing though, they were all trying our admin account and they were all coming from the local machine???

    If you have ever seen this kind of attack normally what you find here is a bunch of common names and account names being tried from various overseas IP addresses. You will see several logins under "john" and "chris" and "root" and "admin" and "local" etc and normally it would not come from the local machine. If you already have malware running on the local machine there are a million better less obvious ways to collect passwords.

    The server had just come back up when my technician got into the remote office. As he walked in, the front desk receptionist said: "hey when you get done with whatever you are here for this machine next to me keeps beeping at me". she waves at the fax server My technician walked up to the fax server, picked up a catalog off of the enter key and then promptly called back to let us know that we are all a bunch of morons.

    submitted by /u/just_some_random_dud
    [link] [comments]

    Yeah, it's on fire (Part 2: The bomb drops)

    Posted: 16 Apr 2018 11:17 PM PDT

    Part 1

    Hi there, TFTS. You can stop harassing me now. This story happened when I worked in a B2B Internet Service Provider as my first real job, a lowly L1 Helldesk, way back in the beginning of this decade.

    As a little more background before I start, $ISP is a multinational company, although our specializing in B2B market means that we are a small player in this country., certainly much smaller than $OtherVendor. $Customer is also a multinational, billion-dollar business with branches all over the globe, they've been using $ISP's services for a couple of years now, before that they exclusively use $OtherVendor.

    it's now Monday morning in $ISP and I'm in my 13th hour working right after night shift. Showering, fresh clothes, and double time certainly perked me up though, as well as the fresh cup of builder's tea I have on my desk.

    $L3 arrives at the Helldesk bullpen

    $L3: Hey $Me, where's that router?

    (I pointed to the charred router beside our spare pile.)

    $L3: Holy cow that looks ugly. To think that $ISPSales asked me whether replacing it was even necessary...

    $Me: Technically it does work, but I think it's best to retire the Terminator from service.

    $L3: Terminator, huh? I can kinda see that. Anyway, you ready? They are already waiting for us.

    $Me: Here or $Site?

    $L3: Neither. We're having a brunch at a nearby restaurant

    So $Me, $L3, and $ISPManager arrived at the restaurant, where $Customer is waiting. $CustTech is there, as I expected, with $CustManager. I didn't know the other person sitting with them.

    $L3: Hi guys, this is $Me, He was the onsite engineer last Saturday.

    $Me: Hi there, good to meet you.

    $CustTech: Hi $L3, $Me, you know $CustManager, and this is $CustDirector, our Director of Operations (!).

    I've met $CustManager and I've seen $CustDirector in email communications. But for him to come? This has to be a big deal. No wonder $ISPManager joined.

    $L3: Thanks for coming here, $CustTech, $CustManager, and $CustDirector. Everyone ordered something yet? $Me, the coffee's amazing here...

    . . . . .

    $L3: This is a bit unorthodox, but I think it's best for $CustTech to explain this meeting's objective to everyone, just to make sure nothing is lost in communication.

    $CustTech: Right, thanks for having us here. I talked a bit about this with $L3 earlier and called this meeting as a follow up to last Saturday's problem.

    $ISPManager: The fire incident? Is there anything we missed?

    $CustManager: No, more like we're missing someone entirely.

    $ISPManager: Not exactly sure what you guys are talking about.

    $CustManager: Let's ask $Me, Did you see $OtherVendor's tech last Saturday?

    $Me: Not that I'll recognize one, but I don't think so. I was the only one working there aside from $CustTech. Are we on the same SLA? (we have 4 hour to recovery Service Level Agreement for $Customer, outside of force majeure)

    $CustTech: Yes, and since the downtime occured simultaneously. You guys should have arrived around the same time. Do you want to guess when $OtherVendor engineer comes on site?

    $CustManager: This afternoon, if we're lucky.

    $ISPEmployees: !

    $CustDirector: That's after setting our rack on fire. And it's not an isolated incident, either. We have a 99.9% uptime guarantee, but they're barely making the second nine.

    $ISPManager: What about your contract?

    $CustManager: They discounted some of our bills after a LOT of pressure, but it's not something we want to do every month.

    $ISPManager: I see... I understand your predicament, but I'm not sure on how we can help you with another company's SLA

    ($L3 grins)

    $CustManager: $CustTech, how many links do we have with $ISP?

    $CustTech: 22 dedicated for our offices, about 40 sprinkled in some of our sites

    $CustManager: How about $OtherVendor?

    $CustTech: 22 dedicated, like $ISP, and about 200 in our sites.

    $CustDirector: We're thinking of migrating all of them to you.

    This is a bombshell. As it is, $Customer is already one of our biggest customer. If they move all of $OtherVendor's site to $ISP, this will make them our biggest customer by a large margin.

    $ISPManager: That's a bombshell. Should I call $ISPSales here?

    $CustDirector: Please don't, let's keep it between us for now. We invited you guys because we need some people to bounce ideas with, to know if it's viable. That's also why this brunch is on us.

    $ISPManager: (leaning back on his chair) We... might... be able to handle all of the small sites, should not be too much of a problem…

    $L3: Your 22 offices, though. I don't think that's a very good idea.

    $CustDirector: Why is that?

    $L3: If we took over your whole network, you will be completely reliant on us. We are proud of our infrastructure, but problems do happen.

    $CustDirector: What do you propose?

    $ISPManager: We can subcontract another ISP and manage their SLA. But if you want to use them directly, we really don't mind.

    I was personally in awe at the discussion. I can troubleshoot most network problems pretty readily. But $L3 and $ISPManager basically redesigned $Customer's network on the back of a napkin that morning.

    Being a lowly L1 operations tech, however, I was not invited to the next few meetings. But from what I heard from $L3, $Customer finally decided to move all of their connections to us, making them our biggest customer.

    $Customer's problem with $OtherVendor has been ongoing for a long time. The fire incident turned out to be the straw that broke the camel's back. Reliability is only one part of the equation. Apparently, our prices are also noticeably cheaper compared to $OtherVendor, which makes the decision all the more popular.

    The Terminator? It had its own place behind $ISPManager's desk for a while. Last thing I know another tech brought it back home for their personal lab. Pity I left the company before they gave it away.

    submitted by /u/really-likes-tea
    [link] [comments]

    Of Sheep and machine

    Posted: 16 Apr 2018 08:35 AM PDT

    Do you like to read in Chronological order? Here is the Index

     

    $Selben: Me! "Technical team lead" previously Tier II helldesk helpdesk technician for a mid-sized company, very skeleton-crew helpdesk 10 of us total for 24 hour coverage (not including supervisors) to support 2500+ company-wide.

    $Ditzy: Front desk administrator (Computers are an obstacle, not a tool)

    $Peer: Random Tier1 technician. (Tier 1 worked base calls and sorted them to other groups, only basic troubleshooting)

    $Sup2: All around great supervisor, worked his way up from the support line, understands how the helpdesk works.

     

    $Selben crossed the parking-lot, entering the building to be pleasantly greeted by $Ditzy, and slightly blinded by both her smile and the reflection of the sun from her shiny new desk. Followed by a grunt from $Peer who was working on her machine with his forehead furrowed. Deciding it was best to let $Peer try to sort it out, he continued to his desk which still smelled like onions and mustard. $Snickers had been given some time off as well from recent events, specifically involving a sandwich.

    It was not long before $Peer came around the corner with shoulders lowered, $Selben let out a sigh.

    $Selben: What's up?

    $Peer: I don't know, $Ditzy's machine wont power up.

    $Selben: Power light?

    $Peer: Nope.

    $Selben: Power cable?

    $Peer: Tested.

    $Selben: Ask hardware?

    $Peer: He said to ask you.

    $Selben: … Any beeps or anything?

    $Peer: Nope.

    $Selben got up and headed over to $Ditzy's desk, the machine was 100% dead. They got her a loaner machine and ordered a replacement, ticket closed. $Selben attempted to return to his normal day, but after a couple hours he got a call from $Sup2.

    $Sup2: Hey, $Ditzy's machine wont power on - can you take a look at it?

    $Selben: I already did, this morning with $Peer.

    $Sup2: Her loaner is doing the same thing.

    $Selben: Huh… Okay, I'll check it out.

    Yet again, same symptoms no power dead machine. $Selben replaced the surge protector and had $Ditzy walk through what she did. Nothing seemed out of the ordinary. She would get up and the machine would just die, they got her another loaner and chalked it up to evil I.T. spirits and all seemed well for the rest of the day.

     

    A few days later $Selben was crossing the parking-lot and rubbed his hands together to keep warm, it was pretty cold. $Ditzy stood at her desk with a frown - apparently the loaner had died as soon as she came in. $Selben and $Sup2 found her new machine was ready so they brought it out to her, naturally she had gotten her manager involved and everyone was upset. $Selben handed her the machine, she set it on the desk and slid forward then reached for the power button. $Selben saw a massive arcing static bolt leap the last inch or so from her finger to the computer.

    $Selben: Well, this ones probably dead now too…

    $Sup2: What?!

    $Ditzy repeatedly pushed the power button, but again nothing.

    $Ditzy: HOW?!

    The desk that $Ditzy had recently received was one of those bent acrylic see through desks. As it was cold she had been wearing wool leggings and a wool skirt which resulted in massive static buildup. She had fried every machine she touched - she was advised to not wear wool skirts if possible and the acrylic desk was removed.

    submitted by /u/Selben
    [link] [comments]

    My Husband is Connected, Therefore, So Am I

    Posted: 16 Apr 2018 09:17 AM PDT

    Tech: Thank you for calling XYZ Helpdesk. Can I have your employee ID number please.
    collect initial info, user can't connect to VPN from home
    Tech: Can you try browsing to google.com and see if that page loads?

    User: No, it says Page can't be Displayed.

    Tech: Can you check your VPN client and tell me the status of the client? Is it connected, connecting, or is there a disconnect button?

    User: It says "connecting," and it's just spinning.

    Tech: OK, please cancel the connection.

    User: OK, done.

    Tech: Now, try to go to google.com.

    User: It says Page can't be Displayed.

    Tech: It appears to be a problem with your home internet and not your VPN. Let's...

    User: No. My home internet is working just fine because my husband is connected.

    Tech: OK, then it's a connection between your computer and your home's internet.

    User: That doesn't sound right. Are you sure you know what you're doing? Is there someone more knowledgeable I could speak to? Obviously, my husband is connected - so it's not my home's internet, and you obviously can't help me.

    Tech: Let's work on this together - how do you connect to the Internet?

    User: audible sigh If you don't even know that, then you obviously don't know what you're doing.

    Tech: Wirelessly or wired?

    User: Wirelessly.

    Tech: What's the name of the network you connect to?

    User: ABCDEFGH

    Tech: Can you please make sure you're connected to ABCDEFGH?

    User: Of course I am.

    Tech: Did you check and make sure?

    User: How do I do that?

    Tech: Do you see the wireless signal strength next to your clock at the bottom right? It looks like cell phone reception.

    User: OK, I clicked it and it's asking for a password.

    Tech: Yes, that's the password to your ABCDEFGH connection.

    User: OK, I put in the password.

    Tech: Now can you go to google.com?

    User: Yes, it's working now.

    Tech: Can you now connect to the VPN.

    User: It's asking for a username and password.

    Tech: Yes, that's your XYZ username and password.

    User: It says my credentials are invalid.

    Tech: What username did you put in?

    User: User@yahoo.com

    Tech: You'll need to put in your XYZ credentials.

    User: OK, it says I'm connected now.
    click

    submitted by /u/megamanxtc
    [link] [comments]

    Dude, I think our problem is solved

    Posted: 16 Apr 2018 10:34 AM PDT

    Another post about fire reminded me of this little gem.
    I used to work in phone support for a company that makes tools for TV and filmmakers. We used to make drive arrays including one that had to have some internal settings made via serial connection before it was used. Customer is a small post house in Colorado somewhere that had bought 4x 5 drive arrays, each unit of the 4 needed to be individually configured before use. As a side note this was the very first support call ever on that array.

    Of course the customer hadn't configured the drives and it wasn't long before the data went corrupt. not lost mind you, all the file names were there but nothing could be read. I spent the whole dang day on the phone with them trying this and that even though I was pretty sure the situation was hopeless.

    The owners of the post house were horrible people, they very rarely spoke, everything was yelled. The editor was a really nice guy, I have no idea why he worked for the jerks. He was a bit of a stoner and I'll never forget the following exchange:

    Editor: Dude, I think our problem is solved.
    Me: Why, what happened?
    Editor: The array is on fire.
    He was so calm when he said it, he might as well have said "My shoe is untied."
    Me: (panic rising) Well shut it off!
    Editor: Not yet.
    Me: Why not?
    Editor: Wait for it.

    Long pause

    Editor: Okay, they're all on fire now, I'll shut them off.

    In his brilliance the editor realized that if all 4 arrays were on fire there would be no chance any data could be recovered and we could finally quit trying.

    Me: I'll go ahead and process your RMA, can I call you Monday to set this one up correctly?

    It turned out there was a design defect in the controller board in the array which would cause every single one of them to catch fire. Eventually the board was redesigned and the arrays were actually pretty good but I'll never forget that calm voice "The array is on fire."

    submitted by /u/curtludwig
    [link] [comments]

    Tales from the Baby Bell: A Very Manic Sunday

    Posted: 16 Apr 2018 07:49 PM PDT

    CAST OF CHARACTERS
    * Me - obviously
    * Supervisor - mine.
    * Erin - A junior member of tier 2.
    * Earl - Member of tier 3.
    * Sam - Earl's boss.
    * Joe - Property manager of University Dorms.


    Our heroine, a lovely young-at-heart woman with freshly dyed blonde hair and naturally blue eyes stolls across the office to her desk. Halfway there she is intercepted by the new supervisor, a giant bear of a man with dark hair with dark eyes and a slight russian accent.

    Him: Hey you!!!

    Startled she freezes.

    Her: What did I do?
    Supervisor: Do you do hugs?
    Her: Umm...sometimes? Is something wrong?

    He envelopes her in a giant -- but gentle -- hug

    record scratch
    FREEZE FRAME!

    Yup, That's me! You're probably wondering how I got here. Well, I'll tell ya. It all started yesterday when...

    Time rolls back 24 hours and the beginning of the scene repeats itself, but this time in an office with only two other people in it. Waving at our inventory guy, I clocked in and then headed over to my coworker's desk.

    Me: Hey Erin, anything going on today?
    Erin: It's a Sunday. Not much ever happens on a Sunday.
    Me: knocks on the wooden desk Please. Don't jinx it!
    Erin (smiling at my tired joke): We've got a video ticket from Homes on the Hill. Guy says all of his channels are out. I've reset his ONT but it didn't make any difference. Sorta wondering if I should contact tier 3.
    Me: Did you check the controller?
    Erin: Yeah. $HDController shows no errors.
    Me: What about the $SDController?
    Erin:Ummm....
    Me: Gotta check 'em all.

    Controller: Error 404: Controller Not Found.

    Me: Well that right there is worth emailing tier 3 about.
    Erin: Is it an outage?
    Me: I don't know. Maybe. It could be that our GUI to the controller is down but the controller itself is still working. Let me take a look at the ONTs

    An ONT is a piece of hardware that funnels video and internet into a person's apartment. They are linked together on a GPON which can be remotely manipulated through a GUI that we don't have a name for. All I know is that it is a ruddy bloody PITA to log into, and if you leave it alone for more than five minutes it will log you out. Welcome to my world.

    The first thing I noticed when the GUI (finally) loaded was a sea of yellow alerts. The second thing I noticed was that every single ONT had a "Video LOS" error on it. LOS stands for "Loss of signal". No signal means no cable.

    Me: We've only gotten one ticket from this property?
    Erin: yeah.
    Me: Strange.
    Erin: So is it an outage?
    Me (long pause): Yeah. This is an outage. I'll start the problem ticket if you do the email.

    By now I've created so many problem tickets that I can do them in my sleep and this one was knocked out in about in 2 minutes. I pass the link over to Erin who is laborously typing up what sounds like a thesis but ends up being two sentences.

    From: Erin To: Tier 3 CC: Admins Subject: Video Outage at Homes on the Hill. We have what appears to be a video outage at The Apartments on the Hill. The Standard Def controller is 404ing and there are LOS error on all ONTs. 

    That was a perfectly acceptable message, minus one little thing: screencaps. The unofficial First Rule of Tier 3 is "Pics or it didn't happen." I was in the process of forwarding Tier 3 the pics I'd attached to the problem ticket when the their reply hit my inbox.

    From: Earl To: Erin, Tier 3 CC: Admins Subject: Re: Video Outage at Homes on the Hill There is only one incident attached to that problem ticket. What makes you think this is an outage? 

    Inner me: Really dude? You noticed that but missed the part about the error messages?

    From: simAlity To: Earl, Erin, Tier 3 CC: Admins Subject: Re: Video Outage at Homes on the Hill All ONTs have LOS errors and the $SDController is offline. Please see the attached screencaps. 

    That was enough to get Earl to pause his game or whatever it was he was doing and take us seriously. Twenty minutes later we got another response:

     From: Earl To: simAlity, Erin, Tier 3 CC: Admins Subject: Re: Video Outage at Homes on the Hill I've powercycled the $SDController. It should be back online. The transponder was offline. I've turned it back on. Screencap attached. 

    After confirming that the SD controller was indeed online, I logged back into the ONT server just in time to watch the LOS messages disappear...and then return. Then they disappeared again...and returned again.

    From: simAlity To: Earl, Erin, Tier 3 CC: Admins Subject: Re: Video Outage at Homes on the Hill SD controller's back online but the Video LOS errors remains. Whatever you did fixed it for a skinny minute but now it is broken again. From: Earl To: simAlity, Erin, Tier 3 CC: Admins, Dispatch Subject: Re: Video Outage at Homes on the Hill The transponder won't stay on. We'll have to get a tech to look at it tomorrow. Passing ticket to dispatch. 

    Sometimes all you can do is all you can do. After adding few more notes about the issue I informed our tier 1 so that they wouldn't send up any more tickets and closed the tab.

    Thirty minutes later the sound of breaking glass filled my ears. A service alert had arrived.

    A service alert is issued anytime anything from a router to an access point goes offline. We get dozens of them every day and it can be easy to tune them out. One of the first things I did after starting at the Baby Bell was to program my email to play a sound effect whenever we got one about a gateway or a router. Even then I have to change the sound effect every other month. This month's effect, BreakingGlass.wav is especially appropriate.

    This service alert announced that a router, two gateways and every frakkin' switch at The Cabins in the Woods had fallen offline. This is known as a site outage.

    Erin: We've lost the Cabins.
    Me: Yeah I see it.
    Erin (pause): Weather is good. No power outages in the area.
    Me (typing one thing and reading another): Equipment is down hard though. I'll call the property. Maybe someone is in the office. Erin: On a Sunday? Me: You never know.

    Erin's pessemism was well placed. Even though the office wasn't supposed to close for another hour, nobody answered the phone. Erin called the carrier and learned that a fiber cut had taken out service to most of the town.

    Erin: Do you need me to stay late?
    Me: Nah, I'm good. See you tomorrow.

    For the next little bit all was quiet on the networking front. I cleared out the ticket queue and took my lunchbreak. When I returned there was a message from tier 1 on my screen.

    Tier1: simAlity? We're getting a lot of calls from The Apartments on the Corner. They're saying that the cable is out. Can you take a look? Me: of course. One sec.

    Unlike Homes on the Hill, The Apartments on the Corner has one controller for all types of channels. Loading it up, I again find myself in a sea of errors. But at least these are things I can fix.

    Me (to tier 1): Confirmed. 40 channels off the air. Working to fix it now. ETR, one hour. If we don't have a problem ticket already, go ahead and create one.
    Tier1: Aye-aye ma'am!

    The next 45 minutes was spent rebooting components and services. While this may sound impressive it mostly involved a bunch of button pushing. The hardest part was sitting on my hands and waiting for the reboots to finish. But finally they did and I had the satisfaction of seeing all channels back online.

    BreakingGlass.wav

    This service alert actually brought good news. The Cabins were back online! Well...mostly online. A brief but through examination of the wifi controller showed a lot of bad connections. This isn't uncommon immediately after a site outage. After monitoring things for fifteen minutes and watching it go from bad to worse I decide that a tier 3 intervention will be required.

    Me (to tier 1): Be aware (if you aren't already) that we have a wifi issue at Cabin in the Woods. We're working on it now.
    Tier 1 Supervisor: Understood. Could you also take a look at The University Dorms?
    Me: Of course. What seems to be the problem?
    Tier 1 Supervisor: They're reporting that the Internet just went out. Me: How many calls?
    Tier 1 Supervisor: Five in the past 15 minutes.
    Me: crap. Let me escalate this Cabins issue and then I'll take a look at the University.

    From: simAlity To: Tier 3 CC: Admins Subject: Wifi Connectivity issue at Cabins In the Woods Cabins in the Woods just came back up after a three hour outage. The wifi controller is having trouble stabilizing. Two hundred out of six hundred devices with no or invalid IP address. Please check and advise. Attached: Screencap.png 

    As I was typing a another service alert arrived.

    Whatever that is, is going to have to wait. I thought as I opened the service alert. It was an empty threat but when you have as much crap going on as I often do, empty threats are a necessary part of keeping your sanity.

    As it happened the newest Service Alert simply reinforced the Tier 1 supervisor's message to me. All the network equipment at University Dorms (a 800 bed student housing complex) was offline. Fun!

    After creating the problem ticket I called the property and left a message on the emergency maintenance line. 99% of the time this is an exercise in futility. Ten minutes later my GM line rang and on the other side was a member of the 1%.

    Me: Thank you for calling the BabyBell. How may I help you?
    Caller: yes, this is Joe, property manager of the University Dorms. I'm down in our server room with a laptop. What can I do to get things back online?
    Me: Fantastic! Okay, first thing we need to do is find the router. It should be labeled r1 Joe: Found it.
    Me: Does it have any lights.
    Joe (sounding surprised): Actually, no. It doesn't.
    Me: Okay. Go around to the back of the rack and find the power cord for the r1. Joe: Done.
    Me: Follow the powercord back to where it plugs in.
    **Joe:
    Okay. It appears to be plugged into some sort of generator thing.
    Me: That's probably the UPS. Does it have lights on it?
    Joe: I see the power light is on but the status light is orange. That's probably a bad sign.
    Me Yeah you can say that. Did y'all have a power outage sometime this evening?
    Joe: Yeah. It went off and came back on. That was 90 minutes ago though. We lost Internet no more than 20 minutes ago. Maybe 30, max.
    Me: Something must be preventing the UPS from going back to accepting power from the outlet. Could be a blown fuse. I'm assuming the lights in the MDF are on, right?
    Joe: Of course.
    Me: Then probably not a bad outlet or a blown fuse. I mean, I guess it could be a blown fuse inside the UPS but that's a bit above my head. So, what I want you to do is unplug the r1 from the UPS and plug it into a wall outlet.
    Joe: What about the rest of the stuff plugged into the UPS?
    Me: Let's take it one thing at a time.
    Joe: Looks like I need to clear a path between the rack and the outlet. This will take a couple of minutes.
    Me: Take your time.

    Against the background sounds of him moving what sounded like bunch of bowling balls, I checked my email. There was one new message.

    From: Earl To: simAlity, Tier 3 CC: Admins Subject: Wifi Connectivity issue at Cabins In the Woods. All devices are connected with good IP addresses. Problem must have resolved itself. In the future please wait ten minutes before contacting tier 3. 

    Gritting my teeth against a strong desire to tell him what I thought of his instructions I pulled up the wifi controller for the Cabins. Twenty seconds later I pound out a reply.

    From: simAlity, To: Earl, Tier 3 CC: Admins Subject: Wifi Connectivity issue at Cabins In the Woods Half the access points are offline. 

    Joe: I can hear you typing all the way over here. Everything alright?
    Me (keeping the bite out of my voice with effort): Yup. Just dealing with another situation.
    Joe: Well I'm about to plug the router into this wall outlet. If this thing electricutes me, you can have my big screen TV.
    Me (ever aware that all calls are monitored): If you don't feel safe...
    Joe: I'm joking!
    Me: Just making sure.
    Joe: Okay I see a lot more lights on the router. They're flashing. Actually a LOT of lights are flashing.
    Me: Excellent! This means we are on the right track. Next you want to find the gateway. It should be right under or over the router.
    Joe: Found it. No Lights. It also appears to be plugged into the UPS thing. Want me to move the power cord?
    Me: Got it in one.

    We did this routine with the core switch and the wifi controller. Finally everything that had been plugged into the UPS was plugged into a wall outlet.

    Me: Give me a minute to check system status. one minute later Gateway is up, wifi controller is up. Switches are up.... Looks like we're back in business. Are you able to connect to the Internet on your phone?
    Joe: I am indeed.

    After a few closing remarks Joe hung up a happy man. Whew!

    After closing the half-dozen tabs that I had opened over the course of Joe's call and then checked the Cabin's WiFi Controller. All access points were back online. In my email was a message from Earl's boss, Sam.

    From: Sam To: Earl, simAlity, Tier 3 CC: Admins Subject: Wifi Connectivity issue at Cabins In the Woods. Restarted the wifi controller. Monitoring the DHCP service to make sure it doesn't overload again. Please let me know if you need anything else. 

    I've worked here too long to believe that Sam's intervention means that he will have a chat with Earl and Earl will apologize for his "mistake". But I do appreciate the intervention none the less.

    The next couple of hours were spent staring at the ceiling hoping against hope that nothing else would go wrong before I went home. My hope was in vain. Half an hour before quitting time (I kid you not) another service alert arrived.

    This one came from The View at The Park (TVATP). Obviously that's not the real name but it is a reasonable approximation. Where apartment complexes come up with the names they use is beyond me. Anyway TVATP is a medium size complex about a hundred miles from the far side of the back of the beyond. It is so far in the boonies that none of the major ISP or carriers service the area and we have to use a little basement business carrier. As in, one guy runs this little ISP like a one-man-band. We'll call this place Boonies Online (BOL). After calling the property (no joy, phones were down) I gave BOL a call and got an answer I will never forget.

    BOL: Hello? Me: Hi. Not sure I have the right person. But this is simAlity calling from the Baby--
    BOL: Look I'm driving through a fking blizzard taking a generator to my data center. I'll get service back up as soon as I can.
    Me: Okay. Umm.... Text me when it is back up?
    BOL: Yeah. Sure. Crap! Gotta drive. click

    I stared at my phone for 30 seconds after he hung up and then burst out laughing like a mad woman. Clearly I wasn't the only one having a craptastic day at work.

    After writing up the call, the ticket, and sending a message to tier 3 about the ongoing outage it was finally -- FINALLY -- time to go home.

    Which brings us back to the present day.

    Supervisor: You did GREAT! I kept seeing those messages coming in and wondering if I should get you some backup but you handled it like a boss!
    Me (flustered): I wouldn't be much of a tech if I couldn't handle a few curve-balls.
    Supervisor: If you say so.

    Buoyed by the (extremely rare) compliment I walked the rest of the way to my desk feeling like I was walking on cloud 9. I was so happy I almost didn't notice the person sitting in the spare chair near my cube until he stood up to greet me.

    Me: Hi Sam! What can I do for you?
    Sam: I was wondering if you would be interested in joining tier 3?


    More stories from me.

    submitted by /u/simAlity
    [link] [comments]

    The Graphics 'Card'

    Posted: 16 Apr 2018 10:31 AM PDT

    This one is from last year and I've just remembered it.

    We had a customer moving offices and me and another colleague were asked to help them. The customer had vans so took their own PCs, monitors, keyboards etc. over to the new office themselves. We went to the old office to get their server, UPS and any switches that were there.

    We got to the new office, got the server connected, flood wired the building so all the network ports were live and then went about setting up everyone's PCs. I set one guy's machine up and was working on a machine opposite him on another desk. Then the guy whose PC I set up piped up:

    $User "Can you set me up with two monitors? I'd like two monitors instead of one."

    I went to have a look at the connections on the back of his PC and there was only one VGA port.

    $Me "Sorry, you've only got one connection on the back of your PC, I'll have to send your machine model to our orders department so we can order a graphics card for it."
    $User "Do you not have any on you?"
    $Me "No, I don't just carry graphics cards around with me, we'll have to order one that will fit into your PC."
    $User "What do you mean you don't carry them around with you? You should have everything you need with you to complete the job you're on."

    I thought he was just messing around at this point so just smiled and nodded, acknowledged that he was joking and carried on setting up the other PC. I looked up after afew seconds and he was giving me the death stare.

    $Me "You aren't joking are you?"
    $User "No. Why haven't you got a graphics card on you?"
    $Me "I can't carry around every single peripheral that could fit into a PC. I've only got a Ford Focus, I'd never fit everything in."
    $User "Well you should have a works van then."
    $Me "Tell that to my boss, I use my own car for my job. I can't have the whole of PC World in it all the time."
    $User "This is ridiculous. You should have these things on you all the time."
    $Me "If you asked us before we set off then we could've brought one and got it working for you."
    $User "Why? They're only little cards."
    $Me "What do you mean? They need to be installed into your PC. There are different connections inside the PCs and depending what connections are on the monitors we need to make sure we get one that will fit and also the leads to connect the screens up."

    This guy then gets his wallet out and puts a credit card on his desk and explains to me that the graphics cards are only this big and you just plug them into your current monitor to add another on to it.

    I told him that's not how it works and we'll order one in and come out when we've got one.

    Got back to the office and told my boss about this guy being a pain in the arse. My boss spoke to his boss and she said "yeah, we've had a few complaints about User before and him being rude to people.

    A few weeks later we had a ticket come in from this customer saying to remove User because he'd been let go.

    submitted by /u/Sprowt
    [link] [comments]

    Here is your PC mr impatient

    Posted: 16 Apr 2018 05:10 PM PDT

    Back in time a few centuries (well, technically around 1998) I worked as a desktop support tech for a company.

    Some background: A manger of another department wandered over last minute on Friday to tell me that they had a new hire starting on Monday afternoon. Some new 20 year old "fantastic manager" hire had been hired away (from a Pizza hut) to be a new manager of a department in a company doing "customer response management".

    I put it on my schedule to image the new guys PC on Monday morning. Plenty of time in spite of the short notice.

    In Monday morning I get told by my manager that me and the other guy I mostly worked with (Mike C, the SysAdmin) were going to drive out immediately to a city 6 hours away (each way) to go install more RAM in a dozen PC's we co located in a client office.

    We go out, install the memory, test the PC's, and all working. Have lunch, drive back.

    At this point, my shift had already ended when we got back. I go upstairs, read some emails before taking off for the day (salary position).

    There new guy wanders into IT, finds my desk."Hey, I did not get my PC today. I need you to give me one NOW. I HATE working at a desk that is not my own!"

    I told him that my shift was over. He did not care. I promised it first thing in the morning. He says "No." And threatened to go to my manager.

    I told him his new PC had not been delivered yet (we were going through an equipment refresh, from using Pentium 1 - 200's to brand new P3's.)

    He insisted I needed to give him a PC NOW or I would be "out on my a$$" and unemployed.

    I told him I would need to give him a temporary PC until the new one was delivered. He settled down. No thank you. Just an entitled expression with a smug attitude.

    I dropped an old Pentium 1 - 60 with minimum RAM and a 200 Mb hard drive on his desk.

    He whined about it. He waited until batch 3 to get his new one for being a jerk.

    (The guy didn't last 2 months at the company.)

    submitted by /u/seanarthurmachado
    [link] [comments]

    Yeah, it's on fire (Part 1)

    Posted: 16 Apr 2018 04:26 AM PDT

    Hi there, TFTS. This story happened a long time ago, I didn't remember the exact dialogue and everything, but the gist of it should be accurate.

    I used to work in a B2B Internet Service Provider as my first real job, a lowly L1 Helldesk, way back in the beginning of this decade.

    I arrived in the office for my Saturday morning shift, expecting a quiet day so I can catch up on my reading. No sooner than after I finished making my morning cup of tea, the dreaded phone rang.

    $Me: $ISP, good morning

    $CustTech: Hi, $Customer at $Site here… uh… Not sure how to say this, but your router caught fire…

    $Me: !

    $CustTech: Yeah, we shut down the power to the rack just now, you should see it soon.

    Sure enough, one tiny dot turned red in our 1-minute-timeout NMS. Ruining the beautiful sea of green and, with it, my day. This is a premium level customer, all troubles goes up the chain to Director level.

    $Me: Yep, got it. I'll bring a replacement unit onsite. Is there anything else we can help you with?

    $CustTech: Not really, thanks.

    $Me: Right, see you in a few.

    I proceeded to create an incident ticket, (Eth0 on fire!) grab a spare 1841 from our pile and pulled the latest configuration backup from our system (I have complaints about this company, but thoroughness is not one of them.)

    After a cursory glance, I see their router has simple configuration, OSPF and IPSEC. Nothing I can't troubleshoot onsite (this was before my first smartphone). A quick 'wr' and I went on a 20-minute drive to $Site.

    As I enter the DC the smell of barbecued technology welcomed me. The fire's not too big and was localized to one rack - luckily a mostly empty one. Fire department didn't really do much besides documenting everything. Once finished, they went to talk with $CustTech, who signalled me to join.

    $Fireman: Is this your equipment? (Pointing to a silver box just 2U below my router)

    $Me: No, mine's the one above. That looked pretty bad.

    $Fireman: Yeah, I thought it was an internal short or overload or something, but looking at the wiring this seems to be an unsafe install

    $Me and $CustTech: !

    $Fireman: You see here? The AC input caught fire and there's nothing much left of it, but the output survived. See those exposed copper? That's a half-assed job if I've ever seen one.

    $Me: Whose equipment is that?

    $CustTech: $OtherVendor I think, that's your backup link. This will be an interesting discussion.

    They cleared us just after 12 and I had the chance to document everything, pull the charred equipment and give it a once-over on a nearby desk.

    Its plastic faceplate melted pretty badly on the blinkenlights side and I see scorch mark all over its bottom plate. That being said, It looked pretty good and I remember it didn't shut down until the power's turned off. So I plugged in a console cable and powered it up (I now know this is a really bad idea, don't worry.) Lo and behold, standard boot sequence. I even pulled the latest configuration from it.

    The router was still definitely toast, so I mounted the replacement, rearranged some cabling around, test the connection and confirm the NMS status is green, asked for customer's signature on my onsite form, and was on my way back to office on time for a late lunch.

    Back at the office, I grabbed my cup of tea (cold) and the phone rang again. $L3, an engineer dedicated solely for $Customer called and asked for an update.

    $L3: How was it at the $Site?

    $Me: Surprisingly uneventful, considering it involved a fire. Pulled latest config, mounted replacement, test, get out.

    $L3: Wait, you can pull the config? The router TURNED ON?

    $Me: grins I was surprised myself, you should see it on Monday.

    $L3: Ha! I'll put it on a pedestal somewhere. What happened anyway? What's the cause of fire?

    $Me: $OtherVendor's equipment, rectifier I think, caught fire

    $L3: $OtherVendor?

    $Me: Yea, the fireman suspects a bad wiring job.

    $L3: Huh... Look, I need to hang up now. I'll get back to you in a moment.

    $Me: Sure, make it 1 hour or so, I'm going out for a lunch.

    $L3: Okay, cool.

    After a lunch and a fresh cup of tea, I updated the ticket with more information and $L3 rang again.

    $L3: Are you available Monday at 10?

    $Me: I have Sunday night, my shift ends at 9 in the morning.

    $L3: I'll ask $Manager to approve you for double time.

    $Me: Fair enough, see you then.

    Hanging up, my only thought at that time was only how long until I can go home. Never will I thought this case will bring a great disturbance in the force.

    Part 2 is up!

    submitted by /u/really-likes-tea
    [link] [comments]

    Password resets are too complicated for me!

    Posted: 16 Apr 2018 11:26 AM PDT

    Today, as I was walking down the hallway, I was beckoned in by one of our Senior (in every sense of the word) accountants who had forgotten her password that she has used every day for the last year. I reset her password temporarily to "password" and had her enter that so that she could create her new password.

    Firstly, she had issues typing "password" and would consistently type "passowrd" even after I corrected her. She then tried several times to put in "password" as her new password. This didn't meet our complexity requirements anyways, but she still tried to type in "password" and got the "passwords do not match" error several times while attempting to do so. After every attempt, she would say something along the lines of "I don't get this." or "Why is this so complicated!". I tried to get her to pass the keyboard over to me several times, but she declined every time, saying "I've got this" when evidently she didn't.

    Finally, after about 10 attempts of inputting the wrong password or not having the new passwords match, I was finally able to get control of the keyboard from her, have her write down what the password was on a sticky note, and then type it in to create her new password. After I successfully set her new password, she says to me "This system is so stupid!".

    TL;DR: PEBKAC

    submitted by /u/derpickson
    [link] [comments]

    Legacy Support of Client’s Computer, Part III ft. Flight of the Valkyries

    Posted: 16 Apr 2018 07:03 PM PDT

    So today is day 3 of this onsite where I am trying to get this ancient DOS computer up and running so the client can reflash this processor that controls a whole production line of stuff.

    It's pouring out today and I gather up my stuff, making sure to bring a folding table and folding chair because I didn't want to spend another day on a factory floor. Once I got there we set up and I took a closer look at the PLC controller thingie (unfortunately I know nothing about this stuff, I just work on computers).

    Turns out they weren't turning the thing on properly before. The big green button saying start wasn't a tip off, until I pressed it. Then the PLC controller turned on and we probably will be able to program it. So got DOS 6.22 out because that is the only reliable installer floppies that I currently have. Got DOS installed and installed their program. Tried to work out how to get the program to connect to the processor without any success as always. Decided to hook up the client's original hard drive and have a peek through it because the last time I didn't get a good look at the files. Upon booting DOS up again and changing directory to D:, I found that everything was intact and the only issue with the drive was that the operating system wasn't loading.

    This got me thinking, could I copy this program over to the new drive... the answer was no. It didn't work copied over, however, turns out it will work from the old hard drive. This was while booted into the new hard drive loaded with DOS 6.22. Bear in mind, the old hard drive had DOS 5.0 installed so how that works, I don't know. Probably something to do with DOS just being so simple in terms of drivers and overall compatibility. So once we started the original program stored on the old hard drive, we found out that the whole issue with us connecting to the processor was down to what is called a KT address. Essentially, a KT address specifies where the program will be sent into the little computer containing the processor to control the process line. If you're lost by now, don't worry, I'm still lost too.

    So after a few minutes of WTF and how TF did we just do this. The employee and I flashed the program onto the processor and began hitting buttons. The PLC2 processor was located under a big board filled with buttons for each piece of machinery on the line. So the satisfying clunk as you pressed each button was, well, satisfying. I began hitting different buttons, start, then another start, and finally the start for the first process, and boom, an alarm goes off. Not sure why the alarm was triggered, but I nearly needed a new pair of pants. Turns out it was just like that without any errors.

    So the employee called over a specific job title guy, that you know, knows how the hell to run these machines without breaking them. Not the tech support guy who is giddy about fixing a very expensive machine and project. All the different machines worked and that was it, I go cold turkey about thinking about all of this and everything is solved. Well, now back to my normal everyday dealing with peoples' boring problems like Outlook isn't working or Windows won't boot.

    Overall, I gave my chances of doing this a 25% chance. I even told the clients this. So either I am a genius or very lucky. I prefer to think both. This was an amazing learning experience for me and I kinda like the balls to wall, work at the problem till it is solved sort of thing. Normally, I don't get to go all out with client's because of price or things like that. With this one, I had full approval to do anything to get it done. Wish every client was like that, but that's not the small tech support/repair company life. Maybe once I upgrade to a big corporation it will be different, but from everything I see on here, it looks like it is just more brown pants moments than anything else.

    I hope you enjoyed. This was my first real multi post experience so let me know what you all think. Learned this in part II, links for both part I and II are below:

    https://www.reddit.com/r/talesfromtechsupport/comments/8c1r9r/legacy_support_of_clients_computer_ongoing/ https://www.reddit.com/r/talesfromtechsupport/comments/8cbvxn/legacy_support_of_clients_computer_part_ii/

    submitted by /u/discusfish99
    [link] [comments]

    Where Did Our Internet Go?

    Posted: 16 Apr 2018 07:58 AM PDT

    Over my nearly 20 years working various tech support gigs I've ran into an assortment of interesting characters. This one perhaps might be close to taking the cake. To put this in perspective tech was a net admin that didn't know how to build a straight through cable, slept at their desk and watched YouTube the remaining part of his day. Only reason they kept the job was because they and the IT Director were bffs

    The Cast

    $Me

    $User

    $Network Admin

    One day $User comes calling to me telling me of the wireless internet is down, asking where did it go and when i t would be back up. $me unsuspecting and just getting the report goes about the usual troubleshooting. Part of this is checking the APs and the internet connection the wireless was connected to.

    Now $Network Admin had the idea of putting wireless on it's own internet circuit(10mbps), but that's not the best part.

    At first check AP is up and broadcasting, I can ping outbound from the AP to the intergoogles all looks good until I dig deeper into the AP network.

    $Network Admin had set themselves up with QOSing for video and internet streaming with max speeds, while looking at the clients setup within this policy were all devices that $Network Admin had on the network, laptop, phone, etc. further looking in, $Network Admin was streaming Youtube and with all devices setup to QOS max speed it was eating up nearly the entire 10mbps circuit leaving others only able to connect to the APs but not route out due to dropping packets via no available inbound/outbound bandwidth.

    After making this discovery and removing the policies for QOSing, users where then able to access the network/internet.

    TL;dr Network admin setup all their devices to QOS over all other traffic including max bandwidth locking out all other users.

    For those who will ask, nothing happened to them after reporting this to higher ups, they continued this practice until years later when I left for a new gig.

    submitted by /u/hidesinserverroom
    [link] [comments]

    No comments:

    Post a Comment

    Fashion

    Beauty

    Travel