"Your program stopped working correctly, after I switched around some of the I/O wires that the coding logic was directly dependent on." Tech Support |
- "Your program stopped working correctly, after I switched around some of the I/O wires that the coding logic was directly dependent on."
- Submit a ticket to tech support? Look here pal, I AM tech support.
- It's a LEGAL issue and a LIABILITY!
- The Hybrid
- May I suggest you actually try the service before complaining?
- Smalltown Insurance Co.
- A Key, a Power Cable, and an Ethernet Port
- The AV Saga Part 4: SouthEast City Interlude
| Posted: 02 Jan 2019 02:04 PM PST During the Christmas holidays, I was finishing overhauling some codes in a Programmable Logic Controller (PLC) after the station was upgraded with new machinery and all of the I/O was wired up. There were a few moments where my code didn't perform as expected, but I implemented some workarounds to get them going. I learned to "trust but verify" when it comes to PLC coding comments (if there were any) and electrical wiring documentation. Today my program crashed, hard. Station had physical damages from the software failure, and somehow the station didn't alarm out until it realized that nothing was happening (because a robotic arm got caught in a transfer mechanism). I asked around if there were changes made to the station, and everyone denied it. A few hours of troubleshooting later, someone mentioned that they saw someone else open up an electrical cabinet and worked on it for a bit before leaving. I tracked down that person and asked what changes did they make. "I realized I wired some of the I/O wrong, so I went in and fixed it." PLC programs can be very low level. The ones I was working with don't support automatic memory allocation; the programmers have to manually allocate the memory bits and manually check that their codes aren't using memory bits that are already in use. There was one numbskull that kept forgetting to check, and cause spectacular crashes or garbage data output. Such as overwriting a part identification tracker's memory with something else, and crash the server that was recording all of the part numbers being reported by the stations, probably because the server software didn't have enough safeguards for random garbage coming in. For example, let's say a code checks if X012 bit lights up, which is the X012 input wire that connects to a mechanical switch that is pressed in by a part if the part enters a certain section of the machine. If someone moves that X012 input wire over to a conveyor system to report whenever #4 motor is firing up, then that code that relies on X012 fails hard. And if there was another chunk of code that relies on the X012 code, then that error is going to propagate throughout the program until it alarms out or something unpredictable happens. It would be like scrambling Ethernet cabling connections in a network/server and expecting it to work without any software reconfiguration, you just don't. And my program was based on the incorrect I/O wiring... [link] [comments]  |
| Submit a ticket to tech support? Look here pal, I AM tech support. Posted: 02 Jan 2019 02:32 PM PST So this just happened and most of it was me being stubborn, stupid and lazy all at once, but also me being new and curious how the system works. Our tech team is massive and consists of multi divisions, groups and people. I'm relatively new and wanted to help out an end user a bit quicker by traversing the bullshit and going to the top a bit quicker than an end user would. I ended up calling the "front desk" of another tech department since I didn't have a direct line to someone higher up who would answer. Them: "tech support how can I help?" Me: "hey, I'm tech support too and need help with an Adobe license" Them: "okay, the best bet is to submit a ticket to tech support for software" Me: "can I just talk to someone who works with (insert other tech who deals with software and is OOF for holidays)?" Them: "sure, I'll need to verify you work for tech support too. Can I get your name and birthday?" Me: "sure" *gives info Them: "okay, please hold" 15 minutes of holding (30 minutes into the call) New Them: "hello, tech support sorry for holding. How can I help you?" Me: "hey been waiting a while. I am tech support and wanted to talk to someone who handles licensing for Adobe software." Them: "oh, you'll have to submit a ticket" Me: "what kind of ticket?" Them: "a tech support ticket. Here, go to this website and fill out this form" Me: *playfully going along just for laughs * "this form?...the first blank asks which tech team to submit this to, who should I pick?" Them: "oh! (Names my tech team)" Me: "I'm sorry man, I'm not going to submit a tech ticket to myself" Them: "oh...you're tech support?" Me: "look, I'll just talk to (manager) when he gets back in" Them: "oh, it'll be faster to submit a ticket" Me: "...to myself?" Them: "...I can show you how" And that is how I ended up not getting an answer after an hour. [link] [comments]  |
| It's a LEGAL issue and a LIABILITY! Posted: 02 Jan 2019 10:29 AM PST This just happened this morning: The Office Manager for one of our clients, a small village office, submits a Sev1 ticket stating she can't open her email. Outlook is throwing a "the data file was not closed properly" error and takes 5ever to scan the file. I remote on, have a look, and decide to just rebuild her Outlook profile, as they have Exchange. I do the diddly, and when I re-open Outlook it's still trying to find the old data file for some reason. So I navigate to the folder and '.old' the data file. While I have the data file location open, the Office Manager (OM) sees three archive.pst files, and immediately proceeds to lose her shit.
I convey to her that I understand her concerns, and promise her we'll have a look at the archive files just as soon as I get her dang Outlook working again. .old-ing the data file worked. Outlook starts normally this time, and begins downloading from the Exchange server. So far, so good. I load up the archive files for OM, and lo and behold, they contain absolutely nothing. However, OM continues her tirade:
She's satisfied with Outlook working again. So I move to close things up.
[link] [comments]  |
| Posted: 02 Jan 2019 06:42 PM PST Greetings TFTS - it's been a while since I've had a tale worthy of your esteemed ears. I've worked for $MSP for half a decade now. Coworkers have come and gone, either on to other teams or to pastures new; old customers have left us and new customers have taken on our services, and I've survived several rebrands and restructuring efforts. One side-effect of working support for the same company for so long is that you get to know the quirks of every environment you support - whether it be something as complex as firewalls blocking access to websites because it detected symbols in your encrypted password and assumed you were trying to run SQL injection attacks, or as simple as the specific order you must follow when restarting certain services, everything eventually becomes paint-by-numbers. So when a "Missing Emails" ticket fell upon my queue on a cold morning in early November, it stood to reason that it was a simple case of user error, something deleted or moved by accident. It certainly couldn't be anything more complex than that. Right? The Hybrid I enter the building, my extremities still numb from the biting cold, nicotine buzz spreading warm fuzzies through my veins. It has been a hectic morning - phone call after phone call, ticket after ticket, but at last I have unshackled myself from my desk and partaken of the sole vice I afford myself. The momentary peace I have enjoyed is broken almost immediately upon my return, in the form of a Skype message from $EasilyConfused, our newest employee. It reads:
I look up, and realise that $EasilyConfused is staring at me from his desk, eagerly anticipating my acknowledgement of his message. His gaze prevents me from sliding under my desk and establishing residence there, and affords me no other option but to bring the ticket up on my screen.
Almost immediately, he replies.
With a weary sigh, I walk the short distance from my desk to his, and am greeted with a view of $VanishingAct's Outlook client.
$EasilyConfused, amazingly, does have these details and so I run a search against the whole inbox. The emails we're looking for don't show up in the results, but there are several other emails in there from the sender in question, so it's clear that this person has emailed him in the past. I direct $EasilyConfused to ask $VanishingAct for another example, which he provides. Another search against the inbox again comes up empty for the specific emails we're looking for, but provides proof that this new sender has also emailed him in the past. So two senders, both from within our organisation, suddenly have their emails not delivering. Caught in the Spam filter, maybe?
Back at my desk, I log in to the Mail Protection utility we employ, which $EasilyConfused does not have access to as yet. I run a trace against the first missing email, and am greeted with a screen informing me that the message was indeed delivered. Same for the second example. So where are the emails? A thought strikes me, then - it's only internal senders that are affected in his examples. Those shouldn't even hit the spam filter as they aren't leaving the Exchange server, though they appear in the tracking as the Exchange message logs are journalled. So they aren't being held... I break out my troubleshooting cap, and fire a quick test email to $VanishingAct. I check in the tracking logs, both on Exchange and the Mail Protection Utility - both report that my email was delivered successfully, so I fire $VanishingAct a message on Skype and ask him if he received my test email.
So that rules out the possibility of him using Inbox rules - my email address has a different suffix to his coworkers, though our mailboxes live on the same server. So where the hell are these emails going? Something very strange is going on here. I ask $VanishingAct when he first noticed the emails weren't coming through, and run a new trace using the Mail Protection Utility, from several days before the issue was noticed, with the first example sender have gave me. I then go through the results, one by one, checking them for differences. It didn't jump out at me immediately, I'll admit - but eventually I was able to notice that, after the date in question, the IP address of the receiving server changes. But that's not the IP address of our Exchange server. Where the hell is it getting that from? I check the second sender, and get the same results. I then check my own test email, and sure enough - the recipient server IP isn't the IP of our Exchange server. But, wait, didn't he say that the only affected senders were internal senders? Why would they be getting a different recipient server IP? They shouldn't even go outside the Exchange server, they should be getting sent internally. A thought strikes me, and I do a search for all emails sent to $VanishingAct on this day - looking for one sent from outside our organisation. The recipient server IP is our Exchange server. What the actual hell? I lean back in my chair, truly stumped now. MX records for our domain are all correct, external senders get their emails delivered to the mailbox without issue. Internal senders report that the email has been delivered successfully, but are nowhere to be seen. As I ponder, lost in thought, I am dragged back to reality by that annoying little bloop Skype makes when you receive a new message. It's $VanishingAct, looking for an update. I fill him in on where we are with this, and how I have no idea what could be causing the issue.
Office 365. Holy shit, Office 365. Here at $MSP, the majority of our mailboxes on On-Prem Exchange mailboxes, and we use DirSync to assign the relevant licenses once the accounts are synchronised with the tenant. Only certain branches of $ParentCompany have actual Office 365 mailboxes. I immediately log into the Admin Center, search for $VanishingAct in Active Users, and check his licensing. There - under the E3 license, right at the very bottom, the toggle for Exchange Online has not been deactivated. I switch to the Exchange server, and log into the Exchange Management Console. I run a search for $VanishingAct's mailbox on the On-Premise tab, and get a hit. I then switch to the Office 365 tab - and there I find my culprit, sitting sweetly as can be, totally unaware of the damage it's causing. Two mailboxes. One on-premise, one cloud. Both with the same SMTP address. "Well," I think to myself, "That's a new one." In the end, I needed to pass the ticket over to $ExchangeAdmin, the sole point of contact for Exchange Administration for $MSP - because both mailboxes had emails in them. Internal senders were getting routed to the O365 mailbox, while anyone external (i.e. reliant on MX records) were getting routed to the On-Prem mailbox. As it was, $ExchangeAdmin had to step in manually, export the O365 mailbox and merge it with the On-Prem one. Afterwards, we switched off the Exchange Online sub-license against $VanishingAct, and the O365 mailbox was no more. This would not be the last time we would experience this issue - we had several more users complain of the same issue, and each time they were found to have an On-Prem/O365 mailbox combo with the same primary SMTP address. Eventually, $ExchangeAdmin got fed up and decided to track down the culprit, by digging through the event logs on the Exchange Server itself, one by one, until at last, he had a name - which is how $EasilyConfused earned his moniker. [link] [comments]  |
| May I suggest you actually try the service before complaining? Posted: 03 Jan 2019 03:13 AM PST Got a customer today who just switched from DSL (30 Mb/s) to fiber optic (500 Mb/s) , and was not happy about this. C: My DSL is not working! I need internet so my daughter can access her plane ticket for today! (there is mobile internet which everyone has, but okay..) M: That's correct, it was turned off last night after your fiber optic was installed so you don't pay for two internet services at the same time. C: The seller said we would keep our DSL for two more weeks so we could try the fiber! M: Sorry, cannot find any note about this. In your order it said to turn the DSL off today, which you have received an email about. C: Well I didn't check that! I was promised it would be on for two more weeks! M: Okay, I'm afraid we simply cannot turn the DSL on again, but since your fiber optic is already activated, you could just use that one instead. C: The internet is already working (so what was the problem with the plane tickets then?), but I cannot use the TV for Netflix! My network cables are not long enough, the local countryside store has none, and the ones I've ordered online have not arrived yet! M: I see, but you can connect your TV to the WIFI instead of using cables. C: No, I cannot do that, the connection is not good enough, so I get bad image quality! M: Okay, but then you could connect a computer with a shorter cable to the router in the meantime and watch Netflix through it instead. C: But I want to watch on the TV! I want my DSL turned on now so I can watch Netflix! I don't want to have to wait for the cables to arrive! I've waited in line for an hour, you need to fix this! M: I'm sorry, but it's just not possible to turn the DSL on like that, a technician would need to be booked again which would take several days. C: Damn it! Fine, I guess I will have to try to connect the fiber router then. M: Excuse me, you have not connected the fiber router yet? C: No, I'm still just using the DSL router which is not working now! M: Uhm... maybe you could try to connect your router to the fiber optic instead and see if it's working if you try to connect your TV to the new WIFI? (which is course is much stronger since she had upgraded both the speed and gotten a more powerful router for the fiber optic) C: Fine, I will do that later. It's not that important anyway, we are pretty busy with going to the airport in the city (where you could also buy the needed Ethernet cables!). M: ...All right then... Long story short; the customer complained about the WIFI being too bad for her smart-TV and demanded us to re-connect the DSL (which would both be very time-consuming and costly), even though she had not even tried to check if the WIFI was working, and on top of that, was not even going to be home to watch the Netflix shows that was so important... [link] [comments]  |
| Posted: 02 Jan 2019 05:29 PM PST This is a story from a few years ago when I worked for a MSP. One of our larger clients was Tri-County Hometown Insurance Company ($THIC), who has recently acquired Small Hometown Insurance Company ($SHIC). One day I received a Critical severity ticket from a man we'll call Richard, as the shortened form of the name describes his personality perfectly. Richard was the Ex-owner of $SHIC and had been transitioned to a VIP role at $THIC, this allowed him to directly submit tickets to MSP. My HUD reported as my monitors turned blood red and klaxons sounded in the distance. A quick status flip from "NEW CRITICAL TICKET" to "In Progress" silenced the doomsday alarm as I opened up the ticket details. After confirming that email was still flowing and that Richard's ticket was not in fact critical, I replied that I was looking into the issue and pulled up their email console. To give some insight on OUR failed email service, MSP was a certified re-seller of Brafish365 cloudhosted email service. $THIC had us migrate all of $SHIC's email hosting to their Brafish365 account once $THIC bought $SHIC. Now, $THIC's email service was about as basic as they come, except for one small email routing modification $THIC had us make. $THIC had an in house marketing and design department who loved to append fancy HTML signatures and footers to each and every email. To do this they used a hosted email forwarder based in Pre-Brexit UK. Now, you may ask, "Why does Pre-Brexit matter?". One of the internet laws imposed by the EU on it's member states was a anti-hate speech and defamation law which we'll call $Chartercode:8675309. Looking at the top level email logs in Brafish365 did in fact show a few bounceback emails from Richard over the past two days. I picked the most recent one and started reading over the bounce back receipt. TL:DR of the bounce back was that Richard was trying to send an email to his buddy's auto shop email. The email sent fine until it hit $THIC's fancy email service in the UK and was sent back. Confused why Richard was getting the EU law based bounce back, I informed my boss I was stuck and got permission to pull down a copy of the original email. I quickly found why the email was blocked. "Oh.... oh no." I thought to myself before immediately assigning my boss as the owner of the ticket, attaching a copy of the bounce back and original email. I walked over to him, told him what I found, and removed myself from the ticket. In the end Richard was still hired at $THIC but had all external email access cut off and most of his duties removed as it was cheaper to keep him hired and have him do nothing than to buy out his remaining time before retirement. [link] [comments]  |
| A Key, a Power Cable, and an Ethernet Port Posted: 02 Jan 2019 09:57 AM PST I work as a tech for an ISP. A few weeks ago I was on-call. I received a call from a customer who said that their internet was down. Looking at our monitoring system, I saw an alarm that usually indicates a failing battery backup, followed by an alarm indicating the customer's ONT was offline. Because of the battery-related alarm, I decided to head into the office (it's closer to home than the warehouse and we keep some stock on hand for the tech team to make quick calls) to grab a spare battery in case it needed to be replaced. I activate the alarm system on my way out, then lock the door. [SNAP] My key snaps in half in the lock. Well that's fun. Give the customer a call and say I'll be another few minutes, then send a message out to notify the team of what's happened. We can't get a locksmith out for a few days, so I have to find another solution. The door didn't lock before my key snapped, and I can't just leave it unlocked. Thankfully the back door locks itself, so I can lock the main door from the inside then run out the back. I tried pulling the broken key out with a pair of pliers but it just made the problem worse. On to the customer. This particular customer has one ONT feeding two routers - one for them, and one for another customer who rents the basement of their home. The ONT was installed in the basement, so we had to get the other customer's permission to enter. Thankfully, we get it pretty quickly. That basement was an absolute disaster. Calling it a pigsty might be an insult to pigs. Looking at the ONT, the problem was immediately obvious. The ONT's power cable had been unplugged. Not only that, but the customer who lived in the basement had attempted to jam a power cable for a completely different model of ONT into one of the ONT's ethernet ports. The other end wasn't connected to anything. Plugging it in correctly resolved the customer's issue, as it so often does, but the mystery of the power cable in the ethernet port still remains. Nobody on the tech team or the installation team knows where that power cable came from. The model of ONT which that power cable is for had never been installed in that home at any point. I also spilled my coffee on myself when I tripped on the stairs at home later. I had great luck that day. [link] [comments]  |
| The AV Saga Part 4: SouthEast City Interlude Posted: 02 Jan 2019 06:35 AM PST Previously: https://redd.it/a7zw8k
Tuesday morning was relatively quiet. I decided against building a replacement PC for SecEng, and instead give him the old PC of our network engineer who had left a couple weeks before. Around 9am I walked past the PM's office.
I turn back and step into PM's doorway.
We walk down 3 offices to CTO's office.
And with that I go back to my desk. I figured screw it, leave today, get there tonight, and have all of Wednesday and Thursday for training, and about a half day or so on Friday, and get home late Friday. Be a bit awkward to go there and train NTech with SETech there, but whatever. I load up our travel webpage and schedule a rental car pickup for 11am, and a hotel in SouthEast City for the next 3 nights. DeskAdm was nice enough to follow me home to drop off my car and bring me back to the office. I decided to stick around for our team conference call before heading out. SETech did call in with NTech. Unsurprisingly, no one else knew until then NTech had been hired. Halfway through, CTO and PM walk up.
A few minutes later in the team meeting.
I could tell no one really understand that SETech was gone. I let DeskTL know right after the meeting he wasn't coming back, and that I would be on-site tomorrow to help NTech for the rest of the week. With that I walked down the street to pick up my rental c-truck. All they had were trucks. They gave me a Dodge Ram with 30 miles on it. Whatever, not my money paying for gas. I made it home by noon. Packed a bag, ate lunch and hit the road by 1pm. I managed to get checked into my hotel by 8pm in SouthEast City. Wednesday I rolled into the SouthEast City office at my normal time, and NTech is already there. After introductions, I see where she's at with the laptops and other AV related things.
It took me about 2 hours to figure even though we thought our Windows 10 image was setup for UEFI and GPT boot settings, it only worked with Legacy, MBR settings. Hasn't been a problem with all the other PC models we use, but this specific model only supports UEFI and GPT. Took me another hour to pull down a copy of our Windows 10 iso from our software share so I could drop it on a flash drive and build the laptops manually. During this Office Manager stopped by, happy that I was now here. Business Manager (BMgr) also stopped by, excited to see me.
The only other thing of note on Wednesday was a suspension form came through our ticketing system for SecEng. Thursday Thursday morning our virtual PC environment crapped the bed. Our virtual PC environment consists of about 300 PCs for a large chunk of our users to login with through zero-clients. We have a server that stores their user profiles. Surprise Surprise, with this server suddenly getting AV Software, it's CPU was pegged at 100% constantly. User's login times to a virtual PC went from less than 5 minutes to 30 minutes plus, if at all. Luckily for me, SouthEast Office doesn't use our virtual system, so I was spared having to participate in its troubleshooting. I spent most of Thursday finishing BMgr's laptop, training NTech, and answering any questions. NTech was finishing Client Manager's laptop. Early Thursday afternoon I have BMgr's laptop finished and have just finished setting it up for her.
And yeah, at the end of the day a termination form came through for Client Manager. DeskEng helped successfully failover the profiles server to its backup, which greatly reduced the login delay for virtual users. That was it for my time in SouthEast City Office. Pretty much just to replace one user's laptop. Although I know CTO greatly values face to face training, so my trip would have happened regardless, just with not as much urgency. To Be Continued… [link] [comments]  |
| You are subscribed to email updates from Tales From Tech Support. To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
| Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States | |
No comments:
Post a Comment