electrician killing off the entire network - twice Tech Support |
- electrician killing off the entire network - twice
- A little indexing goes a long way
- Can’t access the what????
- Can we have some direction/instruction please?
- The Chronicles of IT-Freely and a bo$$Threats and you – when your bo$$ threatens your job because he didn’t like your predecessor
| electrician killing off the entire network - twice Posted: 01 Apr 2019 01:21 AM PDT So you'd guess the electrician provoked a power surge or something, right? Far from it. I worked as a subcontractor for a company providing IT services for medical practices. In this quite large practice (cardiologists, 5 doctors, 3 ultrasound rooms...) they use a Windows workgroup without AD because their ancient patient administration software (a product that leads the market in my country and it's a first world country…) can't handle a modern network. They also have a relatively slow internet uplink. Some years ago the electrician who also likes to dabble in IT just swapped out the professional vpn router for an all-in-one router you usually use at home. One of the boss doctors agreed on him doing it, someone suggested it in the first place, we as IT were neither involved nor informed. What happened is what was to be expected: New router installed with different MAC ID, all the Windows machines' firewalls and the server detected a "new public network", switched to closed clamshell mode and not a single machnine worked. Then they cried for help and started pointing out the reaction times in the contract we have with them as in "you better fix what you fucked up". This was some 3 years ago, there were numerous other incidents of people doing IT things there without informing us as their IT services provider obligated by contract to fix their stuff in a time frame if something goes wrong. Guess what: The exact same electrician did the exact thing AGAIN!!! Swapped out the existing vpn router for a different device as someone let him do the transition from ISDN to VOIP and they needed a new router for that - and the entire network did go down again because of the different router. What an idiot. The funny thing is: The contract with this medical practice was cancelled a couple of days before because we had enough of their shenanigans. I pointed their new IT guy in the right direction of course and he fixed it. Good luck fixing your electrician's crap in the future, you curmudgeons. [link] [comments]  |
| A little indexing goes a long way Posted: 31 Mar 2019 09:51 AM PDT I've got a million of these stories. Someone stop me if they get annoying... For those that remember my "Where did Dave go?" story, you may recall that I had several SQL Server clusters from hell. The most fragile, finicky and difficult to support pieces of garbage I've ever seen. Being supported by a matrixed team doesn't help, and that's where the DBA team comes in. Boss: Cluster 1 is running hot. We want to scale up the app by 100% three months from now. Can we do it? Me: Yes. The performance of that cluster is fucking terrible and I'm sure we can solve that problem in three months. We bring in MS Titanium Premier support (or whatever it was called - big bucks for on site experts). We bring in our datacenter's best hardware consultants. We bring in the dev team, the business owners, and finally the DBA team. Me: That cluster is performing terribly. It often pegs the CPU. The RAM is maxed out. The disk access is running extremely hot. The actual load on the DB is really light though - very few users and a reasonable database size. Everyone: Agreed, but we think it's because of some other team <finger pointing ensues>. I quickly realize that me and the dev team will have to solve it, so we spend the next few weeks checking the most obvious things. We are hampered because we can't actually touch production, so we have to keep asking for config to be pulled and profilers to be run. Meanwhile, all of these experts are recommending the stupidest shit. MS: Tweak this setting, trust us. <complete production outage ensues> Hardware guys: Raise a request for new servers. The new ones are almost 5% faster. DBAs: Can we move the app to the mainframe DB2? We get a profiler trace that shows that the query that is executing the most often is running poorly. The trace doesn't have enough logging to pinpoint the problem, but it's a good place to start. SELECT <col1, col2...> FROM tblX where id>blah order by dateY Me: DBAs, can you confirm that there's an index on id, and on dateY in production? DBAs: Yes, of course there is. Let's tweak the fuck out of the RAM settings instead! <complete production outage> A couple months go by, and one of the devs comes to me and says, "There's no way those indexes exist. It's tablescanning for sure." Me: DBAs, please pull a list of indexes from production again. The list arrives, the indexes are there, so another dead end. Two weeks before scaling... Boss: BigJilm, what do you need to make this happen? This is too important to miss the deadline. Me: I need Admin access to the cluster for two hours. Boss: Done - midnight to 2 am tonight. We amass an army of people for an overnight session, probably 30 people that have been involved and who want to check on their suspicions real-time. The DBAs are there, my devs, hardware, network, business... 12:05, Me: Let's quickly run profiler in that select statement with logging cranked up. 12:06, Me: Why is it tablescanning? 12:07, Me: Why are there no indexes on that table? This is after 10 weeks of full time troubleshooting. I'm fucking furious. Everyone on the call is quiet. Next morning, two new indexes in place, and the cluster is running at about 1% everything. Me: DBAs, why did that report show all of those indexes two months ago? DBAs: That report was from Dev. We aren't allowed to check Prod. Me: #%}%#%! I'll be in the bar, muthafuckas. [link] [comments]  |
| Posted: 31 Mar 2019 03:53 PM PDT We are going way back for this one. I was a new grad, working as a dev on a call center app for a huge national company here in Canada. It was a massive project and when we launched, the team and I did L2/3 support and enhancement for a year or so. Last phase of the project was French translation, where were we snagged all of the English words and sent them to our Québécois colleagues for translation. That's easy enough for the core app, but error messages were tough as they often contained tech jargon. We basically agreed that the jargon would be left in English and we'd translate around it. The app was a client-server beast, using OS/2 named pipes for communication. Whenever we couldn't communicate with the server, we'd pop up a message "Can't access the pipe". Network failures, server crashes, TCP/IP problems, whatever - this error popped up a lot. Our translation team sent back the translation, but they were baffled by the word "pipe", so they decided it was jargon. "ne peut pas accéder au pipe" or similar. I probably remember that wrong - I don't speak French but it was something like that. We moved in to support mode, and I got a bunch of tickets on the translation. This word was wrong, or this sentence would be more clear. I would get a ton of tickets about the pipe message, but my business partner just closed them out and said not to worry about it. I'm guessing a third of the overall tickets were about this message, but they just got closed by her. Year end, we all head to Montreal for a big party to celebrate our success. They are giving out awards, some real like MVP, and some cheeky like "Most Likely to Quit". I win an award for something like "Biggest Bonehead" and as I'm approaching the podium kind of baffled, the announcer says it's for the legendary pipe guy! Every French person is now laughing hysterically!!! After I get the award, I find out that "pipe" means "blowjob" and the error message was "can't access a blowjob". Every user, several times a day, was getting that error... [link] [comments]  |
| Can we have some direction/instruction please? Posted: 31 Mar 2019 07:49 PM PDT So this just happened. Bit of background first. We set up production environments and applications for businesses and provide services on top such as Operations support. A ticket came into our queue for 'user cannot connect to server host'. I check the ticket and it's a standard error we have been seeing. Basics are that one of our applications has two servers (primary and backup) and there is a file that is created when the application is installed that has these listed in it. Here's the problem. The file itself is created when the user first installs the application. The way it works, is that it gets the server name that the user typed into the login page. And it then checks that server to see what the backup is. Do you see the problem? To a user, there is only one server. As far asthey know (or care), serverP (primary) is the only server. Even when P goes down. They still type in P, and the application still works. So. The problem. Occassionally P goes down and B (Backup) takes over as the primary. When P comes back up, it comes up as the new backup until someone manually flips them back. So a user installs the application for the first time. They enter the name of P, the file reads this and puts it as the Primary, then checks it to find the backup. P says it is currently the backup, so the file adds P as backup. As far as this file knows, there actually is, only 1 server. Now, the user tries to log into P, P says it is in backup and they need to conmect to the primary. The apllication checks the file then tries to connect to the other server, which is also P. Then fails. Anyway. This is all getting fixed but slightly less new users are facing issues still and we are going through and fixing these files when they come up. Enter our user. $u - can't connect to server host $me - hey $user. This is a common issue that is being fixed currently. We just need your $file and we can fix the error and it should all be working fine (An hour later) $um (users manager) - can we get this job escalated? We are very busy today and need u working ASAP (note that this is in direct reply to my email earlier) $me - Hey $u/$um. We require $file from you in order to complete this job. This is located in $path. Once we have the file. We can fix it and have $u delete the old file and place the fixed version in the folder where the old one was (Another 2 hours later) $um - $u is still unable to work. Can we get this job looked into. It's been 3 hours since it was opened and we still haven't heard back. (Again. This was in direct reply to my email). Can we get some direction and instruction please? $me - (copy pasting the previous email) $sd (service desk) - is this the file you are talking about? (File attached) I check this and it is the right file, but looks fixed aready, damn. $u must have some otger issue going on $me - that's the file but it's already fixed. $u must have a different issue. I'll look into it further $sd - actually that is a file from MY computer. Another engineer gave it to us to use for fixing users' issues ourselves. Phew. In that case. We seem fine $me - can we have that file placed in the folder for $u? $sd - it is done. Haven't congirmed with $u, but they listed PCxx in the ticket, so we remoted in and fixed it $u (first time replying in 4 hours now) - actually I'm on PCxy. I switched PCs to see if another one worked. We replaced the file on that PC too, and now it works fine. TLDR - simple 5 minute job takes 4 hours because user and manager don't like to read emails that they reply too. [link] [comments]  |
| Posted: 31 Mar 2019 12:39 PM PDT Welcome again! This one is going to be short. But it's another documented issue I ended up talking to HR about. During the very first week of my new job, and throughout my tenure, my boss had this tendency to tell me something whenever he felt something wasn't being done to his satisfaction, or just in a general threat so I would do things exactly as he wanted. Now, keep in mind, bo$$ would constantly ask me to do x, then ask me why I hadn't done f, when f was not even related or part of what we discussed. Or would actually forget what he asked me or claim he never did, etc. I still don't know if it was just total burnout or if he was also gaslighting me, but sufficed it to say, he wasn't kind about it. Whenever these things happened he'd outright tell me: "Remember, $formertech would do things that way, and they aren't here anymore…" Yeah…he wouldn't discuss what was bothering him, if I was actually doing something wrong (and while I own up to all my mistakes, because of how on the edge I ended up being with this guy, would be sure to do things as perfectly as possible). He would just shoot that out at me. As an adult, I tried to let that just slide off, but it still hurts and made me feel like crap. I tried to bring it up before and talk through "issues" with him, but that just never worked. It was do what I want or else. I already knew he had trust issues and was a huge micro manager, but saying something like this to your subordinate is never ok. You either address legitimate issues or get over yourself. I'd go on to document this and present it with a huge list of issues during employee reviews, where in I got to anonymously review my boss. That'll be a fun story I tell soon… Coming soon: Part 6: The keyboard and mouse But first, part 5: Yawning may offend others, and other pearls of wisdom. [link] [comments]  |
| You are subscribed to email updates from Tales From Tech Support. To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
| Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States | |
No comments:
Post a Comment