The Ghost in the Machine Tech Support |
| Posted: 28 Mar 2022 08:07 AM PDT This story is one that I always remember, because it has taught me to take a look at things a little differently. TL:DR at bottom. So a few years back I worked at a Point of Sale (POS) company that provided systems to casinos for their various restaurants and snack areas. One such customer called us in a panic that one of their machines was randomly opening the cash drawer in one of their snack areas and they were about to get in trouble with the Gaming Commission. For those that don't know all casinos are governed by a Casino Gaming Commission that regulates how money, machines and other various aspects of the business are run to keep them "above board". So off I go (about 1.5 hour drive) out to them to take a look. As anyone who has gone to work at a casino security is so much fun! /s. I check in with my ID and am issued a guest pass and then had to wait about 10 minutes for a security guard to come and escort me. First they take me to the security camera room and show me the recordings of the drawer opening on its own both with people around and without. Once I am fully confused then I am shown to the area in question (which is another fun thing - they have to be with you the whole time, no matter how long it takes you). As a side note for anyone that has gotten this far I have already ruled out any viruses or hacks purely because the systems ran on CE and they would have needed to get to passed the main firewall and a few other hurdles just to get to these machines. So I show up and go to work on the machines. I start testing things and the drawers pop up just fine (there were two drawers on this machine). All the connections look fine, so I start taking apart the drawer to see if maybe there is a short in the drawer. Of course as I pull out the drawer there is three $1 bills shoved at the back so I have to stop what I am doing and wait for a second security guard to come and put on a show for the cameras to make sure the whopping $3 is safe! Once that 15 minutes was over I went back to checking everything out. I can't find anything and am completely stumped. And of course the issue does not happen while I am there so I tried the only thing I could and swapped the connections on the drawers. I left and didn't even get to the parking lot that it so happens to go off again, but curiously this time it is the opposite drawer that opened. So now I am beginning to think it is the machine, but of course it is out of warranty and are squabbling about the cost of repair. At this time I decide to place a call to another tech to pick his brain about the issue. As we are talking he mentions "what if it is environment" at which point I pull the line from the title "You mean a ghost in the machine" (It was much funnier at the time). So I start looking around and am checking everything including the power. Now something to note is that we recommend all of our customers to install their systems on dedicated lines so that they are isolated from other items that should be plugged in, including heating/refrigeration units. This is usually denoted by the power outlet being orange. I mention this to my colleague and he laughs and told me that often the electricians will cut corners and just add an orange plug even if it isn't dedicated. And not two seconds later the compressor for the soda fountain machine about 5 feet away kicks on and what do you know, the drawer pops open as well. I bust out laughing because of how simple, yet complicated this has to be. If you think about it there is a tiny surge of electricity that happens every time that compressor kicks on and over time it has come to direct the exact right signal to open the drawer. At this point I let my colleague go and advise the customer of the issue. I install a power conditioner and they sign the bill for it (stating they are going to bill it to the original electricians, good luck!). So for those of you out there that have a head scratcher, as we have seen time and time again on this forum, sometimes it's not something obvious and to check the environment, because today it might be the machine 5 ft away, tomorrow it may be the janitor flipping the power switch tied to the electrical outlet that your bosses machine is on (story for another time). TL:DR - Register was randomly popping open drawer at all times. Turns out the soda fountain machine compressor was causing a surge that opened the drawer. One power conditioner later and never got that call again. [link] [comments] |
| Mystery server reboot serendipitously solved Posted: 28 Mar 2022 04:33 PM PDT Another post just now reminded us that "it's not always obvious, check the environment." This made me think of a story from a friend, Dave, who is now sadly no longer with us, so he can't tell this tale himself. I hope I can do it justice. Dave was called out to a site because one of their servers kept rebooting itself randomly. He pulled out his network sniffing toolbox and ran some tests, but couldn't duplicate the error. After a while, he needed a break, so he set the software to page him if the server rebooted, and took himself off for a snack and a pee. So he's in the bathroom, does his thing, and flushes. Immediately the pager goes off. Dave, being the genius that he is, thinks, what are the odds? He checks the logs. The server has detected a power fluctuation and, ever obedient to its instructions, rebooted. (Don't ask me details as I don't have any.) Dave goes back to the bathroom, starts flushing things. When he gets to that same urinal, boom, the pager goes off. Long story short, the electrical line for the server's power outlet was right next to the water pipe in the wall. Flushing would flex the pipe just enough to touch the wire, which, even with normal insulation, was enough of a temperature differential to cause a slight power fluctuation. I don't remember how they solved it, probably just used a different outlet. But Dave might still be there to this day if he hadn't needed to pee and picked that particular urinal to use. [link] [comments] |
| You are subscribed to email updates from Tales From Tech Support. To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
| Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States | |
No comments:
Post a Comment