Moronic Monday! Networking |
- Moronic Monday!
- Spectrum is rate limiting VOIP/SIP traffic (port 5060). How to find out if you are affected.
- Intra building fiber and optics
- Cisco Netflow Help
- Can a wildcard SSL certificate be used on an ASA 5555 where C2S is terminated?
- 10g Fiber Connection TX and RX
- How is your VOIP connectivity at work?
- Fluke CableIQ CIQ-100 Problem
- Cisco Firepower 1010 portmanager fails
- NSX-T & DC Firewall
- Is Aegism exists in Networking Industry?
- Mutualized Switch database for comparing ?
- GoTeleport SSH bastion server in networking infra
- How are current company branches connected and how could they be connected in the future?
- Meshed WiFi for large building sweet spot
- Help with avoiding SNAT for SSH and dynamic routing
- Hook up 2 cameras 500ft from NVR
| Posted: 06 Mar 2022 04:00 PM PST It's Monday, you've not yet had coffee and the week ahead is gonna suck. Let's open the floor for a weekly Stupid Questions Thread, so we can all ask those questions we're too embarrassed to ask! Post your question - stupid or otherwise - here to get an answer. Anyone can post a question and the community as a whole is invited and encouraged to provide an answer. Serious answers are not expected. Note: This post is created at 01:00 UTC. It may not be Monday where you are in the world, no need to comment on it. [link] [comments] |
| Spectrum is rate limiting VOIP/SIP traffic (port 5060). How to find out if you are affected. Posted: 07 Mar 2022 04:10 AM PST Summary: Spectrum "upgraded" our DOCSIS cable modem and it broke all of our IP phones. I discovered they are rate-limiting inbound port 5060 traffic. Spectrum "support" is worthless and unwilling to help. You might be affected too. I'll show you how to test, and how to exploit this vulnerability. This is a really long nightmare of a story, so stay with me. I am a network engineer with a client who uses IP phones at all of their business locations. Last November, nearly four months ago, Spectrum came out and replaced our old DOCSIS 3.0 cable modem with a DOCSIS 3.1 modem and router pair after we upgraded the service speed. They installed a Hitron EN2251 cable modem and Sagemcom RAC2V1S router. Immediately afterwards I started getting complaints that phones were not working. I've isolated it down to the cable modem and/or the service coming from the CMTS/Head Node. To be technical: Spectrum is rate-limiting all inbound ip4 packets with a source OR destination port of 5060, both UDP and TCP. The rate limit is approximately 15Kbps and is global to all inbound port-5060 packets transiting the cable modem, not session or IP-scoped in any way. Outbound traffic appears to be unaffected. By "inbound" I mean from the internet to CPE. I won't bore you with the tremendous amount of effort and time that was put into troubleshooting and isolating this problem, but I want to make it clear right away that this isn't a problem with our firewall. This isn't a problem with the Sagemcom RAC2V1S router either. This is not a SIP-ALG problem. For those of you who are security conscious and paying attention, yes, this is an exploitable vulnerability. Anyone can send a tiny amount of spoofed traffic to any IP behind one of these cable modems and it will knock out all VOIP services using standard SIP on 5060. Demonstrating the problem. Below I run four iperf3 tests. First I run two baseline tests coming from port 5061 to show what things should look like. Then I the same tests but change the client source port to 5060. I've provide both the client and server stdout. The TCP traffic gets limited down to 14Kbps, and UDP sees 98% packet loss. IP addresses have been changed for privacy. Test #1. TCP baseline test, traffic unaffected. --> iperf3 -c $IPERF_SERVER -p 5201 --cport 5061 -t 10 -b 5M Test #2. UDP baseline test, traffic unaffected. --> iperf3 -c $IPERF_SERVER -p 5201 --cport 5061 -t 10 -b 1M -u Test #3. TCP test, traffic is rate-limited. --> iperf3 -c $IPERF_SERVER -p 5201 --cport 5060 -t 10 -b 5M Test #4. UDP test, traffic is rate-limited. --> iperf3 -c $IPERF_SERVER -p 5201 --cport 5060 -t 10 -b 1M -u How can you find out if you are affected? It's notable that not all Spectrum service seem to be affected. My customer has two other locations in the same city, not even five miles away, with Spectrum service, and both of those are unaffected by this problem. However, those locations have older DOCSIS 3.0 modems (Arris TG862G) on older legacy speed plans. Remember that we didn't have this problem before Spectrum came out and replaced equipment. Suspected affected cable modem models include E31N2V1, E31T2V1, E31U2V1, EN2251, ET2251, EU2251, and ES2251. These are given out for Spectrum's Ultra plans and anything over 300Mbps. I've verified that at least one other Spectrum customer is affected, but I don't know how widespread this is. To test, you will need to use the iperf3 tool to do a rate limit test. iperf is available for Windows, linux, Mac, Android, and more: https://iperf.fr/iperf-download.php You will need both a client and server system. NOTE: If you don't have access to good client system with a public IP address on the internet, set up your server, leave it up, and send me a PM with your IP address and port. I can run a test against it and send you the results. If you are paranoid about security, just use some port like 61235. The server should reside behind the cable modem being tested. The default port is 5201, but you can use any port on the server side as long as it's not 5060. It's okay to port-forward the server to a NAT firewall. The client needs to be out on the internet somewhere and it needs to have a real unique public IP address. It probably can't be behind a NAT firewall because we need to control the source port it uses to send traffic to the server. Pay attention to the client traffic coming into the server side. If the port gets translated to something other than we specify with "--cport" the test won't be valid. The server is really easy to set up. Just do "iperf3 -s" to start the server and leave it running. Add "-p 61235" to specify a different port. The client is where the action is. We want to send traffic to the server and make sure it's received. Run the following four commands on the client system: iperf3 -c $IPERF_SERVER -p 5201 --cport 5061 -t 10 -b 5M iperf3 -c $IPERF_SERVER -p 5201 --cport 5061 -t 10 -b 1M -u iperf3 -c $IPERF_SERVER -p 5201 --cport 5060 -t 10 -b 5M iperf3 -c $IPERF_SERVER -p 5201 --cport 5060 -t 10 -b 1M -u -c is for the client IP. replace the $IPERF_SERVER with your server public IP. -p is the server port and should match the server, the default is 5201. -t is length of test, 10 seconds. -b is bandwidth, limited to 5Mbps for TCP and 1Mbps for UDP. -u is a UDP test, as opposed to the default TCP. --cport is the client traffic source port, and this is where the magic happens. I'm using port 5061 as a baseline measurement port, which should be unaffected by any rate limit, but you could use anything other than 5060. It's normal to see some small (<5%) packet loss on the UDP tests. Also, don't worry if you can't get 5Mbps on the TCP test. Just pay attention the difference between using port source port 5060 and anything else. If Spectrum is rate-liming your traffic, you will notice a substantial difference in the results. You might see 100Mbps on the port 5061 test and then less than 20Kbps on the 5060 test. On UDP you would see nearly 0% packet loss on the UDP baseline test and >80% loss on the 5060 test. Q: If this problem was widespread, other people would have noticed, right? This is the big question I have right now. Why are we are affected, and who is else out there affected as well? You would think that people would notice if all of their SIP phones stopped working, but it turns out the rate limit is just high enough to let a few phones through without trouble. It's possible this problem is limited to certain accounts, or maybe it's regional, the head node/CMTS, or maybe other customers don't have enough phones to notice. I've found one other customer who can reproduce the problem, so I know it's not just us. My testing shows I can get up to 7 of our Yealink phones registered with the SIP server, as long as I stagger their initial connections. With less than 4 phones I can't trigger the issue at all because there isn't enough SIP traffic. Anything past 10 phones causes all of them to constantly lose their registration. The more phones, the more SIP traffic, and the worse the problem gets. Most customers probably don't have as many phones as we do, and this problem only seems to be affecting the newer cable modems and higher-tier service, and not all VOIP providers use ports 5060 for their signaling traffic. So, yes, It's possible this is a national issue and nobody has noticed or been able to figure out what's going on here. Q: So why would Spectrum be doing this? What's their motive? I suspect the answer might be right here: DDoS Attacks: VoIP Service Providers Under Pressure Phone calls disrupted by ongoing DDoS cyber attack on VOIP.ms I think this might be some kind of idiot's Denial of Service policy gone wrong. Back in late September of 2021, just about 30 days before this problem started, a number of VOIP server/carriers were hit with large DDoS attacks. My client's phones were affected by this attack too, and we noticed, but it only lasted a couple of days and then the attack was mitigated. It's possible Spectrum was trying to prevent or mitigate reflection attacks against their customers, or maybe they are being anti-competitive and trying to force customers into using their own VOIP services. Who knows and I don't care. It's noteworthy that the modem also restricts the amount of ICMP traffic it generates (non transit) so heavily that two MTR sessions will cause it to start dropping packets. If they are dumb enough to do that, then I can see them fucking with other types of traffic as well. All other traffic seems to be unaffected, as far as I know, but I wouldn't be shocked to find out something else is limited. I did test a couple of ports common to reflection attacks such as 53 and 123 but they turned up negative. Testing methods and other information. This isn't a problem with any IP allocation, though I didn't test ipv6. We get a /29 from Spectrum, but if you plug directly into the cable modem you can get a public-unique IP address from a completely different subnet via DHCP, but the problem persists. Changing your CPE MAC address causes a new IP address to be allocated, so it's easy to test different addresses. This also makes it clear the problem isn't the Sagemcom RAC2V1S router that Spectrum mandates we use for the IP allocation. I'm fairly certain this isn't a SIP-ALG service in the cable modem, but that's possible. The content of the packets doesn't matter, and I can't find any evidence that SIP traffic is actually being transformed in any way, even after trying. Both MonsterVOIP and RingLOGIX have SIP-ALG test tools and those pass because they don't send enough traffic to trigger the rate limit. We've eliminated all other possibilities at this point. We tested four different firewalls and linux boxes behind the modem. The fact that we have other Spectrum locations in the same city to test from, just miles away, means we ruled out a 3rd party transit provider too. There's literally nothing left but Spectrum to blame here. What about Intel Puma chipsets? While researching this problem I learned all about the issues with Intel Puma chipsets in DOCSIS cable modems. I really don't know if this is the source of problem or if this is some kind of policy administratively imposed. Apparently there are only two DOCSIS 3.1 chipsets currently on the market, the Intel Puma 7 (Intel FHCE2712M) and the Broadcom BCM3390. The older Intel Puma 6 chips are extremely well-known for being terrible. There are countless articles documenting all of the modems they are in, and which to avoid. There's been class action lawsuits. To say they are not good is an understatement. Apparently the newer Puma 7 chips still have latency problems. We've had a Hitron EN2251 and a Sercomm ES2251 installed and both of those modems definitely have an Intel Puma 7 chipset. But we recently got a Technicolor ET2251 installed, and that's supposed to maybe have a Broadcom chip. Unfortunately the port 5060 limiting continues. There are some rumors that the Technicolor and Ubee variants of these modems may have the Broadcom chip, but other rumors say the newer units after 2018 have Intel Puma chips too, and I just don't know what the truth is. Unfortunately this client is far far away so I can't just take a screwdriver and crack the case to find out. Note that my client has a business account and Spectrum will absolutely not let us use our own cable modem. They mandate that they supply the modem, and because we have static IPs, they give us that dumb Sagemcom router too. I've made attempts to procure our own supplied modem but nobody at Spectrum will allow it. Both Spectrum's dispatch techs and support reps say that you can't request specific hardware when requesting a modem swap and that you get whatever the warehouse sends and you'll like it. What to do? There is absolutely zero justification for Spectrum to be fucking with our SIP traffic like this, or any other traffic. To work around this issue I simply routed the SIP traffic out over a VPN tunnel to one of our other nearby locations, which also has Spectrum service, and that makes the problem go away. But, in the long term I don't want to do stupid workarounds like this. If our VOIP provider supported service using a port other than 5060 we could change the phones to use that, but they don't. We plan to ditch our current provider in the next year anyway, so that'll probably take care of the problem too. Beyond the above, we already have some lawyer letters going out to the FCC and state government. If I can't get anyone at Spectrum with two brain cells to rub together here soon, we will file a claim in small claims court, which is something I've done a couple of times before, and it's very effective. When the corporate office lawyers get involved and they have to send an employee to court, shit gets fixed real fast. But I'm definitely open to suggestions. Oh yea, almost forgot, click here for a good time. [link] [comments] |
| Intra building fiber and optics Posted: 07 Mar 2022 12:23 AM PST Hello, I have to pass fiber from the closet of each floor to the data center in the same building to link the access and distribution switches (I have to buy the switch optics too). Until now there was only copper in those uplinks. There are less than 100 m in each path. I would use 10G connections or 40G. At the beggining I thought it was clear I would need multimode fiber because of the distances and the cost, but it seems there are more problems to take into account with MMF: use of MPO connectors with multiples fibers, non standard protocols at 40G over 2 fibers LC, more problems to upgrade to 100G... So at the end I don't see as clear if I have to install MMF or SMF. What do you recommend? Thank you [link] [comments] |
| Posted: 07 Mar 2022 07:05 AM PST Non network guy here that was thrusted into networking duties, and my organization is in a pickle.... Is it easier just to start from scratch, as I would feel more comfortable doing this? Or going in, renaming and modifying the flow that's already on the interfaces. Thank you in advance. [link] [comments] |
| Can a wildcard SSL certificate be used on an ASA 5555 where C2S is terminated? Posted: 07 Mar 2022 03:11 AM PST The client terminates on an ASA 5555, where a C2S VPN is configured. The time has come where the SSL cert needs to be refreshed and the question on my part is does it have to be a dedicated vpn.domain.com cert (currently we have dedicated cert for vpn.domain.com) or can it be a wildcard *.domain.com for example? [link] [comments] |
| 10g Fiber Connection TX and RX Posted: 07 Mar 2022 07:15 AM PST I tried to search for this but I think my answer is so simple I am getting no results. I am looking to confirm that a 10g fiber is capable of 10g tx and 10g rx at the same time. Never put though into this. I know fiber comes in pairs so I would assume this is correct. Seeing if 20g port-channels for my edge Firewall will max out the license. (Firepower 4145 53g throughput) [link] [comments] |
| How is your VOIP connectivity at work? Posted: 06 Mar 2022 10:27 AM PST I been doing networking for 9 years now and I have never managed VOIP before. All I have done is preparing ports for VOIP connectivity but never managed the system. I interviewed for a network job where my one of my responbilities will be VOIP. I asked if they have dedicated circuits for VOIP and they said no. They use the secondary link at their remote sites for the VOIP connectivity. How do you guys do VOIP at your place? [link] [comments] |
| Posted: 06 Mar 2022 10:28 PM PST I think this is the best sub for this, but if anyone can think of a better one, drop a comment and I'll post it there. I have a Fluke CableIQ CIQ-100 tester and a starting a few months ago, the gigabit test would always fail. Whether the cable was factory made or made by me, it failed. the 10/100/VOIP tests always passed. The wire map always showed correct, just always a problem with gigabit. I haven't really used it enough over the last year to put any effort into looking into it, but here I am, have two massive cabling jobs, and well, need to get it solved. I originally suspected the wire mapper adapter as the problem as it takes a lot of abuse in my tool bag, so I purchased a brand new one, same thing, the gigabit test still fails. It has to be the tester itself. Has anyone had this issue and found a solution? I really can't justify $1500 on a new one, but if I have to, I have to. [link] [comments] |
| Cisco Firepower 1010 portmanager fails Posted: 07 Mar 2022 04:27 AM PST Hello, I have a lot of issues with this device. Does anyone have similar problems?
Message: %FTD-2-199014: port-manager: ERROR: portmanager DIED !!!!!
I think all is related to portmanager somehow... [link] [comments] |
| Posted: 07 Mar 2022 03:39 AM PST Hi all , We planning to deploy multiple host VMware cluster with NSX-T and going to use distributor firewall ips/ids for this cluster , in this case i'm confusing , do i still need physical data-center firewall for E/W traffic or not , [link] [comments] |
| Is Aegism exists in Networking Industry? Posted: 07 Mar 2022 03:19 AM PST Hi Members, i just like to know about ageism thing in Networking industry. In software industry, i had heard that one tend to face tough to exist as dev after above 40 to 45. How are things pertaining to Networking/Infrastructure industry? Is it apply common to every field in IT domain? Thank you in Advance.Eager to have your suggestions and advices. [link] [comments] |
| Mutualized Switch database for comparing ? Posted: 07 Mar 2022 03:12 AM PST Hi is there any website where one can compare switches by ports/features across many vendors ? [link] [comments] |
| GoTeleport SSH bastion server in networking infra Posted: 07 Mar 2022 02:06 AM PST Hey I bet some of you have heard of GoTeleport. It is tailored to work best in a client-server model with their own agent, but also works directly with OpenSSH daemon. I've been trying to get this to work with Juniper, but it seems like JunOS is lacking some config options in the cli. Even our VAR does not know of these options. I'm wondering if anyone has been able to get Teleport to work on networking appliances, and if so; how. (I'm aware of Tacacs+, but teleport also would fit niceley in to our existing linux/windows env.) Tnx! [link] [comments] |
| How are current company branches connected and how could they be connected in the future? Posted: 07 Mar 2022 01:53 AM PST Started learning about networking recently, and was really fascinated by how current Branches (shops with many branches basically) are connected to each other and how transactions are made. I'm curious about the fancy things that exist and why they're not implemented (like for example what you think would be a good implementation but won't be feasible for small businesses). From what i've read online the most popular ones seem to be using cloud computing or wifi. what are your thoughts? [link] [comments] |
| Meshed WiFi for large building sweet spot Posted: 06 Mar 2022 05:14 PM PST Hey learned beings. Looking for insight into what the current industry sweet spot is for a decent sized meshed WiFi network. I need to modernise a moderate mixed warehouse/office type environment (mapping indicates it'll likely need 4 or 5 AP's), and want to move the hell away from the current, consumer grade AP's being used to something more robust and business grade which support things like WPA2-Enterprise authentication and roaming across AP's. Previous installations, I've used UniFi UAP AC Pro's, and while they work, I've seen some mixed messages of late in support forums about UniFi and their practises, so I'm putting the feelers out in case there's a better option. Does anyone have a better solution? I'd rather avoid anything which is based on an annual subscription if I can, but I'm not ruling things like Meraki out completely if they're simple enough and work well enough. Thanks in advance [link] [comments] |
| Help with avoiding SNAT for SSH and dynamic routing Posted: 06 Mar 2022 01:54 PM PST Hi all, I currently have a situation where: * external servers receive SSH traffic * The traffic is then 'SNAT'ed and 'DNAT'ed to redirect through a VPN to another server (which is on a completely different network and external traffic is routed completely independently). * Since the initial servers and the forwarded server all have different internet connections, the SNAT becomes required (and because asymmetric routing is the path to madness :D ). However, I lack any visibility of the origin source, which makes logging (for standard logging, anomaly detection and automated blocking) difficult, so I want to change this. There is a firewall sitting in-front of the SSH server and consul is used for DNS for the public IPs of the service. I was thinking about some rather out-there ideas, such as: * Creating an interim consul service, which contained all externally-accessible hosts, * adding BGP (in some hacked up way) to the hosts (though I haven't used BGP extensively, so maybe I'm way off) * and then setting up a small service which would: Monitor the consul service and select a single host and publish to a new consul service (which would become the primary service that users connected to) * then use the service to reconfigure BGP to advertise this single external host and have pfserver use BGP to set the gateway for internet traffic for the single internal SSH server) - this all sounds pretty horrific though. It does mean that only one host would be used for traffic and if a server went-down, it may take a little while for the monitoring service to reconfigure consul and BGP, but that doesn't really matter. Maybe some tunneling software on the edge servers to wrap the SSH traffic, which could communicate with a daemon on the SSH server, which would then sent the traffic directly to the SSH daemon (with intact source IP/port) and manipulate iptables to reroute the return traffic back to the daemon (puke)! Can you see any other nicer alternatives to get around this? Are there routing technologies to automatically do any sort of routing based on the source IP/port for a connection (though I guess this mostly how NAT works), but since as soon as the traffic reaches the SSH server (or the firewall in front of it), the edge server that forwarded the request is basically unknown (once SNAT is removed). I feel like this should be a solved problem, but I guess it's just from having the nicety of setting headers in HTTP that it's relatively easy there. Any thoughts would be greatly appreciated :D Thanks in advanced! [link] [comments] |
| Hook up 2 cameras 500ft from NVR Posted: 06 Mar 2022 12:40 PM PST This is for a wildlife rescue I volunteer at. They want to hook up 2 outdoor animal enclosures with cameras. Their NVR is 500ft away. I know ethernet cables max out at 328ft. I know coax can go 500meters. I know I need poe. I don't think coax can do poe normally. Saving money is a plus Tell me if my options are correct, and if there are better options
For #3 I don't imagine I could have a couple switches with one 500ft coax.. could I? Here is a link to a poe over coax I found. Very expensive. Dualcomm POE-over-Coax Adapter Kit (DECA-200) - Twin Pack https://www.amazon.ca/dp/B076XNYR2H/ref=cm_sw_r_cp_api_glt_i_0YZHE18VP869A12AHH5D Thank you [link] [comments] |
| You are subscribed to email updates from Enterprise Networking Design, Support, and Discussion. To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
| Google, 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States | |
No comments:
Post a Comment