If you’re working with a vWire configuration, there’s a crucial consideration you need to take into account, especially if you have older devices running on your network. This piece of advice is drawn from a personal experience I had while implementing a vWire setup.
Dealing with Legacy Devices
The network I was working with had some medical devices that, in all honesty, should have been decommissioned years earlier. But if you’ve been around IT, there are occasions where it’s out of your control, and you must work with these devices. These devices sent packets that were out of sync, which introduced a rather interesting troubleshooting challenge. Not all devices exhibited this behavior, and even those that did were inconsistent. Nevertheless, the issue occurred often enough to warrant investigation.
Legacy devices often struggle with sending in-order packets for various reasons, and one of them, indeed, can be outdated TCP stack software. The following points delve into this issue more deeply:
1. Outdated TCP Stack: A device’s Transmission Control Protocol (TCP) stack is responsible for ensuring that packets are sent and received in the correct order. Older devices might be running an outdated TCP stack, which might not efficiently handle the ordering of packets. In such scenarios, software updates (if available) can help improve the TCP stack’s performance.
2. Hardware Limitations: Some legacy devices may have physical limitations, such as slower processors, limited memory, or outdated networking components, which can lead to processing inefficiencies, including the mishandling of TCP packets.
3. Outdated Network Drivers: Another common issue with legacy devices is outdated or incompatible network drivers, which can contribute to incorrectly handling TCP packets.
4. Network Conditions: In some cases, out-of-order packets may not be the fault of the device itself but a result of the network conditions. For instance, variable latency can cause packets to arrive at their destination out of order.
5. Compatibility Issues: If the device was designed for a specific network setup, and the current network setup has evolved significantly, compatibility issues may lead to out-of-order packets.
To solve these problems, upgrading or replacing these devices is the best course of action. In the meantime, adjustments in your network configuration, such as the ones mentioned earlier, can help manage these out-of-order packets. However, such changes should be done judiciously, considering the overall network performance and security.
Pinpointing the Problem
The problem wasn’t entirely apparent until we performed a packet capture. That’s when we discovered the culprit devices were not properly completing the TCP 3-way handshake – the essential SYN, SYN-ACK, ACK sequence was interrupted.
To verify the packets were out of sync, the following command was extremely useful:
kcordero@PA5250-A(active)> show counter global filter packet-filter yes
This command gives a global counter, which among other things, shows the number of out-of-sync TCP packets:
tcp_out_of_sync 12134 0 warn tcp pktproc can't continue tcp reassembly because it is out of sync
In addition to the command above, the “delta” option proved to be valuable in showing the difference from the last read, allowing us to observe the increasing number of hits:
show counter global filter packet-filter yes delta yes
Implementing the Fix
This happened quite some time ago, and back then, the necessary configuration to allow this type of traffic had to be done via the CLI. It wasn’t something you could easily find and fix through the GUI. Today, the procedure is accessible through the GUI, making things significantly easier.
For those comfortable with the command line, here’s the necessary configuration:
> configure # set deviceconfig setting tcp asymmetric-path bypass # commit
For GUI users, it’s located in the Zone Protection Profile:
Verification
Before running the command, the output might be:
kcordero@PA5250-A(active)> show running tcp state
TCP State Information:
TCP MSS:
Server-to-Client: 1460
Client-to-Server: 1460
TCP Window Scale:
Server-to-Client: 7
Client-to-Server: 7
Session with asymmetric path: drop packet
Bypass if Out-Of-Order queue limit is reached: no
Favor new segment data: no
Urgent data: clear
Drop if zero after clearing urgent flag: yes
Check Timestamp option: yes
Allow Challenge Ack: no
Remove MPTCP option: yes
After running the command `set deviceconfig setting tcp asymmetric-path bypass`, the `show running tcp state` output would indicate a change in how the firewall handles asymmetric paths:
kcordero@PA5250-A(active)> show running tcp state
TCP State Information:
TCP MSS:
Server-to-Client: 1460
Client-to-Server: 1460
TCP Window Scale:
Server-to-Client: 7
Client-to-Server: 7
Session with asymmetric path: bypass
Bypass if Out-Of-Order queue limit is reached: no
Favor new segment data: no
Urgent data: clear
Drop if zero after clearing urgent flag: yes
Check Timestamp option: yes
Allow Challenge Ack: no
Remove MPTCP option: yes
The main change to note here is the “Session with asymmetric path” line, which changes from `drop packet` to `bypass` after running the command. This indicates that the firewall will now bypass asymmetric path checking, reducing the likelihood of legitimate traffic being dropped.
Preemptive Measures
The key takeaway here is the importance of anticipating this kind of problem. If you come across this issue, you can bypass asymmetric checking while you update, replace, or repair the devices causing trouble. Once the issue is resolved, you can turn the checking back on.
Remember, working with vWire configurations can be complex, but being aware of potential pitfalls and knowing how to navigate around them can make the process much smoother and more efficient.