Examining Packet Loss Using Wireshark

Recently, I was involved in troubleshooting, the issue at hand was that QRadar was unable to send full updates to a Event Processor (EPs) following an upgrade. When the problem was escalated, my first question was, "Do we have packet captures ?" I was happy because I got the copies I needed, especially captures done for this situation. It's moments like these that make troubleshooting feel like a detective's investigation - each piece of evidence bringing us closer to the solution.

Now, I must mention that I can't share the captures or screenshots here due to various data privacy reasons, but what I observed within them was as expected: packet loss and a multitude of duplicate ACKs. I had a clue that packet loss was involved, and this confirmation added another piece to the puzzle. I proceeded with my checks, focusing on factors that could impact the transfer. The window size appeared fine, and I noted the Maximum Segment Size (MSS) established at 1460. The connection from the Console to the EP was established over an IPsec tunnel on internet, and I noticed that the encrypted packet was leaving with the Don't Fragment (DF) bit set. This is a common characteristic of encrypted packets, as they are typically sent with the DF bit set to avoid fragmentation during transit. In my calculation, if the payload length is 1460 bytes, the addition of the overhead from the TCP, IP, and IPsec layers will roughly make it: 1460 (MSS) + 40 (TCP/IP) + 60 (IPsec ESP) + 20 (new IP header) = 1580 bytes. Path MTU Discovery appears to be ineffective, leading me to suspect that it may be blocked.

My hypothesis began to take shape. Was it possible that a packet with a higher MTU was getting stuck in a lower MTU path over the internet and subsequently discarded? With this theory in mind, the solution seemed clear- reduce the MSS to a lower value. We set it to 1400 to be on the safe side. Bingo! That fixed the problem.

This #networktroubleshooting experience using #wireshark made me brush up on a skill I hadn't used in some time, reminding me of the thrill of the chase in solving these problems. It also inspired me to write about dealing with packet loss in Wireshark. I hope this basic writeup might assist someone else playing the role of a 'Sherlock Holmes' in network troubleshooting. So here I had a sample captures gathered from the repository, and I could identify packet loss in these. I thought these would be perfect samples to walk through the basics.

1. Identifying the position of Lost Bytes.

Alert is triggered when #wireshark detects a discrepancy in TCP sequence numbers. In a typical TCP stream, the sequence number of a segment (packet) should be equal to the sequence number of the preceding segment plus the number of bytes received in that preceding segment. If Wireshark observes that the sequence number of the next segment doesn't match this expected value, it generates an alert indicating a potentially lost packet.

This alert, "[TCP previous segment not captured]," signifies that there's a gap in the sequence of TCP segments, suggesting that a packet may have been lost during transmission.

It's an important alert for network administrators and engineers, as packet loss can significantly impact network performance and reliability.? Alternatively, you can apply the display filter 'tcp.analysis.ack_lost_segment' to list all potentially lost segments.

Let's look at the below screen shot.

• An alert on frame 451 stating "[TCP previous segment not captured]." This message indicates that there is a gap in the sequence of TCP segments, suggesting that a packet was lost between frame 450 and frame 451.
• To ascertain the number of bytes lost, a simple calculation can be employed.
• By subtracting the sequence number of the TCP segment on frame 450 from the sequence number of the TCP segment on frame 451, you can determine the number of bytes that were expected but not received.

In this instance, the sequence numbers are as follows:

    Sequence number of TCP segment on frame 450 = 3423566064
    Sequence number of TCP segment on frame 451 = 3423572964
            
        Substituting these values into the formula gives us:
            
            3423572964 - 3423566064 = 6900 bytes 
    
This calculation suggests that 6900 bytes were expected to be received.
However, we know from the data that 1380 bytes were received at 
sequence number 3423572964. 
    
Therefore, the actual number of lost bytes between the sequence numbers
of the TCP segments on frames 450 and 451 is - 
    
            ?6900 bytes - 1380 bytes = 5520 bytes:
                  ?        

This analysis implies that 5520 bytes were lost in the transmission between these frames, providing valuable insights into the packet loss within this TCP stream.??

2. Now lets figure out possible number of lost segments?

Understanding the number of lost segments in a TCP stream requires knowledge of the Maximum Segment Size (MSS) established during the communication. The MSS is the maximum amount of data that a single TCP segment can hold, excluding the TCP headers.

In this specific analysis, the MSS has been determined to be 1380 bytes.

?We've already calculated the total number of bytes lost between frames 
450 and 451 to be 5520 bytes. To estimate the number of lost segments, 
we can divide this total by the MSS

5520 bytes / 1380 bytes per segment = 4 segments:        

This suggests that approximately 4 segments were lost between the two frames 450 and 451.

Please note that this is an approximation. The actual number of lost segments could be slightly different if the segments were not fully loaded to the MSS or if there were additional overheads due to protocol headers or retransmissions. This calculation provides a useful estimate for network troubleshooting and performance analysis.

3. Viewing the lost segment in the missing place -

In the process of analysing packet loss using Wireshark, it's beneficial to verify if the presumed lost packets were subsequently received after the duplicate acknowledgements were issued. Duplicate acknowledgements in TCP serve as triggers for the sender to retransmit the lost segments, thus it's likely that the lost segments might be found later in the capture.

To facilitate this analysis, it is critical to adjust Wireshark's default settings. By default, Wireshark uses relative sequence numbering, which can make it difficult to match sequence numbers with those specified in the TCP headers. For accurate comparison, you need to disable relative sequence numbering.

To do this, navigate to "Edit" -> "Preferences" -> "Protocols" -> "TCP" and uncheck the option for "Relative sequence numbers". With this setting disabled, Wireshark will display the actual TCP sequence numbers, as they appear in the packet headers.

Additionally, to make it easier to view and sort packets by their sequence numbers, add the actual sequence number as a column in the main Wireshark window. Right-click on the column header, select "Column Preferences", and then add a new column with the type set to "Sequence number".

As we are observing duplicate acknowledgement in the direction -?Src: 172.16.16.128, Dst: 72.4.123.180, we can assume the packet loss happens in the opposite direction which is Src: 72.4.123.180, Dst: 172.16.16.128.

The receiver (172.16.16.128) is expecting a packet from the source (72.4.123.180) that it didn't receive, which triggers the duplicate acknowledgements.

• To investigate this, first apply the display filters for the specific TCP stream ID and the source IP address in the direction where packet loss has been observed. By doing this, we isolate the traffic of interest, removing unrelated packets from our view.
• Next, sort the 'Sequence number' column in ascending order. This allows us to see the progression of sequence numbers and easily spot where there might be a gap, indicating potential packet loss.
• In the filtered and sorted display, look for the frames between 450 and 451. If there are missing sequence numbers, these are likely the lost packets. In this particular case, frames 531, 533, 535, and 537 were identified as lost packets, as they were not present in the original ordered sequence.

?However, it's important to note that these packets were eventually retransmitted, as indicated by their later appearance in the stream after the occurrence of duplicate acknowledgements. Duplicate acknowledgements are a clear indication of packet loss, triggering the sender to retransmit the missing packets.

This analysis is made possible by sorting the frames with the sequence number in ascending order. Without this sorting, identifying the lost and retransmitted packets would be more challenging. Understanding these dynamics can provide valuable insights into network performance and potential issues.

In conclusion, this write-up is intended provides a glimpse into the use of Wireshark for identifying packet loss, although it doesn't cover all possible causes. It serves as a refresher for experienced individuals and a guide for beginners, #networkadministration #networkadmin . The journey from diagnosing the packet loss to adjusting the Maximum Segment Size (MSS) showcases the detective-like nature of network troubleshooting and the importance of understanding network protocols.