443 reads

Sending UDP Messages on Linux: What is the Condition to Packets Being Lost?

by AlexeyChashchegorovFebruary 25th, 2022

Too Long; Didn't Read

The Gemscode indicator is a state-of-the-art trading indicator. The indicator works at any time and in any market in real-time; stocks, indices, crypto-currencies, etc. The indicator includes buy and sell signals on the chart, based on two different trend directions to avoid noise, these signals can also be set up as bus/sell alerts. The script uses an exponential smoothing approach to filter erroneous signals and is based on a momentum strategy.

featured image - Sending UDP Messages on Linux: What is the Condition to Packets Being Lost?

UDP losses classification

In developing the distributed software system it might be reasonable to use exchange data over the UDP protocol to minimize exchange time. UDP is quite a simple way to act fast. But this simplicity comes with a lack of guarantee of data being delivered.

There are several sporadic factors of data stream packets loss:

network media errors
network-related hardware overloads
source host-related hardware overloads

Media related losses

Network media errors come with the inability of the media to deliver data stream packets on specific sporadic circumstances. For example: on a magnetic storm at the north pole wi-fi media can be useless for several hours.

Network hardware overload

On network-related hardware overloads, packets can be lost in the case of data streams processing priorities. Network software processes the packets in a “priority queues algorithm”. That means that the most valuable child queue packets will be processed first of all child queues. If the UDP queue has no first priority, then packets are not sent on overload. New incoming UDP packets will make the queue bigger at the same time. The UDP queue will grow over the limit after some overload process duration.

On new UDP packets incoming on an over-limited child queue buffer - some old packets need to be dropped without sending to emplace new packets.

UDP source host-related losses

On source host hardware overload it is also possible to detect UDP packets losing. The matter of these packets losing is equal to losing at intermediate network hardware. But one use case on source host UDP packets lost is specific. UDP queue buffer overflow will happen if we will put packets with rates greater than the rate of packets getting from the queue. It is possible to reproduce on Linux just by actions at one host.

Linux UDP protocol processing implemented over act with one buffer which contains a queue of all packets from all processes. On some system-dependent factors (number of UDP clients processes, incoming packet size, incoming packets rate) this buffer can be over-limited. To change UDP buffer size at Linux it is possible to use the command string:

sysctl -w net.core.rmem_max=<your value>

It is a case that is not widely discussed but can confuse a lot. To minimize the harm it is good to remember that if well-packed data will be transferred better over UDP. It is good to have the UDP stream be split into chunks close to MTU. Packing for smaller-sized packets at one be also preferable. Then it will cover Linux buffer entries as best as possible.

Another factor to work on is data compression. Compression logical way can reduce packet rate on UDP clients side. That is why it is important to have compression as an option for any UDP-related application under Linux.