Time-Sensitive Networking for robotics

Ethernet with TSN standards will become the de facto standard for communications on layers 1 and 2, in robotics.

This article is a writeup of “Time-Sensitive Networking for robotics” available at https://arxiv.org/pdf/1804.07643.pdf and submitted to AHS 2018. Peer written with Carlos San Vicente, Lander Usategui and Irati Zamalloa Ugarte.

Typical response-time of common robotic components. For sensors, the response-time reflects the typical time required to provide digital data of their measurements. For actuators, it states the typical control cycle. Figure shows the diversity of several robot components and their different networking requirements. Motors usually require simple data as parameters, such as set-points like position, velocity, torque; a camera system streams instead a considerably larger amount of data, which can go up to few megabytes per second.

The field of robotics is growing rapidly. New areas such as professional, consumer or industrial robotics are demanding more flexible technologies and a set of standardized policies that facilitate the process of designing, manufacturing and configuring a robot for potentially more than one specific application. Previous work [1], [2] highlighted the relevance of standard interfaces at different levels.

One of the main problems in robotics, as it happens in other industries, is that there is no such thing as a standard communication protocol, but a variety of them. Choosing a communication protocol is not straightforward: the list is large, and each protocol has evolved to meet the needs of a particular application area. Typically, each protocol has been customized for specific applications and, as a result, multiple communication protocols and buses are used to meet different requirements within those more complex use cases.

Classification of real-time Ethernet solutions according to their network layers.

In fact, many industrial protocols have common technological baselines, but customize upper abstraction layers to meet different requirements for real-time Ethernet solutions. In the case of real-time communication protocols, the links and physical layers are commonly modified to achieve a better performance. This leads to hardware incompatibility problems, making communications between devices cumbersome. A common solution is the use of gateways (or bridges), which add cost, complexity and produce a loss in performance. Having a unique standard protocol would improve the interoperability between robots and facilitate the robotic component integration, which is still one of the main hurdles in the robot building process, as stated at [3]. Robotic peripheral manufacturers suffer especially from these problems because they need to support several protocols, further increasing the integration time and costs.

Time-Sensitive Networking (TSN) is a set of standards defined by the Time-Sensitive Networking task group of the IEEE 802.1 working group designed to make Ethernet more deterministic. The TSN sub-standards were created to meet different communication requirements in the industry: automation, automotive, audio, video, etc. Most of the existing real-time Ethernet solutions were created for low data volume applications such as distributed motion control. These solutions are usually very limited in bandwidth and cannot reach the Ethernet bandwidth capabilities. With the growing integration in robotics of Artificial Intelligence (AI), computer vision or predictive maintenance to name a few, there is an increasing need of sensors and actuators streaming high bandwidth data in real-time. The information provided by these components is often integrated in the control system or needs to be monitored in realtime. The common solution is to use a specific bus for real-time control and a separate one for higher bandwidth communications. As more and more high bandwidth traffic is generated, the control process of having two separated communications is inefficient. Adding real-time capabilities to Ethernet, TSN provides a common communication channel for high bandwidth traffic and real-time control traffic.

Time-Sensitive Networking (TSN) will become the de facto standard for real-time communications in robotics.

TSN will also improve the access to the robot components which is especially interesting for predictive maintenance, re-configurability or adaptability[3]. Isolated real-time communications buses difficult the communication with with sensors and actuators inside the robot. In predictive maintenance the components need to be monitorized in real-time to detect possible faults or simply to know the condition of the components. A direct communication with the monitoring systems would enhance the integration of the robot components in monitoring systems.

Time Sensitive Networking (TSN)

Overview of the standards

TSN is composed by a set of standards that aim to make Ethernet more deterministic. Several of these standards treat the problem of how to achieve bounded latencies. Standard Ethernet did not meet the deterministic capabilities for real-time applications. In the past, Ethernet endpoints used to be half-duplex and used hubs for the connection. One of the first problems Ethernet faced was the collision domain problem. Data packets could collide with one another while being sent. To deal with this, an arbitration algorithm called Carrier Sense Multiple Access with Collision Detection (CSMA/CD) was used. With the introduction of full duplex Ethernet switches, the network communications were isolated in different domains, so they do not contend for the same wire. One of the major issues was solved, but the main source causing a lack of determinism got moved to congestion problems inside the switch queues.

The main source of indeterminacy in nowadays’ switches is due to traffic contention to access the media access control (MAC) level. To limit and control the congestion problems at the queues, the IEEE introduced quality of service (QoS) mechanisms, such as traffic prioritization. Packet prioritization was introduced by the IEEE 802.1P task group in the 90’s. This QoS technique, also known as class of service (CoS) consist in a 3-bit field called the Priority Code Point (PCP) within an Ethernet frame header when using VLAN tagged frames. This field allows to specify a priority value between 0 and 7 that can be used to prioritize traffic at the MAC layer.

The Ethernet switch may have one or more transmission queues for each bridge port. Each queue provides storage for frames that await to be transmitted. The frames will be assigned to each queue according their CoS. The switching transmission algorithm will then select the next packet of the queue with the highest priority. Frames stored with lower priority will be transmitted only if higher order queues are empty during the selection process.

Transmission selection of the TSN Time-Aware Shaper

Despite packet prioritization being an effective technique to decrease the traffic interference, it is still not enough to guarantee a deterministic latency. One of the problems is that lower priority traffic still interferes the higher priority traffic. That is, high priority frames will need to wait until lower priority ones finish their transmission. The delay will depend on the size of the packet being transmitted and on the number of switches along the frame path. In applications with low latency requirements and a high number of hops, the problem becomes very significant. Additionally, as Ethernet is asynchronous, the high priority frames sharing the same link can content between them. To solve this problem, the 802.1 TSN task group developed time aware traffic scheduling, defined in 802.1Qbv [4]. The Time-Aware Scheduler (TAS) is based on a TDMA (Time Division Multiple Access) to divide a cycle time into time slots dedicated to a specific CoS. The TAS uses transmission gates for each queue and the gate can open or close the transmission of that queue (F_igure above_). The transmission selection algorithm selects the next frame of the higher order queue, but just from those queues with the gates opened. To prevent the MAC being busy when the scheduled frames arrive, the TAS introduces a guard band (G) in front of every time sensitive traffic time slice. This ensures access without delay to the MAC for time critical traffic.

The gates are programmed specifying a cycle time and a gate control list. The list configures the times slices to open and close the gates of each queue. The gate control mechanism requires time synchronization among all the Time-Aware devices on the TSN network. The most used time synchronization protocol is the IEEE 1588 Precision Time Protocol (PTP), which synchronizes the clocks by exchanging Ethernet frames with synchronization information. The IEEE 802.1 TSN task group is working on a revision of IEEE 802.1AS [5], a profile of IEEE 1588 for audio/video systems. The new revision will add some characteristics needed in other fields, such as industrial control. One of the main requirements of IEEE 802.1AS capable devices in the TSN network is a sub-microsecond synchronization among them.

Another standard developed for deterministic communications is the IEEE 802.1Qbu [6] which provides a frame preemption mechanism. This standard allows a high priority frame to interrupt a low priority frame in transmission. In order to decide if the frame can be preempted, the preemption mechanism needs a minimum necessary fragment of the frame. This fragment can be, in the worst case, 124 bytes. This means that using preemption solely does not guarantee an end-to-end deterministic latency. According to the standard, 802.1Qbu can be used in isolation to reduce latency and jitter or in combination with scheduling. When used with scheduling, it minimizes the protected window or guard band so the available bandwidth for preemptable traffic is optimized. Bandwidth optimization seems to be the main purpose of 802.1Qbu, which can be relevant for 100 Mbps.

Switched Ethernet timing model

In this section we analyze the delays involved in a switched Ethernet network. We will define an analytical model to determine the end-to-end latency in a linear topology compounded by bridged-endpoints in cut-trough mode. This model will help us to understand better the nondeterminism sources of the end-to-end latency, and we will use it to analyze the results of the experiments in a later section.

The proposed model for the end-to-end latency is based on the delays defined in [7] and [8]. First, we define some of the terms used to generalize an equation for the timing model:

Frame transmission delay (d_t): time required to transmit all of the packet’s bits into the link.
Propagation delay (d_l ): time for one bit to propagate from source to destination at propagation speed of the link.
Switch delay (d_S): time for one bit to traverse from the switch input port to the switch output port.
Switch input delay (d_{S_{in}} ): delay of the switch ingress port, including the reception PHY and MAC latency.
Switch output delay (d_{S_{out}}): delay of the switch egress port, including the transmission PHY and MAC latency.
Switch processing delay (d_{S_{p}}): time required to examine the packet’s header and determine where to direct the packet is part of the processing delay.
Switch queuing delay (d_{S_{q}} ): time until a frame waits in the egress port of a switch to start the transmission onto the link.

The propagation delay depends on the distance d between two switches and the propagation speed of the link s. The transmission delay depends on the packet length L and the link capacity C.

In cut-trough the switch delay does not depend on the packet length and can be expressed as shows the following equation

The end-to-end delay from the source endpoint A to the destination endpoint B can be expressed as the sum of the delays of all the switches and links in the path, being n the number of links and n−1 the number of switches along the path.

The key idea is that all the terms of the equation above are deterministic except the queuing delay, which depends on the switch queue occupancy when the frame arrives at time t. This delay is the main problem to bound the latency in switched Ethernet. There are different ways to bound this delay, for example QoS techniques such as Weighted Fair Queueing (QFQ) or Strict priority [9], but there is still certain delay and jitter which limits the real-time performance. Using a TSN time aware shaper and a appropriate schedule, this delay can be completely eliminated. Once this delay disappears, the end-to-end latency becomes deterministic and, combined with cut trough, it is possible to achieve a very low latency.

In the context of robotics, for a simple robot manipulator, the worst case end-to-end latency from the actuators to the robot control will determine the minimum achievable control cycle time and the maximum number of actuators allowed for a fixed cycle time. This is why reducing the queuing delay is critical to achieve low cycle times.

The cycle time is an important metric to measure the performance of the communication protocol. It can be defined as the time necessary to exchange the input and output data between the controller and all the sensors and actuators. The TAS provides real-time performance, which makes Ethernet with TSN comparable in terms of real-time performance with other real-time communication protocols. Jasperneite et al.[10] introduced an analytic method to estimate input and output cycle times for Ethernet technologies. They used this method to compare EtherCAT with Profinet IRT.

Bernier[11] extended the method to Modbus/TCP solution and Ethernet/IP. In his work, he defines the cycle time as the time necessary to exchange the input and output data between the controller and all the sensors and actuators. Bruckner et al.[12] used this method to compare the most common communication technologies performance with the OPC UA over TSN. The conclusions drawn by this last work shows how a TSN based technology using 1 Gbit Ethernet and a frame aggregation approach can outperform other hard real-time industrial protocols. In the following section, we will challenge these results experimentally in the context of robotics.

Experimental results

For the experimental setup, we have selected a typical robotic use case with mixed-critical traffic. In particular, we have chosen a modular robotic arm with a high bandwidth sensor attached at the end (Figure below). Such robot arm is an interesting use case for TSN because it gathers traffic with different characteristics. The actuators require hard real-time low latency traffic while sensors, such as high resolution camera or a laser scanner, generate high volume data. The proposed setup contains two actuators, A1 and A2, a sensor S and a robot controller RC.

Exemplary modular robotic arm used for the experimental setup.

For the experiment we will use two TSN capable bridged endpoints and two PCs, each one with an Intel i210 card.

Experimental setup networking devices overview.

The TSN bridged endpoints simulate the actuators and the PCs the sensor S and the robot controller RC. The sensor S is connected to the last actuator and will send a high bandwidth to the robot controller. As the sensor is connected in a chain topology, the sensor traffic goes through each of the actuators switches. This traffic will contend with the time sensitive traffic from the actuators switch queues, which will produce a queuing delay.

We call F_{A1} to the traffic flow from A1 to RC, F_{A2} to the traffic flow from A2 to RC and FS to the traffic flow from S to RC, as shown in figure 7. The measurements are performed in RC using the hardware time-stamps capabilities of the i210 network card. For the transmission time, we take advantage of the TSN capabilities of the bridged endpoints. We set the transmission times t_{TX_1} and t_{TX_2} of the TSN bridged endpoints by configuring the TAS of each endpoint. For an actuator i = 1, 2, we calculate the delay d_i from A_i to RC by a simple subtraction.

As shown in the Figure above, the hardware time-stamp of RC measures the arrival time of the Start of Frame Delimiter (SFD). This means that the frame transmission delay is not included in our measurements. As the switches are configured in cut-trough, the transmission delay is neither included in the switch delay and hence the measured delay does not depend on the frame length. From the image we can relate the measurements with the delays expressed in previous equations. Notice that in this particular case the sources are switched endpoints. The delays from the endpoint trough the switch d_{SE_i} are not the same of the switch delay d_{S_i} because it does not include the input port delay. Expressing the measured delay in terms of the switch delay, we arrive to the following expressions:

To generate the F_{A1} and F_{A2} traffic, we have used UDP/IP with 256 Bytes payload at a 1 ms rate. For F_S, we used the iperf tool to generate a network load using a 1500 Byte payload with a traffic bandwidth of 900 Mbps for 1 Gbps link capacity and 90 Mbps for 100 Mbps link capacity. For the calculations, we measured 10000 samples during over a 10 seconds period.

Experiment 1.

Same priority blocking: In this experiment, we measure the blocking effect of same priority traffic. All the traffic flows are sent trough lowest priority queue, which is the best-effort queue (BE). F_{A1} and F_{A2} will contend with FS in the port 1 BE queue of the A_1 and A_2 switches. The blocking delay depends on the implemented transmission selection mechanism of the switch.

Timeplot delay measurements for 10s.Terms: C=Link Capacity,. Terms: A1=Actuator1 (Blue), A2=Actuator 2 (Red), BE=Best-effort queue, ST=Scheduled Traffic queue. a) Same priority blocking for C=1 Gbps b) Lower priority blocking for C=1 Gbps c) Lower priority blocking for C=100 Mbps d) Using a TAS for C=1 Gbps

The switches used for this setup use a FIFO occupancy credit based arbitration, from now on referred as CBF. CBF prioritizes the traffic coming from a higher occupancy ingress port queue. This makes the blocking effect highly dependent on the sending rates of the contending sources of traffic. The results (subfigure a) show worst case delays in the order of milliseconds, which clearly shows that the latency is unbounded. Increasing FS bandwidth or decreasing F_{A1} sending rates, we get higher delays and even packet loss. Note that the purpose of this experiment is not to analyze the queuing delay of the CBF but only to illustrate the non-determinism of this arbitration policy due to the same priority blocking.

Experiment 2.

Lower priority blocking: In this experiment, we measure the blocking effect of lower priority traffic using a strict priority transmission. F_{A1} and F_{A2} are sent trough the highest priority queue of the switches, which is the scheduled traffic queue (ST). Previous work [14] analyzed the effect of lower priority blocking. The worst case delay added by each switch is given by the frame transmission time for the maximum frame size allowed by the switch (MTU). For a 1500 Byte MTU and a 100 Mbps link capacity, the worst case added delay would be 120 µs, for 1 Gbps 12 µs. The experimental results are presented in Tables II and III and pictured in the subfigures b and c above. These results confirm the expected worst case delay. In the case of 100 Mbps, we have a maximum delay of 127.22 µs for d_1 and 253.33 µs for d_2. The results show how the effect is accumulative for each bridge. While this queuing delay in this case has an upper bound, it clearly highly limits the real-time performance and the scalability of the system. For 100 Mbps and network load, the delay goes approximately from 3 to 120 µs traffic for A_1 and from 9 to 250 µs for A_2. As F_{A1} and F_{A2} are sent asynchronously, F_{A1} and F_{A2} can content between them. Therefore, in this scenario, apart from the lower priority blocking, the worst case must take into account the same priority blocking of F_{A1} and F_{A2}.

Experiment 3.

Using a TSN Time-Aware Shaper: In this experiment we configure the TAS of A1 and A2 switches egress ports. As all the TAS are synchronized with submicrosecond accuracy, the scheduled traffic is perfectly isolated from lower priority traffic and from the same priority traffic. As shown in subfigure d, the end-to-end latency is highly deterministic, the results with and without lower priority traffic are in the same order and sub-microsecond jitter is achieved.

Timeplot delay measurements for 10s. Link Capacity 100 Mbps. Terms: A1=Actuator1 (Blue), A2=Actuator 2 (Red), BE=Best-effort queue, ST=Scheduled Traffic queue. a) Same priority blocking b) Same priority blocking with network load c) Lower priority blocking d) Lower priority blocking with network load e) Using a TAS f) Using a TAS with network load.

Timeplot delay measurements for 10s. Link Capacity 1 Gbps. Terms: A1=Actuator1 (Blue), A2=Actuator 2 (Red), BE=Best-effort queue, ST=Scheduled Traffic queue. a) Same priority blocking b) Same priority blocking with network load c) Lower priority blocking d) Lower priority blocking with network load e) Using a TAS f) Using a TAS with network load.

Conclusion

In this work we have presented an experimental setup to show the suitability of TSN for real-time robotic applications. We have compared the delays experienced by the queuing delay in Ethernet switches for standard Ethernet, against the delays when using a TSN Time-Aware shaper. The results showed the indeterminacy of Ethernet and how these problems can limit the scalability and performance in real-time robotic applications such as the exemplary modular robot. When the TSN Time-Aware shaper was used, the results showed that the time sensitive traffic was perfectly isolated from lower priority traffic, maintaining low latency and jitter even in the presence of high bandwidth background traffic. These results suggest that it is possible to develop hard real-time motion control systems mixed with high bandwidth sensors, such as LIDARs and high resolution cameras. Based on the presented results, we claim that Ethernet with TSN standards will become the de facto standard for communications on layers 1 and 2, in robotics. We argue that, within robotics, many of the existing real-time industrial solutions will slowly be replaced by TSN. For higher layers, we foresee a contending landscape where the integration of TSN in different middleware solutions focused on interoperability such as OPC-UA and DDS promise to deliver a bottom-up real-time communication solution.

References

[1] V. Mayoral, A. Hernández, R. Kojcev, I. Muguruza, I. Zamalloa, A. Bilbao, and L. Usategui, “The shift in the robotics paradigm ; the hardware robot operating system (h-ros); an infrastructure to create interoperable robot components,” in 2017 NASA/ESA Conference on Adaptive Hardware and Systems (AHS), July 2017, pp. 229–236.

[2] I. Zamalloa, I. Muguruza, A. Hernández, R. Kojcev, and V. Mayoral, “An information model for modular robots: the Hardware Robot Information Model (HRIM),” ArXiv e-prints, Feb. 2018.

[3] V. Mayoral, R. Kojcev, N. Etxezarreta, A. Hernández, and I. Zamalloa, “Towards self-adaptable robots: from programming to training machines,” ArXiv e-prints, Feb. 2018.

[4] “Ieee standard for local and metropolitan area networks — bridges and bridged networks — amendment 25: Enhancements for scheduled traffic,” IEEE Std 802.1Qbv-2015 (Amendment to IEEE Std 802.1Q2014 as amended by IEEE Std 802.1Qca-2015, IEEE Std 802.1Qcd2015, and IEEE Std 802.1Q-2014/Cor 1–2015), pp. 1–57, March 2016.

[5] “Ieee standard for local and metropolitan area networks — timing and synchronization for time-sensitive applications in bridged local area networks,” IEEE Std 802.1AS-2011, pp. 1–292, March 2011.

[6] “Ieee standard for local and metropolitan area networks — bridges and bridged networks — amendment 26: Frame preemption,” IEEE Std 802.1Qbu-2016 (Amendment to IEEE Std 802.1Q-2014), pp. 1–52, Aug 2016.

[7] J. F. Kurose and K. W. Ross, Computer Networking: A Top-Down Approach (6th Edition), 6th ed. Pearson, 2012.

[8] S. Thangamuthu, N. Concer, P. J. L. Cuijpers, and J. J. Lukkien, “Analysis of ethernet-switch traffic shapers for in-vehicle networking applications,” in 2015 Design, Automation Test in Europe Conference Exhibition (DATE), March 2015, pp. 55–60.

[9] J. P. Georges, T. Divoux, and E. Rondeau, “Strict priority versus weighted fair queueing in switched ethernet networks for time critical applications,” in 19th IEEE International Parallel and Distributed Processing Symposium, April 2005, pp. 141–141.

[10] J. Jasperneite, M. Schumacher, and K. Weber, “Limits of increasing the performance of industrial ethernet protocols,” in 2007 IEEE Conference on Emerging Technologies and Factory Automation (EFTA 2007), Sept 2007, pp. 17–24.

[11] C. Bernier, “Industrial robot communication protocols.” [Online]. Available: https://blog.robotiq.com/bid/65821/ Industrial-Robot-Communication-Protocols

[12] M.-P. S. A. A. W. S. D. K. S. S. R. W. K. W. L. L. M. S. R. H. E.-C. L. S. R. D. Bruckner, R. Blair, “Opc ua tsn: A new solution for industrial communication,” 2018. [Online]. Available: https://www.moxa.com/doc/white_papers/opc-ua-tsn.pdf

[13] J. Chen, Z. Wang, and Y. Sun, “Real-time capability analysis for switch industrial ethernet traffic priority-based,” in Proceedings of the International Conference on Control Applications, vol. 1, 2002, pp. 525–529 vol.1.

[14] M. Felser, “Real time ethernet: Standardization and implementations,” in 2010 IEEE International Symposium on Industrial Electronics, July 2010, pp. 3766–3771.