Ethernet goes real-time: a survey on research and technological developments

Ethernet is the most popular LAN technology. Its low price and robustness, resulting from its wide acceptance and deployment, has created an eagerness to expand its responsibilities to the factory-floor, where real-time requirements are to be fulfilled. However, it is difficult to build a real-time control network using Ethernet, because its MAC protocol, the 1-persistent CSMA/CD protocol with the BEB collision resolution algorithm, has unpredictable delay characteristics. Many anticipate that the recent technological advances in Ethernet such as the emerging Fast/Gigabit Ethernet, micro-segmentation and full-duplex operation using switches will also enable it to support time-critical applications. This technical report provides a comprehensive look at the unpredictability inherent to Ethernet and at recent technological advances towards real-time operation

Ethernet - a survey on its fields of application

During the last decades, Ethernet progressively became the most widely used local area networking (LAN) technology. Apart from LAN installations, Ethernet became also attractive for many other fields of application, ranging from industry to avionics, telecommunication, and multimedia. The expanded application of this technology is mainly due to its significant assets like reduced cost, backward-compatibility, flexibility, and expandability. However, this new trend raises some problems concerning the services of the protocol and the requirements for each application. Therefore, specific adaptations prove essential to integrate this communication technology in each field of application. Our primary objective is to show how Ethernet has been enhanced to comply with the specific requirements of several application fields, particularly in transport, embedded and multimedia contexts. The paper first describes the common Ethernet LAN technology and highlights its main features. It reviews the most important specific Ethernet versions with respect to each application field’s requirements. Finally, we compare these different fields of application and we particularly focus on the fundamental concepts and the quality of service capabilities of each proposal

Hierarchical Up/Down Routing Architecture for Ethernet backbones and campus networks

We describe a new layer two distributed and scalable routing architecture. It uses an automatic hierarchical node identifier assignment mechanism associated to the rapid spanning tree protocol. Enhanced up/down mechanisms are used to prohibit some turns at nodes to break cycles, instead of blocking links like the spannning tree protocol does. The protocol performance is similar or better than other turn prohibition algorithms recently proposed with lower complexity O(Nd) and better scalability. Simulations show that the fraction of prohibited turns over random networks is less than 0.2. The effect of root bridge election on the performance of the protocol is limited both in the random and regular networks studied. The use of hierarchical, tree-descriptive addresses simplifies the routing, and avoids the need of all nodes having a global knowleddge of the network topology. Routing frames through the hierarchical tree at very high speed is possible by progressive decoding of frame destination address, without routing tables or port address learning. Coexistence with standard bridges is achieved using combined devices: bridges that forward the frames having global destination MAC addresses as standard bridges and frames with local MAC frames with the proposed protocol.Publicad

Design issues of a back-pressure-based congestion control mechanism

Congestion control in packet-based networks is often realized by feedback protocols -- in this paper we assess the performance under a back-pressure mechanism that has been proposed and standardized for Ethernet metropolitan networks. Relying on our earlier results for feedback fluid queues, we derive explicit expressions for the key perfomance metrics, in terms of the model parameters, as well as the parameters agreed upon in the service level agreement. Numerical experiments are performed to evaluate the main trade-offs of this model (for instance the trade-off between the signaling frequency and the throughput). These can be used to generate design guidelines. The paper is concluded by an elementary, yet powerful, Markovian model that can be used as an approximative model in situations of large traffic aggregates feeding into the system; the trade-offs and guidelines identified for the feedback fluid model turn out to carry over to this more stylized model

PABO: Mitigating Congestion via Packet Bounce in Data Center Networks

In today's data center, a diverse mix of throughput-sensitive long flows and delay-sensitive short flows are commonly presented in shallow-buffered switches. Long flows could potentially block the transmission of delay-sensitive short flows, leading to degraded performance. Congestion can also be caused by the synchronization of multiple TCP connections for short flows, as typically seen in the partition/aggregate traffic pattern. While multiple end-to-end transport-layer solutions have been proposed, none of them have tackled the real challenge: reliable transmission in the network. In this paper, we fill this gap by presenting PABO -- a novel link-layer design that can mitigate congestion by temporarily bouncing packets to upstream switches. PABO's design fulfills the following goals: i) providing per-flow based flow control on the link layer, ii) handling transient congestion without the intervention of end devices, and iii) gradually back propagating the congestion signal to the source when the network is not capable to handle the congestion.Experiment results show that PABO can provide prominent advantage of mitigating transient congestions and can achieve significant gain on end-to-end delay

Ethernet Networks for Real-Time Use in the ATLAS Experiment

Ethernet became today's de-facto standard technology for local area networks. Defined by the IEEE 802.3 and 802.1 working groups, the Ethernet standards cover technologies deployed at the first two layers of the OSI protocol stack. The architecture of modern Ethernet networks is based on switches. The switches are devices usually built using a store-and-forward concept. At the highest level, they can be seen as a collection of queues and mathematically modelled by means of queuing theory. However, the traffic profiles on modern Ethernet networks are rather different from those assumed in classical queuing theory. The standard recommendations for evaluating the performance of network devices define the values that should be measured but do not specify a way of reconciling these values with the internal architecture of the switches. The introduction of the 10 Gigabit Ethernet standard provided a direct gateway from the LAN to the WAN by the means of the WAN PHY. Certain aspects related to the actual use of WAN PHY technology were vaguely defined by the standard. The ATLAS experiment at CERN is scheduled to start operation at CERN in 2007. The communication infrastructure of the Trigger and Data Acquisition System will be built using Ethernet networks. The real-time operational needs impose a requirement for predictable performance on the network part. In view of the diversity of the architectures of Ethernet devices, testing and modelling is required in order to make sure the full system will operate predictably. This thesis focuses on the testing part of the problem and addresses issues in determining the performance for both LAN and WAN connections. The problem of reconciling results from measurements to architectural details of the switches will also be tackled. We developed a scalable traffic generator system based on commercial-off-the-shelf Gigabit Ethernet network interface cards. The generator was able to transmit traffic at the nominal Gigabit Ethernet line rate for all frame sizes specified in the Ethernet standard. The calculation of latency was performed with accuracy in the range of +/- 200 ns. We indicate how certain features of switch architectures may be identified through accurate throughput and latency values measured for specific traffic distributions. At this stage, we present a detailed analysis of Ethernet broadcast support in modern switches. We use a similar hands-on approach to address the problem of extending Ethernet networks over long distances. Based on the 1 Gbit/s traffic generator used in the LAN, we develop a methodology to characterise point-to-point connections over long distance networks. At higher speeds, a combination of commercial traffic generators and high-end servers is employed to determine the performance of the connection. We demonstrate that the new 10 Gigabit Ethernet technology can interoperate with the installed base of SONET/SDH equipment through a series of experiments on point-to-point circuits deployed over long-distance network infrastructure in a multi-operator domain. In this process, we provide a holistic view of the end-to-end performance of 10 Gigabit Ethernet WAN PHY connections through a sequence of measurements starting at the physical transmission layer and continuing up to the transport layer of the OSI protocol stack

Converged IP-over-standard ethernet progress control networks for hydrocarbon process automation applications controllers

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.The maturity level of Internet Protocol (IP) and the emergence of standard Ethernet interfaces of Hydrocarbon Process Automation Application (HPAA) present a real opportunity to combine independent industrial applications onto an integrated IP based network platform. Quality of Service (QoS) for IP over Ethernet has the strength to regulate traffic mix and support timely delivery. The combinations of these technologies lend themselves to provide a platform to support HPAA applications across Local Area Network (LAN) and Wide Area Network (WAN) networks. HPAA systems are composed of sensors, actuators, and logic solvers networked together to form independent control system network platforms. They support hydrocarbon plants operating under critical conditions that — if not controlled — could become dangerous to people, assets and the environment. This demands high speed networking which is triggered by the need to capture data with higher frequency rate at a finer granularity. Nevertheless, existing HPAA network infrastructure is based on unique autonomous systems, which has resulted in multiple, parallel and separate networks with limited interconnectivity supporting different functions. This created increased complexity in integrating various applications and resulted higher costs in the technology life cycle total ownership. To date, the concept of consolidating HPAA into a converged IP network over standard Ethernet has not yet been explored. This research aims to explore and develop the HPAA Process Control Systems (PCS) in a Converged Internet Protocol (CIP) using experimental and simulated networks case studies. Results from experimental and simulation work showed encouraging outcomes and provided a good argument for supporting the co-existence of HPAA and non-HPAA applications taking into consideration timeliness and reliability requirements. This was achieved by invoking priority based scheduling with the highest priority being awarded to PCS among other supported services such as voice, multimedia streams and other applications. HPAA can benefit from utilizing CIP over Ethernet by reducing the number of interdependent HPAA PCS networks to a single uniform and standard network. In addition, this integrated infrastructure offers a platform for additional support services such as multimedia streaming, voice, and data. This network‐based model manifests itself to be integrated with remote control system platform capabilities at the end user's desktop independent of space and time resulting in the concept of plant virtualization

Robustness of Ethernet-Based Real-Time Networked Control System with Multi-Level Client/Server Architecture

The importance of real-time communication at the device level in a factory automation setup is a widely researched area. This research is an effort to experimentally verify if Ethernet can be used as a real-time communication standard in a factory automation setup, by observing the effects of packet delays, packet loss, and network congestion on the performance of a networked control system (NCS). The NCS experimental setup used in this research involves real-time feedback control of multiple plants like DC motors and a magnetic-levitation system connected to one or more controllers. A multi-client-multi-server architecture on a local area network (LAN) was developed using user datagram protocol (UDP) as the communication protocol. Key observations are as follows. (1) The multi-client-single-server system showed the highest packet delays compared to single-client-single-server architecture. (2) In the singleclient- single-server system, as the Ethernet link utilization increased beyond 82 percent, the average packet delays and steady-state error of the DC motor speed-control system increased by 2231 percent and 304 percent, respectively. (3) Even under high link utilization, adding an additional server to the NCS reduced average packet delays considerably. (4) With large packet sizes, higher packet rates were automatically throttled by Ethernet’s flow control mechanism affecting the real-time communication negatively. (5) In the multiclient- multi-server architecture, average packet delays at higher packet rates, and at higher packet lengths were found to be 40 percent lesser than the those of the single-clientsingle- server system and 87.5 percent lesser than those of the multi-client-single-server system

Investigating Performance and Reliability of Process Bus Networks for Digital Protective Relaying

To reduce the cost of complex and long copper wiring, as well as to achieve flexibility in signal communications, IEC 61850 part 9-2 proposes a process bus communication network between process level switchyard equipments, and bay level protection and control (P&C) Intelligent Electronic Devices (IEDs). After successful implementation of Ethernet networks for IEC 61850 standard part 8-1 (station bus) at several substations worldwide, major manufacturers are currently working on the development of interoperable products for the IEC 61850-9-2 based process bus. The major technical challenges for applying Ethernet networks at process level include: 1) the performance of time critical messages for protection applications; 2) impacts of process bus Ethernet networks on the reliability of substation protection systems. This work starts with the performance analysis in terms of time critical Sampled Value (SV) messages loss and/or delay over the IEC 61850-9-2 process bus networks of a typical substation. Unlike GOOSE, the SV message is not repeated several times, and therefore, there is no assurance that each SV message will be received from the process bus network at protection IEDs. Therefore, the detailed modeling of IEC 61850 based substation protection devices, communication protocols, and packet format is carried out using an industry-trusted simulation tool OPNET, to study and quantify number of SV loss and delay over the process bus. The impact of SV loss/delay on digital substation protection systems is evident, and recognized by several manufacturers. Therefore, a sample value estimation algorithm is developed in order to enhance the performance of digital substation protection functions by estimating the lost and delayed sampled values. The error of estimation is evaluated in detail considering several scenarios of power system relaying. The work is further carried out to investigate the possible impact of SV loss/delay on protection functions, and test the proposed SV estimation algorithm using the hardware setup. Therefore, a state-of-the-art process bus laboratory with the protection IEDs and merging unit playback simulator using industrial computers on the QNX hard-real-time platform, is developed for a typical IEC 61850-9-2 based process bus network. Moreover, the proposed SV estimation algorithm is implemented as a part of bus differential and transmission line distance protection IEDs, and it is tested using the developed experimental setup for various SV loss/delay scenarios and power system fault conditions. In addition to the performance analysis, this work also focuses on the reliability aspects of protection systems with process bus communication network. To study the impact of process bus communication on reliability indices of a substation protection function, the detailed reliability modeling and analysis is carried out for a typical substation layout. First of all, reliability analysis is done using Reliability Block Diagrams (RBD) considering various practical process bus architectures, as well as, time synchronization techniques. After obtaining important failure rates from the RBD, an extended Markov model is proposed to analyze the reliability indices of protection systems, such as, protection unavailability, abnormal unavailability, and loss of security. It is shown with the proposed Markov model that the implementation of sampled value estimation improves the reliability indices of a protection system