System and Method for Deterministic I/O with Ethernet Based Industrial Networks
A networking system is discussed. The system may be used for industrial networks, where deterministic behavior is often valued. Bounded message travel times may be achieved for a first set of network traffic. Additional traffic may be routed over the networking system without interfering with the message travel times associated with the first set of network traffic. Systems and methods for assigning priority to various sets of network traffic are discussed.
The present application is a continuation patent application of U.S. patent application Ser. No. 13/357,762, filed Jan. 25, 2012, which is hereby incorporated by reference.BACKGROUND
Many environments are sensitive to the amount of time it takes for messages to travel between two networked machines. For example, in many industrial applications it is essential to know the maximum time it will take for a message to travel from one device to another. Most widespread communications protocols, such as TCP/IP, cannot provide the consistent, predictable behavior needed for environments that are sensitive to the time it takes for information to be communicated between devices. Consequently, these environments have used specialized communications hardware to achieve predictable message travel times.SUMMARY
Some aspects of the disclosure relate to methods for transmitting messages with predictable message travel times using standard Ethernet hardware and message routing that is compatible with established standards such as IEEE 802.1D/Q.
Additional aspects of the disclosure relate to determining the application response time and/or message travel time for various networks and how changes in network topology may affect these times.
According to still further aspects of the disclosure, devices that transmit low-priority traffic may be attached to a network, such as an industrial network, without affecting the application response time and/or message travel time for higher-priority traffic.
Additional aspects of the disclosure relate to how devices on a network may add, remove, or alter priority tags on received traffic. These behaviors may ensure that devices that transmit low-priority traffic do not interfere with higher-priority traffic.
The preceding presents a simplified summary in order to provide a basic understanding of some aspects of the disclosure. The summary is not an extensive overview of the disclosure. It is intended neither to identify key or critical elements of the disclosure nor to delineate the scope of the disclosure. The summary merely presents some concepts of the disclosure in a simplified form as a prelude to the description below.
The present disclosure is illustrated by way of example and is not limited in the accompanying figures.
In the following description of various illustrative embodiments, reference is made to the accompanying drawings, which form a part hereof, and in which is shown, by way of illustration, various embodiments in which aspects of the disclosure may be practiced. It is to be understood that other embodiments may be utilized, and structural and functional modifications may be made, without departing from the scope of the present disclosure.
In the example of
Many industrial processes rely on inputs being processed and converted to outputs in a deterministic fashion. For example, the process controlling the devices of
In some systems, an equipment failure, such as a cable failure or any other violation of the expected operating conditions, may cause an ART limit to be violated. Some failures may cause the system to stop meeting the ART limit until the failure is repaired. Other failures may cause the system to stop meeting the ART limit only momentarily. For example, a single cable failure may cause a maximum of 100 ms of additional message travel delay as the devices of the system reconfigure themselves. If this is the case, the 250 ms ART limit of the example above may be violated once, but the system may resume meeting the 250 ms ART limit thereafter.
Some failures may not cause a system to stop working as designed at all. For example, if a single cable failure may cause a maximum additional delay of 80 ms, then the 250 ms ART limit of the example above may still be met even when a cable failure occurs. (In the example above, the maximum message travel and processing delay was 170 ms. 170 ms+80 ms≦250 ms.)
The number of devices that are connected together may affect the maximum message travel time. Continuing the above example, it may take up to 20 ms to communicate a message across each of Ethernet cables 141-144. Thus, the maximum delay for communications between controller 100 and valve motor interface 110 is 20 ms if the message is transmitted across Ethernet cable 141. If the message is instead transmitted across Ethernet cables 144, 143, and 142, the maximum delay is 60 ms, because three hops, each taking up to 20 ms, were used instead of one. A message may be transmitted along this longer route due to, for example, a cable failure or message routing rules. For instance, the message transmitted to valve motor interface 110 may also need to be received by sensor interface 130, or by some other device. Connecting more devices together may increase the maximum message travel time by increasing the number of hops each message may traverse. The delay at each hop may be caused by, for example, transmission delay (also known as store-and-forward or switch-and-forward delay, which is the time it takes to transmit a message), propagation delay (the time it takes for the message to travel across a cable), processing delay (the time it takes for a device to process the message), and queuing delay (the time a message may need to wait for other messages). These delays will be discussed in more detail below.
Although network interfaces for sensors and a valve motor were identified in
The maximum total ART may be calculated by summing all of the delays that may occur under normal operating conditions. These delays will now be discussed in the order they may occur. First, an I/O device generates a message. This message may be, for example, a reading from a sensor. Many devices accept new messages at a predefined time interval, which may be known as a scan rate. The new message may not be generated until just after a scan begins. Thus, the message may not be accepted by the network interface of the I/O device until the next scan begins. Thus, a delay of one input scan may be possible before the message is accepted for transmission.
Once a message is accepted for transmission by the network interface of the I/O device, the message may not be transmitted immediately. A message may be delayed until other traffic that is to be output from the I/O device on the same port has cleared. Assuming the message is prioritized above all other traffic, the maximum delay is the delay that results from the message being received just after an earlier message begins being transmitted. This time may be referred to as a queue delay, as it is the maximum time the message may be queued before its transmission begins.
If a message is not prioritized above all other traffic, then the length of the queue delay would depend on the volume of higher-priority traffic to be transmitted before the message. For this reason, messages related to applications with ART limits may be given the highest priority. Queue delay can also be limited by controlling the total amount of traffic that may be transmitted on the network. In some cases, only high-priority traffic is limited. For example, the number of devices capable of transmitting high-priority traffic may be limited such that the devices, even when transmitting at their maximum rate, do not transmit enough high-priority messages to cause a backup of high-priority messages.
In addition to queuing delay, a processing delay may exist. One example of a processing delay is that, in network interfaces configured to produce packets at a predefined time interval, and a message may be delayed until the beginning of the next interval. Another example of a processing delay is the time required to examine a received message and determine what to do with it.
The next delay is the time it takes to transmit the message. This transmission delay is affected by the size of the message and the speed of the link.
Once a message is transmitted from one I/O device, the next delay is the time it takes for the message to be relayed back to the controller. This time depends on the number of hops between a device and the controller. At each hop, the message may be delayed by the time it takes for the message to be received, placed in the output queue, and transmitted to the next hop. Each of these steps was described above with respect to the message being received, placed in the output queue, and transmitted from the device that originates the message.
A switch, such as switch 220, may receive several messages simultaneously due to it having several ports. If incoming messages are received simultaneously on several of these ports, and several of the messages are to be transmitted on the same port, the queuing delay for the switch may become greater than the queueing delay for devices with fewer ports. Each additional port that a switch contains adds the possibility that the size of the message queue will increase by one message.
Once the message reaches the controller, it may be delayed by the time it takes the controller to receive the message (i.e. by the length of the controller's scan interval). Once received, the message will then be processed and used by the controller to generate a second message. For instance, in the example of
The second message must then be output from the controller to the network. This may result in delays as the second message is received by the controller's network interface, queued within the controller's network interface, and then transmitted from the controller's network interface. In some embodiments, the controller may generate messages such that there is no queuing delay at this stage. Also, in some embodiments the time required for the second message to be received by the controller's network interface may be very limited. This can be accomplished by immediately transmitting the second message regardless of the scan cycle. In some embodiments the technique of immediately transmitting a message regardless of the scan cycle may be applied to other devices in the network besides the controller, including I/O devices.
Once the second message is transmitted from the controller's network interface, the next delay is the time it takes for the message to be transmitted to the appropriate I/O device(s). Like the delay for a message traveling from an I/O device to the controller, this delay depends on the number of hops between the device(s) receiving the second message and the controller. At each hop, the message may be delayed by the time it takes for the message to be received, placed in the output queue, and transmitted to the next hop.
Once the second message is received by the destination I/O device's network interface, the next delay is the time it takes to pass the message from the network interface to the device itself. Similar to the controller and the network interface itself, the device may accept new messages with a pre-defined scan rate. Thus, the second message may be delayed by up to the length of a scan.
Summing each of the delays above allows the ART and message travel time for a particular message path to be calculated. In order to allow for cable failures, the ART and/or message travel time may also be calculated by assuming that all messages will travel around each daisy-chain loop in the direction that includes the most hops. The calculation can be simplified further by setting a maximum number of devices per daisy-chain loop and a maximum number of loops. Instead of identifying the actual number of hops that exist, the maximum number of hops and switch traversals that are allowed by these maximums may be assumed. Making this assumption may yield an ART and/or message travel time that is greater than the real maximum. The ART and/or message travel time calculated with this assumption may be useful because it provides an upper bound on what the real ART and/or message travel time may be. Further, this upper bound may apply even if devices are added to the network or the network is otherwise reconfigured.
A device's network interface may be integral with or separate from the device itself. For example, a device's network interface may be a separate module that connects to the device using, for example, a group of wires. Although
Industrial networks may carry traffic that is not as time sensitive as the traffic for which an ART needs to be known. This less-time-sensitive traffic may be carried on the same Ethernet cables used to transmit the messages for which an ART needs to be known. Carrying both of these types of traffic on the same Ethernet cables can be accomplished by prioritizing the less-time-sensitive messages behind the more-time-sensitive messages. For example, a device may have four messages queued for transmission. The device may select the highest priority message for transmission first, regardless of when that message arrived relative to the other messages in the queue.
I/O device 310n includes four Ethernet ports. Computer 352 is attached on one of the ports. Server 353 is attached to another of the ports. These attached devices may serve any purpose. Examples may include logging and storing information about the operation of the industrial network. Other examples may be unrelated to the operation of the industrial network. For example, network 370 may be a corporate intranet. Computer 352 and/or server 353 may connect to the industrial network in order to reach network 370. Similarly, network 360 is also connected to the industrial network, and the industrial network may be used to transport messages between these networks. Computers 351 and 352, server 353, and networks 360 and 370 are not illustrated as being connected using daisy-chain loops. Unlike a daisy-chain loop, these connections may fail if only one Ethernet cable fails. These devices could be connected using a daisy-chain loop to reduce the impact of a cable failure. Similarly, the devices that are in a daisy-chain loop may be connected using another topology, such as a simple daisy-chain (without a loop) or hub-and-spoke connections.
More-time-sensitive messages may be prioritized above less-time-sensitive messages by using the tags specified by the IEEE 802.1Q standard. These tags are part of the Ethernet headers defined the IEEE 802.1Q standard. The tags may also be known as 802.1D/Q tags, which is a reference to the IEEE 802.1D standard that may be implemented concurrently with the IEEE 802.1Q standard. The tags specified by IEEE 802.1Q include three bits that represent the priority of an Ethernet frame. By using the 802.1Q standard and the techniques described herein, message transmission with a bounded delay can be achieved for high-priority messages without requiring any special-purpose hardware. Instead, all hardware in the network may be compliant with one or more of the IEEE 802.3 standards for wired Ethernet. These techniques may be combined with the EtherNet/IP standard. The “IP” in this standard stands for “Industrial Protocol.”
Other tags may also be used to indicate the priority of messages transmitted within an industrial network. For example, if messages are contained in IP (internet protocol) packets, then differentiated services control point (DSCP) tags, which are 6-bit fields of IP headers, may be used. Further, a non-standard tag may be used to identify the priority of messages within an industrial network.
In some embodiments, tags for priority may be included only on the daisy-chain loops (or other network topologies) over which the most time-sensitive messages travel. In other embodiments, tags for priority may be included in messages that travel over some or all of the other Ethernet cables, such as the cable connecting computer 351 to I/O device 310a. Where tags that indicate priority are included in messages that travel over cables that connect to devices that generate lower-priority traffic, those tags may be checked in order to ensure they do not include high-priority tags. If high-priority tags are discovered coming from ports that are designated for lower-priority traffic, the high-priority tags may be replaced with tags indicating a lower level of priority.
In the example of
In the example of
The priority tag for traffic received on ports 2 or 3 is not altered if that traffic will also be transmitted on ports 2 or 3. If traffic is received on ports 2 or 3 without a tag, the traffic may be tagged as low priority if it will be transmitted on ports 2 or 3. Traffic received on ports 2 or 3 that will be transmitted on the internal port may mirror this behavior. As mentioned above with respect to traffic received from port 1, the internal port may store the priority levels of the messages without storing a tag in the messages themselves.
Messages from the internal port that are destined for ports 2 or 3 may already be tagged based on, for example, the content of the messages. If a tag already exists, the priority value of the tag may be maintained. If messages received on the internal port do not already include a tag, then a tag may be generated with a priority level that is determined based on the content of the message. For example, messages related to a process with a maximum ART may be given the highest priority, and other messages may be given lower priorities. The priority to assign to messages received on the internal port may be determined based on other priority tags included within the message. For example, if 802.1Q tags are being used and the message content contains a DSCP tag, an 802.1Q tag may be generated with a priority level that corresponds to the priority level of the DSCP tag. A default priority level may be used for messages received on the internal port without tags. Because the internal port may be more likely to receive time-sensitive messages than port 1, this default priority level may be a higher level than the low priority level given to traffic from port 1.
The behavior illustrated in
Other allocations of the priority levels are possible. For instance, only the highest priority level may be reserved for use by the devices connected to ports 2 or 3. This could be achieved by tagging traffic received on port 1 with a priority level that reflects the priority tag with which the traffic was received. But, if the priority tag that is received on port 1 is the highest priority, the tag would be re-written to the next lower priority level (i.e. the second-highest priority level). Traffic received on port 1 without a tag may be given a default priority level, which may or may not be the lowest priority level.
Another possible allocation of the priority levels is for an intermediate number of the priority levels to be reserved for use by the devices connected to ports 2 or 3. Traffic with the reserved priority levels received on port 1 may be assigned the highest non-reserved priority level. Alternatively, a mapping of the received priority levels to output priority levels may be established. For example, the received priority levels may be divided by two. The threshold between the priority level(s) reserves for the second and third ports and the priority level(s) available to the first port may be configurable.
In the example of
The example of
If the message was received from a low priority port, then it is determined, at step 610, whether the message is to be output from a low priority port or a high priority port. If the message is to be output from a low priority port, then it is output with no tag in step 611. If the message is to be output from a high priority port, then it is output with a tag indicating low priority in step 612.
If step 602 determines that the message was received from a port that is not designated as a low priority port, then the method continues to step 620, where it is determined whether the message was received from an internal port.
If the message was not received from an internal port, then whether the destination port is a high or low priority port is determined at step 630. If the destination port is a low priority port, then the message is output from the low priority port without a tag at step 631. If the destination port is a high priority port, then whether a tag is included in the received message is determined in step 632. If a tag is included in the received message, then the message is output with a tag indicating the same priority as the tag included in the received message in step 633. If a tag is not included in the received message, then the message is output with a tag indicating low priority in step 634.
If the message was received from an internal port, then whether the destination port is a high or low priority port is determined at step 640. If the destination port is a low priority port, then the message is output from the low priority port without a tag at step 641. If the destination port is a high priority port, then whether a tag is included in the received message is determined in step 642. If a tag is included in the received message, then the message is output with a tag indicating the same priority as the tag included in the received message in step 643. If a tag is not included in the received message, then the message is output with a tag in step 644. The priority level indicated by the tag used in step 644 is determined based on the content of the message, as discussed above with reference to
The method of
One or more aspects of the disclosure may be embodied in computer-usable or readable data and/or executable instructions, such as in one or more program modules, executed by one or more processors or other devices as described herein. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types when executed by a processor in a computer or other device. The modules may be written in a source code programming language that is subsequently compiled for execution, or may be written in a scripting language such as (but not limited to) HTML or XML. The computer executable instructions may be stored on a computer readable medium, as described above. As will be appreciated by one of skill in the art, the functionality of the program modules may be combined or distributed as desired in various illustrative embodiments. In addition, the functionality may be embodied in whole or in part in firmware or hardware equivalents such as integrated circuits, field programmable gate arrays (FPGA), and the like. Particular data structures may be used to more effectively implement one or more aspects of the disclosure, and such data structures are contemplated within the scope of executable instructions and computer-usable data described herein.
Aspects of the disclosure have been described in terms of illustrative embodiments thereof. While illustrative systems and methods as described herein embodying various aspects of the present disclosure are shown, it will be understood by those skilled in the art, that the disclosure is not limited to these embodiments. Modifications may be made by those skilled in the art, particularly in light of the foregoing teachings. For example, each of the features of the aforementioned illustrative examples may be utilized alone or in combination or subcombination with elements of the other examples. For example, any of the above described systems and methods or parts thereof may be combined with the other methods and systems or parts thereof described above. For example, one of ordinary skill in the art will appreciate that the steps described above may be performed in other than the recited order, including concurrently, and that one or more steps may be optional in accordance with aspects of the disclosure. It will also be appreciated and understood that modifications may be made without departing from the true spirit and scope of the present disclosure. The description is thus to be regarded as illustrative instead of restrictive on the present disclosure.
1. A device comprising:
- a first, a second, and a third Ethernet port; and
- a network interface connected to each of said Ethernet ports, the network interface configured to: for Ethernet frames received on the first port and destined for the second port or third port, ensure that any priority tags of the frames do not indicate a priority that exceeds a predetermined threshold upon transmission from the second port or third port; and for Ethernet frames received on the second port or third port and destined for the second port or third port, preserve the priority level indicated in any priority tags of the received frames upon transmission from the second port or third port,
- for Ethernet frames received on the first port and destined for the second port or third port, ensure that any priority tags of the frames do not indicate a priority that exceeds the predetermined threshold upon transmission from the second port or third port by setting the priority level of the frames to a default value.
Filed: Sep 1, 2014
Publication Date: Dec 18, 2014
Inventor: Vijay Vallala (North Andover, MA)
Application Number: 14/474,295
International Classification: H04L 12/935 (20060101); H04L 12/931 (20060101);