DetNet H. Wang Internet Draft P. Wang Intended status: Standards Track H. Yang Expires: May 15, 2018 Chongqing University of Posts and Telecommunications November 11, 2017 Joint Real-Time Scheduling Methods for Deterministic Industrial Field/Backhaul Networks draft-wang-detnet-joint-scheduling-02 Abstract In industrial field/backhaul networks, joint real-time scheduling method is important to make end-to-end flows meet their deadline. This document proposes four joint scheduling methods, and these methods have considered four scenarios: time-slotted industrial backhaul network, regarding industrial backhaul network as a black box system, ignoring delay of industrial backhaul and establishing latency model of industrial backhaul network. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on May 15, 2018. Wang, et al. Expires May 15, 2018 [Page 1] Internet-Draft DetNet Joint Scheduling November 2017 Copyright Notice Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction ................................................. 2 2. Deterministic Industrial Field/Backhaul Network Requirement .. 4 3. Deterministic Industrial Field/Backhaul Network Joint Scheduling Key Technology ............................................... 5 3.1. End-to-end Network Data Stream .......................... 5 3.2. Network Communication Resource .......................... 5 3.3. Network Time Slot Scheduling ............................ 6 4. Joint Real-Time Scheduling Methods for Deterministic Industrial Field-Backhaul Network ....................................... 6 4.1. Time-Slotted Industrial Backhaul Networks ............... 6 4.2. Consider Industrial Backhaul Network as a Black Box .... 10 4.3. Ignore the Delay of Industrial Backhaul Network ........ 11 4.4. Build Delay Model of Industrial Backhaul Network ....... 11 5. Security Considerations ..................................... 11 6. IANA Considerations ......................................... 11 7. References .................................................. 12 7.1. Normative References ................................... 12 7.2. Informative References ................................. 12 Authors' Addresses ............................................. 13 1. Introduction Industrial field network is a network that can be deployed in process control industry to monitor industrial field equipment for achieving the target of control and management. Industrial field network can improve production efficiency, reduce human intervention and decrease cost. It is significant for industrial modernization. Wang, et al. Expires May 15, 2018 [Page 2] Internet-Draft DetNet Joint Scheduling November 2017 Industrial field bus and industrial Ethernet are two kinds of common solutions of industrial automation, while they are both a wired network. With the development of industrial wireless technology, Wireless Sensor Networks (WSN), which is a typical industrial wireless network, has been applied to industrial network gradually. WSN can free traditional field devices from the limits of abundant cables, and it is easy and flexible to deploy in industrial environment. Wireless network can be applied to building automation, process automation, and industrial automation. Currently, There are three major industrial wireless networks international standards: ISA100.11a[IEC62734], WirelessHART[IEC62591], WIA-PA[IEC62601]. Industrial backhaul network is a transition network, which combines industrial field network with high-level network to achieve the goal of interconnection. It mainly solves the problem that makes the sensor or control data from industrial field network transmit to high-level network. Generally, industrial field network is limited to a specific region, such as a plant. Through industrial backhaul network, data of industrial field network can be transferred to internet or other industrial field networks. Industrial backhaul network is a medium-sized network, which can cover from a few kilometers to tens of kilometers. The major technology of industrial wireless backhaul network consists of Wi-Fi, WiMAX and LTE. In order to adapt to the burgeoning industry 4.0, which aims to elevate the level of manufacturing, industrial field network should not be confined to a plant network only. Therefore, it is necessary to introduce the technology of industrial backhaul network to break the restrictions of interconnection between different networks, and construct a hybrid industrial network. Figure 1 indicates a typical network architecture of the hybrid industrial network. It is a type of architecture of industrial deterministic network that was illustrated with use cases in the drafts proposed by DetNet Workgroup of IETF of [I-D.bas-usecase-detnet] and [I-D.finn-detnet- architecture]. Wang, et al. Expires May 15, 2018 [Page 3] Internet-Draft DetNet Joint Scheduling November 2017 +-----------------------------------+ | | | | | Backhaul network | | | | | +-----------------------------------+ / \ / \ +-------------------------------+ +-------------------------------+ | | | | | | | | | Field network | | Field network | | | | | | | | | +-------------------------------+ +-------------------------------+ Figure 1. Typical industrial field/backhaul network In the hybrid network architecture, field network may be an ISA100.11a, which is an industrial WSN protocol. In Figure 1, a node deployed in a plant can communicate with a node deployed in another plant through backhaul network. 2. Deterministic Industrial Field/Backhaul Network Requirement The draft of [I-D.finn-detnet-problem-statement], proposed by DetNet Workgroup of IETF, has described the requirements of deterministic network and deterministic scheduling partially. Because industrial field network directly faces the monitoring of industrial process, there is a difference between industrial field network and general network. Industrial field network has high demands on the deterministic delay bounds. In a field network, the delay of data transmission will affect productivity, and even generate industrial accidents when happening high packet loss ratio and transmission latency. For example, real-time measure and control of liquid level is required to avoid overfilling of oil tanks, because overflow may lead to serious economic loss and environmental threats. Therefore, it requires a deterministic joint scheduling method to guarantee the deterministic transmission of data stream in the new network architecture. Wang, et al. Expires May 15, 2018 [Page 4] Internet-Draft DetNet Joint Scheduling November 2017 3. Deterministic Industrial Field/Backhaul Network Joint Scheduling Key Technology 3.1. End-to-end Network Data Stream In industrial field/backhaul network, end-to-end data stream indicates a complete transmission path that a source node of field network transfers to destination node located in another field network through an industrial backhaul network. Industrial field/backhaul network data stream has following features: o Period. Every data stream generates data with periodicity. o Deterministic. Every data stream has a deadline, and scheduling methods should ensure each data stream arrives at destination node before its deadline. o Sequential. A path of an end-to-end data stream contains some transmission links. In the process of scheduling, it must be scheduled in the order of sequence of links on the path. o Priority. End-to-end data stream has a priority. When data streams with different priorities occur collisions, the data streams with lower priority should be delayed by higher priority data streams. 3.2. Network Communication Resource In deterministic industrial field/backhaul network architecture, network communication resources include time slot, channel and link. If backhaul network adopts Software Defined Network (SDN) architecture, then the SDN controller can schedule the bandwidth and cache of switch. Therefore, bandwidth and cache resources can be included in schedulable communication resources. o Time slot. Time slot is the basic transmission unit in the network communications based on Time Division Multiple Access (TDMA). In the entire network, the length of time slots is fixed and stays the same. Only one sending packet and its corresponding ACK can be accommodated in one time slot. o Channel. In order to increase network throughput, industrial field network provides a number of channels with different frequencies. Wang, et al. Expires May 15, 2018 [Page 5] Internet-Draft DetNet Joint Scheduling November 2017 o Link. Link refers to a direct packet transmission between two nodes that located in a communication radius of each other. A data stream comprises many links. 3.3. Network Time Slot Scheduling In TDMA-based industrial field network, time is divided into time slots with the same length. In the time-slot scheduling process, it will cause link collision when a node wants to transmit and receive simultaneously, and it will cause channel collision when the same channel is used within a certain range. As shown in Figure 2, the time-slot scheduling process should avoid such collisions. +---+ +---+ +---+ +---+ +---+ +---+ +---+ | A |-->| B |-->| C | | A |-->| B | | C |-->| D | +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---------+------------+ +---------+------------+ |Time slot| Time slot 0| |Time slot| Time slot 0| +---------+------------+ +---------+------------+ |Channel 0| A->B | |Channel 0| A->B | +---------+------------+ | | C->D | |Channel 1| B->C | +---------+------------+ +---------+------------+ Figure 2. Link Collision & Channel Collision 4. Joint Real-Time Scheduling Methods for Deterministic Industrial Field-Backhaul Network Joint real-time scheduling methods for industrial field/backhaul networks intend to solve the deterministic problem of industrial field/backhaul networks. Due to the investigative architecture includes backhaul network, the deterministic scheduling algorithm needs to collaborate with backhaul network to conduct joint scheduling to ensure data deterministic transmission. The proposed joint scheduling methods are described as follows. 4.1. Time-Slotted Industrial Backhaul Networks In order to ensure determinism, industrial field networks utilize TDMA to make the network time-slotted. If the industrial backhaul network can also be time-slotted, then the deterministic scheduling algorithm can jointly schedule with small modification. Industrial backhaul network can be a variety of network standards such as WIFI, WiMAX, and LTE. WiMAX and LTE are high cost and poor feasibility, thus we assume the IEEE 802.11 as backhaul network. Wi-Fi network has various operating modes, such as peer-to-peer mode, point to Wang, et al. Expires May 15, 2018 [Page 6] Internet-Draft DetNet Joint Scheduling November 2017 multi-point networking mode and the relay network mode. Here we consider the hierarchical network architecture in a way of point to multi-point networking mode, as shown in Figure 3. +----------------------------------------+ | | | +--------+ | | +-------| Head AP|-------+ | | | +--------+ | | | | | | | +--------+ +--------+ | +---+---| AP1 | | AP2 |---+---+ | | +--------+ +--------+ | | | +----------------------------------------+ | | | +---------------------------------++----------------------------------+ |ISA100.11a field wireless network||ISA100.11a field wireless network | +---------------------------------++----------------------------------+ Figure 3. Industrial Backhaul Network consisting of WIFI Although IEEE 802.11 supports multiple channels, but AP is not able to perform channel hopping between transmission timeslots, which means that the AP cannot use a channel in the current time slot and use another channel the next time slot. We assume that AP1 and AP2 in Figure 3 can transmit packets simultaneously as long as their transmission tasks do not contain the same AP, i.e. head AP. For example, when a data stream of field network is transmitting packets to AP1 in a time slot, AP2 is able to receive packets from head AP, or send packets to field network in the same time slot. Therefore, the backhaul network framework with wireless APs can be considered as a single-channel linear network, which is shown in Figure 4. +---------+ +--------+ +--------+ +--------+ +---------+ | Gateway |--> | AP |--> | AP |--> | AP |--> | Gateway | +---------+ +--------+ +--------+ +--------+ +---------+ Figure 4. A single-channel linear network Therefore, the data stream in industrial field/backhaul network can be seen as equivalent to the data stream in field network, except that each data stream needs to flow through the WIFI. The scheduling process is illustrated as follows: Wang, et al. Expires May 15, 2018 [Page 7] Internet-Draft DetNet Joint Scheduling November 2017 1. Abstract end-to-end data stream in the entire network, and allocate a priority for each stream. 2. Establish the delay model of network data stream. If collisions happened between different priority data stream, the low priority data stream will be delayed by high priority, so a model can be built under the worst circumstances that the low-priority data streams impacted by all higher priority data streams. 3. Estimate the network schedulability. A data stream is schedulable when the minimum time for the data stream to complete its once transmission task plus the worst delay time caused by higher priority data streams is less than or equal to its deadline, In the current priority allocation scheme, if each data stream is schedulable, the network can be considered as schedulable. If the data stream cannot be scheduled, then try to change the priority allocation scheme and estimate again until a corresponding scheme is found or return no schedulable results. 4. Allocate time slot and channel for each data stream. Traverse data streams according to their priorities, and each data stream should allocate link that is about to be released in a time slot. According to the rule that low priority data streams should give way to high priority data streams, the channels can be utilized if it is not unoccupied. However, if collisions happened between data streams of different priority, then the lower priority data stream should be placed in the next time slot until there are no unallocated higher priority data streams. Repeat these steps until the whole network scheduling is completed. The scheduling process is described in Figure 5: Wang, et al. Expires May 15, 2018 [Page 8] Internet-Draft DetNet Joint Scheduling November 2017 +----------+ | Begin | +----------+ | | +---------------------------+ | Initial the priority of | | each data stream | +---------------------------+ |<--------------------------------------+ | | +--------------------+ +------------------------------+ / Traverse every data \ no | If the data stream cannot be | / stream and estimate the\--------->| scheduled, then change the | \ schedulablity according/ | priority allocation scheme | \ to delay model / | and estimate again | +--------------------+ +------------------------------+ | |yes +-----------------------------------+ | Traverse data streams according to| | their priority, each data stream | | should allocate the next link that| | is about to be released in each | | time slot to the greatest extent | +-----------------------------------+ | | +-----------------------------------+ | The spare channels can be utilized| | if there is no collision. If | | collisions happened, then the | | lower priority data stream should | | be placed in the next time slot | +-----------------------------------+ | | +-------+ | End | +-------+ Figure 5. Scheduling of times-slotted industrial backhaul network Wang, et al. Expires May 15, 2018 [Page 9] Internet-Draft DetNet Joint Scheduling November 2017 Further, if the backhaul network can support TDMA mechanism like the industrial field network completely, the deterministic scheduling methods in field network can be applied in industrial field/backhaul networks. For backhaul network using wired technology, time-sensitive network based on Ethernet is preferred for industrial scenarios. Time- sensitive network can provide dedicated slots for scheduled traffic, so above scheduling method can be used in this kind of backhaul network to guarantee the deterministic performance for data flows across field and backhaul networks. 4.2. Consider Industrial Backhaul Network as a Black Box In order to solve the deterministic problem of industrial network, backhaul network can be regarded as a black box so that we can only consider its delay impacts and ignore its internal details. When the packet passes through the industrial backhaul network, we can give it a timestamp at the application layer and read it after the transmission is ready to leave the backhaul network. Delay caused in backhaul network can be calculated, and a fitting curve of delay can be obtained by collecting large amount of data. It has been verified experimentally that the delay is concentrated in a numerical range despite its randomness. Therefore, we can estimate the approximate delay time caused by industrial backhaul network. A main scheduling path can be configured according to the average delay of the backhaul network. Some redundant paths should be pre- configured in case the delay of the main path is too high. The scheduling process of industrial field/backhaul network can be divided into three sections, as shown in Figure 6: +--------------------+ +-----------------+ +------------------------+ | Scheduling of | | Delay of | | Scheduling of | |source field subnet |->| backhaul network|->|destination field subnet| | (deterministic) | |(indeterministic)| |( deterministic dynamic)| +--------------------+ +-----------------+ +------------------------+ Period 1 Period 2 Period 3 Figure 6. Three periods of scheduling In source field subnet we can apply the deterministic scheduling algorithm of field network to get the time spent by each data stream before entering the source subnet. Then the data stream enters the backhaul network, which will cause indeterministic delay in a numerical range. When the data stream leaves the backhaul network, Wang, et al. Expires May 15, 2018 [Page 10] Internet-Draft DetNet Joint Scheduling November 2017 the timestamp should be parsed. If the deadline is missed, it indicates that the packet has gone through poor network and needs to be retransmitted. If there is free time after leaving the backhaul network, scheduling path can be dynamically selected at downward gateway to get the schedulability of the end-to-end data stream. 4.3. Ignore the Delay of Industrial Backhaul Network Since the field network is slow-speed (250 KB/s), while industrial backhaul network is high-speed, if the industrial backhaul networks adopt IEEE 802.11 protocol, gigabit wireless routers supporting IEEE 802.11 ac can make the delay of industrial backhaul network quite low. As a result, the joint deterministic scheduling of the entire network only needs to consider the field networks. 4.4. Build Delay Model of Industrial Backhaul Network If industrial backhaul network is constructed with IEEE 802.11, the network access delay test model in IEEE 802.11 Distributed Coordination Function (DCF) mode can be established by using Markov chain or queue theory. While the model in IEEE 802.11 Point Coordination Function (PCF) mode can be established based on queue theory. Therefore, the field network needs to build a delay model, while backhaul network follows another delay model, then the total transmission scheduling delay will have certain regularity. The total transmission delay will meet delay requirements with specified probability by scheduling, in other words, the unsuccessful scheduling is acceptable. 5. Security Considerations 6. IANA Considerations This memo includes no request to IANA. Wang, et al. Expires May 15, 2018 [Page 11] Internet-Draft DetNet Joint Scheduling November 2017 7. References 7.1. Normative References 7.2. Informative References [IEC62734] ISA/IEC, "ISA100.11a, Wireless Systems for Automation, also IEC 62734", 2011, . [IEC62591] IEC, "Industrial Communication Networks - Wireless Communication Network and Communication Profiles - WirelessHART - IEC 62591", 2010, [IEC62601] IEC, "Industrial networks - Wireless communication network and communication profiles - WIA-PA - IEC 62601", 2015, < https://webstore.iec.ch/preview/info_iec62601%7Bed2.0%7Db .pdf> [I-D.finn-detnet-problem-statement] Finn, N. and P. Thubert, "Deterministic Networking Problem Statement", draft-finn-detnet-problem-statement-05 (work in progress), March 2016. [I-D.finn-detnet-architecture] Finn, N., Thubert, P., and M. Teener, "Deterministic Networking Architecture", draft-finn-detnet-architecture-08 (work in progress), August 2016. [I-D.bas-usecase-detnet] Kaneko, Y., Toshiba and Das, S, "Building Automation Use Cases and Requirements for Deterministic Networking", draft- bas-usecase-detnet-00 (work in progress), October 2015. Wang, et al. Expires May 15, 2018 [Page 12] Internet-Draft DetNet Joint Scheduling November 2017 Authors' Addresses Heng Wang Chongqing University of Posts and Telecommunications 2 Chongwen Road Chongqing, 400065 China Phone: (86)-23-6248-7845 Email: wangheng@cqupt.edu.cn Ping Wang Chongqing University of Posts and Telecommunications 2 Chongwen Road Chongqing, 400065 China Phone: (86)-23-6246-1061 Email: wangping@cqupt.edu.cn Hang Yang Chongqing University of Posts and Telecommunications 2 Chongwen Road Chongqing, 400065 China Phone: (86)-23-6246-1061 Email: 18716322620@163.com Wang, et al. Expires May 15, 2018 [Page 13]