Techniques herein define a simple, but very useful, extension to hop-by-hop signaling that can be utilized to determine a failed node in a network, which may help to reduce fault detection time.  In one instance, techniques described herein may involve multicast Label Distribution Protocol (mLDP)-based signaling, however, other replication technologies that involve underlay signaling may be utilized in accordance with techniques described herein

Budhiraja, Anuj

Kumar, Nitin

Mishra, Mankamana

Santhanam, Sridhar

English

Technical Disclosure Common

Technical Disclosure Commons Defensive Publications Series June 2021 INBAND MULTICAST FAULT DETECTION TO REDUCE SERVICE COST Mankamana Mishra Anuj Budhiraja Nitin Kumar Sridhar Santhanam Follow this and additional works at: https://www.tdcommons.org/dpubs_series Recommended Citation Mishra, Mankamana; Budhiraja, Anuj; Kumar, Nitin; and Santhanam, Sridhar, "INBAND MULTICAST FAULT DETECTION TO REDUCE SERVICE COST", Technical Disclosure Commons, (June 30, 2021) https://www.tdcommons.org/dpubs_series/4418 This work is licensed under a Creative Commons Attribution 4.0 License. This Article is brought to you for free and open access by Technical Disclosure Commons. It has been accepted for inclusion in Defensive Publications Series by an authorized administrator of Technical Disclosure Commons.  1 6648 INBAND MULTICAST FAULT DETECTION TO REDUCE SERVICE COST  AUTHORS:   Mankamana Mishra Anuj Budhiraja Nitin Kumar Sridhar Santhanam   ABSTRACT Techniques herein define a simple, but very useful, extension to hop-by-hop signaling that can be utilized to determine a failed node in a network, which may help to reduce fault detection time.  In one instance, techniques described herein may involve multicast Label Distribution Protocol (mLDP)-based signaling, however, other replication technologies that involve underlay signaling may be utilized in accordance with techniques described herein.  DETAILED DESCRIPTION Multicast technologies are deployed throughout many networks.  However, multicast distribution trees (MDTs) can be difficult to debug when network failures occur.  Figure 1, below, illustrates how current network failures are often handled. 1. User experiences service interruption2. Calls service provider customer service3. Customer service escalates matter to network admin4. Network admin involves equipment provider for helpEquipment Provider Figure 1: Example Network Failure Handling Flow 2Mishra et al.: INBAND MULTICAST FAULT DETECTION TO REDUCE SERVICE COSTPublished by Technical Disclosure Commons, 2021 2 6648  As shown in Figure 1, a network failure typically involves: 1. An end user experiencing a service disruption; 2. After some time (potentially after many failures), the user decides to call customer service of a service provider; 3. The customer service identifies a potential network-related issue and reaches out to a local operations team; and 4. The local operations team contacts an equipment provider, which starts trying to understand the potential issue (e.g., asking for logs from different network locations).  In some instances, the above process can take many days before an actual network node is identified as having an issue.  For example, in some instances, for a very large network, more than one week of debugging may be involved such that initial debugging may start at a local data center network and, once operation there is confirmed, the debugging may continue to different network segments.  In some instances, different parts of a network may also involve debugging from different equipment providers/vendors, which can also lead to increased debugging time.  Accordingly, it can be inherently difficult to debug network failures involving multicast traffic, as opposed to unicast traffic, because there can be many fork points in a network at which issues can be potentially introduced.  As business landscapes and competitors change among network equipment provides/vendors, it becomes increasingly important to improve quality of service as a competitive differentiator.  Further, there is also often a need for network operators to be able to detect failures within their networks as soon as possible in order to identify the actual node(s) having issues so that quick recovery/resolution is possible.  Further, network failures can impact end user experience (e.g., for streaming video, etc.) and may be costly in terms of service cost (e.g., help desk time, network operator troubleshooting costs, equipment provider/vendor troubleshooting costs) and/or lost revenue, which further motivates network operators and equipment providers/vendors to identify/resolve network issues as quickly as possible.  When considering various solutions that may be utilized to address the challenges noted above, two types of deployment scenarios may considered that vary in scale, which 3Defensive Publications Series, Art. 4418 [2021]https://www.tdcommons.org/dpubs_series/4418 3 6648 can impact potential in-band versus out-of-band fault detection solutions.  For example, a first deployment scenario may be considered in which the number of multicast flows may be much higher than the number of nodes in a given network deployment (e.g., > 350,000+ flows) and a second scenario may be considered in which the number of multicast flows may be much lower than the number of nodes in a given network deployment (e.g., flows expected to be on the order of a few thousand).  When considering network deployments under the second scenario in which the number of nodes may be much higher than the number of flows (e.g., potentially on the order of millions of nodes), it can become very difficult to utilize an out-of-band telemetry-based solution that involves exporting data to a centralized location that would detect traffic loss and take some corrective action.  Although such a centralized solution may work well for lower scale deployments (e.g., fewer network nodes), it can still be challenging to handle large scale data for out-of-band solutions.  Further, given the massive scale of many Internet Protocol (IP) networks, it may be difficult to maintain an out-of-band solution.  Rather, different types of solutions may be needed to handle different issues.   This proposal provides a distributed scale solution that can help to identify problematic network nodes, potential identifying a problem on the order of a number of minutes, which can save many weeks of cost and/or lost revenue for network operators, as well as for equipment providers/vendors.  Broadly, techniques of this proposal may involve: 1. Detecting the traffic loss at an Egress node using available hardware capabilities; 2. Once traffic loss has been detected, an in-band signaling mechanism is utilized to notify an upstream node about traffic loss for given flow; and 3. The upstream node checks for traffic loss. If there is traffic loss, the upstream node notifies its own upstream node or notifies a network administrator as this node to be responsible for traffic loss.  Consider Figure 2, below, which illustrates various example flow characteristics. 4Mishra et al.: INBAND MULTICAST FAULT DETECTION TO REDUCE SERVICE COSTPublished by Technical Disclosure Commons, 2021 4 6648  Figure 2: Example Flow Characteristics  Referring to Figure 2, many multicast deployments use the term 'Data multicast distribution tree (MDT)', which could potentially refer to an aggregated representation of multiple customer flows for a given Virtual Routing and Forwarding (VRF) element.  In case of any traffic loss, there would not be per-flow traffic loss considering the core network is not even aware of individual flows. So any traffic loss detection would be for all flows associated with a given Data MDT. Various example details are discussed herein with reference to an example multicast over Virtual Private Network (mVPN) profile, labeled 'profile-X', however it is to be understood that techniques of this proposal may be utilized in conjunction with any profile and multicast technology, which can be enhanced using appropriate type-length-value (TLV) objects and associated protocol extensions (e.g., mLDP, Protocol Independent Multicast (PIM), etc.). Consider example mVPN profile-X flow Data MDT signaling, as shown in Figure 3, below. 5Defensive Publications Series, Art. 4418 [2021]https://www.tdcommons.org/dpubs_series/4418 5 6648  Figure 3: Example mVPN profile-X flow Data MDT Signaling  During operation, once a flow meets the criteria of being moved to the Data MDT (either a configured threshold is met or an immediate switch is configured), the Ingress router notifies each of the Egress Provider Edge (PE) routers participating in given VRF about the flow to Data MDT mapping.   For an implementation in which the core is a Multiprotocol Label Switching (mPLS) core, the Ingress router may send a Global ID and root to use for the mLDP Forwarding Equivalence Class (FEC).  For example once receiver behind router-4 (R4) sends multicast join for all of the groups behind router-1 and if Data MDT conditions are met. Router-4 may obtain the Data MDT information as, shown in Table 1, below.  6Mishra et al.: INBAND MULTICAST FAULT DETECTION TO REDUCE SERVICE COSTPublished by Technical Disclosure Commons, 2021 6 6648 TABLE 1: Data MDT Information Obtained by Router-4 For Example Flows   Even though there are 4 different multicast flows in the present example, the information would be sent as one label mapping to an upstream node considering all of the flows belong to the same Data MDT. Thus, router-4 may store/maintain the Data MDT, as shown in Table 2, below.  TABLE 2: Data MDT Maintained by Router-4 For Example Flows    From the data base, as shown in Table 2, it is clear that a join should be sent to next hop 3.3.3.3, as shown in Table 3, below. 7Defensive Publications Series, Art. 4418 [2021]https://www.tdcommons.org/dpubs_series/4418 7 6648 TABLE 3: Downstream Neighbor of Router-4    Router-3 (R3) may receive an mLDP join for the Data MDT, which would be aggregated for all of the flows, as shown in TABLE 4, below.  TABLE 4: Data MDT Maintained by Router-3 For Example Flows    8Mishra et al.: INBAND MULTICAST FAULT DETECTION TO REDUCE SERVICE COSTPublished by Technical Disclosure Commons, 2021 8 6648  A network administrator can configure certain flows to be in a monitoring category at the Egress node to enable failure detection at the Egress node.  For example, as shown below in Figure 4, traffic is expected to be received on a virtual tunnel from a source and Egress router-4 can send individual flows out on appropriate interfaces. Once a network administrator marks particular flow for monitoring, the associated Data MDT flow can be monitored. Based on nature of Data MDT, either loss occurs for the whole data MDT (impacting each flow) or not at all.  DATA FLOWLmdtRED 4 Figure 4: Detecting a Failure at the Egress Router   Once the Data MDT is monitored, flows can also be monitored for Ingress statistics, depending on configuration.  For example, as shown below via Figure 5, the network administrator can mark flows to monitor at the Ingress router and, once monitored, if there is traffic loss detected, the next hop can be signaled using mLDP signaling. Monitor flow Data MDT ‐ XNo Loss Traffic Received 0Do Nothing, Mark Status as HealthymLDP Upstream Notification Regarding Traffic Loss Associated with FEC Figure 5: Example Ingress Router Monitoring 9Defensive Publications Series, Art. 4418 [2021]https://www.tdcommons.org/dpubs_series/4418 9 6648   Consider an example in which traffic loss is detected at Egress router-4, as shown below in Figure 6.   Figure 6: Traffic Loss Example   The intention of this proposal is to cover cases in which traffic is dropped completely. In one example, the Egress node can monitor its counters per "X" seconds, where value of X can be configured based on user need and platform capability. Once Egress detects a complete traffic loss it would start signaling the upstream node (router-3) regarding the loss, as discussed in further detail, below.  In another example, overlay signaling can be extended to report a traffic rate range for a given flow and the Egress node can keep monitoring for some %-deviation before determining that a traffic loss has occurred.  If there is a complete traffic drop there are likely two reasons:  1. The source stopped sending traffic, or 2. There is an actual traffic drop in network. In either case, it would be still good for service provider to know of such traffic losses in advance, before an end-user reaches out to the service provider. 10Mishra et al.: INBAND MULTICAST FAULT DETECTION TO REDUCE SERVICE COSTPublished by Technical Disclosure Commons, 2021 10 6648 Once traffic loss is detected for the example as shown in Figure 6, mLDP can be utilized to notify the next hop about the loss, as shown below in Table 5.  TABLE 5: Example Loss Notification   As shown in Table 6, the following FEC mapping is received on router-3 at which an attribute is set that indicates that there is traffic loss.  TABLE 6: Router-3 FEC Mapping  11Defensive Publications Series, Art. 4418 [2021]https://www.tdcommons.org/dpubs_series/4418 11 6648   Since router-3 received a signal from downstream node, an internal check is triggered for router-3 in which router-3 also determines that it is not obtaining traffic from the upstream node.  Following the determination, an upstream notification is triggered for router-3, as shown in Table 7, below.  TABLE 7: Upstream Notification Triggered for Router-3    Thereafter, router-3 can send an in-band notification to an upstream node, as shown in Table 8, below. 12Mishra et al.: INBAND MULTICAST FAULT DETECTION TO REDUCE SERVICE COSTPublished by Technical Disclosure Commons, 2021 12 6648 TABLE 8: Upstream Notification Provided by Router-3    Upon obtaining the notification, router-2 (R2) can verify its incoming rates and determine that it is receiving traffic; thus, router-2 would notify the network administrator that it is the node introducing a traffic loss.  An example notification is shown below in Table 9.  TABLE 9: Example Notification from the Node at which Loss is Occurring    The amount of information that could be provided to the network administrator, as shown in Table 9, would be sufficient to start debugging the node.  For example, automated scripts can be provided that can examine traces and commands related to the above information that may provide enough context to identify the possible root cause of the issue. As noted above, traffic is being monitored on a per-Default MDT or Data MDT and is based on detecting a 100% traffic loss.  Although there may be considerations concerning false positives and potentially unnecessary signaling if some flows are removed from the 13Defensive Publications Series, Art. 4418 [2021]https://www.tdcommons.org/dpubs_series/4418 13 6648 Default MDT, if data rate is reduced (e.g., variable bitrate), or when a source stops sending data, these cases may be addressed through different configurations that can be managed for a given implementation/deployment. Broadly, techniques of this proposal can operationally be realized as follows: 1. Traffic for a multicast flow starts flowing. 2. Initially, it flows in a Default MDT and then finally settles in a Data MDT. 3. Now, the service provider/network operator decides which all Data MDTs or Default MDT it wants to monitor. 4. The Egress node starts polling counters every "X" seconds/minutes (depending on hardware capability and user tolerance). 5. If the Egress detects traffic rate as 0 for those "X" seconds, it starts the procedures as discussed above with reference to Figure 6.  Based on the above, consider various issues that can be addressed through network configuration.  For example, regarding bursty flows, there are many ways to 'break' a deployment. Consider the case where a network administrator configures 10 seconds as the timer to poll and detect traffic loss and the source sends traffic for 2 seconds and stops for the next 11 seconds. In this example, a loss would be detected.  Theoretically, it may be true that additional signaling may be triggered for bursty traffic utilizing the techniques herein, but such a bursty traffic example is a non-practical use-case. If there is some source behaving like this for traffic such as IP television (IPTV), such bursty traffic is a bigger problem that should be addressed rather than detecting traffic loss. Regarding potential false positives for instances in which a source has stopped sending traffic for a long period of time, the only case where false positives could potentially occur is with bursty flows, which is not a valid case (as explained above). If the source really stops sending traffic, it is only a one-time signaling end to end. The network administrator would receive a loss report and there is no additional signaling would occur since the traffic itself has stopped. In reality, with the IPTV use case, this may be a scenario that may occur every now and then. So, this can be considered as noise. Regarding flow removal from Default MDT to Data MDT, since monitoring is per MDT not inside the MDT, it does not matter if flows are being moved around. By nature, 14Mishra et al.: INBAND MULTICAST FAULT DETECTION TO REDUCE SERVICE COSTPublished by Technical Disclosure Commons, 2021 14 6648 there is no way to have traffic loss for only one flow in an MDT if multiple flows are aggregated. When considering data rates on different nodes, depending on how they are sampled, there may be some deviations; however, techniques of this proposal involve comparing data rates from zero to non-zero, so sampling may not be a concern.  Consider an example, as shown below in Figure 7. R5R4R3R2R1  Check traffic rate for  last 10 secondsTraffic rate for last 10 seconds is 0Signal Next Hop Figure 7: Data Rate Consideration Example    Consider for the above figure an example in which router-1 (R1) is sending traffic downstream that is being received by router-5 (R5) in which the following occurs: 1. Router-5 detects traffic loss for 10 seconds at time t0. 2. Router-5 signals the next hop for this flow stating that there is traffic 0 for the last 10 seconds. 3. By the time the signaling reaches router-4 (R4) an amount of time has already elapsed. 4. Thus, sampling at router-4 will start only at t0+delta. It also takes a sample for the last 10 seconds. So, there are only two possible cases: a. Router-4 is receiving traffic, which means the issue is between router-4 and router-5, or b. Router-4 also has zero traffic, so it needs to signal next hop.  15Defensive Publications Series, Art. 4418 [2021]https://www.tdcommons.org/dpubs_series/4418 15 6648 As illustrated through the above example, the intention is to detect a complete traffic loss; thus, there won't be any deviation unless the source is bursty in nature (sending traffic for every "X" seconds and pausing for the next "Y" seconds) which, as discussed above, may not be a realistic use-case.  Further, the innovation of the proposal is discussed with reference to mLDP but it is not limited to just one protocol. The techniques described herein remain the same irrespective of different tree-building in-band protocols; the only changes may involve respective TLVs to be defined in other respective protocols.  The primary novelty of the techniques of this proposal involve the application and set of procedures to achieve fault detection in a network environment. Thus, a mechanism is provided through which rest of a network could poll data on-demand, only if there is real loss detected in the last-hop router. In summary, techniques herein define a simple, but very useful, extension to hop-by-hop signaling that can be utilized to determine a failed node in a network, which may help to reduce fault detection time as well as reduce service provider/network operator cost and equipment provider/vendor cost in debugging network failures.   16Mishra et al.: INBAND MULTICAST FAULT DETECTION TO REDUCE SERVICE COSTPublished by Technical Disclosure Commons, 2021

INBAND MULTICAST FAULT DETECTION TO REDUCE SERVICE COST

https://www.tdcommons.org/cgi/viewcontent.cgi?article=5503&context=dpubs_series

INBAND MULTICAST FAULT DETECTION TO REDUCE SERVICE COST

Abstract

Similar works

Full text

Available Versions

Technical Disclosure Common