US20110051727A1 - Network based multicast stream duplication and merging - Google Patents
Network based multicast stream duplication and merging Download PDFInfo
- Publication number
- US20110051727A1 US20110051727A1 US12/584,008 US58400809A US2011051727A1 US 20110051727 A1 US20110051727 A1 US 20110051727A1 US 58400809 A US58400809 A US 58400809A US 2011051727 A1 US2011051727 A1 US 2011051727A1
- Authority
- US
- United States
- Prior art keywords
- packet
- duplicate
- received
- receiver
- received packet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 claims abstract description 26
- 238000004891 communication Methods 0.000 claims abstract description 13
- 230000004044 response Effects 0.000 claims 1
- 230000005540 biological transmission Effects 0.000 description 4
- 238000005538 encapsulation Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- RGNPBRKPHBKNKX-UHFFFAOYSA-N hexaflumuron Chemical compound C1=C(Cl)C(OC(F)(F)C(F)F)=C(Cl)C=C1NC(=O)NC(=O)C1=C(F)C=CC=C1F RGNPBRKPHBKNKX-UHFFFAOYSA-N 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000035755 proliferation Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 239000004557 technical material Substances 0.000 description 1
- 230000005641 tunneling Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/02—Details
- H04L12/16—Arrangements for providing special services to substations
- H04L12/18—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast
- H04L12/1863—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast comprising mechanisms for improved reliability, e.g. status reports
- H04L12/1877—Measures taken prior to transmission
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/16—Multipoint routing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/24—Multipath
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/28—Routing or path finding of packets in data switching networks using route fault recovery
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/02—Details
- H04L12/16—Arrangements for providing special services to substations
- H04L12/18—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast
- H04L12/185—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast with management of multicast group membership
Definitions
- the present disclosure relates generally to communication networks, and more particularly, to multicast routing.
- IP Internet Protocol
- a host sends packets to a single host (unicast transmission) or to all hosts (broadcast transmission).
- IP Internet Protocol
- a third routing technique has evolved, multicast routing.
- multicast routing a host sends packets to a subset of all hosts as a group transmission.
- Multicast routing is widely deployed and continues to gain popularity with the advent of multimedia programs and the proliferation of Internet applications. Many multicast-based applications require delivery of large amounts of time sensitive and loss sensitive data simultaneously to large audiences. For these applications, a high degree of multicast resilience is desired to provide near-zero packet loss. In order to achieve this resilience, multicast traffic is split into multiple data streams which are sent over diverse paths. In the case of a failure on one path, the multicast traffic is still available via another path. The multiple data streams, however, result in duplicate packets which increase bandwidth usage and need to be properly discarded when the streams merge.
- FIG. 1 illustrates an example of a network in which embodiments described herein may be implemented.
- FIG. 2 depicts an example of a network device useful in implementing embodiments described herein.
- FIG. 3 illustrates data packets transmitted over the network of FIG. 1 .
- FIG. 4 is a flowchart illustrating a process for multicast stream duplication at a source edge router.
- FIG. 5 is a flowchart illustrating a process for setting up diverse paths and merging multicast streams at a receiver edge router.
- a method generally comprises receiving a join message from a receiver at a receiver edge router, the join message identifying a source and a group, and creating at least two forwarding trees for transmitting multicast data streams comprising duplicate data to the receiver edge router.
- the forwarding trees include a root at a source edge router in communication with the source and different network paths.
- the method further includes receiving a packet from one of the forwarding trees and processing the received packet to determine if the packet is a duplicate of another packet received at the receiver edge router. If the packet is a duplicate packet, it is discarded. If the packet is not a duplicate, it is forwarded to the receiver.
- an apparatus generally comprises a processor configured to receive a join message from a receiver, create at least two forwarding trees for transmitting multicast data streams comprising duplicate data to the apparatus, the forwarding trees comprising a root at a source edge router in communication with a source, and different network paths, process a packet received from one of said forwarding trees to determine if the received packet is a duplicate of another received packet, discard the received packet if the packet is a duplicate packet, and forward the received packet to the receiver if the packet is not a duplicate packet.
- the apparatus further includes memory for storing data from received packets for use in determining if the received packet is a duplicate packet.
- Networks such as those for use in transporting digitized television programs, Service Provider (SP) Multicast Virtual Private Network (MVPN) enabled networks that carry IP/TV traffic in Multicast Virtual Route Forwarding (MVRF), or dedicated or MVPN enabled networks for use in delivering market data information, for example, require a high degree of multicast resilience. This is due in part to the sensitivity of applications to packet loss and low latency requirements.
- SP Service Provider
- MVPN Multicast Virtual Private Network
- MVRF Multicast Virtual Route Forwarding
- MVPN dedicated or MVPN enabled networks for use in delivering market data information
- a network that guarantees zero (or near-zero) packet loss in the case of a single network failure is referred to herein as a “live-live” network.
- This may be accomplished using, for example, an MT (multi-topology) IGP (Interior Gateway Protocol) capable network or different BGP (Border Gateway Protocol) loopbacks (update-source) with separate IGP processes to build diverse paths for multicast.
- the live-live network may also be created using ECMP (Equal-Cost Multi-Path), MoFRR (Multicast only Fast Reroute), RSVP/TE (Resource Reservation Protocol—Traffic Engineering), or other protocols.
- the primary traffic in the network is IP multicast. It is to be understood that these networks are only examples and the embodiments described herein may be implemented in different networks, without departing from the scope of the embodiments.
- duplicate streams of data are sent over diverse paths.
- the duplicate streams may be transmitted at a source and merged at a receiver.
- this process has drawbacks.
- the Service Provider may have no control over what the source can or will send, thus, it is not desirable to require the source to send multiple streams at all times.
- the bandwidth available between the SP network and the receiver may be limited.
- the embodiments described herein duplicate and merge data streams within a transport network. As described in detail below, the embodiments utilize tunnels within the transport network to transmit the duplicate data streams. This allows the content source to send only one data stream and eliminates the need for the content receiver to merge the streams. The embodiments provide minimum latency and bandwidth for traffic entering and exiting the transport network.
- the embodiments described herein operate in the context of a data communication network 10 including multiple network elements ( FIG. 1 ).
- Some of the elements in a network that employs the embodiments may be routers, switches, gateways, or other network devices.
- some of the nodes may be specially configured routers such as those available from Cisco Systems, Inc. of San Jose, Calif.
- the term router refers to devices that forward packets based on network and higher layer information.
- the router may include, for example, a master central processing unit (CPU), interfaces, and a bus (e.g., a PCI bus).
- the CPU is responsible for such router tasks as routing table computations, network management, and general processing of packets. It preferably accomplishes all of these functions under the control of software including an operating system and any appropriate application software.
- the network device is implemented on a general purpose network host machine as described below with respect to FIG. 2 .
- FIG. 2 depicts a network device 40 that may be used to implement embodiments described herein.
- Network device 40 is configured to implement all of the network protocols and extensions thereof described herein.
- network device 40 is a programmable machine that may be implemented in hardware, software, or any combination thereof.
- Logic may be encoded in one or more tangible media for execution by a processor.
- processor 42 may execute codes stored in a program memory 44 .
- Program memory 44 is one example of a computer-readable medium.
- Program memory 44 can be a volatile memory.
- Another form of computer-readable medium storing the same codes is a type of non-volatile storage such as floppy disks, CD-ROMs, DVD-ROMs, hard disks, flash memory, etc.
- Network device 40 interfaces with physical media via a plurality of linecards (network interfaces) 46 .
- Linecards 46 may incorporate Ethernet interfaces, DSL interfaces, Gigabit Ethernet interfaces, 10-Gigabit Ethernet interfaces, SONET interfaces, etc.
- packets As packets are received, processed, and forwarded by network device 40 , they may be stored in a packet memory 48 .
- linecards 46 may incorporate processing and memory resources similar to those discussed above in connection with the network device as a whole. It is to be understood that the network device 40 shown in FIG. 2 and described above is only one example and that different configurations of network devices may be used.
- the network 10 implementing the embodiments described herein is configured to use IP multicast, which simultaneously delivers a stream of information to numerous recipients.
- IP multicast which simultaneously delivers a stream of information to numerous recipients.
- Multicast operation is based on the concept of a group.
- a multicast group is an arbitrary group of receivers that expresses an interest in receiving a particular data stream.
- An IP multicast address, or a portion thereof, specifies a particular group.
- IGMP Internet Group Management Protocol
- MLD Multicast Listener Discovery
- Multicast-capable routers create distribution trees that control the path that IP multicast traffic takes through the network in order to deliver traffic to all receivers. Members of multicast groups can join or leave at any time; therefore the distribution trees are dynamically updated.
- Protocol Independent Multicast (PIM) is used to dynamically create a multicast distribution tree to ensure distribution to intended receivers while limiting distribution so that network segments that are not in the path between the source and receivers are not burdened with unnecessary traffic. It is to be understood that PIM is only one example and that other multicast protocols may be used without departing from the scope of the embodiments.
- the network 10 includes a source 12 , receiver 14 , Source Edge Router (SER) 16 , Receiver Edge Router (RER) 18 , and intermediate nodes 20 , 22 , connected through communication links.
- the SER 16 is a network device that is connected to the content source or a network through which the content source can be reached.
- the RER 18 is a network device that is connected to the receiver or a network via which the receiver can be reached.
- the SER 16 and RER 18 are preferably located along an edge of transport network 34 .
- the protection domain for multicast is between the SER 16 and RER 18 .
- Source 12 may be a node (e.g., server, computer, or any other type of network device that can source IP media, such as video, audio, voice, data, etc.) or a network in communication with the source node.
- the source 12 transmits a multicast data stream 28 to SER 16 , which transmits the data stream to one or more receivers 14 over the network 34 .
- the receiver 14 may be a node that receives and stores or renders multicast stream 30 , a network in communication with the receiver node, or a gateway to another network.
- the receiver 14 may be, for example, a computer terminal, personal computer, television with IP interface, Set Top Box (STP), Digital Video Recorder (DVR), Voice over IP (VoIP) phone, cell phone, eQAM (edge Quadrature Amplitude Modulation) device, Personal Digital Assistant (PDA), etc.
- STP Set Top Box
- DVR Digital Video Recorder
- VoIP Voice over IP
- cell phone cell phone
- eQAM edge Quadrature Amplitude Modulation
- PDA Personal Digital Assistant
- the SER 16 transmits two data streams 24 , 26 on tree T 1 and tree T 2 , respectively.
- Packets for data stream 24 go through intermediate node 20 wherein packets for data stream 26 go through intermediate node 22 .
- the data streams 24 , 26 may pass through any number of nodes in transport network 34 .
- the tree topologies T 1 , T 2 preferably have SER-to-RER disjointness. The use of different network paths increases the likelihood that packets from at least one of the two data streams 24 , 26 successfully arrive at the RER 18 . If the two paths 24 , 26 are completely disjoint (i.e., do not share any network segment), the data is protected from any single failure in the transport network 34 .
- FIG. 3 illustrates an example of multicast data packets 50 , 58 sent over the transport network.
- the data streams are sent over RTP (Realtime Transport Protocol) tunnels using different IP headers 52 , 60 , which allows the routing system to differentiate the data streams and send the streams over separate paths.
- the addresses in the IP headers 52 , 60 specify separate multicast groups, as described below.
- the packets 50 , 58 contain the same RTP header 54 and data (payload) 56 .
- Information in the RTP header (e.g., sequence number, timestamp) is used to identify duplicate packets.
- the RTP header 54 and outer IP header 52 , 60 are inserted at the SER 16 and removed at the RER 18 .
- the RTP header 54 may be inserted between the original IP header and payload 56 or a new IP header and RTP header may be used to encapsulate the packet, including its original IP header. If the original IP header is used, the protocol type field, total length field, header checksum, and TTL are preferably modified.
- RTP is only one example of a protocol that may be used.
- Other protocols such as GRE (Generic Routing Encapsulation) or L2TPv3 (Layer 2 Tunneling Protocol Version 3), may also be used for encapsulation.
- FIGS. 4 and 5 are flowcharts illustrating an overview of a process for multicast stream duplication and merging in the transport network.
- FIG. 4 illustrates operation at the SER 16 .
- the SER 16 receives a multicast packet from the source 12 and destined for a group including the receiver 14 .
- the SER 16 duplicates the incoming packets into two or more flows (step 64 ) and transmits the packets over diverse data paths (step 66 ).
- the SER 16 when the SER 16 receives a packet (S->G) (wherein S is a source identifier and G is a group identifier), from the content source 12 , it duplicates the packet into two packets 50 , 58 , each with the same RTP header 54 , but different IP headers 52 , 60 (e.g., (S 1 ->G 1 ) and (S 2 ->G 2 )) ( FIG. 3 ).
- the resulting packets 50 , 58 have the same RTP sequence number and timestamp, but different outer IP headers 52 , 60 .
- the packets 50 , 58 are forwarded along their respective trees (T 1 , T 2 ) to the RER 18 ( FIG. 1 ).
- FIG. 5 illustrates a process at the RER 18 for setting up paths for the multicast data streams and merging duplicate data streams.
- the RER 18 receives a join message 32 from the receiver 14 .
- the join message identifies a group that the receiver 14 wants to join.
- Forwarding trees are created for transmitting multiple (e.g., two or more) multicast streams (step 72 ).
- Each tree comprises a common root (e.g., SER 16 ) and different network paths.
- SER 16 e.g., SER 16
- (S 1 , G 1 ) and (S 2 , G 2 ) are configured such that either S 1 or S 2 may be the same as S, either G 1 or G 2 may be the same as G, and (S 1 , G 1 ) and (S 2 , G 2 ) are distinguishable from one another.
- This allows the network to build two forwarding trees for (S 1 , G 1 ) and (S 2 , G 2 ) in such a way that the two forwarding trees do not share any network segment and both forwarding trees are rooted at the same SER 16 . This provides zero (or near-zero) packet loss in the event of a single failure within the protection domain.
- the RER 18 receives a packet from one of the forwarding trees and determines if the packet is a duplicate of another received packet. If the received packet is a duplicate packet, the RER 18 discards the packet (step 78 ). If the received packet is not a duplicate packet, the RER 18 forwards the packet to the receiver 14 (step 80 ).
- the RER 18 inspects packets using RTP information in the RTP header 54 to detect and discard duplicate packets. For every (S, G, SER), the RER 18 maintains a base sequence number (SEQ), a timestamp from the router from which the base sequence number is sent, and a window of X. The RER 18 uses this information to detect and discard duplicate packets. If a packet is not a duplicate, its RTP header 54 will be stripped off and the packet forwarded to the receiver 14 .
- SEQ base sequence number
- the base sequence number is 1000 and the window size is 500.
- a packet arrives with a sequence number 1001, and this sequence number has not been seen by the RER 18 .
- the sequence number 1001 is stored at the RER 18 and the packet is forwarded to the receiver 14 .
- a packet then arrives with a sequence number of 1003. Even though a packet with sequence number 1002 has not been received at the RER 18 , as long as a packet with sequence number 1003 has not been received, the packet is forwarded.
- a packet with sequence number 1002 then arrives at the RER 18 . This packet is also forwarded. If another packet arrives at the RER 18 with a sequence number of 1001, 1002, or 1003, it is dropped. If a packet arrives with a sequence number of less than 1000 or larger than 1500, it is forwarded by the RER 18 .
- the base sequence number is advanced by software.
- the embodiments described herein provide numerous advantages.
- the embodiments allow an “in-the-network” zero (or near-zero) packet loss solution that can handle fragmentation or misordering.
- An SP or MSO (MultiService Operator) customer can deploy the embodiments without having to depend on traffic behavior of their customer.
Abstract
Description
- The present disclosure relates generally to communication networks, and more particularly, to multicast routing.
- Traditional Internet Protocol (IP) communication allows a host to send packets to a single host (unicast transmission) or to all hosts (broadcast transmission). To support a demand to provide applications such as audio and video conference calls, audio broadcasting, and video broadcasting that involve high data rate transmission to multiple hosts, a third routing technique has evolved, multicast routing. In multicast routing, a host sends packets to a subset of all hosts as a group transmission.
- Multicast routing is widely deployed and continues to gain popularity with the advent of multimedia programs and the proliferation of Internet applications. Many multicast-based applications require delivery of large amounts of time sensitive and loss sensitive data simultaneously to large audiences. For these applications, a high degree of multicast resilience is desired to provide near-zero packet loss. In order to achieve this resilience, multicast traffic is split into multiple data streams which are sent over diverse paths. In the case of a failure on one path, the multicast traffic is still available via another path. The multiple data streams, however, result in duplicate packets which increase bandwidth usage and need to be properly discarded when the streams merge.
-
FIG. 1 illustrates an example of a network in which embodiments described herein may be implemented. -
FIG. 2 depicts an example of a network device useful in implementing embodiments described herein. -
FIG. 3 illustrates data packets transmitted over the network ofFIG. 1 . -
FIG. 4 is a flowchart illustrating a process for multicast stream duplication at a source edge router. -
FIG. 5 is a flowchart illustrating a process for setting up diverse paths and merging multicast streams at a receiver edge router. - Corresponding reference characters indicate corresponding parts throughout the several views of the drawings.
- In one embodiment, a method generally comprises receiving a join message from a receiver at a receiver edge router, the join message identifying a source and a group, and creating at least two forwarding trees for transmitting multicast data streams comprising duplicate data to the receiver edge router. The forwarding trees include a root at a source edge router in communication with the source and different network paths. The method further includes receiving a packet from one of the forwarding trees and processing the received packet to determine if the packet is a duplicate of another packet received at the receiver edge router. If the packet is a duplicate packet, it is discarded. If the packet is not a duplicate, it is forwarded to the receiver.
- In another embodiment, an apparatus generally comprises a processor configured to receive a join message from a receiver, create at least two forwarding trees for transmitting multicast data streams comprising duplicate data to the apparatus, the forwarding trees comprising a root at a source edge router in communication with a source, and different network paths, process a packet received from one of said forwarding trees to determine if the received packet is a duplicate of another received packet, discard the received packet if the packet is a duplicate packet, and forward the received packet to the receiver if the packet is not a duplicate packet. The apparatus further includes memory for storing data from received packets for use in determining if the received packet is a duplicate packet.
- The following description is presented to enable one of ordinary skill in the art to make and use the embodiments. Descriptions of specific embodiments and applications are provided only as examples and various modifications will be readily apparent to those skilled in the art. The general principles described herein may be applied to other embodiments and applications without departing from the scope of the disclosure. Thus, the present disclosure is not to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features described herein. For purpose of clarity, features relating to technical material that is known in the technical fields related to the embodiments have not been described in detail.
- Networks, such as those for use in transporting digitized television programs, Service Provider (SP) Multicast Virtual Private Network (MVPN) enabled networks that carry IP/TV traffic in Multicast Virtual Route Forwarding (MVRF), or dedicated or MVPN enabled networks for use in delivering market data information, for example, require a high degree of multicast resilience. This is due in part to the sensitivity of applications to packet loss and low latency requirements.
- A network that guarantees zero (or near-zero) packet loss in the case of a single network failure is referred to herein as a “live-live” network. This may be accomplished using, for example, an MT (multi-topology) IGP (Interior Gateway Protocol) capable network or different BGP (Border Gateway Protocol) loopbacks (update-source) with separate IGP processes to build diverse paths for multicast. The live-live network may also be created using ECMP (Equal-Cost Multi-Path), MoFRR (Multicast only Fast Reroute), RSVP/TE (Resource Reservation Protocol—Traffic Engineering), or other protocols. The primary traffic in the network is IP multicast. It is to be understood that these networks are only examples and the embodiments described herein may be implemented in different networks, without departing from the scope of the embodiments.
- In order to achieve high multicast resilience, duplicate streams of data are sent over diverse paths. The duplicate streams may be transmitted at a source and merged at a receiver. However, this process has drawbacks. For example, in some cases, the Service Provider may have no control over what the source can or will send, thus, it is not desirable to require the source to send multiple streams at all times. Also, the bandwidth available between the SP network and the receiver may be limited.
- The embodiments described herein duplicate and merge data streams within a transport network. As described in detail below, the embodiments utilize tunnels within the transport network to transmit the duplicate data streams. This allows the content source to send only one data stream and eliminates the need for the content receiver to merge the streams. The embodiments provide minimum latency and bandwidth for traffic entering and exiting the transport network.
- The embodiments described herein operate in the context of a
data communication network 10 including multiple network elements (FIG. 1 ). Some of the elements in a network that employs the embodiments may be routers, switches, gateways, or other network devices. For example, some of the nodes may be specially configured routers such as those available from Cisco Systems, Inc. of San Jose, Calif. As used herein the term router refers to devices that forward packets based on network and higher layer information. The router may include, for example, a master central processing unit (CPU), interfaces, and a bus (e.g., a PCI bus). When acting under the control of appropriate software or firmware, the CPU is responsible for such router tasks as routing table computations, network management, and general processing of packets. It preferably accomplishes all of these functions under the control of software including an operating system and any appropriate application software. In one embodiment, the network device is implemented on a general purpose network host machine as described below with respect toFIG. 2 . -
FIG. 2 depicts anetwork device 40 that may be used to implement embodiments described herein.Network device 40 is configured to implement all of the network protocols and extensions thereof described herein. In one embodiment,network device 40 is a programmable machine that may be implemented in hardware, software, or any combination thereof. Logic may be encoded in one or more tangible media for execution by a processor. For example,processor 42 may execute codes stored in aprogram memory 44.Program memory 44 is one example of a computer-readable medium.Program memory 44 can be a volatile memory. Another form of computer-readable medium storing the same codes is a type of non-volatile storage such as floppy disks, CD-ROMs, DVD-ROMs, hard disks, flash memory, etc. -
Network device 40 interfaces with physical media via a plurality of linecards (network interfaces) 46.Linecards 46 may incorporate Ethernet interfaces, DSL interfaces, Gigabit Ethernet interfaces, 10-Gigabit Ethernet interfaces, SONET interfaces, etc. As packets are received, processed, and forwarded bynetwork device 40, they may be stored in apacket memory 48. To implement functionality according to the system, linecards 46 may incorporate processing and memory resources similar to those discussed above in connection with the network device as a whole. It is to be understood that thenetwork device 40 shown inFIG. 2 and described above is only one example and that different configurations of network devices may be used. - The
network 10 implementing the embodiments described herein is configured to use IP multicast, which simultaneously delivers a stream of information to numerous recipients. A brief discussion of multicast routing is provided to help introduce concepts used in the embodiments described herein. - Multicast operation is based on the concept of a group. A multicast group is an arbitrary group of receivers that expresses an interest in receiving a particular data stream. An IP multicast address, or a portion thereof, specifies a particular group. Hosts (receivers) that are interested in receiving data flowing to a particular group join the group by transmitting a join message, using Internet Group Management Protocol (IGMP) or Multicast Listener Discovery (MLD), for example.
- Multicast-capable routers create distribution trees that control the path that IP multicast traffic takes through the network in order to deliver traffic to all receivers. Members of multicast groups can join or leave at any time; therefore the distribution trees are dynamically updated. In one embodiment, Protocol Independent Multicast (PIM) is used to dynamically create a multicast distribution tree to ensure distribution to intended receivers while limiting distribution so that network segments that are not in the path between the source and receivers are not burdened with unnecessary traffic. It is to be understood that PIM is only one example and that other multicast protocols may be used without departing from the scope of the embodiments.
- Referring again to
FIG. 1 , thenetwork 10 includes asource 12,receiver 14, Source Edge Router (SER) 16, Receiver Edge Router (RER) 18, andintermediate nodes SER 16 is a network device that is connected to the content source or a network through which the content source can be reached. TheRER 18 is a network device that is connected to the receiver or a network via which the receiver can be reached. TheSER 16 andRER 18 are preferably located along an edge oftransport network 34. The protection domain for multicast is between theSER 16 andRER 18. -
Source 12 may be a node (e.g., server, computer, or any other type of network device that can source IP media, such as video, audio, voice, data, etc.) or a network in communication with the source node. Thesource 12 transmits amulticast data stream 28 toSER 16, which transmits the data stream to one ormore receivers 14 over thenetwork 34. Thereceiver 14 may be a node that receives and stores or rendersmulticast stream 30, a network in communication with the receiver node, or a gateway to another network. Thereceiver 14 may be, for example, a computer terminal, personal computer, television with IP interface, Set Top Box (STP), Digital Video Recorder (DVR), Voice over IP (VoIP) phone, cell phone, eQAM (edge Quadrature Amplitude Modulation) device, Personal Digital Assistant (PDA), etc. - As described in detail below, the
SER 16 transmits twodata streams data stream 24 go throughintermediate node 20 wherein packets fordata stream 26 go throughintermediate node 22. The data streams 24, 26 may pass through any number of nodes intransport network 34. The tree topologies T1, T2, preferably have SER-to-RER disjointness. The use of different network paths increases the likelihood that packets from at least one of the twodata streams RER 18. If the twopaths transport network 34. -
FIG. 3 illustrates an example ofmulticast data packets different IP headers IP headers packets same RTP header 54 and data (payload) 56. Information in the RTP header (e.g., sequence number, timestamp) is used to identify duplicate packets. TheRTP header 54 andouter IP header SER 16 and removed at theRER 18. TheRTP header 54 may be inserted between the original IP header andpayload 56 or a new IP header and RTP header may be used to encapsulate the packet, including its original IP header. If the original IP header is used, the protocol type field, total length field, header checksum, and TTL are preferably modified. - It is to be understood that RTP is only one example of a protocol that may be used. Other protocols, such as GRE (Generic Routing Encapsulation) or L2TPv3 (Layer 2 Tunneling Protocol Version 3), may also be used for encapsulation.
-
FIGS. 4 and 5 are flowcharts illustrating an overview of a process for multicast stream duplication and merging in the transport network. -
FIG. 4 illustrates operation at theSER 16. Atstep 62, theSER 16 receives a multicast packet from thesource 12 and destined for a group including thereceiver 14. TheSER 16 duplicates the incoming packets into two or more flows (step 64) and transmits the packets over diverse data paths (step 66). For example, when theSER 16 receives a packet (S->G) (wherein S is a source identifier and G is a group identifier), from thecontent source 12, it duplicates the packet into twopackets same RTP header 54, butdifferent IP headers 52, 60 (e.g., (S1->G1) and (S2->G2)) (FIG. 3 ). The resultingpackets outer IP headers packets FIG. 1 ). -
FIG. 5 illustrates a process at theRER 18 for setting up paths for the multicast data streams and merging duplicate data streams. Atstep 70, theRER 18 receives ajoin message 32 from thereceiver 14. The join message identifies a group that thereceiver 14 wants to join. Forwarding trees are created for transmitting multiple (e.g., two or more) multicast streams (step 72). Each tree comprises a common root (e.g., SER 16) and different network paths. For example, when theRER 18 receives an (S, G) join from thereceiver 14, it turns the (S, G) join into two joins (S1, G1) and (S2, G2). (S1, G1) and (S2, G2) are configured such that either S1 or S2 may be the same as S, either G1 or G2 may be the same as G, and (S1, G1) and (S2, G2) are distinguishable from one another. This allows the network to build two forwarding trees for (S1, G1) and (S2, G2) in such a way that the two forwarding trees do not share any network segment and both forwarding trees are rooted at thesame SER 16. This provides zero (or near-zero) packet loss in the event of a single failure within the protection domain. - At
steps RER 18 receives a packet from one of the forwarding trees and determines if the packet is a duplicate of another received packet. If the received packet is a duplicate packet, theRER 18 discards the packet (step 78). If the received packet is not a duplicate packet, theRER 18 forwards the packet to the receiver 14 (step 80). - The following describes one example of a process for identifying duplicate packets at the
RER 18. In one embodiment, theRER 18 inspects packets using RTP information in theRTP header 54 to detect and discard duplicate packets. For every (S, G, SER), theRER 18 maintains a base sequence number (SEQ), a timestamp from the router from which the base sequence number is sent, and a window of X. TheRER 18 uses this information to detect and discard duplicate packets. If a packet is not a duplicate, itsRTP header 54 will be stripped off and the packet forwarded to thereceiver 14. - In one example, the base sequence number is 1000 and the window size is 500. A packet arrives with a sequence number 1001, and this sequence number has not been seen by the
RER 18. The sequence number 1001 is stored at theRER 18 and the packet is forwarded to thereceiver 14. A packet then arrives with a sequence number of 1003. Even though a packet with sequence number 1002 has not been received at theRER 18, as long as a packet with sequence number 1003 has not been received, the packet is forwarded. A packet with sequence number 1002 then arrives at theRER 18. This packet is also forwarded. If another packet arrives at theRER 18 with a sequence number of 1001, 1002, or 1003, it is dropped. If a packet arrives with a sequence number of less than 1000 or larger than 1500, it is forwarded by theRER 18. In one embodiment, the base sequence number is advanced by software. - As can be observed from the foregoing, the embodiments described herein provide numerous advantages. For example, the embodiments allow an “in-the-network” zero (or near-zero) packet loss solution that can handle fragmentation or misordering. An SP or MSO (MultiService Operator) customer can deploy the embodiments without having to depend on traffic behavior of their customer.
- Although the method and system have been described in accordance with the embodiments shown, one of ordinary skill in the art will readily recognize that there could be variations made to the embodiments without departing from the scope of the present embodiments. Accordingly, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/584,008 US8184628B2 (en) | 2009-08-28 | 2009-08-28 | Network based multicast stream duplication and merging |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/584,008 US8184628B2 (en) | 2009-08-28 | 2009-08-28 | Network based multicast stream duplication and merging |
Publications (2)
Publication Number | Publication Date |
---|---|
US20110051727A1 true US20110051727A1 (en) | 2011-03-03 |
US8184628B2 US8184628B2 (en) | 2012-05-22 |
Family
ID=43624817
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/584,008 Active 2030-07-24 US8184628B2 (en) | 2009-08-28 | 2009-08-28 | Network based multicast stream duplication and merging |
Country Status (1)
Country | Link |
---|---|
US (1) | US8184628B2 (en) |
Cited By (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120059946A1 (en) * | 2010-09-03 | 2012-03-08 | Hulu Llc | Bandwidth allocation with modified seek function |
US20120201244A1 (en) * | 2011-02-04 | 2012-08-09 | Cisco Technology, Inc. | Optimizing otv multicast traffic flow for site local receivers |
US20140204939A1 (en) * | 2013-01-23 | 2014-07-24 | Alcatel-Lucent Usa Inc. | Method to setup protocol independent multicast trees in the presence of unidirectional tunnels |
US20140280752A1 (en) * | 2013-03-15 | 2014-09-18 | Time Warner Cable Enterprises Llc | System and method for seamless switching between data streams |
US8861520B1 (en) * | 2011-02-24 | 2014-10-14 | Directpacket Research, Inc. | Packet cloning for enhanced delivery of communication from a source device to one or more destination devices |
US8861340B1 (en) | 2012-03-12 | 2014-10-14 | Juniper Networks, Inc. | Fast reroute using maximally redundant trees |
US8958286B1 (en) * | 2012-03-12 | 2015-02-17 | Juniper Networks, Inc. | Fast reroute for multicast using maximally redundant trees |
US20150063211A1 (en) * | 2013-08-29 | 2015-03-05 | Samsung Electronics Co., Ltd. | Method and apparatus for applying nested network cording in multipath protocol |
US9100328B1 (en) | 2012-03-12 | 2015-08-04 | Juniper Networks, Inc. | Forwarding mechanisms for fast reroute using maximally redundant trees |
US9270426B1 (en) | 2012-09-11 | 2016-02-23 | Juniper Networks, Inc. | Constrained maximally redundant trees for point-to-multipoint LSPs |
US9397965B2 (en) * | 2011-09-22 | 2016-07-19 | Kakao Corp. | Conversational message service operating method for providing acknowledgement |
US9571387B1 (en) | 2012-03-12 | 2017-02-14 | Juniper Networks, Inc. | Forwarding using maximally redundant trees |
US10320954B2 (en) * | 2017-02-03 | 2019-06-11 | Microsoft Technology Licensing, Llc | Diffusing packets to identify faulty network apparatuses in multipath inter-data center networks |
US10554425B2 (en) | 2017-07-28 | 2020-02-04 | Juniper Networks, Inc. | Maximally redundant trees to redundant multicast source nodes for multicast protection |
US20210385169A1 (en) * | 2017-02-12 | 2021-12-09 | Mellanox Technologies Ltd. | Communication apparatus generating and eliminating redundant data packets |
US20230073121A1 (en) * | 2020-03-06 | 2023-03-09 | Huawei Technologies Co., Ltd. | SR Policy Issuing Method and Apparatus and SR Policy Receiving Method and Apparatus |
WO2023177471A1 (en) * | 2022-03-15 | 2023-09-21 | Microsoft Technology Licensing, Llc | Packet level redundancy in distributed computing systems |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8411129B2 (en) | 2009-12-14 | 2013-04-02 | At&T Intellectual Property I, L.P. | Video conference system and method using multicast and unicast transmissions |
US9166902B1 (en) | 2012-09-14 | 2015-10-20 | Cisco Technology, Inc. | Per-port multicast processing |
Citations (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6097720A (en) * | 1998-04-07 | 2000-08-01 | 3Com Corporation | Enabling multicast distribution efficiencies in a dialup access environment |
US6269080B1 (en) * | 1999-04-13 | 2001-07-31 | Glenayre Electronics, Inc. | Method of multicast file distribution and synchronization |
US20020150094A1 (en) * | 2000-10-27 | 2002-10-17 | Matthew Cheng | Hierarchical level-based internet protocol multicasting |
US20030112749A1 (en) * | 2001-12-18 | 2003-06-19 | Brian Hassink | Methods, systems, and computer program products for detecting and/or correcting faults in a multiprotocol label switching network by using redundant paths between nodes |
US20030139929A1 (en) * | 2002-01-24 | 2003-07-24 | Liang He | Data transmission system and method for DSR application over GPRS |
US20050265284A1 (en) * | 2003-10-10 | 2005-12-01 | Hsu Liangchi Alan | Apparatus, and associated method, for facilitating communication handoff in multiple-network radio communication system |
US20060050643A1 (en) * | 2004-09-06 | 2006-03-09 | Hitachi Communication Technologies, Ltd. | Router for multicast redundant routing and system for multicast redundancy |
US20060120396A1 (en) * | 2004-12-02 | 2006-06-08 | Kddi Corporation | Communication system, delay insertion server, backup server and communication control apparatus |
US20060221975A1 (en) * | 2005-04-05 | 2006-10-05 | Alton Lo | Transporting multicast over MPLS backbone using virtual interfaces to perform reverse-path forwarding checks |
US20070177593A1 (en) * | 2006-01-30 | 2007-08-02 | Juniper Networks, Inc. | Forming multicast distribution structures using exchanged multicast optimization data |
US20080095160A1 (en) * | 2006-10-24 | 2008-04-24 | Cisco Technology, Inc. | Subnet Scoped Multicast / Broadcast Packet Distribution Mechanism Over a Routed Network |
US20080101360A1 (en) * | 2006-10-25 | 2008-05-01 | At&T Corp. | Method and apparatus for providing congruent multicast and unicast routing |
US20080225850A1 (en) * | 2007-03-14 | 2008-09-18 | Cisco Technology, Inc. | Unified transmission scheme for media stream redundancy |
US20080267078A1 (en) * | 2007-04-26 | 2008-10-30 | Cisco Technology, Inc. | Multicast fast reroute |
US20080285452A1 (en) * | 2007-05-14 | 2008-11-20 | Cisco Technology, Inc. | Remote monitoring of real-time internet protocol media streams |
US20090201803A1 (en) * | 2008-02-12 | 2009-08-13 | Cisco Technology, Inc. | Multicast fast reroute for network topologies |
US20090245248A1 (en) * | 2008-03-31 | 2009-10-01 | Peter Arberg | Method and apparatus for providing resiliency in multicast networks |
US20100202452A1 (en) * | 2009-02-11 | 2010-08-12 | Corrigent Systems Ltd | High-speed processing of multicast content requests |
US7830825B2 (en) * | 2004-03-24 | 2010-11-09 | Huawei Technologies Co., Ltd. | Method for realizing the multicast service |
-
2009
- 2009-08-28 US US12/584,008 patent/US8184628B2/en active Active
Patent Citations (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6097720A (en) * | 1998-04-07 | 2000-08-01 | 3Com Corporation | Enabling multicast distribution efficiencies in a dialup access environment |
US6269080B1 (en) * | 1999-04-13 | 2001-07-31 | Glenayre Electronics, Inc. | Method of multicast file distribution and synchronization |
US20020150094A1 (en) * | 2000-10-27 | 2002-10-17 | Matthew Cheng | Hierarchical level-based internet protocol multicasting |
US20030112749A1 (en) * | 2001-12-18 | 2003-06-19 | Brian Hassink | Methods, systems, and computer program products for detecting and/or correcting faults in a multiprotocol label switching network by using redundant paths between nodes |
US20030139929A1 (en) * | 2002-01-24 | 2003-07-24 | Liang He | Data transmission system and method for DSR application over GPRS |
US20050265284A1 (en) * | 2003-10-10 | 2005-12-01 | Hsu Liangchi Alan | Apparatus, and associated method, for facilitating communication handoff in multiple-network radio communication system |
US7830825B2 (en) * | 2004-03-24 | 2010-11-09 | Huawei Technologies Co., Ltd. | Method for realizing the multicast service |
US20060050643A1 (en) * | 2004-09-06 | 2006-03-09 | Hitachi Communication Technologies, Ltd. | Router for multicast redundant routing and system for multicast redundancy |
US20060120396A1 (en) * | 2004-12-02 | 2006-06-08 | Kddi Corporation | Communication system, delay insertion server, backup server and communication control apparatus |
US20060221975A1 (en) * | 2005-04-05 | 2006-10-05 | Alton Lo | Transporting multicast over MPLS backbone using virtual interfaces to perform reverse-path forwarding checks |
US20070177593A1 (en) * | 2006-01-30 | 2007-08-02 | Juniper Networks, Inc. | Forming multicast distribution structures using exchanged multicast optimization data |
US20080095160A1 (en) * | 2006-10-24 | 2008-04-24 | Cisco Technology, Inc. | Subnet Scoped Multicast / Broadcast Packet Distribution Mechanism Over a Routed Network |
US7626984B2 (en) * | 2006-10-25 | 2009-12-01 | At&T Corp. | Method and apparatus for providing congruent multicast and unicast routing |
US20080101360A1 (en) * | 2006-10-25 | 2008-05-01 | At&T Corp. | Method and apparatus for providing congruent multicast and unicast routing |
US20080225850A1 (en) * | 2007-03-14 | 2008-09-18 | Cisco Technology, Inc. | Unified transmission scheme for media stream redundancy |
US20080267078A1 (en) * | 2007-04-26 | 2008-10-30 | Cisco Technology, Inc. | Multicast fast reroute |
US20080285452A1 (en) * | 2007-05-14 | 2008-11-20 | Cisco Technology, Inc. | Remote monitoring of real-time internet protocol media streams |
US20090201803A1 (en) * | 2008-02-12 | 2009-08-13 | Cisco Technology, Inc. | Multicast fast reroute for network topologies |
US20090245248A1 (en) * | 2008-03-31 | 2009-10-01 | Peter Arberg | Method and apparatus for providing resiliency in multicast networks |
US20100202452A1 (en) * | 2009-02-11 | 2010-08-12 | Corrigent Systems Ltd | High-speed processing of multicast content requests |
Cited By (26)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8832293B2 (en) * | 2010-09-03 | 2014-09-09 | Hulu, LLC | Bandwidth allocation with modified seek function |
US20120059946A1 (en) * | 2010-09-03 | 2012-03-08 | Hulu Llc | Bandwidth allocation with modified seek function |
US9112711B2 (en) | 2011-02-04 | 2015-08-18 | Cisco Technology, Inc. | Optimizing OTV multicast traffic flow for site local receivers |
US20120201244A1 (en) * | 2011-02-04 | 2012-08-09 | Cisco Technology, Inc. | Optimizing otv multicast traffic flow for site local receivers |
US8774076B2 (en) * | 2011-02-04 | 2014-07-08 | Cisco Technology, Inc. | Optimizing OTV multicast traffic flow for site local receivers |
US8861520B1 (en) * | 2011-02-24 | 2014-10-14 | Directpacket Research, Inc. | Packet cloning for enhanced delivery of communication from a source device to one or more destination devices |
US9397965B2 (en) * | 2011-09-22 | 2016-07-19 | Kakao Corp. | Conversational message service operating method for providing acknowledgement |
US9571387B1 (en) | 2012-03-12 | 2017-02-14 | Juniper Networks, Inc. | Forwarding using maximally redundant trees |
US8861340B1 (en) | 2012-03-12 | 2014-10-14 | Juniper Networks, Inc. | Fast reroute using maximally redundant trees |
US8958286B1 (en) * | 2012-03-12 | 2015-02-17 | Juniper Networks, Inc. | Fast reroute for multicast using maximally redundant trees |
US9100328B1 (en) | 2012-03-12 | 2015-08-04 | Juniper Networks, Inc. | Forwarding mechanisms for fast reroute using maximally redundant trees |
US9270426B1 (en) | 2012-09-11 | 2016-02-23 | Juniper Networks, Inc. | Constrained maximally redundant trees for point-to-multipoint LSPs |
US9401859B2 (en) * | 2013-01-23 | 2016-07-26 | Alcatel Lucent | Method to setup protocol independent multicast trees in the presence of unidirectional tunnels |
US20140204939A1 (en) * | 2013-01-23 | 2014-07-24 | Alcatel-Lucent Usa Inc. | Method to setup protocol independent multicast trees in the presence of unidirectional tunnels |
US10567489B2 (en) * | 2013-03-15 | 2020-02-18 | Time Warner Cable Enterprises Llc | System and method for seamless switching between data streams |
US20140280752A1 (en) * | 2013-03-15 | 2014-09-18 | Time Warner Cable Enterprises Llc | System and method for seamless switching between data streams |
US10462043B2 (en) * | 2013-08-29 | 2019-10-29 | Samsung Electronics Co., Ltd. | Method and apparatus for applying nested network cording in multipath protocol |
US20150063211A1 (en) * | 2013-08-29 | 2015-03-05 | Samsung Electronics Co., Ltd. | Method and apparatus for applying nested network cording in multipath protocol |
US20190273810A1 (en) * | 2017-02-03 | 2019-09-05 | Microsoft Technology Licensing, Llc | Diffusing packets to identify faulty network apparatuses in multipath inter-data center networks |
US10320954B2 (en) * | 2017-02-03 | 2019-06-11 | Microsoft Technology Licensing, Llc | Diffusing packets to identify faulty network apparatuses in multipath inter-data center networks |
US10931796B2 (en) * | 2017-02-03 | 2021-02-23 | Microsoft Technology Licensing, Llc | Diffusing packets to identify faulty network apparatuses in multipath inter-data center networks |
US20210385169A1 (en) * | 2017-02-12 | 2021-12-09 | Mellanox Technologies Ltd. | Communication apparatus generating and eliminating redundant data packets |
US10554425B2 (en) | 2017-07-28 | 2020-02-04 | Juniper Networks, Inc. | Maximally redundant trees to redundant multicast source nodes for multicast protection |
US11444793B2 (en) | 2017-07-28 | 2022-09-13 | Juniper Networks, Inc. | Maximally redundant trees to redundant multicast source nodes for multicast protection |
US20230073121A1 (en) * | 2020-03-06 | 2023-03-09 | Huawei Technologies Co., Ltd. | SR Policy Issuing Method and Apparatus and SR Policy Receiving Method and Apparatus |
WO2023177471A1 (en) * | 2022-03-15 | 2023-09-21 | Microsoft Technology Licensing, Llc | Packet level redundancy in distributed computing systems |
Also Published As
Publication number | Publication date |
---|---|
US8184628B2 (en) | 2012-05-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8184628B2 (en) | Network based multicast stream duplication and merging | |
CN107276904B (en) | Method and network device for distributing multicast service | |
US7684316B2 (en) | Multicast fast reroute for network topologies | |
US9497110B2 (en) | Multicast-only fast re-route processing for point-to-multipoint pseudowire | |
US7826348B2 (en) | Multicast fast reroute | |
US9118564B2 (en) | Providing PIM-SM support for mRSVP-TE based multicast virtual private networks | |
US8422364B2 (en) | Multicast label distribution protocol node protection | |
US8830999B2 (en) | Dynamic updating of a label switched path | |
US9509636B2 (en) | Multicast traffic management within a wireless mesh network | |
US8848709B2 (en) | Source rooted multicast (SRM) | |
US9992099B2 (en) | Protocol independent multicast designated router redundancy | |
US8611346B1 (en) | Multicast sparse-mode source redundancy | |
US9160683B2 (en) | Providing PIM-SSM support for MRSVP-TE based multicast virtual private networks | |
US9203631B2 (en) | Multicast distribution trees for mRSVP-TE based multicast virtual private networks | |
US10382217B1 (en) | Single actual-forwarder election in multicast VPNs | |
US20140314079A1 (en) | Efficient Multicast Delivery to Dually Connected (VPC) Hosts in Overlay Networks | |
US9240942B2 (en) | Bandwidth utilization for equal cost multiple paths | |
EP2567510B1 (en) | Source selection by routers | |
US11811509B2 (en) | Upstream multicast hop (UMH) extensions for anycast deployments | |
Shihab et al. | PIM-SM based Multicast Comparison for IPv4 verses IPv6 using GNS3 and JPERF | |
CA2902402A1 (en) | Multicast traffic management within a wireless mesh network | |
Raahemi et al. | A minimum-cost resilient tree-based VPLS for digital TV broadcast services |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: CISCO TECHNOLOGY, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CAI, YIQUN;ECKERT, TOERLESS;OU, HEIDI;SIGNING DATES FROM 20090826 TO 20090827;REEL/FRAME:023222/0471 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 12 |