|Publication number||US4799215 A|
|Application number||US 06/916,429|
|Publication date||Jan 17, 1989|
|Filing date||Oct 7, 1986|
|Priority date||Oct 7, 1985|
|Also published as||CA1266318A, CA1266318A1, DE3686629D1, DE3686629T2, EP0219049A2, EP0219049A3, EP0219049B1|
|Publication number||06916429, 916429, US 4799215 A, US 4799215A, US-A-4799215, US4799215 A, US4799215A|
|Original Assignee||Nec Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (3), Non-Patent Citations (2), Referenced by (114), Classifications (19), Legal Events (4)|
|External Links: USPTO, USPTO Assignment, Espacenet|
The present invention relates to a packet-switched communications system which is capable of high speed, high throughput switching operations to handle a variety of traffic patterns by efficient utilization of system resources.
According to a prior art packet switched communications system, flow control with a higher-level data link control and error recovery retransmission control are effected on a link-by-link throughout the network. These techniques are needed because the transmission speed of the prior art packet switching system is low, typically in the range between several Kbps and several tens of Kbps and the bit error rate is relatively high. Since these techniques involve a large number of complicated procedures, they require a lengthy period execution time. In addition, the execution of such procedures must be repeated at locations where packets are retransmitted over the network, resulting in packets arriving at delayed times.
The introduction of optical fibers as a high speed high quality transmission medium allows reduction of packet transmission delay. However, the amount of delay involved in the execution of link-by-link flow control and retransmission is becoming a dominant factor of the total delay time. On the other hand, the frequency at which the retransmission process must be effected has decreased significantly with the reduction of transmission bit error rate and the control overhead for link-by-link retransmission increases significantly. As a result, a high speed, high throughput packet switched communications system cannot be implemented with the introduction of high speed, high quality transmission media without improvement of the switching speed of the packet switching system,
More specifically, FIG. 1 shows a packet transfer sequence effected between end terminals according to a prior art packet switched communications system. In FIG. 1, character L represents the execution of flow control and retransmission control, on a link-by-link basis, L(S) indicates a transmission execution and L(R) indicates a reception execution. Character N represents a packet switching operation by means of which a route is determined for each packet. D#n represents the packet and ACK(L) and ACK#n represent link-by-link acknowledgement and end-to-end acknowledgement, respectively. RGJ(L)#n represents a request for retransmission between links and ERR indicates that a packet has been affected by an error during transmission. The number of frames which can be continuously transmitted between switching offices, or "nodes" and the number of frames which can be continuously transmitted between end users are both assumed to be "8". Since no acknowledgment ACK is returned to packets D#0 to D#14 generated by a terminal PT(A) and eight packets can be continuously retransmitted, the first eight packets D#0 through D#7 are allowed entry to the network. Acknowledgement ACK(L) is returned at each link and acknowledgement ACK#n is returned to the source terminal to indicate that the destination terminal has correctly received packets D#0 through D#n. Assume that an error has occurred in packet D#4 during transmission between nodes PS(A) and PS(B) and node PS(B) returns a request for retransmission REJ(L)#4 to node PS(A) and the latter retransmits packets D#4 through D#7 to node PS(B). When acknowledgment ACK#4 is returned to source terminal PT(A), the remaining four data packets are allowed entry to the network.
As can be seen from FIG. 1, the amount of processing time at each node is substantial particularly when speech signal is transmitted. Because of the redundancy of information in the speech signal, no retransmission is required even if packets are noise affected at low frequency. Although buffer overflow in an interface node can be avoided, it is impossible to directly restrict the traffic at the entry point of the network when overflow occurs in an intermediate node. As a result, the traffic congestion in an intermediate node is likely to migrate to neighboring nodes.
It is therefore an object of the present invention to provide a high-speed packet switched communications system and a high-speed packet switching system incorporated in the communications system to serve as a switching node.
The foregoing problems are eliminated by the packet switched communications system of the invention by the employment of end-to-end flow control and retransmission instead of the link-to-link method of flow control and retransmission.
The communications system includes a plurality of switching nodes for serving a plurality of terminals through transmission links. Each of said terminals transmits a packet of data link layer control protocol including a network layer control protocol to one of said switching nodes. Each of the switching nodes comprises means responsive to the network control protocol of the packet for routing the packet to one of the transmission links defined by the network layer control protocol of the packet.
Preferably, the routing means comprises a plurality of line controllers associated respectively to the transmission links, each of the line controllers including means for appending a physical address of said packet at the defined transmission link, updating the network layer control protocol of the packet with a logical address of the packet at the defined transmission link and sending the updated packet to one of the line controllers corresponding to the defined transmission link. A switching network interconnects the line controllers.
According to another aspect of the present invention, each of said terminals is capable of returning an acknowledgment to a source terminal when a packet is correctly received therefrom, transmitting a series of packets without receiving the acknowledgment and retransmitting to a destination terminal a packet which has not been correctly received. Each switching nodes detects the degree of congestion of traffic in a buffer which receives packets from an adjacent node or terminal and transmits a traffic control signal instructing the cessation of transmission of packets when said detected degree of congestion exceeds a predetermined level, and halts the transmission of a packet which is to be transmitted to the adjacent switching node or terminal upon receipt of the traffic control signal.
According to a second aspect of the invention, the communications system is of a virtual circuit type in which a call is established through a logical channel between source and destination terminals in response to a control packet of a multipacket message and the logical channel is permanently established until the call terminates to allow message packets of the multipacket message to follow the established channel. Each terminal is capable of retransmitting a packet to a destination terminal which is not correctly received by the destination terminal. Each of switching node comprises means for detecting the degree of congestion of traffic in a buffer which receives packets from an adjacent switching node or terminal and transmitting a traffic control signal indicating a cessation of transmission of packets when the detected degree of congestion exceeds a predetermined level. In response to the traffic control signal the transmission of a packet which is to be transmitted to the adjacent switching node or terminal is stopped and a timeout signal is generated when the traffic control signal is present for a predetermined time interval. The establishment of a logical channel to the adjacent switching node is prevented in response to the timeout signal. The logical channel established between the adjacent switching node and the destination terminal is determined and the source terminal is identified in accordance with the detected logical channel to prevent the transmission of packets from the source terminal. An alternate route is then established for routing the call from the source terminal to the destination terminal.
According to a third aspect of the invention, a packet switching system is provided having a plurality of line controllers and a central processor which are connected by a switching network, wherein the switching system handles a multipacket message comprising a control packet and one or more message packets, each of the packets having a logical channel number. The switching system determines a logical channel in response to the control packet and the relationship between the route number of an outgoing transmission link through which the logical channel passes and the channel number of the logical channel and sends a routing signal representing the determined relationship through the network to each of the line controllers. Each of the line controllers comprises a memory in which the routing information is stored. A logical channel number is extracted from a message packet as it is received by the line controller to access the memory to read the route number of an outgoing transmission link through which the message packet is to be transmitted and the logical channel number of the packet at the outgoing transmission link. The message packet is processed by updating its logical channel number with the detected logical channel number and by appending the detected route number to the message packet and routed through the network to one of the line controllers associated with said outgoing transmission link.
FIG. 1 is an illustration of packet transfer sequences according to the link-by-link method of flow control and retransmission employed in a prior art packet switched communications system;
FIG. 2 is a schematic illustration of a packet switched communications system embodying the present invention, which is useful for description of the end-to-end flow control and alternate routing operations;
FIG. 3 is an illustration of a data format according to the present invention;
FIG. 4 is a block diagram of a packet switching system embodying the present invention;
FIG. 5 is a block diagram of each of the line controllers of FIG. 4;
FIG. 6 is a block diagram of the data-link layer controller of FIG. 5;
FIG. 7 is a block diagram of the switching control memory of FIG. 5;
FIG. 8 is a block diagram of the header updating circuit of FIG. 5;
FIGS. 9 and 10 are details of the receive buffer and transmit buffer of FIG. 5, respectively;
FIG. 11 is an illustration of details of the central processor of FIG. 4;
FIGS. 12A, 12B, 12C are illustrations of the contents of the call setup, outgoing route and incoming route tables, respectively, of FIG. 7 and FIG. 12D is an illustration of the contents of the overflow control table of FIG. 7;
FIG. 13 is a schematic illustration useful for describing the operation of intranode overflow control;
FIG. 14 is a schematic illustration useful for describing the operation of internode overflow control;
FIG. 15 is an illustration of the contents of the routing table of FIG. 11; and
FIGS. 16 and 17 are illustrations of packet transfer sequences according to the end-to-end flow control and retransmission of the present invention.
A virtual-circuit packet-switched communications system embodying the present invention is shown in FIG. 2. The communications system comprises a plurality of packet switching offices or "nodes" PS and end users or "terminals" PT(A), PT(B) and PT(C) which terminate at interface nodes PS(A), PS(B) and PS(C), respectively. For purposes of illustration, the terminal PT(C) is shown as having an alternate transmission link to the interface node PS(D). In the virtual-circuit network, the transmission capacity is assigned on an "as needed" basis, but all packets of a multipacket message follow the same route through the network. Before interactive communication begins, a route is established through the network in response to a call-establishing control packet, and all participating nodes are informed of the connection and how to route the individual packets that follow. The route so established is a virtual circuit, or "logical channel" as is interchangeably used in data communications art. At the end of a connection, a virtual circuit is released by a call-clearing control packet propagating through the network.
To eliminate the prior art link-by-link method of flow control to achieve high speed packet switching, frame and packet formats are provided shown in FIG. 3 which respectively correspond to data-link layer and network layer protocols. A network-layer packet 105 is transmitted between a terminal and an interface switching node and between adjacent switching nodes in the form of a frame 101 by appending to the network layer packet a data-link layer header field which comprises start and closing flags 102, a receive-ready/receive-not-ready subfield 103 and a frame check sequence 104. This data-link layer header is processed by participating switching nodes in the network and the terminals. The packet comprises a logical channel number subfield 107, a packet identifier 108 designated "TYP", a retransmission enable/disable subfield 109 designated "RT/NRT", and subfields 110 and 111 designated N(S) and N(R), with N(S) indicating the sequence number of packets sent on a common logical channel from a source node and N(R) indicating the sequence number of packets sent to the source node from a receiving node.
The packet identifier identifies a packet with a particular characterization including control, message, traffic congestion (CONG), congestion-free (FREE), acknowledgment (ACK), and retransmission request (REJ). The subfield RT/NRT with a binary "0" enables the retransmission request packet (REJ) to be transmitted and with a binary "1" indicating the disablement of the retransmission packet.
As illustrated in FIG. 4, each packet swtiching node comprises a central processor 1, a plurality of line controllers 2 to which associated end users or adjacent nodes are connected by way of transmission links 3 which may be four-wire lines or full-duplex facilites. Each line controller 2 operates as an incoming line controller or an outgoing line controller. A control packet is transferred from an incoming line controller through a control bus 6 to the central processor 1 where it is processed and transferred to an outgoing line controller whenever a connection is set up or released. Central processor 1 stores routing information regarding the incoming and outgoing route numbers and incoming and outgoing logical channel numbers whenever a connection is established by address data contained in the control packet. Message packets are transferred between incoming and outgoing line controllers 2 over a data bus 5 under the control of a bus controller 7.
Details of line controller 2 are shown in FIG. 5. When controller 2 operates as an incoming line controller, the link 3 is the incoming route and packets received on link 3 from a source user are passed through a data-link layer controller 10 and a header updating circuit 11 to a receive buffer 14 for transfer to an outgoing line controller 2. When packets are transferred from an incoming line controller and sequentially stored into the transmit buffer 16, the line controller operates as an outgoing line controller for transmitting the packets through the data-link layer controller 10 on link 3 to a destination user.
Header updating circuit 11 has an input from the data-link layer controller 10 to examine the header field of a packet received from the controller 10 and extract the logical channel number from the LCN subfield 107 and the packet identifyer from the TYP subfield 108 of the packet. Updating circuit 11 uses the extracted data as an address to access a switching control member 12 which is updated by a memory controller 13 with routing information supplied from the central processor 1 in response to the establishment of a connection by each of the line controllers 2. During the period following a call setup, the data read out of the memory 12 by the header updating circuit 11 indicates the outgoing route number RN and outgoing logical channel number LCN. Updating circuit 11 inserts the outgoing route number to a subfield 106 of a message packet and the outgoing logical channel number into the LCN subfield of the packet so that it appears as shown at 112 in FIG. 3 at the input of the receive buffer 14. If the packet is a control packet, the logical channel number of the packet remains unchanged and the outgoing route number is updated such that the control packet is routed to the central processor 1 through receive buffer 14 and an interface 15. The updated message packet is routed to an outgoing line controller identified by the outgoing route number and logical channel number indicated by the header field of the message packet.
Receive buffer 14 has a control input from the switching and overflow control memory 12 to receive a node overflow signal and a control output leading to the data-link layer controller 10 to supply to it a buffer-overflow signal (RNR). As shown in FIG. 9, the receive buffer 14 includes a memory 140 having a data input terminal connected to the output of the header updating circuit 11 and an output terminal connected to the bus interface 15. An address controller 141 controls the memory 140 with write and read enable signals at proper timing in response to a system clock under normal conditions and examines the storage level of the memory 140 to generate the buffer overflow signal when it is filled with more than normal packets. Address controller 141 responds to the node overflow signal by disabling the read/write operation.
Transmit buffer 16 has a control input connected from the data-link layer controller 10 to receive a hald-and-discard signal and a control output leading to the controller 13 to supply to it a transmit buffer overflow signal. As shown in FIG. 10, the transmit buffer 16 includes a memory 160 having a data input terminal connected to the bus interface 15 and an output terminal connected to the data-link layer controller 10. An address controller 161 controls the memory 160 with write and read enable signals at proper timing in response to a system clock under normal conditions and examines the storage level of the memory 160 to generate the transmit buffer overflow signal when it is filled with more than normal packets. Address controller 161 responds to the halt-and-discard signal by disabling the read/write operation.
Specifically, in FIG. 6, the data-link layer controller 10 provides data-link layer process by detecting start and closing flags, a frame check sequence and an RR/RNR subfield in a packet received through a line interface 20 and by appending such data-link layer subfields to a packet to be sent through the interface 20. To this end, a framing circuit 21, an FCS error detector 22 and an RR/RNR (receive-ready/ receive-not-ready) detector 23 are connected in a series circuit from the interface 20 and the header updating circuit 11. An RR/RNR generator 24, an FCS generator and a framing circuit 26 are connected in series from the output of transmit buffer 16 and the line interface 20 to provide the data-link-layer process. The RR/RNR detector 23 applies an output signal to a link overflow controller 27 when the RR/RNR subfield indicates that a "receive-not-ready" condition, or traffic overflow has occurred in an adjacent node.
The function of the link overflow controller 27 includes the detection of the presence of a "receive-not-ready" signal returned from the adjacent node, the detection of its continuity exceeding a predetermined time interval and the application of a link-overflow signal to the controller 13. This is to allow an intermediate switching node to determine the source terminal and send to it a congestion (CONG) signal to prevent the outgoing traffic at entry point of the network in order to reduce the volume of traffic from an intermediate node to an adjacent node. At the same time, link overflow controller 27 causes the transmit buffer 16 to stop sending packets to the source terminal. If the storage level of the buffer 16 exceeds a predetermined value as a result of the cessation of packet transmission, controller 27 causes buffer 16 to discard the first of packets that form a queue in the buffer 16 waiting to be served. Further, link overflow controller 14 has an input from the receive buffer 14 to respond to a "buffer overflow" signal by enabling the RR/RNR generator 24 to reduce the amount of traffic from the adjacent node or terminal by indicating that a receive buffer is overflowd. The RR/RNR generator 24 operates when the traffic in the opposite direction is overflowed.
In FIG. 7, the switching control memory 12 comprises a call setup table 30, an outgoing route table 31, an incoming route table 32 and an overflow control table 33. The outgoing and incoming route tables 31 and 32 are updated by the memory controller 13 by way of the central processor 1. More specifically, the call setup table 30 stores an outgoing route number (which is permanently indicative of the address of the central processor 1) in a storage location addressable as a function of the packet identifier (TYP) and incoming logical channel number (ILCN) subfields of a control packet together with the incoming route number (IRN) which identifies the line controller handling the control packet (see FIG. 12A). During call setup phase, the call setup table 30 is accessed by the header updating circuit 11 to direct it to the central processor 1 where it is processed for call establishment in a manner as will be described later. The outgoing route table 31 stores a set of outgoing route number (ORN) and outgoing logical channel number (OLCN) in a location addressable as a function of the incoming route number (IRN) and incoming logical channel number (ILCN) subfields of a message packet (FIG. 12B). Header updating circuit 11 accesses the outgoing route table 31 to update the incoming logical channel number (ILCN) of a message packet and append to it the outgoing route number to direct the message packet to an outgoing line controller. Incoming route table 32 stores a set of incoming route number (IRN), incoming logical channel number (ILCN) and the address of a source terminal (PT) into a location addressable as a function of the outgoing route number (ORN) and outgoing logical channel number (OLCN) subfields of either control or message packet (FIG. 12C). During a read mode, the incoming route table 32 is accessed by the controller 13 to read out the stored incoming routing data and source terminal address to provide traffic control.
It is to be noted that there is a one-to-one correspondence between the outgoing and incoming route tables 31 and 32 except for the source terminal address so that a set of incoming route number and incoming logical number can be determined by a set of outgoing route number and outgoing logical channel number, and vice versa. This permits the memory controller 13 to obtain routing information as to the source terminal to impose access restriction at the entry of the network.
Overflow control table 33 stores overflow control data as shown in FIG. 12D which indicates the traffic conditions of outgoing routes. This table is used by incoming-mode line controllers to determine whether the outgoing route for an incoming logical channel number is overflowed or not whenever it receives a packet.
Table 33 is updated under control of the memory controller 13 of the incoming line controller when it is informed of an incoming logical channel number by an outgoing line controller 2 of which the outgoing route is overflowed. This occurs in response to the generation of an overflow signal by the transmit buffer 16 of the outgoing line controller when the storage level of the buffer 16 exceeds a predetermined value.
Specifically, when the transmit buffer 16 of a given outgoing line controller 2 is overflowed, the memory controller 13 of this line controller communicates this fact to the incoming line controller or controllers that extend their incoming logical channels through the overflowed outgoing route.
The determination of the incoming line controller or controllers by the outgoing line controller is accomplished by use of its incoming route table 32 by the memory controller 13. For example, incoming route table 32, FIG. 12C, indicates that an incoming logical channel ILCN=0 is passing through an outgoing route ORN2 on an outgoing logical channel OLCN=1. ILCN=0 is used by the incoming line controller to access the overflow control table 33 to update the storage location of ILCN=0 with a congestion signal CONG. As will be described later, the overflow control table 33 of each incoming line controller is accessed by the header updating circuit 11 as a function of the incoming logical channel number of a packet it receives to control its traffic if it passes through the overflowed outgoing route.
When the traffic congestion is relieved, the transmit buffer 16 of the outgoing line controller notifies this fact to the incoming line controller or controllers to cause them to update their overflow control table 33 accordingly.
Header updating circuit 11 is shown in detail in Fig. 8. This circuit comprises a timing circuit 40 that supplies timing signals to latches 41, 42 and 43 and to a multiplexer 44, and a shift register 45. Incoming packets from data-link layer controller 10 are applied to latch 41 and shift register 45. Latch 45 extracts the logical channel number and packet identifier for coupling to switching control memory 12 in response to a timing pulse from the timing circuit 40. The outgoing route number and outgoing logical channel number from the switching control memory 12 are stored into latches 42 and 43. Multiplexer 44 is properly timed to multiplex the outputs of latches 42 and 43 with the output of shift register 45, whereby the logical channel number subfield of the packet is updated and the outgoing route number subfield is appended to the packet for it to be processed within the switching node.
Central processor 1, shown in FIG. 11, comprises a central processing unit 50 which determines the outgoing logical channel number in accordance with the address of the destination terminal contained in a control packet it receives from an incoming line controller by way of data bus 5, data bus inerface 52 and receive buffer 53. Central prcessing unit 50 determines the outgoing route number of the control packet and appends it to the control packet. A control-packet header updating circuit 51 updates the logical channel number subfield of the control packet with the outgoing logical channel number determined by the CPU 50. The control packet with the LCN subfield updated and an outgoing route number being appended thereto is sent through transmit buffer 54, interface 52 and data bus 5 to the line controller 2 specified by the outgoing route number. A memory 55 serves an aid for the CPU 50 to process control packets and stores the routing information of each packet in response to the establishment of a connection. The routing information is also transferred through a bus interface 56 and control bus 6 to all the line controllers 2 to update their outgoing and incoming route tables 31 and 32. Further included in the central processor 1 is a routing table 57 which is used to determine an outgoing route for leach call or an alternate route when link overflow is encountered on the initial route.
A call setup procedure is initiated upon arrival of a call-establishing control packet of a multi-packet message at the data-link layer controller 10 of an incoming line controller through the transmission link 3. Following the processing of data-link layer control including framing, frame-check-sequence error detection, and RR/RNR detection, the control packet is supplied to the header updating circuit 11, FIG. 8. The logical channel number LCN and the packet identifier TYP of the control packet are sent from latch 41 to the outgoing route table 30 of switching control memory 12. As illustrated in FIG. 12A, the outgoing route is permanently fixed to "0" which causes the control packet to be transferred to the central processor 1. An outgoing logical channel number and an outgoing route number are determined by the central processor in accordance with the address of a sink terminal indicated by the control packet to subsequently transfer the control packet to the desired outgoing route. Central processor 1 proceeds to rewrite the outgoing route table 31 of the incoming line controller 2 with the determined outgoing logical channel number and outgoing route number and rewrite the incoming route table 32 of the outgoing line controller 2 with the incoming route number and incoming logical channel number. Simultaneously, the head updating circuit 51 of central processor 1 updates the logical channel number subfield of the control packet and appends the determined outgoing route number to it as a new subfield forming a packet 112 as shown in FIG. 2 and sends it to the outgoing line controller of the determined route.
After the connection is established, message packets, either data or voice, are sequentially supplied to the data-link layer controller 10. Controller 10 provides the same data-link layer control on message packets as it does on the control packet. Header updating circuit 11 extracts the incoming logical channel number and packet identifier from the message packet it received from controller 10 and utilizes the extracted data to look up the outgoing route table 31 of the switching control memory 12 to read out the outgoing logical channel number (OLCN) and outgoing route number (ORN) which have been established by the preceding control packet. The LCN subfield of the message packet is updated with the LCN data read out of the outgoing route table 31 and the RN data is appended to it as a new subfield by the header updating circuit 11 for routing it to a desired outgoing link, as mentioned previously. All the message packet thus follow the same route as established by the control packet.
When a connection is to be released, a call-clearing control packet is sent from the source terminal and is treated so that the stored outgoing and incoming routing information relating to the connection are erased.
As can be seen from the above that according to the present invention control packets and message packets are are handled respectively by the central processor for call-establishment and call-clearing procedures and by individual line controllers without seeking assistance to the central processor using separate control and data buses. The separation of the switching functions and the separation of the switching network into control and message handling subnetworks result in a packet switching system capable of operating at a significantly high speed and high throughput.
During the call setup and subsequent packet switching phases, the header updating circuit 11 of each line controller accesses the overflow control table 33 as well as to call setup or outgoing route table in order to check for the presence of an overflow traffic on the desired outgoing route.
Referring to FIG. 13, it is assumed that an incoming link identified by route number IRN1 is connected through a line controller 2-1 through data bus to line controllers 2-2 and 2-3 to establish virtual circuits identified by incoming logical channel numbers ILCN=1, ILCN=2, a common incoming route number IRN1, outgoing logical channel numbers OLCN1, OLCN2 and outgoing routes ORN1, ORN2, as illustrated. If the outgoing route ORN2 and hence the transmit buffer 16 of line controller 2-3 is overflowed. Transmit buffer 16 of the outgoing line controller now sends an overflow signal to its memory controller 13. In response to this overflow signal, the memory controller 13 accesses the incoming route table 32 to determine which incoming line controller or controllers the overflow traffic is to be communicated. This is done by accessing the incoming route number IRN1 and incoming logical channel number ILCN=2 of the incoming route table 32 as a function of the outgoing route number ORN2. Memory controller 13 informs the incoming logical channel number ILCN=2 through control bus 6 to the memory controller 13 of incoming line controller 2-1 identified by the incoming route number IRN1 to allow it to update its overflow control table 33 by rewriting the memory location of the incoming logical channel ILCN=2 with a congestion (CONG) signal. As a result, when the line controller 2-1 receives a packet, its header updating circuit 11 accesses the overflow control table 33 as a function of the incoming logical channel number subfield of the packet. If the logical channel number of the packet is ILCN=2, overflow control table 33 supplies a "discard" instruction to the receive buffer 14 (see FIG. 7). Packets destined to the outgoing route ORN2 are therefore denied entry to the receive buffer of line controller 2-1 to prevent it from becoming overflowed with long-waiting packets, allowing packets destined to the outgoing route ORN1 to be passed to the receive buffer of line controller 2-2.
When the overflow condition is relieved, a sequence of events similar to that described above is reinitiated by the transmit buffer of line controller 2-3 with the exception that the line controller 2-1 updates its overflow table 33 with an indication (FREE) that the incoming logical channel ILCN=2 is allowed entry to receive buffer of line controller 2-1.
Assume that switching nodes PS(A) and PS(B) are participating in a connection between source and sink terminals as schematically illustrated in FIG. 14 and the traffic from node PS(A) to node PS(B) is overflowed at the entry to the receive buffer 14 of node PS(B). In the switching node PS(B), an overflow signal is supplied from the receive buffer 14 to the link overflow controller 27, which causes the RR/RNR generator 24 to update the RR/RNR (receive-ready/receive-not-ready) subfield of a packet from the transmit buffer 16 so that it signifies that the node PS(B) is not ready to receive traffic from node PS(A). The updated packet is sent to a line controller 2a of node PS(A) where it is received by the RR/RNR detector 23 of data-link layer controller 10, FIG. 6. Detector 23 provides an output to the link overflow controller 27, which starts monitoring the continuity of such overflow condition and supplies a halt-and-discard signal to the transmit buffer 16 if such a condition continues over a predetermined time interval and sends an overflow signal to the memory controller 13. Thus, the outgoing line controller 2a stops sending packets to the switching node PS(B). If the storage level of transmit buffer 16 of line controller 2a reaches a predetermined value following the cessation of transmission, transmit buffer 16 discards the overflowed packets.
Upon receipt of the overflow signal from the overflow controller 27, the memory controller 13 of the outgoing line controller 2a looks up the incoming route table 32 to read the incoming routing information including a source terminal address as well as incoming route number and incoming logical channel number and sends the retrieved incoming route information as well as the outgoing routing information to the central processor 1.
Central processor 1 rejects the establishment of a call through the overflowed outgoing route now informed by the outgoing line controller 2a when it receives a call-establishing control packet that is destined to the overflowed route. At the same time, the central processor 1 uses the incoming route number to identify the incoming line controller 2b and proceeds to generate a congestion packet (CONG) and transmits it to the source terminal through the incoming line controller 2b.
Upon receipt of a congestion packet, the source terminal PT(A) stops sending packets. If the congestion is relieved, the central processor of the switching node PS(A) sends a congestion-free (FREE) packet to the source terminal PT(A). If the congestion-free packet is received within a predetermined time interval, the terminal PT(A) reinitiates transmission of packets. However, if the congestion-free packet is not received within the predetermined time interval, the terminal PT(A) sends a control packet requesting the reestablishment of a connection through an alternate transmission link.
FIG. 15 illustrates the contents of the routing table 57 of central processor 1 in respect of a switching node PS(B) in the network of FIG. 1. Switching node PS(B) has primary routes and alternate routes. With resect to the priorty route, the switching node PS(B) has outgoing routes #3, #0, #1 and #2 for terminal destinations PT(A), PT(B), PT(C) and PT(D), respectively. The alternate routes available for switching node PS(B) are route #2 for destinations PT(A) and PT(C) and route #3 for destination PT(D).
In FIG. 1, it is assumed that a connection is established between terminals PT(A) and PT(C) with nodes PS(A), PS(B) and PS(C) participating in the connection. If a congestion occurs in the route #1, the central processor 1 of switching node PS(B) updates the routing table 57 so that a congestion indication is given to the route #1 with the other routes being given a congestion-free indication. Likewise, congestion-free indications are initially given to all the routes available for alternate routing. When the re-routing request is received, the central processor of the switching node PS(B) accesses the routing table 57 to find an alternate route. Since the destination PT(C) is involved in the congestion, the route #2 to switching node PS(D) is selected to reestablish a connection between the node PS(B) and terminal PT(C). To provide the re-routing operation, the terminal sends a call-disconnect request packet to the switching node PS(B) to clear the route #1. Immediately following the disconnection, the central processor sends a call-establishing control packet to the switching node PS(D) over the alternate route #2.
As described previously, the present invention provides a high speed, high throughput packet-switched communications system by end-to-end flow control and packet retransmission by eliminating the prior art link-by-link method of flow control and packet transmission.
FIGS. 16 and 17 are illustrations of end-to-end packet transfer sequences according to the present invention.
FIG. 16 illustrates packet transfer sequences in which packets D#0 through D#14 are generated sequentially by a terminal and an FCS error occurs in packet D#4 during transmission between nodes PS(A) and PS(B) as in FIG. 1. When an FCS error in packet D#4 is detected by a node PS(B), it is discarded. When destination terminal PT(B) receives a packet D#5, it recognizes that packet D#4 contains an error and sends a retransmission request REJ#4 to the source terminal PT(A) by writing a binary "0" into the RT/NRT field. Retransmission begins with packet D#4 and ends with packet D#11. Retransmission may be initiated upon the receipt of packet REJ#n or in response to a time out oepration. For example, if a packet D#0 is received but no acknowledgment is received within a preset period of time from the node PT(B), node PS(A) retransmits packet D#0.
FIG. 17 is an illustration of sequences in which the RT/RNT subfield of the received packet is "1" indicating that retransmission of REJ packets is disabled. It is assumed that the receive buffer of destination node PS(B) is congested when it receives packet D#3 and node PS(B) returns an RNR frame to source node PS(A). FIG. 17 shows that the source node PS(A) forcibly discards packets D#4 and D#5. Upon the termination of the overflow of receive buffer of destination node PS(B), an RR frame is returned to the source node to allow retransmission of packets D#6 and D#7. The destination terminal PT(B) detects a sequence error in packets D#4 and D#5 and an acknowledgment ACK#7 is returned. Thus, the retransmission of packets D#6 through D#10 does not results in an increase in the propagation delay.
As described above, the present invention eliminates the need for acknowledgment and retransmission on a link-by-link basis with respect to message packets and causes the source and destination terminals to perform compensation for packets lost in the network due to overflows or bit errors, whereby the packet switching operation performed by each node and hence the processing time is significantly reduced and the throughput of a node is increased.
The traffic control at an entry point of the network effectively avoids a networkwide traffic congestion.
The foregoing description shows only preferred embodiments of the present invention. Various modifications are apparent to those skilled in the art without departing from the scope of the present invention which is only limited by the appended claims. Therefore, the embodiments shown and described are only illustrative, not restrictive.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4484326 *||Nov 4, 1982||Nov 20, 1984||At&T Bell Laboratories||Packet load monitoring by trunk controllers|
|US4621359 *||Oct 18, 1984||Nov 4, 1986||Hughes Aircraft Company||Load balancing for packet switching nodes|
|US4630259 *||Nov 14, 1984||Dec 16, 1986||At&T Bell Laboratories||Lockup detection and recovery in a packet switching network|
|1||*||Puzin, Methods Tools & Observations on Flow Control In Pkt. Switched Networks, IEEE, vol. COM 29, 4, Apr. 1981.|
|2||Puzin, Methods Tools & Observations on Flow Control In Pkt. Switched Networks, IEEE, vol. COM-29, #4, Apr. 1981.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US4912703 *||Sep 29, 1988||Mar 27, 1990||Kabushiki Kaisha Toshiba||Apparatus and method for data flow control in broadcast data transmission system|
|US4937817 *||Dec 29, 1988||Jun 26, 1990||American Telephone And Telegraph Company||Packet selection for packet distribution arrangements|
|US4956839 *||Jul 20, 1989||Sep 11, 1990||Hitachi, Ltd.||ATM switching system|
|US4958278 *||Apr 8, 1988||Sep 18, 1990||International Business Machines Corporation||Method for loading data or program to a plurality of terminal stations|
|US4964119 *||Apr 6, 1989||Oct 16, 1990||Hitachi, Ltd.||Method and system for packet exchange|
|US4965793 *||Feb 3, 1989||Oct 23, 1990||Digital Equipment Corporation||Method and apparatus for interfacing a system control unit for a multi-processor|
|US4995032 *||Aug 22, 1989||Feb 19, 1991||Cselt - Centro Studi E Laboratori Telecomunicazioni S.P.A.||Label-switching and control interface for asynchronous fast-packet switching|
|US5007051 *||Mar 3, 1989||Apr 9, 1991||Hewlett-Packard Company||Link layer protocol and apparatus for data communication|
|US5010546 *||Jun 15, 1989||Apr 23, 1991||Fujitsu Limited||Packet switching system|
|US5042029 *||Feb 21, 1989||Aug 20, 1991||Nec Corporation||Congestion control method and apparatus for end-to-end packet communication|
|US5090011 *||Dec 21, 1989||Feb 18, 1992||Hitachi, Ltd.||Packet congestion control method and packet switching equipment|
|US5119367 *||Oct 25, 1989||Jun 2, 1992||Oki Electric Industry Co., Ltd.||Method and a node circuit for routing bursty data|
|US5146564 *||Feb 3, 1989||Sep 8, 1992||Digital Equipment Corporation||Interface between a system control unit and a service processing unit of a digital computer|
|US5151899 *||Feb 11, 1991||Sep 29, 1992||Digital Equipment Corporation||Tracking sequence numbers in packet data communication system|
|US5153876 *||Dec 13, 1989||Oct 6, 1992||Digital Equipment Corporation, Pty. Limited, Inc.||Communication protocol for statistical data multiplexers arranged in a wide area network|
|US5168497 *||Sep 19, 1989||Dec 1, 1992||Hitachi, Ltd.||Packet communication processing method|
|US5265092 *||Mar 18, 1992||Nov 23, 1993||Digital Equipment Corporation||Synchronization mechanism for link state packet routing|
|US5313458 *||Jun 2, 1992||May 17, 1994||Fujitsu Limited||Traffic control system|
|US5361388 *||Apr 7, 1992||Nov 1, 1994||Nec Corporation||Message relaying system for a distributed processing system|
|US5377327 *||Jan 21, 1994||Dec 27, 1994||Digital Equipment Corporation||Congestion avoidance scheme for computer networks|
|US5442624 *||Oct 29, 1993||Aug 15, 1995||At&T Corp.||Dynamic access control for an ATM network|
|US5444706 *||Dec 26, 1990||Aug 22, 1995||Kabushiki Kaisha Toshiba||Packet exchange network and method of controlling discard of packets in packet exchange network|
|US5481735 *||Dec 28, 1992||Jan 2, 1996||Apple Computer, Inc.||Method for modifying packets that meet a particular criteria as the packets pass between two layers in a network|
|US5491801 *||Aug 23, 1994||Feb 13, 1996||Digital Equipment Corporation||System for avoiding network congestion by distributing router resources equally among users and forwarding a flag to those users utilize more than their fair share|
|US5524250 *||Dec 23, 1994||Jun 4, 1996||Silicon Graphics, Inc.||Central processing unit for processing a plurality of threads using dedicated general purpose registers and masque register for providing access to the registers|
|US5546600 *||Dec 14, 1992||Aug 13, 1996||Sharp Kabushiki Kaisha||Data driven computer producing inhibit signal for inhibiting merging external provided data pieces with internal data pieces when number of processing data exceeds reference value|
|US5560036 *||May 1, 1995||Sep 24, 1996||Mitsubishi Denki Kabushiki Kaisha||Data processing having incircuit emulation function|
|US5668951 *||Nov 20, 1995||Sep 16, 1997||Digital Equipment Corporation||Avoiding congestion system for reducing traffic load on selected end systems which utilizing above their allocated fair shares to optimize throughput at intermediate node|
|US5675742 *||Jun 26, 1995||Oct 7, 1997||Digital Equipment Corporation||System for setting congestion avoidance flag at intermediate node to reduce rates of transmission on selected end systems which utilizing above their allocated fair shares|
|US5737626 *||Mar 14, 1996||Apr 7, 1998||Allen-Bradley Company, Inc.||Deterministic communication network for industrial control|
|US5774877 *||Sep 20, 1994||Jun 30, 1998||Papyrus Technology Corp.||Two-way wireless system for financial industry transactions|
|US5793301 *||Dec 10, 1996||Aug 11, 1998||Paryrus Technology Corp.||Assured two-way wireless communication system for financial industry transactions|
|US5797002 *||Jun 7, 1995||Aug 18, 1998||Papyrus Technology Corp.||Two-way wireless system for financial industry transactions|
|US5905714 *||Feb 28, 1995||May 18, 1999||Nokia Telecommunications Oy||Method for rerouting a packet-mode data connection|
|US5915245 *||Feb 23, 1998||Jun 22, 1999||Papyrus Technology Corp.||Two-way wireless system for financial industry transactions|
|US5982743 *||Feb 18, 1997||Nov 9, 1999||Nec Corporation||Performance monitor system, transmission device and performance monitor method of transmission line|
|US6021124 *||Aug 19, 1997||Feb 1, 2000||Telefonaktiebolaget Lm Ericsson||Multi-channel automatic retransmission query (ARQ) method|
|US6067408 *||Feb 22, 1996||May 23, 2000||Advanced Micro Devices, Inc.||Full duplex buffer management and apparatus|
|US6130881 *||Apr 20, 1998||Oct 10, 2000||Sarnoff Corporation||Traffic routing in small wireless data networks|
|US6170022 *||Apr 3, 1998||Jan 2, 2001||International Business Machines Corporation||Method and system for monitoring and controlling data flow in a network congestion state by changing each calculated pause time by a random amount|
|US6198722 *||Feb 27, 1998||Mar 6, 2001||National Semiconductor Corp.||Flow control method for networks|
|US6243358 *||Feb 4, 1998||Jun 5, 2001||France Telecom||Process and device for allocating resources in a packet transmission digital network|
|US6247058 *||Mar 30, 1998||Jun 12, 2001||Hewlett-Packard Company||Method and apparatus for processing network packets using time stamps|
|US6337852||Dec 29, 1997||Jan 8, 2002||International Business Machines Corporation||Flow control system using control information of a message for initiating retransmission of data portion when buffer is available|
|US6338091 *||Dec 29, 1997||Jan 8, 2002||International Business Machines Corporation||System for optimistic transmission flow control including receiver data discards upon inadequate buffering condition|
|US6434114 *||Feb 26, 1998||Aug 13, 2002||At&T Corp||Enhancing the intelligent service capability of a network after a data flow has been established|
|US6452934 *||Sep 14, 1999||Sep 17, 2002||Hitachi, Ltd.||Packet forwarding apparatus|
|US6480897||Dec 29, 1997||Nov 12, 2002||International Business Machines Corporation||Optimistic transmission flow control including receiver data discards upon inadequate buffering condition|
|US6539362||Sep 22, 2000||Mar 25, 2003||Papyrus Technology Corp.||Two-way wireless system for financial industry transactions|
|US6549538||Dec 31, 1998||Apr 15, 2003||Compaq Information Technologies Group, L.P.||Computer method and apparatus for managing network ports cluster-wide using a lookaside list|
|US6570850 *||Apr 23, 1998||May 27, 2003||Giganet, Inc.||System and method for regulating message flow in a digital data network|
|US6665304||Dec 31, 1998||Dec 16, 2003||Hewlett-Packard Development Company, L.P.||Method and apparatus for providing an integrated cluster alias address|
|US6671273 *||Dec 31, 1998||Dec 30, 2003||Compaq Information Technologies Group L.P.||Method for using outgoing TCP/IP sequence number fields to provide a desired cluster node|
|US6768981||Aug 1, 2002||Jul 27, 2004||Papyrus Corporation||Method for executing a cross-trade in a two-way wireless system|
|US6791592||Jun 4, 2003||Sep 14, 2004||Laserink||Printing a code on a product|
|US6829000||Nov 13, 2003||Dec 7, 2004||Laserink||Printing a code on a product|
|US6938091 *||Mar 8, 2002||Aug 30, 2005||Hewlett-Packard Development Company, L.P.||Static end to end retransmit apparatus and method|
|US7046267||Dec 19, 2003||May 16, 2006||Markem Corporation||Striping and clipping correction|
|US7167194||Nov 14, 2003||Jan 23, 2007||Laserink||Printing a code on a product|
|US7209448 *||Feb 20, 2002||Apr 24, 2007||Juniper Networks, Inc.||In-line packet processing|
|US7228538 *||May 2, 2000||Jun 5, 2007||International Business Machines Corporation||Method, system, and program for updating firmware to a storage system comprised of multiple controllers|
|US7283471 *||Mar 11, 2003||Oct 16, 2007||Emulex Design & Manufacturing Corporation||System and method for regulating message flow in a digital data network|
|US7373322||May 21, 2004||May 13, 2008||Papyrus Technology Corporation||Two-way wireless communication system for financial industry transactions|
|US7394479||Mar 2, 2005||Jul 1, 2008||Marken Corporation||Pulsed laser printing|
|US7630304 *||Dec 8, 2009||Hewlett-Packard Development Company, L.P.||Method of overflow recovery of I2C packets on an I2C router|
|US7630311 *||Dec 8, 2009||Electronics And Telecommunications Research Institute||Location management server and ethernet-based wireless LAN distribution system having local management server, and embodiment method thereof|
|US7733868 *||Jan 26, 2006||Jun 8, 2010||Internet Broadcasting Corp.||Layered multicast and fair bandwidth allocation and packet prioritization|
|US7760647 *||Dec 21, 2006||Jul 20, 2010||Cisco Technology, Inc.||Out of band flow control|
|US7801140||Sep 21, 2010||Juniper Networks, Inc.||In-line packet processing|
|US7804833||Sep 28, 2010||Juniper Networks, Inc.||In-line packet processing|
|US7898946 *||Nov 24, 2003||Mar 1, 2011||Samsung Electronics Co., Ltd.||Communication system and method capable of improving data transmission efficiency of TCP in asymmetric network environments|
|US8036122 *||Apr 3, 2003||Oct 11, 2011||Alcatel Lucent||Initiation of network treatment for data packet associated with real-time application different from network treatment applicable to data packet non-associated with the real-time application|
|US8077724||Dec 13, 2011||Juniper Networks, Inc.||In-line packet processing|
|US8320239 *||Nov 27, 2012||Nokia Siemens Networks Oy||Receiver-driven layered multicast rate control protocol|
|US8514718||Jun 17, 2009||Aug 20, 2013||Blitz Stream Video, Llc||Layered multicast and fair bandwidth allocation and packet prioritization|
|US8750320||Jan 21, 2003||Jun 10, 2014||Broadcom Corporation||Fibre channel arbitrated loop bufferless switch circuitry to increase bandwidth without significant increase in cost|
|US8767756||Nov 19, 2008||Jul 1, 2014||Broadcom Corporation||Fibre channel arbitrated loop bufferless switch circuitry to increase bandwidth without significant increase in cost|
|US8774005 *||Jul 24, 2009||Jul 8, 2014||Telefonaktiebolaget L M Ericsson (Publ)||Congestion control method and devices|
|US8774199||Jan 21, 2003||Jul 8, 2014||Broadcom Corporation||Fibre channel arbitrated loop bufferless switch circuitry to increase bandwidth without significant increase in cost|
|US8798091||Apr 30, 2008||Aug 5, 2014||Broadcom Corporation|
|US8867543||Nov 11, 2011||Oct 21, 2014||Juniper Networks, Inc.||In-line packet processing|
|US8958426||Sep 14, 2012||Feb 17, 2015||Blitz Stream Video, Llc||Layered multicast and fair bandwidth allocation and packet prioritization|
|US20020163916 *||Feb 20, 2002||Nov 7, 2002||Oskouy Rasoul Mirzazadeh||In-line packet processing|
|US20030108061 *||Jan 21, 2003||Jun 12, 2003||Black Alistair D.|
|US20030169753 *||Jan 21, 2003||Sep 11, 2003||Black Alistair D.|
|US20030172181 *||Mar 8, 2002||Sep 11, 2003||Sharma Debendra Das||Static end to end retransmit apparatus and method|
|US20030174720 *||Jan 21, 2003||Sep 18, 2003||Black Alistair D.|
|US20030206227 *||Jun 4, 2003||Nov 6, 2003||Laserink, A California Corporation||Printing a code on a product|
|US20040073683 *||Oct 2, 2003||Apr 15, 2004||Beck Paul R.||Method and apparatus for providing an integrated cluster alias address|
|US20040105449 *||Apr 22, 2003||Jun 3, 2004||Hee-Young Jung||Location management server and ethernet-based wireless LAN distribution system having local management server, and embodiment method thereof|
|US20040109477 *||Nov 24, 2003||Jun 10, 2004||Samsung Electronics Co., Ltd.||Communication system and method capable of improving data transmission efficiency of TCP in asymmetric network environments|
|US20040141052 *||Nov 13, 2003||Jul 22, 2004||Laserink, A California Corporation||Printing a code on a product|
|US20040196786 *||Apr 3, 2003||Oct 7, 2004||Subhasis Laha||Initiation of network treatment for data packet associated with real-time application different from network treatment applicable to data packet non-associated with the real-time application|
|US20040252642 *||Jun 12, 2003||Dec 16, 2004||Larson Thane M.||Method of overflow recovery of I2C packets on an I2C router|
|US20050088510 *||Oct 24, 2003||Apr 28, 2005||Shlomo Assa||Low angle optics and reversed optics|
|US20050134678 *||Dec 19, 2003||Jun 23, 2005||Kevin Franklin||Striping and clipping correction|
|US20050210324 *||Sep 16, 2004||Sep 22, 2005||Hideji Abe||Data relay apparatus, data relay method, and computer product|
|US20050256899 *||Nov 18, 2004||Nov 17, 2005||Bea Systems, Inc.||System and method for representing hierarchical data structures|
|US20060268871 *||Jan 26, 2006||Nov 30, 2006||Erik Van Zijst||Layered multicast and fair bandwidth allocation and packet prioritization|
|US20070147257 *||Mar 5, 2007||Jun 28, 2007||Juniper Networks, Inc.||In-line packet processing|
|US20080031252 *||Aug 20, 2007||Feb 7, 2008||Oskouy Rasoul M||In-line packet processing|
|US20080151919 *||Dec 21, 2006||Jun 26, 2008||Cisco Technology, Inc.||Out of band flow control|
|US20080205421 *||Apr 30, 2008||Aug 28, 2008||Broadcom Corporation|
|US20090074408 *||Nov 19, 2008||Mar 19, 2009||Broadcom Corporation|
|US20090257448 *||Jun 17, 2009||Oct 15, 2009||Internet Broadcasting Corporation||Layered multicast and fair bandwidth allocation and packet prioritization|
|US20090296708 *||Dec 3, 2009||Internet Broadcasting Corporation||Layered multicast and fair bandwidth allocation and packet prioritization|
|US20090303997 *||Dec 10, 2009||Internet Broadcasting Corporation||Layered multicast and fair bandwidth allocation and packet prioritization|
|US20100309916 *||Aug 13, 2010||Dec 9, 2010||Juniper Networks, Inc.||In-line packet processing|
|US20110255410 *||Jul 24, 2009||Oct 20, 2011||Soner Yamen||Congestion control method and devices|
|EP0669778A2 *||Dec 21, 1994||Aug 30, 1995||Advanced Micro Devices Inc.||Fast-packet switching networks|
|WO1999054830A1 *||Apr 23, 1999||Oct 28, 1999||Giganet, Inc.||System and method for regulating message flow in a digital data network|
|WO1999055031A1 *||Apr 20, 1999||Oct 28, 1999||Sarnoff Corporation||Traffic routing in small wireless data networks|
|WO2006081454A2 *||Jan 26, 2006||Aug 3, 2006||Internet Broadcasting Corporation||Layered multicast and fair bandwidth allocation and packet prioritization|
|WO2006081454A3 *||Jan 26, 2006||Oct 30, 2008||Internet Broadcasting Corp||Layered multicast and fair bandwidth allocation and packet prioritization|
|U.S. Classification||370/227, 370/236, 370/237, 370/397, 370/231, 370/469|
|International Classification||H04L12/835, H04L12/825, H04L12/801|
|Cooperative Classification||H04L47/30, H04L47/10, H04L47/18, H04L47/263, H04L47/16|
|European Classification||H04L47/26A, H04L47/30, H04L47/10, H04L47/18, H04L47/16|
|Dec 9, 1986||AS||Assignment|
Owner name: NEC CORPORATION, 33-1, SHIBA 5-CHOME, MINATO-KU, T
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST.;ASSIGNOR:SUZUKI, HIROSHI;REEL/FRAME:004641/0668
Effective date: 19860710
Owner name: NEC CORPORATION, JAPAN
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SUZUKI, HIROSHI;REEL/FRAME:004641/0668
Effective date: 19860710
|Jun 23, 1992||FPAY||Fee payment|
Year of fee payment: 4
|Jun 28, 1996||FPAY||Fee payment|
Year of fee payment: 8
|Jul 10, 2000||FPAY||Fee payment|
Year of fee payment: 12