|Publication number||US7697519 B2|
|Application number||US 11/590,714|
|Publication date||Apr 13, 2010|
|Filing date||Oct 31, 2006|
|Priority date||Oct 31, 2006|
|Also published as||US20080101354|
|Publication number||11590714, 590714, US 7697519 B2, US 7697519B2, US-B2-7697519, US7697519 B2, US7697519B2|
|Inventors||Manfred R. Arndt|
|Original Assignee||Hewlett-Packard Development Company, L.P.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (11), Referenced by (8), Classifications (10), Legal Events (3)|
|External Links: USPTO, USPTO Assignment, Espacenet|
Computing systems can include multiple computing devices such as servers, desktop PCs, laptops, and workstations, among other peripheral devices, (e.g., printers, facsimile devices, and scanners). In some systems, these network devices can be networked together across a local area network (LAN) and/or wide area network (WAN) to form a computing device network. A LAN and/or WAN uses clients and servers that have network-enabled operating systems such as Windows, Mac, Linux, and Unix.
Data communication between the different devices of the computing device network can include voice, video, still images, and data traffic. All have widely varying needs in terms of propagation delay (or latency) during transit through the network. Various systems and devices, both in hardware and in software, have attempted to deal with the large data flow requirements present in computing device networks.
One such scheme consists of attempting to allocate resources and police the traffic within the router or switch connecting multiple networks in the typical computing device network at either the data link or network function levels. Such schemes attempt to provide fair allocation of data throughput capacity (bandwidth) by allocating packet buffer and/or queue space according to the type of packets and arrival rate in each flow stream received.
A particular problem in network traffic policing arises from the variety of traffic sources or flows presented to the router/switching device. These flows each consist of multiple packets of data, in a variety of sizes and presented at a variety of rates. Additionally, flows may be presented using different networking protocols, such as the Transmission Control Protocol/Internet Protocol (TCP/IP) and the related User Datagram Protocol (UDP) over which application protocols, such as File Transfer Protocol (FTP), Terminal Emulation Protocol (Telnet), Session Initiation Protocol (SIP), Real Time Protocol (RTP), and Hypertext Transfer Protocol (HTTP) are layered on top of.
Embodiments of the present disclosure provide for packet processing on a network device. One method embodiment includes parsing incoming packets in a number of received packet streams to extract information from various packet fields and associate them into unique flows. A number of bandwidth meters, e.g., hardware values, are used to track application bandwidth usage over a particular time interval in association with each flow. The bandwidth meters are referenced with a hash algorithm. Using the bandwidth meters, and a number of additional factors, a drop probability for a particular packet is adjusted. In various embodiments, the adjusted drop probability factor is compared to a random number. When the random number is less than the adjusted drop probability factor, the packet is dropped. When the random number is not less than the adjusted drop probability factor, the packet is placed in an appropriate queue.
The embodiments provide a resource friendly implementation to automatically rate limit high-bandwidth applications to avoid running out of packet buffers during congestion in a manner that is well suited for high performance hardware logic implementations within networking equipment, e.g., devices, with a limited number of hardware based priority queues and very high speed packet buffer memory used to support full line rate performance across all ports at gigabit and 10 GbE speeds. Embodiments can improve responsiveness for interactive applications, improve fairness between applications, and may provide reduced packet loss for low bandwidth applications like voice over IP (VoIP), without the involvement of complex and error-prone user interactions.
Although the terms router and/or switch will be used in this disclosure, those skilled in the art will realize that other related devices may be used, in addition to routers or switches, to perform analogous functions. Accordingly, the disclosure is not limited to any particular type of device, router, or switch. Also, although the disclosure discusses Internet Protocol (IP) packet flows, those skilled in the will art realize that protocols and flows other than IP, such as Ethernet, can be benefit from the present disclosure and its various embodiments. Accordingly, the disclosure is not limited to a particular type of protocol or packet format.
The example network of
The designators “N” and “M” are used to indicate that a number of fat or thin clients can be attached to the network 100. The number that N represents can be the same or different from the number represented by M. The embodiment of
As one of ordinary skill in the art will appreciate, many of these devices include processor and memory hardware. By way of example and not by way of limitation, the network management station 112 will include a processor and memory as the same are well known to one of ordinary skill in the art. Similarly, the network devices of routers, 116-1, 116-2, 116-3, and 116-4, and hubs and/or switches 118-1, 118-2, 118-3, 118-4, and 118-5 may include processor and memory resources, along with hardware logic or application specific integrated circuits (ASICs). Embodiments of the invention are not limited, for the various devices in the network, to the number, type, or size of processor and memory resources.
Program instructions (e.g., computer executable instructions), as described in more detail below, can reside on the various network devices. For example, program instructions in the form of firmware, software, etc., can be resident on the network 100 in the memory of a network management station 112 and/or one or more routers, 116-1, 116-2, 116-3, and 116-4, and hubs and/or switches 118-1, 118-2, 118-3, 118-4, and 118-5, and be executable by the processor(s) thereon. As the reader will appreciate, program instructions can be resident in a number of locations on various network devices in the network 100 as employed in a distributed computing network.
As one of ordinary skill in the art will appreciate, each network device in the network is physically associated with a port of a switch to which it is connected. Data frames, or packets, are transferred between network devices by means of a switch's logic link control (LLC)/media access control (MAC) circuitry, or “engines”, associated with each switch port. The network switch forwards data frames received from a transmitting network device to a destination network device based on the header information in the received data frame. The switch/router can also forward packets from a given network to other networks through one or more pre-configured ports.
As the packets arrive they are buffered in a buffer pool 234, which may be very high speed memory with the network chips or external random access memory (RAM). Buffering is accomplished according to the directives of a controller 236, coupled to memory 237, and a buffer manager 238. The controller 236 and the buffer manager 238 can process packets, used in the network device's operation, which are received by network chips on the device 230.
The flows are sent to the proper output port 240 by way of a set of output queues 242 and a port scheduler 244, discussed herein. Controller 236, buffer manager 238, and port scheduler 244 can be implemented as one or more processors with associated interface circuitry and logic or can be implemented within a single ASIC to achieve the functions of the embodiments described herein.
Some flows through the network are well-behaved in the event of traffic congestion: when faced with packet drops (i.e., packets discarded deliberately by a networking device due to congestion control algorithm or due to insufficient resources at that device), these “good” (robust) flows reduce their flow rates and send less packets per unit of time, until they detect that the congestion goes away. Other flows, however, are not well-behaved. These non-adaptive “aggressive” flows (NAFs) do not throttle back the flow of packets to the network when they experience packet drops. This may be because the NAFs do not recognize the congestion, sometimes due to the protocol implementation, or sometimes because they actually are trying to capture more network bandwidth. The latter situation arises particularly in flows sent by sources that consider themselves higher priority than all others (hence the term “aggressive”); such priority assumptions by one flow are often in conflict in the modern, highly heterogeneous networks seen today.
Ethernet networks often do not run at a high average utilization, but do experience microburst of congestion. This may be compounded by link speed mismatches that can overwhelm limited buffer pools on switches and routers, e.g., device 230. When multiple applications share the same egress priority queue, a single non responsive or aggressive application may monopolize the buffer queue 242 and negatively impact all well-behaved applications sharing the same priority queue due to packet loss. Most modern applications can tolerate a limited amount of packet loss, but if packets are lost in burst, the impact may be significant. For TCP applications, this can result in slow TCP retransmission timeouts that have a big impact on aggregate throughput or in some cases can even result in the TCP connection to time out and the application fail. Avoiding intervals where a TCP connection does not get any link capacity for extended intervals, e.g., 100 milliseconds, can significantly improve performance. For interactive business applications, bursts of packet loss may result in slow application responsiveness and for voice over IP (VoIP) applications may result in poor voice quality.
When the aforementioned congestion situation occurs regularly, network operators often rely on manual interactions to identify the offending applications and configure the appropriate quality of service (QoS) policy and access control lists (ACLs). ACLs filter network traffic by controlling whether packets are forwarded or blocked at an interface or port. The network device examines each packet to determine whether to forward or drop the packet, based on the criteria specified within the access control lists. Access control list criteria could be the source address of the traffic, the destination address of the traffic, the upper-layer protocol, or other information. The disadvantage to doing this, however, is that it is time consuming, complex, and error prone. In addition, this approach is only viable for sophisticated users and may not be practical when the number and types of applications in use changes frequently.
Advanced QoS implementations on high end core routers and load balancing network devices address the above issues by using a large number of individual per-flow buffer queues to insure fairness between applications and prevent a handful of applications from consuming all the resources. This approach works, but is very resource intensive and challenging to be implemented in ASIC based hardware.
Previous policing schemes include both queue-based and buffer-based approaches. QoS policy and ACLs are an example of a queue-based scheme in which incoming flows are classified according to their actual priority, as determined by the receiving network device, and assigned accordingly to output queues within the network device. High priority flows, such as time-sensitive voice traffic, are placed in a queue that is service much more often. Low priority flows, such as file transfer protocol (FTP) or hypertext transfer protocol (HTTP) flows, are placed in queues that are serviced at a slower rate. Per-flow queuing is an example of a queue-based scheme which assigns one queue per input flow and typically limits the number of buffers that any flow can consume at a given time. Queues are service according to statistically fair scheduling process, such as round-robin scheduling. In round-robin scheduling, one packet is serviced out of each queue, one queue at a time, servicing again from the first queue only when one packet has been serviced from every other queue with packets buffered. Per-flow queuing operates well and insures fairness between applications when the number and variety of input flows is small, but becomes inefficient as the number of flows grows. A high number of flows require a large number of queues, consuming a proportionally larger amount of resources, both in hardware and in operational complexity. More memory and more processing overhead is required to set up and tear down the associated resources when flows begin and end. In the context of the modern networks seen today, this extra cost and complexity is undesirably inefficient.
A number of network switches and routers try to improve application fairness by using congestion avoidance algorithms, like RED (Random Early Detection) and WRED (Weighted Random Early Detection). By dropping some packets early rather than waiting until the buffer is full, RED tries to get the TCP applications to slow down their transmit rate, since TCP will reduce the advertised window size in response to packet loss and their associated retransmissions. RED and WRED can improve fairness, but are only useful with well behaved applications and provides no benefit for non-responsive applications like streaming video. That is, many certain NAFs do not reduce their packet rate and thus continue to take a disproportionate amount of network bandwidth, simply because they do not respond to packet drops. The “good” flows get less and less throughput as they reduce their transmit rate in response to drops while the NAFs capture more bandwidth. In addition, careful tuning of several RED or WRED parameters is needed based on the number and type of applications sharing a link, otherwise overall aggregate performance can actually be worse.
As a further drawback, the random packet drops sometimes hit a fragile flow. These flows contain time-critical or loss sensitive traffic, such as voice data. Fragile flows have the lowest tolerance for drops and delay, so even a low number of packet drops from a random packet drop management scheme can have a highly undesirable effect on them.
To overcome the issues described above, embodiments of the present disclosure provide additional logic to the buffer manager 238 and port scheduler 244 of the network device 230. This logic executes to process packets using a number of bandwidth meters to adjust a drop probability based on a combination of several factors. As will be described in more detail in connection with
According to various embodiments, the number of bandwidth meters can be a dedicated per port resource or shared across multiple ports. Each bandwidth meter tracks application bandwidth usage over a particular time interval in association with each flow, e.g., count packets or bytes over a time interval in association with a particular application or group of applications. Hence, a given bandwidth meter can be shared across multiple applications identified as a particular flow, based on a hash algorithm applied to information extracted from header fields parsed from incoming packets. The values associated with each of the bandwidth meters can be stored in a table of bandwidth meters. The bandwidth meters are referenced using the flow id from the hash algorithm.
In one example embodiment, an n-tupple hash algorithm, as the same will be recognized by one of skill in the art, is used to reference the bandwidth meters. For example, an n-tupple hash algorithm may reference a particular bandwidth meter from a table of bandwidth meters by computing a hash index value based on a combination of information extracted from the source IP address field 320, the destination IP address field 340, the application source port field 350, and the destination port field 360. Embodiments are not limited to this particular example and other combinations may also be used. Thus, as the reader will appreciate, multiple applications may share a hash index value, e.g., hash to the same value. However, by using a sufficiently large number of bandwidth meters, a reasonable distribution of hash index values will be achieved.
The controller and the buffer manager can process packets, used in the network device's operation, which are received to the device. As described above, a number of bandwidth meters are associated with the received flows. In various embodiments, additional logic is provided to the buffer manager which can allocate the bandwidth meters in a dedicated fashion, e.g., as a per port resource, or as a shared resource across multiple ports.
According to embodiments, these bandwidth meters track, e.g., count, application bandwidth usage over a particular time interval in association with each flow 400, e.g., count packets or bytes in association with a particular application or group of applications. Hence, a given bandwidth meter can be shared across multiple applications within a particular flow. Logic can increment a particular bandwidth meter as a packet 401 associated with the meter is added to a particular queue and logic can decrement the bandwidth meter based on the time interval since the last meter update. The total time over which to track bandwidth usage should be fairly short, e.g., on the order of 100 milliseconds or so. Embodiments, however, are not limited to the total time over which to track bandwidth usage given in this example.
The values associated with each of the number of bandwidth meters can be stored and referenced in a table of bandwidth meters 404. The example embodiment of
As shown in
As shown at 406, the parsed information is used by a hash algorithm to reference the bandwidth meter values as stored in the table of bandwidth meters 404. As one example embodiment, an n-tupple hash algorithm is used to compute a hash index value from the parsed packet header information to generate a hash index value to reference the table of bandwidth meters 404. The flow identifying information contained in the packet header data is hashed in order to reduce the range of packet header values into a single compact, easily manipulated field having a far smaller range of values. Various methods can be used to combine the range of packet header values and create the hash index value. Examples include, but are not limited to, applying exclusive OR (XOR) logic and sixteen bit cyclical redundancy checking (CRC16) logic.
One of ordinary skill in the art will appreciate that such hashing may be accomplished with hardware or software means or a combination thereof. Additionally, although an IP packet, e.g., 401, is described in connection with
In the n-tupple hash algorithm example above, a particular bandwidth meter from the table of bandwidth meters 404 is referenced by computing a hash index value (flow id) based on a combination of information extracted from the source IP address field, the destination IP address field, the application source port field, and the destination port field. Embodiments are not limited to this particular example and other combinations may also be used. Multiple applications in one or more packet streams may hash to the same flow id value. However, by using a sufficiently large number of bandwidth meters, e.g., 1920 in this example, a reasonable distribution of hash index values will be achieved.
As shown at 408, logic operates on the parsed information to classify a packet, e.g., 401, as belonging to a particular QoS priority, based on predefined QoS policies. In the example embodiment of
As describe above, logic is provided to process packets using the values associated with the number of bandwidth meters to adjust a drop probability based on a combination of a number of factors. In the example embodiment shown in
In the example embodiment of
Hence according to embodiments, logic can operate such that when a low bandwidth flow is detected a drop probability for packets in that flow is reduced. A single or multiple thresholds can be used. In various embodiments, reducing the drop probability will lessen the likelihood of a random number comparison producing a random number which is less than the drop probability and hence packets associated with the particular flow will be more likely to be buffered and placed on an appropriate queue based on other existing quality of service (QoS) policies. Hence, low bandwidth flows, e.g., containing applications like VoIP, will be protected and experience less packet loss. According to certain example embodiments, such as shown in
As shown in the example embodiment of
In various embodiments the logic can operate so as to never drop a packet when a destination priority queue size is below an established, e.g., predetermined, queue size threshold. Hence, embodiments will operate to prevent the starvation of ports with few packets queued, even though the available buffer pool may be low due to other ports and flows consuming lots of packet buffers.
As mentioned above, additional factors combined by logic to adjust a drop probability logic using the values associated with the number of bandwidth meters include the factor of remaining available buffer pool. Here logic will operate such that as the available buffer pool (e.g., as shared across multiple ports) decreases, the drop probability factor increases. Again, a given bandwidth meter can be used to increment a value as packets or bytes associated with one or more applications identified as a particular flow enter the buffer pool and to decrement based on the time interval since the last meter update. Increasing the drop probability will increase the likelihood of a random number comparison producing value which is less than the drop probability and hence packets associated with flows having high bandwidth usage, e.g., according to a particular threshold or multiple thresholds, will be more likely to be discarded.
In various embodiments the logic can operate so as to never drop a packet when the remaining available buffer pool size is above a particular threshold. Hence, in a shared memory buffer allocation scheme across multiple ports, the logic will operate to support deep queue sizes when only a few ports are busy and prevent global buffer exhaustion when many queues are congested. Additionally, in a share memory buffer allocation scheme across multiple ports, the logic will operate to fairly distribute buffers across multiple ports, e.g., ports with many simultaneous applications will get more buffers than ports with only a few applications and/or users.
According to embodiments, another of the additional factors combined by logic to adjust a drop probability logic using the bandwidth meter value associated to the flow id includes the factor of recent bandwidth usage. Here logic will operate to increase the drop probability for flows with high recent bandwidth usages. Again, a single or multiple thresholds can be used. As described herein, each bandwidth meter tracks the bandwidth usage over a particular time interval associated with one or more applications for each flow, incrementing and decrementing the bandwidth meter value based on the bandwidth usage over a particular period of time. Increasing the drop probability for packets associated with one or more applications for a given flow will increase the likelihood of a random number comparison producing a random number which is less than the drop probability and hence packets associated with one or more applications for a given flow with a bandwidth meter reflecting high recent bandwidth usage, will be more likely to be discarded.
Additionally, according to embodiments, another of the additional factors combined by logic to adjust a drop probability logic using the values associated with the number of bandwidth meters includes the factor of administrator configuration. Here logic can operate such that the various queue size thresholds, bandwidth thresholds, and drop probability factors described herein can be established, e.g., configured on a per-priority queue basis. That is, in the example embodiment of
As shown in
As described herein, logic can execute to allocate multiple bandwidth meters per queue, shared across multiple queues or shared across multiple ports. These meters track bandwidth usage for packets associated with one or more applications over a fairly short interval, e.g., on the order of 100 ms or so. The meters are referenced using the flow id where multiple applications may hash to the same value. According to embodiments, logic can operate to share bandwidth meters referenced in the table of bandwidth meters across one or more queues or across multiple ports. Sharing the meters allows flexibility for a wide variety of constantly changing flow types. Alternatively, a predetermined number of meters can be assigned from the fixed size table of bandwidth meters to each port. For each queue there can be a global table of drop probabilities.
The reader will appreciate that embodiments of the present disclosure can provide many of the benefits for a per-flow queuing implementation, but with a significantly better cost versus performance trade-off and also a much more scalable implementation. For example, benefits over current 802.1D based priority queuing (e.g., up to 8 queues per port) include: better aggregate TCP performance by avoiding slow start retransmissions; avoiding applications failures from excessive TCP retransmission timeouts; automatically rate limiting bandwidth hogs fairly during congestion; automatically protecting low bandwidth flows even without QoS enabled (e.g., insures excellent voice quality and responsive interactive applications); providing consistent TCP performance, even with many connections (e.g., faster DNS lookup, 3 way TCP handshake, slow start/congestion recovery); preventing a small number of non-responsive applications from consuming the majority of the bandwidth; and eliminating the need to manually configure responsive and unresponsive traffic to be classified into separate priority queues or blocked by ACL filters.
An additional benefit of the various embodiments when used with a shared memory buffer allocation scheme across multiple ports includes: supporting deep queue sizes when only a few ports are busy, but preventing global buffer starvation when many queues are congested; and fairly distributing buffers across multiple ports (e.g., ports with many simultaneous applications getting more buffers than ports with only a few applications.
Although specific embodiments have been illustrated and described herein, those of ordinary skill in the art will appreciate that an arrangement calculated to achieve the same techniques can be substituted for the specific embodiments shown. This disclosure is intended to cover adaptations or variations of various embodiments of the invention.
It is to be understood that the above description has been made in an illustrative fashion, and not a restrictive one. Combination of the above embodiments, and other embodiments not specifically described herein will be apparent to those of skill in the art upon reviewing the above description. The scope of the various embodiments of the invention includes other applications in which the above structures and methods are used. Therefore, the scope of various embodiments of the invention should be determined with reference to the appended claims, along with the full range of equivalents to which such claims are entitled.
In the foregoing Detailed Description, various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the embodiments of the invention require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US6377546||May 12, 1998||Apr 23, 2002||International Business Machines Corporation||Rate guarantees through buffer management|
|US6515963||Jan 27, 1999||Feb 4, 2003||Cisco Technology, Inc.||Per-flow dynamic buffer management|
|US6606301 *||Mar 1, 1999||Aug 12, 2003||Sun Microsystems, Inc.||Method and apparatus for early random discard of packets|
|US6829217||Dec 2, 2002||Dec 7, 2004||Cisco Technology, Inc.||Per-flow dynamic buffer management|
|US6985442||Jul 26, 2000||Jan 10, 2006||Lucent Technologies Inc.||Technique for bandwidth sharing in internet and other router networks without per flow state record keeping|
|US7092357||Nov 13, 2001||Aug 15, 2006||Verizon Services Corp.||Anti-flooding flow-control methods and apparatus|
|US7382793 *||Jul 30, 2002||Jun 3, 2008||Juniper Networks, Inc.||Systems and methods for determining the bandwidth used by a queue|
|US7466703 *||Apr 30, 1999||Dec 16, 2008||Alcatel-Lucent Usa Inc.||Scalable high speed router apparatus|
|US20040114520||Dec 12, 2002||Jun 17, 2004||Alcatel Canada Inc.||Bandwidth management of resilient packet ring network|
|US20040143655||Dec 29, 2003||Jul 22, 2004||Narad Charles E.||Accessing transmission control protocol (TCP) segments|
|US20060221835||Mar 30, 2005||Oct 5, 2006||Cisco Technology, Inc.||Converting a network device from data rate traffic management to packet rate|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7941578 *||Oct 1, 2008||May 10, 2011||Hewlett-Packard Development Company, L.P.||Managing command request time-outs in QOS priority queues|
|US8239567 *||Aug 7, 2012||Marvell International Ltd.||Filtering superfluous data fragments on a computer network|
|US8456995 *||Jun 4, 2013||Hitachi, Ltd.||Packet transfer system, network management apparatus, and edge node|
|US8543725||Aug 6, 2012||Sep 24, 2013||Marvell International Ltd.||Filtering superfluous data fragments on a computer network|
|US9160664||Sep 20, 2013||Oct 13, 2015||Marvell International Ltd.||Determining a network node whether a received packet fragment should be transmitted|
|US9172655 *||Nov 15, 2012||Oct 27, 2015||Qlogic, Corporation||Systems and methods for quality of service in networks|
|US20100082856 *||Oct 1, 2008||Apr 1, 2010||Kimoto Christian A||Managing Command Request Time-outs In QOS Priority Queues|
|US20100322072 *||Jun 21, 2010||Dec 23, 2010||Hitachi, Ltd.||Packet Transfer System, Network Management Apparatus, and Edge Node|
|U.S. Classification||370/389, 370/395.32|
|International Classification||H04L12/56, H04L12/28|
|Cooperative Classification||H04L69/22, H04L47/23, H04L47/10|
|European Classification||H04L47/23, H04L47/10, H04L29/06N|
|Oct 31, 2006||AS||Assignment|
Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P., TEXAS
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ARNDT, MANFRED R.;REEL/FRAME:018484/0014
Effective date: 20061030
Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P.,TEXAS
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ARNDT, MANFRED R.;REEL/FRAME:018484/0014
Effective date: 20061030
|Sep 24, 2013||FPAY||Fee payment|
Year of fee payment: 4
|Nov 9, 2015||AS||Assignment|
Owner name: HEWLETT PACKARD ENTERPRISE DEVELOPMENT LP, TEXAS
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P.;REEL/FRAME:037079/0001
Effective date: 20151027