US 20030163593 A1 Abstract A system and method for dynamic bandwidth allocation is provided. The method provides one or more nodes to compute a simple lower bound of temporally and spatially aggregated virtual time using per-ingress counters of packet (byte) arrivals. Thus, when information is propagated along the ring, each node can remotely approximate the ideal fair rate for its own traffic at each downstream link. In this way, flows on the ring rapidly converge to their ring-wide fair rates while maximizing spatial reuse.
Claims(42) 1. A method for allocating bandwidth in a multi-node packet ring network, comprising the steps of:
at each node of the packet ring network, calculating a proxy to obtain a fair rate, the proxy calculated on the basis of per-ingress measurements of traffic on the packet ring network; distributing to upstream nodes of the packet ring network, the calculated proxy for the node; and wherein each upstream node modulates the rate of its traffic according to the bandwidth demands of the downstream nodes of the packet ring network. 2. The method of 3. The method of 4. The method of 5. The method of 6. The method of 7. The method of 8. The method of 9. A method for determining the rate of traffic flow at a node of a multi-node packet ring network, comprising the steps of:
at each node, determining an aggregated traffic flow associated with the node by calculating a traffic flow rate on the basis of per-ingress measurements of traffic on the packet ring; communicating the calculated traffic flow to at least one upstream node of the packet ring network; and adjusting the traffic flow rate at each node on the basis of the downstream traffic demands of the packet ring network. 10. The method of 11. The method of 12. A multi-node packet ring network,
wherein each node of the network calculates a traffic flow rate on the basis of the data stream originating at the node; and wherein each node of the network manages its traffic flow rate as a function of the traffic flow rates of downstream nodes in the packet ring network. 13. A method for establish ring ingress aggregated fairness in a multi-node packet ring network, comprising the steps of:
calculating, for at least one node of the packet ring network, a proxy, the proxy calculated on the basis of per-ingress measurements of traffic on the packet ring; distributing to at least one upstream node of the packet ring network, the calculated proxy for the node; and wherein each upstream node modulates the rate of its traffic according to the bandwidth demands of the downstream nodes of the packet ring network. 14. The method of 15. The method of 16. A method for allocating bandwidth in a multi-node packet ring network, comprising the steps of:
constructing, by at least one of said nodes, a proxy to determine a fair rate of a aggregate flow granularity. 17. The method of 18. The method of 19. The method of 20. The method of 21. The method of 22. The method of receiving by said node, information from one or more other nodes; computing a fair rate for a downstream node based upon said information. 23. The method of rate controlling said node's per-destination station traffic to a ring ingress aggregated with spatial reuse (RIAS) fairness rate. 24. The method of throttling traffic, by said node, when said information indicates congestion in a downstream node. 25. The method of 26. The method of 27. The method of computing a fair rate for said pre-determined time interval. 28. The method of generating a control message, said control message containing said fair rate for said pre-determined time interval for said node. 29. The method of sending said control message to another of said nodes. 30. The method of determining a rate controller value. 31. The method of sub-allocating a per-link fair rate to the flow with at least one egress node. 32. The method of 33. The method of 34. The method of 35. The method of 36. The method of 37. The method of 38. The method of 39. The method of at least one station transmit buffers operative with said rate controllers; at least one transit buffer; and a scheduler, operative with said station transit buffers and said transmit buffer, said scheduler further operative with said traffic monitor. 40. The method of at least one rate controller, said rate controller constructed and arranged to receive ingress traffic; a fair bandwidth allocator operative with said rate controller, said fair bandwidth allocator constructed and arranged to send a control message; a traffic monitor operative with said rate controller and said fair bandwidth allocator; at least one station transmit buffers operative with said rate controllers; at least one transit buffers, said transit buffers constructed and arranged to receive transit in signals; a scheduler operative with said traffic monitor, said scheduler constructed and arranged to receive signals from said station transmit buffers and said transit buffers, said scheduler further constructed and arranged to send transit out signals. 41. The method of 42. The method of Description [0001] This application is a conversion of U.S. Provisional Application No. 60/359,386 entitled “DESIGN, ANALYSIS, AND IMPLEMENTATION OF DISTRIBUTED VIRTUAL TIME SCHEDULING IN RINGS: AN ENHANCED PROTOCOL FOR PACKET RINGS” that was filed on Feb. 25, 2002. [0002] 1. Field of the Invention [0003] The present invention is related to computer networks. More specifically, the present invention is related to a fair, high performance protocol for packets on a distributed virtual-time scheduling of bandwidth within a resilient packet ring. [0004] 2. Description of the Related Art [0005] The overwhelmingly prevalent topology for metro networks is a ring. The primary reason is fault tolerance: all nodes remain connected with any single failure of a bi-directional link span. Moreover, rings have reduced deployment costs as compared to star or mesh topologies as ring nodes are only connected to their two nearest neighbors vs. to a centralized point (star) or multiple points (mesh). [0006] Unfortunately, current technology choices for high-speed metropolitan ring networks provide a number of unsatisfactory alternatives. A SONET ring can ensure minimum bandwidths (and hence fairness) between any pair of nodes. However, use of circuits prohibits unused bandwidth from being reclaimed by other flows and results in low utilization. On the other hand, a Gigabit Ethernet (GigE) ring can provide full statistical multiplexing, but suffers from unfairness as well as bandwidth inefficiencies due to forwarding all traffic in the same direction around the ring as dictated by the spanning tree protocol. For example, in the topology of FIG. 1, GigE nodes [0007] The IEEE 802.17 Resilient Packet Ring (RPR) working group was formed in early 2000 to develop a standard for bi-directional packet metropolitan rings. Unlike legacy technologies, the protocol supports destination packet removal so that a packet will not traverse all ring nodes and spatial reuse can be achieved. However, allowing spatial reuse introduces a challenge to ensure fairness among different nodes competing for ring bandwidth. Consequently, the key performance objective of RPR is to simultaneously achieve high utilization, spatial reuse, and fairness. Additional objectives of the present invention is a 50 msec fault recovery similar to that of SONET. [0008] To illustrate spatial reuse and fairness, consider the depicted scenario in FIG. 2 in which four infinite demand flows share link [0009] The key technical challenge of RPR is design of a bandwidth allocation algorithm that can dynamically achieve such rates. Note that to realize this goal, some coordination among nodes is required. For example, if each node performs weighted fair queuing a local operation without coordination among nodes, flows ( [0010] The RPR standard defines a fairness algorithm that specifies how upstream traffic should be throttled according to downstream measurements, namely, how a congested node will send fairness messages upstream so that upstream nodes can appropriately configure their rate limiters to throttle the rate of injected traffic to its fair rate. The standard also defines the scheduling policy to arbitrate service among transit and station (ingress) traffic as well as among different priority classes. The RPR fairness algorithm has several modes of operation including aggressive/conservative modes for rate computation and single-queue and dual-queue buffering for transit traffic. [0011] Unfortunately, we have found that the RPR fairness algorithm has a number of important performance limitations. First, it is prone to severe and permanent oscillations in the range of the entire link bandwidth in simple “unbalanced traffic” scenarios in which all flows do not demand the same bandwidth. Second, it is not able to fully achieve spatial reuse and fairness. Third, for cases where convergence to fair rates does occur, it requires numerous fairness messages to converge (e.g., 500) thereby hindering fast responsiveness. [0012] The goals of this discussion are threefold. In the detailed description of the invention, we first provide an idealized reference model termed Ring Ingress Aggregated with Spatial reuse (RIAS) fairness. RIAS fairness achieves maximum spatial reuse subject to providing fair rates to each ingress-aggregated flow at each link. We argue that this fairness model addresses the specialized design goals of metro rings, whereas proportional fairness and flow max-min fairness do not. We use this model to identify key problematic scenarios for RPR algorithm design, including those studied in the standardization process (e.g., “Parking Lot”) and others that have not received previous attention (e.g., “Parallel Parking Lot” and “Unbalanced Traffic”). We then use the reference model and these scenarios as a benchmark for evaluating and comparing fairness algorithms, and to identify fundamental limits of current RPR control mechanisms. [0013] Second, we develop a new dynamic bandwidth allocation algorithm termed Distributed Virtual-time Scheduling in Rings (DVSR). Like current implementations, DVSR has a simple transit path without any complex operations such as fair queuing. However, with DVSR, each node uses its per-destination byte counters to construct a simple lower bound on the evolution of the spatially and temporally aggregated virtual time. That is, using measurements available at an RPR node, we compute the minimum cumulative change in virtual time since the receipt of the last control message, as if the node was performing weighted fair queuing at the granularity of ingress-aggregated traffic. By distributing such control information upstream, we show how nodes can perform simple operations on the collected information and throttle their ingress flows to their ring-wide RIAS fair rates. [0014] Finally, we study the performance of DVSR and the standard RPR fairness algorithm using a combination of theoretical analysis, simulation, and implementation. In particular, we analytically bound DVSR's unfairness due to use of delayed and time-averaged information in the control signal. We perform ns-2 simulations to compare fairness algorithms and obtain insights into problematic scenarios and sources of poor algorithm performance. For example, we show that while DVSR can fully reclaim unused bandwidth in scenarios with unbalanced traffic (unequal input rates), the RPR fairness algorithm suffers from utilization losses of up to 33% in an example with two links and two flows. We also show how DVSR's RIAS fairness mechanism can provide performance isolation among nodes' throughputs. For example, in a Parking Lot scenario (FIG. 5) with even moderately aggregated TCP flows from one node competing for bandwidth with non-responsive UDP flows from other nodes, all ingress nodes obtain nearly equal throughput shares with DVSR, quite different from the unfair node throughputs obtained with a GigE ring. Finally, we develop a 1 Gb/sec network processor implementation of DVSR and present the results of our measurement study on an eight-node ring. [0015] The remainder of this discussion is organized as follows. In Section II we present an overview of the RPR node architecture and fairness algorithms. Next, in Section III we present the RIAS reference model for fairness. In Section IV, we present a performance analysis of the RPR algorithms and present oscillation conditions and expressions for throughput degradation. In Section V, we present the DVSR algorithm and in Section VI we analyze DVSR's fairness properties. Next, we provide extensive simulation comparisons of DVSR, RPR, and GigE in Section VII, and in Section VIII, we present measurement studies from our network processor implementation of DVSR. Finally, we review related work in Section IX and conclude in Section X. [0016] A more complete understanding of the present disclosures and advantages thereof may be acquired by referring to the following description taken in conjunction with the accompanying drawings wherein: [0017]FIG. 1 is an illustration of a resilient packet ring according to the prior art. [0018]FIG. 2 is a block diagram illustrating a parallel parking lot flow problem according to the prior art. [0019]FIG. 3 is a block diagram illustrating a generic resilient packet ring node architecture according to the teachings of the present invention. [0020]FIG. 4 is a block diagram illustrating a parallel parking lot flow situation implementing a ring ingress aggregated with spatial reuse (RIAS) fairness according to the teachings of the present invention. [0021]FIG. 5 is a block diagram illustrating a parallel parking lot topology according to the teachings of the present invention. [0022]FIG. 6 is a block diagram illustrating a tow-exit parking lot topology according to the teachings of the present invention. [0023]FIG. 7 is a block diagram of an oscillation scenario according to the teachings of the present invention. [0024]FIG. 8 is a block diagram of an upstream parallel parking lot situation according to the teachings of the present invention. [0025]FIG. 9 [0026]FIG. 9 [0027]FIG. 10 is a plot of throughput loss versus flow rate for a resilient packet ring in aggressive mode according to the teachings of the present invention. [0028]FIG. 11 is a plot of throughput loss versus flow rate for a resilient packet ring in conservative mode according to the teachings of the present invention. [0029]FIG. 12 is a plot of remote fair queuing according to the teachings of the present invention. [0030]FIG. 13 [0031]FIG. 13 [0032]FIG. 13 [0033]FIG. 14 is a block diagram illustrating a single node model for a distributed virtual-time scheduling in rings (DVSR) according to the teachings of the present invention. [0034]FIG. 15 is a plot of fairness versus time illustrating the fairness bound according to the teachings of the present invention. [0035]FIG. 16 is a plot of normalized throughput versus flow for a parking lot example according to the teachings of the present invention. [0036]FIG. 17 is a plot of normalized throughput versus flow for a DVSR's TCP and UDP flow bandwidth shares according to the teachings of the present invention. [0037]FIG. 18 is a plot of normalized throughput versus flow illustrating a DVSR's throughput for TCP micro-flows according to the teachings of the present invention. [0038]FIG. 19 is a plot of normalized throughput versus flow illustrating the spatial reuse in the parallel parking lot example according to the teachings of the present invention. [0039]FIG. 20 illustrates convergence times for the DVSR, and the resilient packet ring in both aggressive mode and conservative mode according to the teachings of the present invention. [0040]FIG. 21 is a block diagram illustrating the testbed configuration according to the teachings of the present invention. [0041] The present invention may be susceptible to various modifications and alternative forms. Specific embodiments of the present invention are shown by way of example in the drawings and are described herein in detail. It should be understood, however, that the description set forth herein of specific embodiments is not intended to limit the present invention to the particular forms disclosed. Rather, all modifications, alternatives and equivalents falling within the spirit and scope of the invention, as defined by the appended claims, are to be covered. [0042] II. BACKGROUND ON IEEE 802.17 RPR [0043] In this section, we describe the basic operation of the Resilient Packet Ring (RPR) fairness algorithm. Due to space constraints, our description necessarily omits many details and focuses on the key mechanisms for bandwidth arbitration. Readers are referred to the standards documents for full details and pseudocode. [0044] Throughout, we consider committed rate (Class B) and best effort (Class C) traffic classes in which each node obtains a minimum bandwidth share (zero for Class C) and reclaims unused bandwidth in a weighted fair manner, here considering equal weights for each node. We omit discussion of Class A traffic that has guaranteed rate and jitter, as other nodes are prohibited from reclaiming unused Class A bandwidth. [0045] A. RPR Node Architecture [0046] The architecture of a generic RPR node is illustrated in FIG. 3. For convenience, the generic RPR node [0047] Next, RPR nodes have measurement modules (byte counters) to measure demanded and/or serviced station and transit traffic. These measurements are used by the fairness algorithm to compute a feedback control signal to throttle upstream nodes to the desired rates. Nodes that receive a control message use the information in the message, perhaps together with local information, to set the bandwidths for the rate controllers [0048] The final component is the scheduling algorithm that arbitrates service among station and transit traffic. In single-queue mode, the transit path consists of a single FIFO queue referred to as the Primary Transit Queue (PTQ). In this case, the scheduler employs strict priority of transit traffic over station traffic. In dual-queue mode, there are two transit path queues, one for guaranteed Class A traffic (PTQ), and the other for Class B and C traffic, called Secondary Transit Queue (STQ). In this mode, the scheduler always services Class A transit traffic first from PTQ. If this queue is empty, the scheduler employs round-robin service among the transit traffic in STQ and the station traffic until a buffer threshold is reached for STQ. If STQ reaches the buffer threshold, STQ transit traffic is always selected over station traffic to ensure a lossless transit path. In other words, STQ has strict priority over station traffic once the buffer threshold is crossed; otherwise, service is round robin among transit and station traffic. [0049] In both cases, the objective is to ensure hardware simplicity (for example, avoiding expensive per-flow or per-ingress queues on the transit path) and to ensure that the transit path is lossless, i.e., once a packet is injected into the ring, it will not be dropped at a downstream node. [0050] B. RPR Fairness Algorithm [0051] The dynamic bandwidth control algorithm that determines the station rate controller values, and hence the basic fairness and spatial reuse properties of the system is the primary aspect in which the RPR fairness algorithm and DVSR differ and is the focus of the discussion below as well as throughout the discussion. [0052] There are two modes of operation for the RPR fairness algorithm. The first, termed Aggressive Mode (AM), evolved from the Spatial Reuse Protocol (SRP) currently deployed in a number of operational metro networks. The second, termed Conservative Mode (CM), evolved from the Aladdin algorithm. Both modes operate within the same framework described as follows. A congested downstream node conveys its congestion state to upstream nodes such that they will throttle their traffic and ensure that there is sufficient spare capacity for the downstream station traffic. To achieve this, a congested node transmits its local fair rate upstream, and all upstream nodes sending to the link must throttle to this same rate. After a convergence period, congestion is alleviated once all nodes' rates are set to the minimum fair rate. Likewise, when congestion clears, stations periodically increase their sending rates to ensure that they are receiving their maximal bandwidth share. [0053] There are two key measurements for RPR's bandwidth control, forward_rate and add_rate. The former represents the service rate of all transit traffic and the latter represents the rate of all serviced station traffic. Both are measured as byte counts over a fixed interval length aging_interval. Moreover, both measurements are low-pass-filtered using exponential averaging with parameter 1/LPCOEF given to the current measurement and 1-1/LPCOEF given to the previous average. In both cases, it is important that the rates are measured at the output of the scheduler so that they represent serviced rates rather than offered rates. [0054] At each aging_interval, every node checks its congestion status based on conditions specific to the mode AM or CM. When node n is congested, it calculates its local_fair_rate[n], which is the fair rate that an ingress-based flow can transmit to node n. Node n then transmits a fairness control message to its upstream neighbor that contains local_fair_rate [n]. [0055] If upstream node (n-1) receiving the congestion message from node n is also congested, it will propagate the message upstream using the minimum of the received local_fair_rate [n] and its own local_fair_rate [n-1]. The objective is to inform upstream nodes of the minimum rate they can send along the path to the destination. If node (n-1) is not congested but its forward_rate is greater than the received local_fair_rate [n], it forwards the fairness control message containing local_fair rate [n] upstream, as this situation indicates that the congestion is due to transit traffic from further upstream. Otherwise, a null-value fairness control message is transmitted to indicate a lack of congestion. [0056] When an upstream node i receives a fairness control message advertising local_fair_rate [n], it reduces its rate limiter values, termed allowed_rate [i][j], for all values of j, such that n lies on the path from i to j. The objective is to have upstream nodes throttle their own station rate controller values to the minimum rate it can send along the path to the destination. Consequently, station traffic rates will not exceed the advertised local_fair_rate value of any node in the downstream path of a flow. Otherwise, if a null-value fairness control message is received, it increments allowed_rate by a fixed value such that it can reclaim additional bandwidth if one of the downstream flows reduces its rate. Moreover, such rate increases are essential for convergence to fair rates even in cases of static demand. [0057] The main differences between AM and CM are congestion detection and calculation of the local fair rate which we discuss below. Moreover, by default AM employs dual-queue mode and CM employs single-queue mode. [0058] C. Aggressive Mode (AM) [0059] Aggressive Mode is the default mode of operation of the RPR fairness algorithm and its logic is as follows. An AM node n is said to be congested whenever STQ_depth[n]>low_threshold [0060] or forward_rate[n]+add_rate[n]>unreserved_rate, [0061] where as above, STQ is the transit queue for Class B and C traffic. The threshold value low_threshold is a fraction of the transit queue size with a default value of ⅛ of the STQ size. The unreserved_rate is the link capacity minus the reserved rate for guaranteed traffic. As we consider only best-effort traffic, unreserved_rate is the link capacity used for the remainder of this discussion. [0062] When a node is congested, it calculates its local_fair_rate as the normalized service rate of its own station traffic, add_rate, and then transmits a fairness control message containing add_rate to upstream nodes. [0063] Considering the parking lot example in FIG. 5, if a downstream node advertises add_rate below the true fair rate (which does indeed occur before convergence), all upstream nodes will throttle to this lower rate; in this case, downstream nodes will later become uncongested so that flows will increase their allowed_rate. This process will then oscillate more and more closely around the targeted fair rates for this example. [0064] D. Conservative Mode (CM) [0065] Each CM node has an access timer measuring the time between two consecutive transmissions of station packets. As CM employs strict priority of transit traffic over station traffic via single queue mode, this timer is used to ensure that station traffic is not starved. Thus, a CM node n is said to be congested if the access timer for station traffic expires or if forward_rate[n]+add_rate[n]>low threshold. [0066] Unlike AM, low_threshold for CM is a rate-based parameter that is a fixed value less than the link capacity, 0.8 of the link capacity by default. In addition to measuring forward_rate and add_rate, a CM node also measures the number of active stations that have had at least one packet served in the past aging_interval. [0067] If a CM node is congested in the current aging_interval, but was not congested in the previous one, the local_fair_rate is computed as the total unreserved rate divided by the number of active stations. If the node is continuously congested, then local_fair_rate depends on the sum of forward_rate and add_rate. If this sum is less than low_threshold, indicating that the link is under utilized, local_fair_rate ramps up. If this sum is above high_threshold, a fixed parameter with a default value that is 0.95 of the link capacity, local_fair_rate will ramp down. [0068] Again considering the parking lot example in FIG. 5, when the link between nodes [0069] III. A FAIRNESS REFERENCE MODEL FOR PACKET RINGS [0070] For flows contending for bandwidth at a single network node, a definition of fairness is immediate and unique. However, for multiple nodes, there are various bandwidth allocations that can be considered to be fair in different senses. For example, proportional fairness allocates a proportionally decreased bandwidth to flows consuming additional resources, i.e., flows traversing multiple hops, whereas max-min fairness does not. Moreover, any definition of fairness must carefully address the granularity of flows for which bandwidth allocations are defined. Bandwidth can be granted on a per-micro-flow basis or alternately to particular groups of aggregated micro-flows. [0071] In this section, we define Ring Ingress Aggregated with Spatial Reuse (RIAS) fairness, a reference model for achieving fair bandwidth allocation while maximizing spatial reuse in packet rings. The RIAS reference model is now incorporated into the IEEE 802.17 standard's targeted performance objective. We justify the model based on the design goals of packet rings and compare it with proportional and max-min fairness. We then use the model as a design goal in DVSR's algorithm design and the benchmark for general RPR performance analysis. [0072] A. Ring Ingress Aggregated with Spatial Reuse (RIAS) Fairness [0073] RIAS Fairness has two key components. The first component defines the level of traffic granularity for fairness determination at a link as an ingress-aggregated (IA) flow, i.e., the aggregate of all flows originating from a given ingress node, but not necessarily destined to a single egress node. The targeted service model of packet rings justifies this: to provide fair and/or guaranteed bandwidth to the networks and backbones that it interconnects. Thus, our reference model ensures that an ingress node's traffic receives an equal share of bandwidth on each link relative to other ingress nodes' traffic on that link. The second component of RIAS fairness ensures maximal spatial reuse subject to this first constraint. That is, bandwidth can be reclaimed by IA flows (that is, clients) when it is unused either due to lack of demand or in cases of sufficient demand in which flows are bottlenecked elsewhere. [0074] Below, we present a formal definition that determines if a set of candidate allocated rates (expressed as a matrix R) is RIAS fair. For simplicity, we define RIAS fairness for the case that all ingress nodes have equal weight; the definition can easily be generalized to include weighted fairness. Furthermore, for ease of discussion and without loss of generality, we consider only traffic forwarded on one of the two rings, and assume fluid arrivals and services in the idealized reference model, with all rates in the discussion below referring to instantaneous fluid rates. We refer to a flow as all uni-directional traffic between a certain ingress and egress pair, and we denote such traffic between ring ingress node i and ring egress node j as flow (i,j) as illustrated in FIG. 2. Such a flow is typically composed of aggregated micro-flows such as individual TCP sessions, although other flows are possible. To simplify notation, we label a tandem segment of N nodes and N=1 links such that flow (i,j) traverses node n if i≦n≦j, and traverses link n if i≦n≦j. [0075] Consider a set of infinite-demand flows between pairs of a subset of ring nodes, with remaining pairs of nodes having no traffic between them. Denote R [0076] Let C be the capacity of all links in the ring. Then we can write the following constraints on the matrix of allocated rates R={ij}: R F [0077] A matrix R satisfying these constraints is said to be feasible. Further, let IA(i) denote the aggregate of all flows originating from ingress node i such that IA(i)=Σ [0078] Given a feasible rate matrix R, we say that link n is a bottleneck link with respect to R for flow (i,j) crossing link n, and denote it by B [0079] Definition 1: A matrix of rates R is said to be RIAS fair if it is feasible and if for each flow (i, j), R R IA(i′) [0080] when IA(i′),IA(i)>0 at both B IA(i′)≦IA(i) otherwise. (6) [0081] We distinguish three cases in Definition 1. First, in Equation (4) since flows (i,j) and (i′,j′) have the same ingress node, the inequality ensures fairness among an IA flow's sub-flows to different egress nodes. Second, in Equation (5), flows (i,j) and (i′,j′) have different ingress nodes and different bottleneck links, but B [0082]FIG. 4 illustrates the above definition. Assuming that capacity is normalized and all demands are infinite, the RIAS fair shares are as follows: R [0083] Proposition 1: A feasible rate matrix R is RIAS-fair if and only if each flow (i,j) has a bottleneck link with respect to R. [0084] Proof: Suppose that R is RIAS-fair, and to prove the proposition by contradiction, assume that there exists a flow (i,j) with no bottleneck link. Then, for each link n crossed by flow (i,j) for which F [0085] where δ [0086] For the second part of the proof, assume that each flow has a bottleneck with respect to R. To increase the rate of flow (i,j) at its bottleneck link while maintaining feasibility, we must decrease the rate of at least one flow from IA(i′) (by definition we have F [0087] We make three observations about this definition. First, observe that on each link, each ingress node's traffic will obtain no less than bandwidth C/N if its demanded bandwidth is at least C/N. Note, if the tandem segment has N nodes, the ring topology has 2N nodes: if flows use shortest-hop-count paths, each link will be shared by at most half of the total number of nodes on the ring. Secondly, note that these minimum bandwidth guarantees can be weighted to provide different bandwidths to different ingress nodes. Finally, we note that RIAS fairness differs from flow max-min fairness in that RIAS simultaneously considers traffic at two granularities: ingress aggregates and flows. Consequently, as discussed and illustrated below, RIAS bandwidth allocations are quite different that flow max-min fairness as well as proportional fairness. [0088] B. Discussion and Comparison with Alternate Fairness Models [0089] Here, we illustrate RIAS fairness in simple topologies and justify it in comparison with alternate definitions of fairness. [0090] Consider the classical “parking lot” topology of FIG. 5. In this example, we have 5 nodes and 4 links, and all flows sending to the right-most node numbered [0091] In contrast, a proportional fair allocation scales bandwidth allocations according to the total resources consumed. In particular, since flow ( [0092] Second, consider the Parallel Parking Lot topology of FIG. 2, which contains a single additional flow between nodes [0093] Finally, consider the “two exit” topology of FIG. 6. Here, we consider an additional node [0094] IV. PERFORMANCE LIMITS OF RPR [0095] In this section, we present a number of important performance limits of the RPR fairness algorithm in the context of the RIAS objective. [0096] A. Permanent Oscillation with Unbalanced Constant-Rate Traffic Inputs [0097] The RPR fairness algorithm suffers from severe and permanent oscillations for scenarios with unbalanced traffic. There are multiple adverse effects of such oscillations, including throughput degradation and increased delay jitter. The key issue is that the congestion signals add_rate for Aggressive Mode and (C/number of active stations) for Conservative Mode do not accurately reflect the congestion status or true fair rate and hence nodes oscillate in search of the correct fair rates. [0098] A.1Aggressive Mode [0099] Recall that without congestion, rates are increased until congestion occurs. In AM, once congestion occurs, the input rates of all nodes contributing traffic to the congested link are set to the minimum input rate. However, this minimum input rate is not necessarily the RIAS fair rate. Consequently, nodes over-throttle their traffic to rates below the RIAS rate. Subsequently, congestion will clear and nodes will ramp up their rates. Under certain conditions of unbalanced traffic, this oscillation cycle will continue permanently and lead to throughput degradation. Let r [0100] Proposition 2. For a given RIAS rate matrix R, demanded rates rand congested link j, permanent oscillations will occur in RPR-AM if there is a flow (n,i) crossing link j such that following two conditions are satisfied:
[0101] Moreover, for small buffers and zero propagation delay, the range of oscillations will be from r [0102] For example, consider Aggressive Mode with two flows such that flow ( [0103] Since the aggregate traffic arrival rate downstream is C+ε, the downstream link will become congested. Thus, a congestion message will arrive upstream containing the transmission rate of the downstream flow, in this case ε. Consequently, the upstream node must throttle its flow from rate C to rate ε. At this point, the rate on the downstream link is 2ε so that congestion clears. Subsequently, the upstream flow will increase its rate back to C−ε upon receiving null congestion messages. Repeating the cycle, the upstream flow's rate will permanently oscillate between C−ε and the low rate of the downstream flow ε. [0104] Observe from Proposition 2 that oscillations also occur with balanced input rates but unbalanced RIAS rates. An example of such a scenario is depicted in FIG. 8 in which each flow has identical demand C. In this case, flow ( [0105] A.2 Conservative Mode [0106] Unbalanced traffic is also problematic for Conservative Mode. With CM, the advertised rate is determined by the number of active flows when a node first becomes congested for two consecutive aging_intervals. If a flow has even a single packet transmitted during the last aging_interval, it is considered active. Consequently, permanent oscillations occur according to the following condition. [0107] Proposition 3: For a given RIAS rate matrix R, demanded rates r, and congested link j, let n [0108] Moreover, the lower limit of the oscillation range is C/n [0109] For example, consider a two-flow scenario similar to that above except with the upstream flow ( [0110] B. Throughput Loss [0111] As a consequence of permanent oscillations, RPR-AM and RPR-CM suffer from throughput degradation and are not able to fully exploit spatial reuse. [0112] B.1Aggressive Mode [0113] Here, we derive an expression for throughput loss due to oscillations. For simplicity and without loss of generality, we consider two-flow cases as depicted in FIG. 7. We ignore low pass filtering and first characterize the rate increase part of a cycle, denoting the minimum and maximum rate by r [0114] Note that r [0115]FIG. 9( [0116] From this characterization of an oscillation cycle, we can compute the throughput loss for the flow oscillating between rates r [0117] where R is the RIAS fair rate. [0118]FIG. 10 depicts throughput loss vs. the downstream flow ( [0119] B.2 Conservative Mode [0120] Throughput loss for Conservative Mode has two origins. First, as described in Section II, the utilization in CM is purposely restricted to less than high_threshold, typically 95%. Second, similar to AM, permanent oscillations occur with CM under unbalanced traffic resulting in throughput degradation and partial spatial reuse. We derive an expression to characterize CM throughput degradation in a two-flow scenario as above. Let r [0121] where r [0122] Notice that link [0123]FIG. 9( [0124] Finally, to analyze the throughput loss of RPR-CM, we consider parking lot scenarios with N unbalanced flows originating from N nodes sending to a common destination. For a reasonable comparison, the sum of the demanding rate of all flows is 605 Mbps, which is less then the link capacity. The 1 [0125]FIG. 11 depicts throughput loss obtained from simulations as well as the above model using Equation (8). We find that the throughput loss with RPR-CM can be up to 30%, although the sum of the offered load is less than the link capacity. Finally, observe that the analytical model is again quite accurate and matches the simulation results within 3% [0126] A. Convergence [0127] Finally, the RPR algorithms suffer from slow convergence times. In particular, to mitigate oscillations even for constant rate traffic inputs as in the example above, all measurements are low pass filtered. However, such filtering, when combined with the coarse feedback information, has the effect of delaying convergence (for scenarios where convergence does occur). We explore this effect using simulations in Section VII. [0128] V. DISTRIBUTED VIRTUAL TIME SCHEDULING IN RINGS (DVSR) [0129] In this section, we devise a distributed algorithm to dynamically realize the bandwidth allocations in the RIAS reference model. Our technique is to have nodes construct a proxy of virtual time at the Ingress Aggregated flow granularity. This proxy is a lower bound on virtual time temporally aggregated over time and spatially aggregated over traffic flows sharing the same ingress point (IA flows). It is based on simple computations of measured IA byte counters such that we compute the local bandwidth shares as if the node was performing IA-granularity fair queuing, when in fact, the node is performing FIFO queuing. By distributing this information to other nodes on the ring, all nodes can remotely compute their fair rates at downstream nodes, and rate control their per-destination station traffic to the RIAS fair rates. [0130] We first describe the algorithm in an idealized setting, initially considering virtual time as computed in a generalized processor sharing (“GPS”) fluid system with an IA flow granularity. We then progressively remove the impractical assumptions of the idealized setting, leading to the network-processor implementation described in Section VIII. [0131] We denote r [0132] A. Distributed Fair Bandwidth Allocation [0133] The distributed nature of the ring bandwidth allocation problem yields three fundamental issues that must be addressed in algorithm design. First, resources must be remotely controlled in that an upstream node must throttle its traffic according to congestion at a downstream node. Second, the algorithm must contend with temporally aggregated and delayed control information in that nodes are only periodically informed about remote conditions, and the received information must be a temporally aggregated summary of conditions since the previous control message. Finally, there are multiple resources to control with complex interactions among multi-hop flows. We next consider each issue independently. [0134] A.1Remote Fair Queuing [0135] The first concept of DVSR is control of upstream rate-controllers via use of ingress-aggregated virtual time as a congestion message received from downstream nodes. For a single node, this can be conceptually viewed as remotely transmitting packets at the rate that they would be serviced in a GPS system, where GPS determines packet service order according to a granularity of packets' ingress nodes only (as opposed to ingress and egress nodes, micro-flows, etc.). [0136]FIG. 12 illustrates remote bandwidth control for a single resource. In this case, RIAS fairness is identical to flow max-min fairness so that GPS server ρ [0137] when v [0138] For example, consider the four-flow parking lot example of Section III. Suppose that the system is initially idle so that ρi(0)=1, and that immediately after time 0, flows begin transmitting at infinite rate (i.e., they become infinitely backlogged flows). As soon as the multiplexer depicted in FIG. 12( [0139] Suppose, at some later time, the 4th flow shuts off so that the fair rates are now ⅓. As the 4th flow would no longer have packets (fluid) in the multiplexer, v(t) will now have slope ⅓ and the rate limiters are set to ⅓. Thus, by monitoring virtual time, flows can increase their rates to reclaim unused bandwidth and decrease it as other flows increase their demand. Note that with 4 flows, the rate controllers will never be set to rates below ¼, the minimum fair rate. [0140] Finally, notice that in this ideal fluid system with zero feedback delay, the multiplexer is never more than infinitesimally backlogged, as the moment fluid arrives to the multiplexer, flows are throttled to a rate equal to their GPS service rates. Hence, all buffering and delay is incurred before service by the rate controllers. [0141] A.2 Delayed and Temporally Aggregated Control Information [0142] The second key component of distributed bandwidth allocation in rings is that congestion and fairness information shared among nodes is necessarily delayed and temporally aggregated. That is, in the above discussion we assumed that virtual time is continually fed back to the rate controllers without delay. However, in practice feedback information must be periodically summarized and transmitted in a message to other nodes on the ring. Thus, delayed receipt of summary information is also fundamental to a distributed algorithm. [0143] For the same single resource example of FIG. 12, and for the moment for Δ=0, consider that every T seconds the multiplexer transmits a message summarizing the evolution of virtual time over the previous T seconds. If the multiplexer is continuously backlogged in the interval [t-T,t], then information can be aggregated via a simple time average. If the multiplexer is idle for part of the interval, then additional capacity is available and rate controller values may be further increased accordingly. Moreover, v(t) should not be reset to 0 when the multiplexer goes idle, as we wish to track its increase over the entire window T. Thus, denoting b as the fraction of time during the previous interval T that the multiplexer is busy serving packets, the rate controller value should be ρ [0144] The example depicted in FIG. 13 illustrates this time averaged feedback signal and the need to incorporate b that arises in this case (but not in the above case without time averaged information). Suppose that the link capacity is 1 packet per second and that T=10 packet transmission times. If the traffic demand is such that six packets arrive from flow [0145] Finally, consider that the delay to receive information is given by Δ>0. In this case, rate controllers will be set at time t to their average fair rate for the interval [t-T-Δ, t-Δ]. Consequently, due to both delayed and time averaged information, rate controllers necessarily deviate from their ideal values, even in the single resource example. We consider such effects of Δ and T analytically in Section VI and via simulations in Section VII. [0146] A.3 Multi-node RIAS Fairness [0147] There are three components to achieving RIAS fairness encountered in multiple node scenarios. First, an ingress node must compute its minimum fair rate for the links along its flows' paths. Thus, in the parking lot example, node [0148] Second, if an ingress node has multiple flows with different egress nodes sharing a link, it must sub-allocate its per-link IA fair rate to these flows. For example, in the Two Exit Parking Lot scenario of FIG. 6, node [0149] where ρ [0150] Finally, we observe that in certain cases, the process often requires multiple iterations to converge, even in this still idealized setting, and hence multiple intervals T to realize the RIAS fair rates. The key reason is that nodes cannot express their true “demand” to all other nodes initially, as they may be bottlenecked elsewhere. For example, consider the scenario illustrated in FIG. 8 in which all flows have infinite demand. After an initial window of duration T, flow ( [0151] B. DVSR Protocol [0152] In the discussion above, we presented DVSR's conceptual operation in an idealized setting. Here, we describe the DVSR protocol as implemented in the simulator and testbed. We divide the discussion into four parts: scheduling of station vs. transit packets, computation of the feedback signal (control message), transmission of the feedback signal, and rate limit computation. [0153] B.1 Scheduling of Station vs. Transit Packets [0154] As described in Section II, the high speed of the transit path and requirements for hardware simplicity prohibit per-ingress transit queues and therefore prohibit use of fair queuing or any of its variants, even at the IA granularity. Consequently, we employ first-in first-out scheduling of all offered traffic (station or transit) in both the simulator and implementation. [0155] Recall that the objective of DVSR is to throttle flows to their ring-wide RIAS-fair rate at the ingress point. Once this is achieved and steady state is reached, queues will remain empty and the choice of the scheduler is of little impact. Before convergence (typically less than several ring propagation times in our experiments) the choice of the scheduler impacts the jitter and short-term fairness properties of any fairness algorithm. While a number of variants on FIFO are possible, especially when also considering high priority class A traffic, we leave a detailed study of scheduler design to future work and focus on the fairness algorithm. [0156] B.2 Feedback Signal Computation [0157] As inputs to the algorithm, a node measures the number of arriving bytes from each ingress node, including the station, over a window of duration T. Thus, the measurements used by DVSR are identical to those of RPR. We denote the measurement at this node from ingress node i as l [0158] First, we observe that the exact value of v(t)−v(t-T) cannot be derived only from byte counters as v(t) exposes shared congestion whereas byte counts do not. For example, consider that two packets from two ingress nodes arrive in a window of duration T. If the packets arrive back-to-back, then v(t) increases by 1 over an interval of 2 packet transmission times. On the other hand, if the packets arrive separately so that their service does not overlap, then v(t) increases from 0 to 1 twice. Thus, the total increase in the former case is 1 and in the latter case is 2, with both cases having a total backlogging interval of 2 packet transmission times. [0159] However, a lower bound to v(t)−v(t-T) can be computed by observing that the minimum increase in v(t) occurs if all packets arrive at the beginning of the interval. This minimum increase will then provide a lower bound to the true virtual time, and is used in calculation of the control message's rate. We denote F as v(t)−v(t−T)/T+(1−b) at a particular node. Moreover, consider that the byte counts from each ingress node are ordered such that l1≦l2≦ . . . ≦k for k flows transmitting any traffic during the interval. Then F is computed every T seconds as given by the pseudo code of Table I. For simplicity of explanation, we consider the link capacity C to be in units bytes/sec and consider all nodes to have equal weight.
[0160] Note that when b<1 (the link is not always busy over the previous interval), the value of F is simply the largest ingress-aggregated flow transmission rate l [0161] Implementation of the algorithm has several aspects not yet described. First, b is easily computed by dividing the number of bytes transmitted by CT the maximum number of bytes that could be serviced in T. Second, ordering the byte counters such that l1≦l2≦ . . . ≦1 [0162] B.3 Feedback Signal Transmission [0163] We next address transmission of the feedback signal. In our implementation, we construct a single N-byte control message containing each node's most recently computed value of F such that the message contains F [0164] An alternate messaging approach more similar to RPR is to have each node periodically transmit messages with a single value F [0165] B.4 Rate Limit Computation [0166] The final step is for nodes to determine their rate controller values given their local measurements and current values of F [0167] C. Discussion [0168] We make several observations about the DVSR algorithm. First, note that if there are N nodes forwarding traffic through a particular transit node, rate controllers will never be set to rates below 1/N, the minimum fair rate. Thus, even if all bandwidth is temporarily reclaimed by other nodes, each node can immediately transmit at this minimum rate; after receiving the next control message, upstream nodes will throttle their rates to achieve fairness at timescales greater than T; until T, packets are serviced in FIFO order. [0169] Next, observe that by weighting ingress nodes, any set of minimum rates can be achieved, if the sum of such minimum rates is less than the link capacity. [0170] Third, we note that the DVSR protocol is a distributed mechanism to compute the RIAS fair rates. In particular, to calculate the RIAS fair rates, we first estimate the local IA-fair rates using local byte counts. Once nodes receive their locally fair rates, they adapt their rate limiter values converging to the RIAS rates. [0171] Finally, we observe that unlike the RPR fairness algorithm, DVSR does not low pass filter control signal values at transit nodes nor rate limiter values at stations. One important reason is that the system has a natural averaging interval built in via periodic transmission of control signals. By selecting a control signal that conveys a bound on the time-averaged increase in IA virtual time as opposed to the station transit rate, no further damping is required. [0172] VI. ANALYSIS OF DVSR FAIRNESS [0173] There are many factors of a realistic system that will result in deviations between DVSR service rates and ideal RIAS fair rates. Here, we isolate the issue of temporal information aggregation and develop a simple theoretical model to study how T impacts system fairness. The technique can easily be extended to study the impact of propagation delay, an issue we omit for brevity. [0174] A. Scenario [0175] We consider a simplified but illustrative scenario with remote fair queuing and temporally aggregated feedback as in FIG. 12. We further assume that the multiplexer is an ideal fluid GPS server, and that the propagation delay is ?=0. We consider two flows i and j that have infinite demand and are continuously backlogged. For all other flows, we consider the worst case traffic pattern that maximizes the service discrepancy between flows i and j. Thus, FIG. 14 depicts the analysis scenario [0176] We say that a flow node-backlogged if the buffer at its ingress node's rate controller is non-empty and that a flow is scheduler-backlogged if the (transit/station) scheduler buffer is non-empty. Moreover, whenever the available service rate at the GPS multiplexer is larger than the rate limiter value in DVSR, the flow is referred to as over-throttled. Likewise, if the available GPS service rate is smaller than the rate limiter value in DVSR, the flow is under-throttled. Note that as we consider flows with infinite demand, flows are always node-backlogged such that traffic enters the scheduler buffer at the rate controllers' rates. Observe that the scheduler buffer occupancy increases in under-throttled situation. However, while an over-throttled situation may result in a flow being under-served, it may also be over-served if the flow has traffic queued previously. [0177] B. Fairness Bound [0178] To characterize the deviation of DVSR from the reference model for the above scenario, we first derive an upper bound on the total amounts of over- and under-throttled traffic as a function of the averaging interval T. [0179] For notational simplicity, we consider fixed size packets such that time is slotted, and denote v(k) as the virtual time at time kT. Moreover, let b(k) denote the total non-idle time in the interval [kT, (k+1)T] and denote the number of flows (representing ingress nodes) by N. The bound for under-throttled traffic is derived as follows. [0180] Lemma 1: A node-backlogged flow in DVSR can be under throttled by at most (1/−1/N)CT. [0181] Proof: For a node-backlogged flow i, an under-throttled situation occurs when the fair rate decreases, since the flow will temporarily be throttled using the previous higher rate. In such a case, the average slope of v(t) decreases between times kT and (k+1)T. For a system with N flows, the worst case of under-throttling occurs when the slope repeatedly decreases for N consecutive periods of duration T. Otherwise, if the fair rate increases, flow i will be over throttled, and the occupancy of the scheduler buffer is decreasing during that period. Thus, assuming flow i enters the system at time 0, and denoting U [0182] since v(k+1)−v(k) is the total service obtained during slot kT for flow i as well as the total throttled traffic for slot (k+1)T. The last step holds because for a flow with infinite demand, v(k)−v(k−1) is between 1/N CT and CT during an under-throttled period. [0183] Similarly, the following lemma establishes the bound for the over-throttled case. Lemma 2: A node-backlogged flow in DVSR can be over throttled by at most (1−1/N)CT. [0184] Proof: For a node backlogged flow i, over throttling occurs when the available fair rate increases. In other words, a flow will be over throttled when the average slope of v(t) increases from kT to (k+1)T. The worst case is when this occurs for N consecutive periods of duration T. For over-throttled situations, the server can potentially be idle. According to DVSR, the total throttled amount for time slot (k+1) will be v(k+1)−v(k)+(1−b(k))CT. Thus, assuming flow i enters the system at time 0, and denoting O [0185] where the last step holds since (v(k)−v(k−1)+(1−b(k−1))CT is no less than 1/N CT. [0186] Lemmas 1 and 2 are illustrated in FIG. 15. Let f(t) (labelled “fair share”) denote the cumulative (averaged) fair share for flow i in each time slot given the requirements in this time slot. Let p(t) (labelled “rate controller”) denote the throttled traffic for flow i. Lemmas 1 and 2 specify that p(t) will be within the range of (1−1/N)CT of f(t). [0187] Furthermore, let s(t) (labelled “service obtained”) denote the cumulative service for flow i. Then DVSR guarantees that if flow i has infinite demand, s(t) will not be less than f(t)−(1−1/N)CT. This can be justified as follows. As long as s(t) is less than p(t) (i.e., flow i is scheduler backlogged), flow i is guaranteed to obtain a fair share of service. Hence, the slope of s(t) will be no less than that of f(t). Otherwise, flow i would be in an over-throttled situation, and s(t)=p(t), and from Lemma 2, p(t) is no less than f(t)−(1−1/N)CT. Also notice that s(t) can be no larger than p(t), so that the service s(t) for flow i is within the range of (1−1/N)CT of f(t) as well. [0188] From the above analysis, we can easily derive a fairness bound for two flows with infinite demand as follows. [0189] Lemma 3: The service difference during any interval for two flows i and j with infinite demand is bounded by 2(C−1/N C)T under DVSR. [0190] Proof: Observe that scheduler-backlogged flows will get no less than their fair shares due to the GPS scheduler. Therefore, for an under-throttled situation, each flow will receive no less than its fair share. Hence, unfairness only can occur during over-throttling. In such a scenario, a flow can only obtain additional service of its under-throttled amount. On the other hand, a flow can at most be under-served by its over-throttled amount. From Lemmas 1 and 2, this amount can at most 2(C−1/N C)T . [0191] Finally, note that for the special case of T=0, the bound goes to zero so that DVSR achieves perfect fairness without any over/under throttling. [0192] C. Discussion [0193] The above methodology can be extended to multiple DVSR nodes in which each flow has one node buffer (at the ingress point) but multiple scheduler buffers. In this case, under-throttled traffic may be distributed among multiple scheduler buffers. On the other hand, for multiple nodes, to maximize spatial reuse, DVSR will rate control a flow at the ingress node using the minimum throttling rate from all the links. By substituting the single node-throttling rate with the minimum rate among all links, From Lemmas 1 and 2 can be shown to hold for the multiple node case as well. [0194] Despite the simplified scenario for the above analysis, it does provide a simple if idealized fairness bound of 2(C−1/N C)T. For a 1 Gb/sec ring with 64 nodes and T=0.5 msec, this corresponds to a moderate maximum unfairness of 125 kB, i.e., 125 kB bounds the service difference between two infinitely backlogged flows under the above assumptions. [0195] VII. SIMULATION EXPERIMENTS [0196] In this section, we use simulations to study the performance of DVSR and provide comparisons with the RPR fairness algorithm. Moreover, as a baseline we compare with a Gigabit Ethernet (GigE) Ring that has no distributed bandwidth control algorithm and simply services arriving packets in first-in first-out order. [0197] We divide our study into two parts. First, we study DVSR in the context of the basic RPR goals of achieving spatial reuse and fairness. We also explore interactions between TCP congestion control and DVSR's RIAS fairness objectives. Second, we compare the convergence times of DVSR and RPR. [0198] We do not further consider scenarios with unbalanced traffic that result in oscillation and throughput degradation for RPR as treated in Section IV. [0199] All simulation results are obtained with our publicly available ns-2 implementations of DVSR and RPR. Unless otherwise specified, RPR simulations refer to the default Aggressive Mode. We consider 622 Mbps links (OC-12), 200 kB buffer size, 1 kB packet size, and 0.1 msec link propagation delay between each pair of nodes. For a ring of N nodes, we set T to be 0.1 N msec such that one DVSR control packet continually circulates around the ring. [0200] A. Fairness and Spatial Reuse [0201] A.1 Fairness in the Parking Lot [0202] We first consider the parking lot scenario with a ten-node ring as depicted in FIG. 5 and widely studied in the RPR standardization process. Four constant-rate UDP flows ( [0203] We make the following observations about the figure. First, DVSR as well as RPR-AM and RPR-CM (not depicted) all achieve the correct RIAS fair rates ({fraction (622/4)}) to within ±1%. In contrast, without the coordinated bandwidth control of the RPR algorithms, GigE fails to ensure fairness, with flow ( [0204] A.2 Performance Isolation for TCP Traffic [0205] Unfairness among congestion-responsive TCP flows and non-responsive UDP flows is well established. However, suppose one ingress node transmits only TCP traffic whereas all other ingress nodes send high rate UDP traffic. The question is whether DVSR can still provide RIAS fair bandwidth allocation to the node with TCP flows, i.e., can DVSR provide inter-node performance isolation? An important issue is whether DVSR's reclaiming of unused capacity to achieve spatial reuse will hinder the throughput of the TCP traffic. [0206] To answer this question, we consider the same parking lot topology of FIG. 5 and replace flow ( [0207] Ideally, the TCP traffic would obtain throughput 0.25, which is the RIAS fair rate between nodes [0208] A.3 RIAS vs. Proportional Fairness for TCP Traffic [0209] Next, we consider the case that each of the four flows in the parking lot is a single TCP micro-flow, and present the corresponding throughputs for DVSR and GigE in FIG. 18. As expected, with a GigE ring the flows with the fewest number of hops and lowest round trip time receive the largest bandwidth shares (cf. Section III). However, DVSR seeks to eliminate such spatial bias and provide all ingress nodes with an equal share. For DVSR and a single flow per ingress this is achieved to within approximately ±8%. This margin narrows to ±1% by 10 TCP micro-flows per ingress node (not shown). Thus, with sufficiently aggregated TCP traffic, a DVSR ring appears as a single node to TCP flows such that there is no bias to different RTTs. [0210] A.4 Spatial Reuse in the Parallel Parking Lot [0211] We now consider the spatial reuse scenario of the Parallel Parking Lot (FIG. 2) again with each flow offering traffic at the full link capacity (and hence, “balanced” traffic load). As described in Section III, the rates that achieve IA fairness while maximizing spatial reuse are 0.25 for all flows except flow ( [0212]FIG. 19 shows that the average throughput for each flow for DVSR is within ±1% of the RIAS fair rates. RPR-AM and RPR-CM can also achieve these ideal rates within the same range when using the per-destination queue option. In contrast, as with the Parking Lot example, GigE favors downstream flows for the bottleneck link [0213] B. Convergence Time Comparison [0214] In this experiment, we study the convergence times of the algorithms using the parking lot topology and UDP flows with normalized rate 0.4 (248.8 Mbps). The flows' starting times are staggered such that flows ( [0215]FIG. 20 depicts the throughput over windows of duration T for the three algorithms. Observe that DVSR converges in two ring times, i.e., 2 msec, whereas RPR-AM takes approximately 50 msec to converge, and RPR-CM takes about 18 msec. Moreover, the range of oscillation during convergence is significantly reduced for DVSR as compared to RPR. However, note that the algorithms have a significantly different number of control messages. RPR's control update interval is fixed to 0.1 msec so that RPR-AM and RPR-CM have received 180 and 500 respective control messages before converging. In contrast, DVSR has received 2 control messages. [0216] For each of the algorithms, we also explore the sensitivity of the convergence time to the link propagation delay and feedback update time. We find that in both cases, the relationships are largely linear across the range of delays of interest for metropolitan networks. For example, with link propagation delays increased by a factor of 10 so that the ring time is 10 msec, DVSR takes approximately 22 msec to converge, slightly larger than 2T. [0217] Finally, we note that RPR algorithms differ significantly in their ability to achieve spatial reuse with unbalanced traffic. As described in Section IV, RPR-AM and RPR-CM suffer from permanent oscillations and throughput degradation in cases of unbalanced traffic. In contrast DVSR achieves rates within 0.1% of the RIAS rates in simulations of all unbalanced scenarios presented in Section IV. [0218] VIII. NETWORK PROCESSOR IMPLEMENTATION [0219] The logic of each node's dynamic bandwidth allocation algorithm depicted in FIG. 3 may be implemented in custom hardware or in a programmable device such as a Network Processor (NP). We adopt the latter approach for its feasibility in an academic research lab as well as its flexibility to re-program and test algorithm variants. In this section, we describe our implementation of DVSR on a 2 Gb/sec Network Processor testbed. The DVSR algorithm is implemented in assembly language in the NP, utilizing the rate controllers and output queuing system of the NP in the same way that a hardware-only implementation would. The result allows an accurate emulation of DVSR behavior in a realistic environment. DVSR assembly language modules are available at http://www.ece.rice.edu/networks/DVSR. [0220] A. NP Scenario [0221] The DVSR implementation is centered around a Vitesse IQ2000™ NP, which is available from Vitesse Semiconductor Corporation of Camarillo, Calif. The IQ2000™ has four 200 MHz 32-bit RISC processing cores, each running four user contexts and including 4 KB of local memory. This allows up to 16 packets to be processed simultaneously by the NP. For communication interfaces, it has four 1 Gbps input and output ports with eight communication channels each, one of which is connected to an eight port 100 Mbps Ethernet MAC (also available from Viesse Semiconductor Corporation). Its memory capacity is 256 MB of external DRAM memory and 4 MB of external SRAM memory. [0222] As described in Section V, the inputs to the DVSR bandwidth control algorithm are byte counts of arriving packets. In the NP, these byte counts are kept per destination for station traffic and per ingress for transit traffic, and are updated with each packet arrival and stored in SRAM. Using these measurements as inputs, the main steps to computing the IA fair bandwidth as given in Table I are written in a MIPS-like assembly language and performed by the RISC processors. [0223] In our implementation, a single control packet circulates continuously around the ring. The control packet contains N 1-byte virtual-time fair rate values F [0224] The output modules for each of the ports contain eight hardware queues per output channel, and each of these queues can be assigned a separate rate limit. Hence, for our 8-node ring, we use these hardware rate limiters to adaptively shape station traffic according to the fairness computation by writing the computed values of the station throttling rates to the output module. [0225] Finally, on the data path, the DRAM of the NP contains packet buffers to hold data on the output queues, with a separate queue for transit vs. station traffic, and transit traffic scheduled alternately with the rate-limited station traffic. [0226] Thus, considering the generic RPR node architecture of FIG. 3, the dynamic bandwidth allocation algorithm and forwarding logic is programmed on the NP, and all other components are hardware. On the transit path, the DVSR rate calculation algorithm is implemented in approximately 171 instructions. Moreover, the logic for nodes to compute their ingress rate controller values given a received control signal contains approximately 40 instructions, plus 37 to write the values to hardware. These operations are executed every T seconds. In our implementation, the NP also contains forwarding logic that increases the NP workload. [0227] B. Testbed [0228] In our testbed configuration [0229] As illustrated in FIG. 21, the eight Ethernet interfaces of the MAC 2102 are connected to port C provide the eight station connections. Each connection (C [0230] There are several factors in the emulation that may differ from the behavior of a true packet ring. Since the “connections” between nodes are wires within a single chip, the link propagation delay is negligible. In order to have increased latency as in a realistic scenario, the emulation includes a mechanism for delaying a packet by a tightly controlled amount of time before it is transmitted. In the experiments below, we have set these values such that the total ring propagation delay (and hence 7) is 0.6 msec. [0231] Since all nodes reside in the same physical chip, all information (particularly the rate counters) is accessible to the emulation of all nodes. However, to ensure accurate emulation, all external memory accesses are indexed by the number of the current node, and all control information is read and written to the control packet only. [0232] C. Results [0233] We performed experiments in two basic scenarios: the parking lot and unbalanced traffic. For the parking lot experiments, we first use an 8-node ring and configure a parking lot scenario with 2 flows originating from nodes [0234] In future work, we plan to configure the testbed with 1 Gb/sec interfaces and perform a broader set of experiments to study the impact of different workloads (including TCP flows), configurations (including the Parallel Parking Lot), and many of the scenarios explored in Section VII. [0235] IX. RELATED WORK [0236] The problem of devising distributed solutions to achieve high utilization, spatial reuse, and fairness is a fundamental one that must be addressed in many networking control algorithms. Broadly speaking, TCP congestion control achieves these goals in general topologies. However, as demonstrated in Section VII, a pure end-point solution to bandwidth allocation in packet rings results in spatial bias favoring nodes closer to a congested gateway. Moreover, end-point solutions do not provide protection against misbehaving flows. In addition, the goals of RPR are quite different than TCP: to provide fairness at the ring ingress-node granularity vs. TCP micro-flow granularity; to provide rate guarantees in addition to fairness, etc. Similarly, ABR rate control, and other distributed fairness protocols can achieve max-min fairness, and as with TCP, provides a natural mechanism for spatial reuse. However, packet rings provide a highly specialized scenario (fixed topology, small propagation delays, homogeneous link speeds, a small number of IA flows, etc.) so that algorithms can be highly optimized for this environment, and avoid the longer convergence times and complexities associated with end-to-end additive-increase multiplicative-decrease protocols. [0237] The problem also arises in specialized scenarios such as wireless ad hoc networks. Due to the finite transmission range of wireless nodes, spatial reuse can be achieved naturally when different sets of communicating nodes are out of transmission range of one another. However, achieving spatial reuse and high utilization is at odds with balancing the throughputs of different flows and hence in achieving fairness. Distributed fairness and medium access algorithms to achieve max-min fairness and proportional fairness can be found in the prior art. While sharing similar core issues as RPR, such solutions are unfortunately quite specialized to ad hoc networks and are not applicable in packet rings, as the schemes exploit the broadcast nature of the wireless medium. [0238] Achieving spatial reuse in rings is also a widely studied classical problem in the context of generalizing token ring protocols. A notable example is the MetaRing protocol, which we briefly describe as follows. MetaRing attained spatial reuse by replacing the traditional token of token rings with a ‘SAT’ (satisfied) message designed so that each node has an opportunity to transmit the same number of packets in a SAT rotation time. In particular, the algorithm has two key threshold parameters K and L, K=L. A station is allowed to transmit up to K packets on any empty slot between receipt of any two SAT messages (i.e., after transmitting K packets, a node cannot transmit further until receiving another SAT message.) Upon receipt of the SAT message, if the station has already transmitted L packets, it is termed “satisfied” and forwards the SAT message upstream. Otherwise, if the node has transmitted fewer than L packets and is backlogged, it holds the SAT message until L packets are transmitted. While providing significant throughput gains over token rings, the coarse granularity of control provided by holding a SAT signal limits such a technique's applicability to RPR. For example, the protocol's fairness properties were found to be highly dependent on the parameters K and L as well as the input traffic patterns; the SAT rotation time is dominated by the worst case link prohibiting full spatial reuse; etc. [0239] X. CONCLUSIONS [0240] In this discussion, we presented Distributed Virtual-time Scheduling in Rings, a dynamic bandwidth allocation algorithm targeted to achieve high utilization, spatial reuse, and fairness in packet rings. We showed through analysis, simulations, and implementation that DVSR overcomes limitations of the standard RPR algorithm and fully exploits spatial reuse, rapidly converges (typically within two ring times), and closely approximates our idealized fairness reference model, RIAS. Finally, we note that RIAS and the DVSR algorithm can be applied to any packet ring technology. For example, DVSR can be used as a separate fairness mode for RPR or as a control mechanism on top of Gigabit Ethernet used to ensure fairness in Metro Ethernet rings. [0241] The invention, therefor, is well adapted to carry out the objects and to attain the ends and advantages mentioned, as well as others inherent therein. While the invention has been depicted, described and is defined by reference to exemplary embodiments of the invention, such references do not imply a limitation on the invention, and no such limitation is to be inferred. The invention is capable of considerable modification, alternation and equivalents in form and function, as will occur to those ordinarily skilled in the pertinent arts and having the benefit of this disclosure. The depicted and described embodiments of the invention are exemplary only, and are not exhaustive of the scope of the invention. Consequently, the invention is to be limited only by the spirit and scope of the appended claims, giving full cognizance to equivalents in all respects. Patent Citations
Referenced by
Classifications
Legal Events
Rotate |