|Publication number||US7426570 B2|
|Application number||US 10/627,324|
|Publication date||Sep 16, 2008|
|Filing date||Jul 25, 2003|
|Priority date||Jul 25, 2003|
|Also published as||US20050021831|
|Publication number||10627324, 627324, US 7426570 B2, US 7426570B2, US-B2-7426570, US7426570 B2, US7426570B2|
|Inventors||Artur Andrzejak, Sven Graupner|
|Original Assignee||Hewlett-Packard Development Company, L.P.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (23), Non-Patent Citations (13), Referenced by (10), Classifications (9), Legal Events (3)|
|External Links: USPTO, USPTO Assignment, Espacenet|
The following applications disclose related subject matter: U.S. application Ser. No. 10/627,274, filed Jul. 25, 2003 and entitled, “Determination of One or More Variables to Receive Value Changes in Local Search Solution of Integer Programming Problem”; and U.S. application Ser. No. 10/627,883, filed Jul. 25, 2003 and entitled, “Incorporating Constraints and Preferences for Determining Placement of Distributed Application onto Distributed Resource Infrastructure”; the contents of all of which are hereby incorporated by reference.
The present invention relates to the field of placing a distributed application onto a distributed resource infrastructure. More particularly, the present invention relates to the field of placing a distributed application onto a distributed resource infrastructure where the distributed application and the distributed resource infrastructure have arbitrary communication topologies.
A distributed application includes a plurality of services. Each of the services performs a task or tasks as part of the distributed application. Often the distributed application is placed on a network of computers. The network of computers forms a distributed resource infrastructure where each of the computers forms a node. Performance of the distributed application depends on optimizing a placement of the services onto the nodes.
A first method of the prior art uses parameters for individual nodes to determine a placement of the services onto the nodes. Such parameters include processing and storage capabilities of the nodes.
The first method does not consider relationships among the nodes or among the services in the determination of the placement of the services onto the nodes. A second method of the prior art considers topologies between the services and between the nodes but requires that the topologies be fixed in certain configurations. The second method does not determine a placement of the services onto the nodes where an arbitrary topology exists between the nodes or between the services.
What is needed is a method of determining a placement of services of a distributed application onto nodes of a distributed resource infrastructure taking into account arbitrary topologies between the nodes and between the services.
The present invention is a method of determining a placement of services of a distributed application onto nodes of a distributed resource infrastructure. In an embodiment of the present invention, the method comprises first, second, and third steps. The first step forms communication constraints between node pairs. The communication constraints ensure that a sum of transport demands between a particular node pair does not exceed a transport capacity between the particular node pair. Each term of the sum comprises a product of a first placement variable, a second placement variable, and the transport demand between the services associated with the first and second placement variables. The second step forms an objective. The communication constraints and the objective comprise an integer program. The third step employs a local search solution to solve the integer program, which determines the placement of the services onto the nodes.
These and other aspects of the present invention are described in more detail herein.
The present invention is described with respect to particular exemplary embodiments thereof and reference is accordingly made to the drawings in which:
The present invention determines a placement of a distributed application onto a distributed resource infrastructure. The distributed application comprises a plurality of services. The distributed resource infrastructure comprises a plurality of nodes.
A distributed application embodiment is illustrated schematically in
A distributed resource infrastructure embodiment is illustrated schematically in
A preferred method of the present invention is illustrated as a block diagram in
A first alternative method of the present invention is illustrated as a block diagram in
An alternative distributed application embodiment is illustrated schematically in
Since services do not communicate with themselves over a network, the transport demands along a matrix diagonal have no values. Further, depending upon a particular implementation it may be sufficient to characterize the transport demands without reference to direction in which case the transport demands below the matrix diagonal would also have no values.
Each of the first through Sth services, 501 . . . 505, of the alternative distributed application 500 is also characterized with a processing demand and a storage demand. For example, the first service 501 has a first processing demand dp1 and a first storage demand ds1. A processing demand vector Dp and a storage demand vector Ds list the processing demands and the storage demands of the first through Sth servers, 501 . . . 505, as follows.
An alternative distributed resource infrastructure is illustrated schematically in
Since nodes do not communicate with themselves over a network, the transport capacities along a matrix diagonal have no values. Further, depending upon a particular implementation it may be sufficient to characterize the transport capacities without reference to direction in which case the transport capacities below the matrix diagonal would also have no values.
Each of the first through Nth nodes, 601 . . . 605, of the alternative distributed resource infrastructure 600 is also characterized with a processing capacity and a storage capacity. For example, the first node 601 has a first processing capacity cp1 and a first storage capacity cs1. A processing capacity vector Cp and a storage capacity vector Cs list the processing capacities and the storage capacities of the first through Nth nodes, 601 . . . 605, as follows.
In some situations, the distributed application under consideration operates solely on the distributed resource infrastructure. In this situation, the transport, processing, and storage capacities represent absolute capacities for the nodes. In other situations, the distributed application is one of a plurality of distributed applications operating on the distributed resource infrastructure. In these other situations, the transport, processing, and storage capacities represent available capacities for the nodes.
In the alternative distributed application embodiment 500 and the alternative distributed resource infrastructure embodiment 600, the transport and storage demands, Dt and Ds, as well as the transport and storage capacity, Ct and Cs, are normalized according to standard parameters of data per unit time and data, respectively. In an embodiment of the present invention, the processing demand Dp and the processing capacity Cp are normalized according to a processing criterion. Preferably, the processing criterion is a transaction speed especially when the distributed application forms a database application. Alternatively, the processing criterion is an algorithm speed. In another embodiment of the present invention, various processors are listed in a look-up table with associated processing capacities that have been normalized by the processing criterion. In this embodiment, a system implementing the present invention would go to the look-up table to find the processing capacity for a particular node when needed.
Applying the first alternative method 400 (
The second task forms the communication constraints according to a communication constraint equation, which is given as follows.
The second step 304 forms the objective, which according to an embodiment of the present invention minimizes communication traffic between the nodes and balances processing loads on the nodes. The latter is accomplished by minimizing the mathematical variance of the processing loads. The objective is given as follows.
for where α provides a relative weight between minimizing the communication traffic and balancing the processing loads, A provides a normalizing factor, Prox1j accounts for distances between the nodes, and N is a number of the nodes. The third step 306 then employs the local search solution to solve the integer program comprising the communication constraints and the objective.
Since the communication constraints account for a distributed application topology according to the transport demands and for a distributed resource infrastructure topology according to the transport capacities, the present invention allows arbitrary topologies for both the distributed application and the distributed resource infrastructure. This allows the present invention to be used for determining placement of applications onto infrastructures in wide variety of situations. Examples of such situations include placing an application onto nodes in a data center and placing a different application onto nodes in geographically distributed data centers.
A second alternative method of the present invention adds processing constraints to the integer program. The processing constraints ensure that a sum of the processing demands for a specific node does not exceed the processing capacity of the specific node. In an embodiment of the present invention, the processing constraints are formed according to a processing constraint equation, which is given as follows.
A third alternative method of the present invention adds storage constraints to the integer program. The storage constraints ensure that a sum of the storage demands for a specific node does not exceed the storage capacity of the specific node. In an embodiment of the present invention, the storage constraints are formed according to a storage constraint equation, which is given as follows.
A fourth alternative method of the present invention adds placement constraints to the integer program. The placement constraints ensure that each of the services is placed on one and only one of the nodes. In an embodiment of the present invention, the placement constraints are formed according to a placement constraint equation, which is given as follows.
A fifth alternative method of the present invention recognizes that, once the services have been placed onto the nodes, a rearrangement of the services onto the nodes comes at a cost. In the fifth alternative method, reassignment penalties are assessed when a service placement differs from an existing assignment of the service. According to an embodiment of the fifth alternative method, a second objective is added to the integer program. The second objective seeks to minimize the reassignment penalties.
In the present invention, the communication constraints include terms which comprise products of two placement variables. Depending on the embodiment of the present invention, the objective also includes products of two placement variables. Thus, the communication constraints and possibly the objective are quadratic equations, i.e., equations having polynomial terms of second order. Solving integer programs that include polynomial terms of second or higher order is particularly difficult. In an embodiment of the present invention, a local search solution is employed according to a local search solution method disclosed in U.S. patent application Ser. No. 10/627,274 filed on Jul. 25, 2003, which is incorporated by reference in its entirety.
An embodiment of the local search solution method is illustrated as a flow chart in
The first solution step 702 defines a problem model as an overconstrained integer programming problem. In an embodiment of the present invention, the problem model comprises data, variables, and constraints. The data comprises the processing demands and capacities, the storage demands and capacities, and the transport demands and capacities. The variables comprise the placement variables, which are Boolean variables where a zero value indicates that a particular service is not located on a particular node and where a one value indicates that the particular service is located on the particular node. In the overconstrained integer programming problem, the constraints comprise hard constraints and at least one soft constraint. The hard constraints comprise the processing constraints, the storage constraints, the placement constraints, and the storage constraints. The soft constraint is the objective, which comprises minimizing the communication traffic between the nodes and balancing the processing loads on the nodes.
In the second solution step 704, the placement variables are randomly initialized. The third solution step 706 selects an unsatisfied constraint. The fourth solution step 708 creates stores in memory for each of the placement variables in the unsatisfied constraint. The fifth solution step 710 parses the unsatisfied constraint by term. For each of the placement variables in the term, an associated store is updated with a change in the unsatisfied constraint due to flipping the value of the placement variable while holding other placement variables constant. In the sixth solution step 712, the placement variable that is to have its value flipped is chosen according to an improvement criterion, such as the placement variable which most improves the unsatisfied constraint or the placement variable which most improves an overall solution while also improving the unsatisfied constraint.
In the seventh solution step 714, assigned values are compared to optimality criteria to determine whether a solution has been found. The optimality criteria for the overconstrained integer programming problem are no violation of the hard constraints and a near optimum solution for the soft constraint. If the optimality criteria are not met, the local search solution method 700 continues in the eighth solution step 716 with a determination of whether an additional iteration is to be performed. If so, the local search solution method 700 returns to the third solution step 706 of selecting another unsatisfied constraint to determine another placement variable which is to have its value flipped. If not, a ninth solution step 718 determines whether to restart the local search solution method 700 by reinitializing the variables. If the optimality criteria are met in the seventh solution step 714, a final value assignment for the placement variables is output as a result in the tenth solution step 720. If the ninth step 210 determines to not restart the first alternative method 200, a “no solution found” message is output in the tenth solution step 720.
Preferably, the local search solution method is implemented using AMPL, a modeling language for optimization problems. Alternatively, the local search solution method 700 is implemented using an alternative modeling language.
An embodiment of a system for solving an integer program of the present invention is illustrated schematically in
Employing the system 800 to solve the integer program according to the local search solution method 700 begins with the processing unit 806 reading the problem model into the memory 810. The processing unit 806 then initializes the variables by randomly assigning values to the variables. Next, the processing unit 806 randomly selects the unsatisfied constraint. Following this, the processing unit 806 creates stores in the memory 810 for the allowable changes of the variables in the unsatisfied constraint. The processing unit 806 then parses the unsatisfied constraint by term updating individual stores associated with the term while maintaining other variables constant. Following this, the processing unit 806 chooses the placement variable to have its value flipped according to the improvement criterion. The processing unit 806 then determines whether the optimality condition has been met and, if not, determines whether to perform more iterations or whether the local search solution method 700 should be restarted.
In an embodiment of the present invention, computer code resides on a computer readable memory, which is read into the system 800 by one of the input/output devices 804. Alternatively, the computer readable memory comprises the storage device 808 or the memory 810. The computer code provides instructions for the processing unit 806 to form the problem model and to solve it according to an embodiment of the present invention. The computer readable memory is selected from a group including a disk, a tape, a memory chip, or other computer readable memory.
The foregoing detailed description of the present invention is provided for the purposes of illustration and is not intended to be exhaustive or to limit the invention to the embodiments disclosed. Accordingly, the scope of the present invention is defined by the appended claims.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US5511167 *||Feb 14, 1991||Apr 23, 1996||Hitachi, Ltd.||Program processing method and apparatus for producing a data flow type program|
|US5734825 *||Jul 18, 1994||Mar 31, 1998||Digital Equipment Corporation||Traffic control system having distributed rate calculation and link by link flow control|
|US5878224 *||Apr 30, 1997||Mar 2, 1999||Bell Communications Research, Inc.||System for preventing server overload by adaptively modifying gap interval that is used by source to limit number of transactions transmitted by source to server|
|US5918021 *||Jun 3, 1996||Jun 29, 1999||Intel Corporation||System and method for dynamic distribution of data packets through multiple channels|
|US5993038 *||Nov 17, 1995||Nov 30, 1999||Bull S.A.||Distributed application load distribution aid tool|
|US6031984||Mar 9, 1998||Feb 29, 2000||I2 Technologies, Inc.||Method and apparatus for optimizing constraint models|
|US6032188 *||Mar 12, 1997||Feb 29, 2000||Microsoft Corporation||Method and system for controlling data flow|
|US6052712 *||Nov 24, 1997||Apr 18, 2000||International Business Machines Corporation||System for barrier synchronization wherein members dynamic voting controls the number of synchronization phases of protocols and progression to each subsequent phase|
|US6125397 *||Jun 3, 1998||Sep 26, 2000||Fuji Xerox Co., Ltd.||Data transfer apparatus and method using congestion recovery-type and congestion avoidance-type data transfers|
|US6125400 *||Nov 10, 1997||Sep 26, 2000||International Business Machines Corporation||Method and system of running object oriented programs across a network through compression and distillation of remote method invocation|
|US6167029 *||Oct 26, 1998||Dec 26, 2000||Xaqti Corporation||System and method for integrated data flow control|
|US6366931 *||Nov 20, 1998||Apr 2, 2002||Hewlett-Packard Company||Apparatus for and method of non-linear constraint optimization in storage system configuration|
|US6473801 *||Mar 23, 1999||Oct 29, 2002||Lsi Logic Corporation||System and method for arbitrating bandwidth on segmented network topologies|
|US6507844 *||Nov 12, 1999||Jan 14, 2003||International Business Machines Corporation||Method and system for minimizing network traffic|
|US6526420||Nov 16, 2001||Feb 25, 2003||Hewlett-Packard Company||Non-linear constraint optimization in storage system configuration|
|US6574669 *||Aug 31, 1998||Jun 3, 2003||Nortel Networks Limited||Method and apparatus for routing traffic within a network utilizing linear optimization|
|US6782527 *||Aug 30, 2000||Aug 24, 2004||Networks Associates, Inc.||System and method for efficient distribution of application services to a plurality of computing appliances organized as subnets|
|US6834310 *||Feb 26, 2002||Dec 21, 2004||Science Applications International Corp.||Preventing packet flooding of a computer on a computer network|
|US6928482 *||Jun 29, 2000||Aug 9, 2005||Cisco Technology, Inc.||Method and apparatus for scalable process flow load balancing of a multiplicity of parallel packet processors in a digital communication network|
|US6978394 *||Feb 22, 2002||Dec 20, 2005||Cisco Technology, Inc.||Linear program-based technique for placing FRR TE tunnels with bandwidth guarantee|
|US7089299 *||Jan 15, 2002||Aug 8, 2006||International Business Machines Corporation||Distributed application deployment using program characteristics and environment characteristics|
|US20030005132 *||May 16, 2001||Jan 2, 2003||Nortel Networks Limited||Distributed service creation and distribution|
|US20030055868 *||Sep 19, 2001||Mar 20, 2003||International Business Machines Corporation||Building distributed software services as aggregations of other services|
|1||Artur Andrzejak, Sven Graupner, Vadim Kotov, Holger Trinks, Algorithms for Self-Origanization and Adaptive Service Placement in Dynamic Distributed Systems, HP Labs Technical Report, HPL-2002-259, Hewlett-Packard Company, Palo Alto, Sep. 2002. <http://www.hpl.hp.com/techreports/2002/HPL-2002-259.pdf>.|
|2||Bart Selman, Greedy Local Search in MIT Encyclopedia of the Cognitive Sciences, MIT Press, Cambridge, 1999.|
|3||Bart Selman, Hector Levesque, David Mitchell, A New Method for Solving Hard Satisfiability Problems, Proceedings of the Tenth National Conference on Artificial Intelligence Jul. 12-16, 1992, San Jose, California, pp. 440-446, AAAI Press, Menlo Park, 1992.|
|4||Bart Selman, Henry A. Kautz, Bram Cohen, Local Search Strategies for Satifiabily Testing Presented at the Second DIMACS Challenge on Cliques, Coloring, and Satisfiability, Oct. 11-13, 1993 Piscataway, New Jersey.|
|5||Bart Selman, Henry A. Kautz, Bram Cohen, Noise Strategies for Improved Local Search, Proceedings of the Twelfth National Conference on Artificial Intelligence, Aug. 1-4, 1994, Seattle, Washington, pp. 337-343, AAAI Press, Menlo Park, 1994.|
|6||Christodoulous A. Floudas, V. Visweswaran, Quadratic Optomization, Handbook of Global Optimization, pp. 217-269, Kluwer Academic Publishers, Boston, 1995.|
|7||Ciprianpo Santos, Xiaoyun Zhu, Harlan Crowder, A Mathematical Optimization Approach for Resource Allocation in Large Scale Data Centers, HP Labs Technical Report, HPL-2002-64R1, Hewlett-Packard Company, Palo Alto, 2002. <http://www.hpl.hp.com/techreports/2002/HPL-2002-64R1.pdf>.|
|8||David Abramson, Henry Dang, Mohan Krishnamoorthy, A Comparison of Two Methods for Solving 0-1 Integer Programs Using a General Purpose Simulated Annealing Algorithim, Annals of Operations Research, v. 63, pp. 129-150, Baltzer Science, Amsterdam, Netherlands, 1996.|
|9||Joachim P. Walser, Solving Linear Pseudo-Boolean Constraint Problems with Local Search, Proceedings of the Fourteenth National Conference on Artificial Intelligence and the Ninth Innovative Applications of Artificial Intelligence Conference, Jul. 27-31, 1997, Providence, Rhode Island, pp. 269-274, AAAI Press, Menlo Park, 1997.|
|10||Joachim Paul Walser, Domain-Independent Local Search for Linear Integer Optomization, Dissertation, Progamming Systems Lab, Universitat de Saarlandes, Saarbrucken, Germany, Oct. 1998.|
|11||Parameswaran Ramanathan, Suresh Chalasani, Resource Placement with Multiple Adjacency Constraints in K-ary n-Cubes, IEEE transactions on Parallel & Distributed Systems, May 1995, vol. 6, No. 5, pp. 511-519, IEEE Press, Los Alimitos, 1995.|
|12||Sven Graupner, Vadim Kotov, Artur Andrsejak, Holger Trinks, Control Architecture for Service Grids in a Federation of Utility Data Centers, HP Labs Technical Report HPL-2002-235, Hewlett-Packard Company, Palo Alto, 2002. <http://www.hpl.hp.com/techreports/2002/HPL-2002-235.pdf>.|
|13||Sven Graupner, Vadim Kotov, Holger Trinks, Resource-Sharing and Service Deployment in Virtual Data Centers, The 22nd International Conference on Distributed Computing Systems Workshops, Vienna Austria, Jul. 2-5, 2002, pp. 666-671, IEEE Computer Society, Los Alamitos, 2002.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7788379 *||Aug 7, 2007||Aug 31, 2010||Fujitsu Limited||Network system and information processing method|
|US8281012||Jan 30, 2008||Oct 2, 2012||International Business Machines Corporation||Managing parallel data processing jobs in grid environments|
|US8370490||Jul 1, 2010||Feb 5, 2013||International Business Machines Corporation||Cloud service cost-optimal data center assignment|
|US8380960||Nov 4, 2008||Feb 19, 2013||Microsoft Corporation||Data allocation and replication across distributed storage system|
|US8589530 *||Mar 28, 2005||Nov 19, 2013||Riverbed Technology, Inc.||Method and system for managing a distributed network of network monitoring devices|
|US8856386 *||Aug 21, 2012||Oct 7, 2014||Cisco Technology, Inc.||Cloud resource placement using placement pivot in physical topology|
|US8935702||Sep 4, 2009||Jan 13, 2015||International Business Machines Corporation||Resource optimization for parallel data integration|
|US8954981||Feb 24, 2012||Feb 10, 2015||International Business Machines Corporation||Method for resource optimization for parallel data integration|
|US20060253566 *||Mar 28, 2005||Nov 9, 2006||Dimitris Stassinopoulos||Method and system for managing a distributed network of network monitoring devices|
|US20140059178 *||Aug 21, 2012||Feb 27, 2014||Cisco Technology, Inc.||Cloud resource placement using placement pivot in physical topology|
|U.S. Classification||709/235, 709/232, 709/226, 709/230, 709/223|
|International Classification||G06F9/50, G06F15/16|
|Oct 2, 2003||AS||Assignment|
Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P., TEXAS
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ANDRZEJAK, ARTUR;GRAUPNER, SVEN;REEL/FRAME:014023/0226;SIGNING DATES FROM 20030724 TO 20030725
|Jul 21, 2009||CC||Certificate of correction|
|Sep 23, 2011||FPAY||Fee payment|
Year of fee payment: 4