|Publication number||US7331048 B2|
|Application number||US 10/406,985|
|Publication date||Feb 12, 2008|
|Filing date||Apr 4, 2003|
|Priority date||Apr 4, 2003|
|Also published as||US8136115, US20040199918, US20080052715|
|Publication number||10406985, 406985, US 7331048 B2, US 7331048B2, US-B2-7331048, US7331048 B2, US7331048B2|
|Inventors||Joseph F. Skovira|
|Original Assignee||International Business Machines Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (12), Non-Patent Citations (7), Referenced by (19), Classifications (14), Legal Events (5)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This invention relates, in general, to the backfill scheduling of applications, and in particular, to a backfill scheduling capability that takes into consideration data of the applications in backfill scheduling the applications.
Scheduling techniques are used to schedule applications in a computing environment such that the resources of the environment are efficiently utilized. One type of scheduling technique is a backfill scheduling technique, which allows applications to run out of order as long as they do not affect the start time of an application already scheduled to execute. In particular, backfill scheduling techniques determine when a free set of processors will be available at a future time and schedule applications to be executed at that time.
Backfill scheduling techniques have progressed over the years to become more efficient. However, further enhancements are still desired. For example, a need exists for a backfill scheduling capability that considers the data of the applications in scheduling those applications.
The shortcomings of the prior art are overcome and additional advantages are provided through the provision of a method of facilitating backfill scheduling of applications of a computing environment. The method includes, for instance, determining on which computing unit of a plurality of computing units of the computing environment an application is to be scheduled to execute, the determining being based, at least in part, on data of the application; and backfill scheduling the application for execution, in response to the determining.
System and computer program products corresponding to the above-summarized method are also described and claimed herein.
Additional features and advantages are realized through the techniques of the present invention. Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention.
The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other objects, features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:
In accordance with an aspect of the present invention, a backfill scheduling capability is provided that takes into consideration data of applications in scheduling those applications for execution. For example, a determination is made as to which computing unit of a plurality of computing units an application is to be scheduled to execute, and that determining is based, at least in part, on data of the application.
One embodiment of a computing environment to incorporate and use one or more aspects of the present invention is described with reference to
To efficiently utilize the resources of the computing environment, scheduling techniques are used to schedule execution of applications of the environment. One type of scheduling technique is a backfill scheduling technique that determines when resources will be available at a later time for use by waiting applications. One example of a backfill scheduling technique is described in “The EASY-LoadLeveler API Project”, Job Scheduling Strategies for Parallel Processing, edited by Dror G. Feitelson and Larry Rudolph, pp. 41-47, Springer-Verlag, 1996, Lecture Notes on Computer Science Vol. 1162, which is hereby incorporated herein by reference in its entirety.
In one example, there are three phases to a backfill scheduling technique. One phase includes starting applications immediately on available resources (free node phase); another phase includes computation of when the next waiting application is to run and what resources it needs (shadow time computation, further described below); and a third phase indicates which jobs may run on available resources without impacting the start of the next job (backfill). The backfill scheduling technique may indicate that an application is to be moved from one computing unit to another computing unit to be scheduled for execution thereon. When an application is moved, the data associated with the application is also moved. However, the movement of the data may take longer than the application, and therefore, the application may have to wait a significant amount of time for the data to arrive before being executed. This is counterproductive and may signify that the application should not have been moved.
Thus, in order to facilitate determining whether an application is to be moved from a source computing unit to a destination computing unit, the backfill scheduling technique is enhanced, in accordance with an aspect of the present invention, to consider the data associated with the application. In one embodiment, the backfill scheduling technique considers one or more aspects relating to the data of the application. These aspects include, for instance, how much data there is to move (e.g., input (code and data) and results); the availability of communication channels between the source and destination sites; and/or the availability of data storage resources at the destination unit.
One embodiment of the logic associated with determining on which computing unit of a plurality of computing units an application is to be scheduled to execute is described with reference to
One step includes determining a local shadow time, which indicates how long the application needs to wait for available local resources, STEP 208. That is, a determination is made as to the estimated completion time for the one or more applications preventing execution of the waiting application. For instance, assume a local computing unit has 8 nodes and a local job queue includes the following: an application J1, which is running on 8 nodes and has a 20 hour completion time; and an application J2, which is waiting for 4 nodes and has an estimated completion time of 1 hour. In this example, the local shadow time (LST) is 20 hours, which is the estimated completion time of J1. It should be noted that in other examples there may be more than one application running that may affect the local shadow time.
A determination is also made as to the local resources needed, STEP 210. This represents what resources the waiting application will run on in the future, and is determined by checking the requested amount of resources for the particular application (e.g., J2 requests 4 nodes).
Additionally, a determination is made as to a local completion time (LCT), which is equal to the local shadow time plus the wallclock estimate for the waiting application (i.e., the estimated time to complete execution of the waiting application), STEP 212. For instance, the local completion time for the above example is equal to the local shadow time (e.g., 20 hours) plus the wallclock estimate for the waiting application (e.g., 1 hour), which is 21 hours.
In addition to the above, a remote shadow time is determined, STEP 214. In one example, the remote shadow time (RST) is the greater of the shadow time on the remote unit or the estimated data transfer time to the remote unit. For instance, assume there is another unit with 8 nodes that has the following remote job queue: an application J100, which is running on 4 nodes with a 10 hour completion time. Since J100 is only using 4 out of the 8 nodes, there are 4 available nodes. Further, since J2 (the waiting application) only needs 4 nodes, the shadow time on the remote unit is equal to 0, since the resources are available. The transfer time indicates how long it takes to transfer the input data. Thus, assume that the computing units are coupled by a 100 kbyte/sec link and that the waiting application, J2 has 20 MBytes of total input data. In that case, the transfer time is 200 seconds. Therefore, the remote shadow time is 200 seconds, the greater of the shadow time on the remote unit and the estimated data transfer time.
Further, a result transfer time (RTT) is determined, which is the time employed to return results to the local machine, STEP 216. As one example, the result transfer time for J2, which has 10 MBytes of total output data coming over a 100 kbyte/sec link, is 100 seconds.
Additionally, a remote completion time (RCT) is determined, STEP 218. As one example, the remote completion time is set equal to the remote shadow time (e.g., 200 seconds) plus the wallclock estimate (e.g., 1 hour) plus the result transfer time (e.g., 100 seconds). Thus, in this example, the remote completion time is 1 hour, 5 minutes.
Subsequently, a determination is made as to whether the remote completion time is less than the local completion time, INQUIRY 220. If the remote completion time is not less than the local completion time, then the application remains on the local unit to be processed when the resources become available, STEP 222. However, if the remote completion time is less than the local completion time, then a further determination is made as to whether remote storage is available, INQUIRY 224. If not, then the application remains on the local unit, STEP 222. Otherwise, a hold is placed on the local application, and a reservation is issued for the application at the remote unit, STEP 226. Thereafter, the data transfer begins, STEP 228.
On the remote machine, a reservation for J2, 200 seconds in the future is added to the job mix, and the transferred data for J2 is stored. After the data arrives, J2 may be started. When J2 runs to completion, the 10 MBytes of results is transferred to the local unit. Then, the input data, code and results are removed from the remote unit. The results are then available on the local unit.
As described above, the backfill scheduling technique is enhanced to perform scheduling based on data considerations. In one embodiment, the technique takes into consideration how much data there is to move (input (code and data) and results); the availability of any communication channels between the source and destination units; and the availability of data storage resources at the destination unit. However, these considerations are only examples. In other embodiments, one or more of the considerations are taken into account. Further, the considerations may be modified and/or other considerations may also be considered. For instance, the consideration of the amount of data may only consider input data. Other changes, additions, and/or deletions are possible.
The backfill scheduling technique provides an estimated start time for a waiting application. However, the estimated start time is just that, an estimate. If applications complete early (e.g., underestimated wallclock time, unexpected job end, etc.), the related terms of shadow time and estimated start time can be changed to earlier time estimates. However, there is still a fixed setup component of an application running on a destination machine, which includes the data transfer time that is still to expire before the application may start. Thus, the application has to wait longer to start (in effect, the data transfer time defines the shadow time for those situations in which an application completes early).
In one embodiment, it is assumed that the entire network bandwidth is available (e.g., full availability of communication channels). However, this may not be true. Thus, in a further embodiment, the estimated data transfer times are improved. For example, one or more adjustment factors are included in the backfill scheduling technique to account for variability in available network bandwidth. As one example, the estimated data transfer time to the remote unit (of STEP 214) includes an adjustment to underestimate available transfer time. For example, the actual transfer time is theoretically established or experimentally measured, and then, that value is adjusted to account for bandwidth variations. The adjustment can be based on historical measurements of the network performance, which may give some idea about how much bandwidth the scheduler can expect, while moving the application in question. For instance, the tracking of a 24 hour period of activity may yield periodic trends, which could be taken into account. In one example, the adjustment, is equal to, for instance, a percentage of the available transfer time. Thus, the estimated data transfer time is equal to the actual estimate plus an adjustment.
Since it is recognized that the tracking of an available resource, such as a network connection, can be difficult, in another embodiment, the data transfer mechanism is guaranteed some reserved bandwidth for the duration of the data transfer.
As a further example of providing an adjustment factor, the result transfer time (216) is adjusted by setting it equal to the actual estimate plus an adjustment, which is similarly determined, as described above, but may or may not be the same as the adjustment for the data transfer time.
The backfill computation is performed periodically. For instance, because of the non-linear nature with which the job stream changes, the backfill computation is performed periodically to adapt to these changes (e.g., anywhere from a continuous rate to a defined interval, such as once per minute). The available bandwidth can be monitored and the start time of the waiting application can be adjusted accordingly. Since the bandwidth estimate is computed with some degree of uncertainty, shrinking bandwidth may be contained up to a certain point. However, if bandwidth continues to deteriorate, eventually, the start time will be pushed into the future. Since this is undesirable for the backfill technique, an alternative would be to begin employing alternative communication channels, if available, or re-examining the possibility of running the applications locally. Thus, it is possible to respond to changes in available bandwidth over time.
There is also a backfill random variable to consider, in another embodiment. A measurement of how accurate shadow time estimates have been in the past can be obtained. By adapting to this measure, it is possible to predict more accurately when a job slot would appear on a remote machine (independent of network availability). This would provide a better opportunity to accurately schedule available network bandwidth for a particular application.
As the job queue develops over time, it may become clear that the waiting application can now be run on available resources on the source machine. If this should happen, data transfer can be suspended, the partial copy at the destination unit discarded, and the application waits to run on the local unit. The impact of this is a temporary use of network bandwidth and remote storage, which might have been used for alternative tasks.
Although in the embodiments described above, the process has been described for the first waiting task on a machine, the process can be extended to applications waiting further in the job queue. For example, in one embodiment, a search is made for tasks in the queue which have relatively modest data requirements (since this information is associated with job metadata). For those deep in the queue with small data requirements, the decision might be made to apply these to an idle, remote machine. Note that deep and small data requirements are relative terms which would be quantified, either by static or administrative settings, or in another embodiment, adjusted dynamically depending on a variety of factors (such as available network bandwidth, or queue size). In a further extension, the backfill technique can be executed for jobs deeper within the queue to determine approximate starting times. Once these are determined, they can be compared with estimated start times on remote units, including data transfer computations, as described herein. If an application might start much sooner on a remote unit, it could be moved to the destination unit for execution.
As a further enhancement, an application may commence executing on a remote unit, in response to some of the data arriving at the remote unit, but before all of the data has arrived. This is particularly useful for an application that traverses the data in a predictable manner. This computation is added to the estimation of shadow time.
Described in detail above is a technique for integrating data movement with scheduling techniques. Advantageously, a predictable approach to running remote applications, which effectively hides the process of transferring data to a remote unit, is provided.
The present invention can be included in an article of manufacture (e.g., one or more computer program products) having, for instance, computer usable media. The media has embodied therein, for instance, computer readable program code means for providing and facilitating the capabilities of the present invention. The article of manufacture can be included as a part of a computer system or sold separately.
Additionally, at least one program storage device readable by a machine, tangibly embodying at least one program of instructions executable by the machine to perform the capabilities of the present invention can be provided.
The flow diagrams depicted herein are just examples. There may be many variations to these diagrams or the steps (or operations) described therein without departing from the spirit of the invention. For instance, the steps may be performed in a differing order, or steps may be added, deleted or modified. As one particular example, the check for remote storage can be performed earlier in the process. Further, the various determinations can be performed in a differing order. All of these variations are considered a part of the claimed invention.
Although preferred embodiments have been depicted and described in detail herein, it will be apparent to those skilled in the relevant art that various modifications, additions, substitutions and the like can be made without departing from the spirit of the invention and these are therefore considered to be within the scope of the invention as defined in the following claims.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4918595||Jul 31, 1987||Apr 17, 1990||International Business Machines Corp.||Subsystem input service for dynamically scheduling work for a computer system|
|US5349682 *||Jan 31, 1992||Sep 20, 1994||Parallel Pcs, Inc.||Dynamic fault-tolerant parallel processing system for performing an application function with increased efficiency using heterogeneous processors|
|US5867716||Jun 12, 1995||Feb 2, 1999||Hitachi, Ltd.||Distributed computer system and method of generating automatic operation schedule for the same|
|US6076174 *||Feb 19, 1998||Jun 13, 2000||United States Of America||Scheduling framework for a heterogeneous computer network|
|US6098091 *||Dec 30, 1996||Aug 1, 2000||Intel Corporation||Method and system including a central computer that assigns tasks to idle workstations using availability schedules and computational capabilities|
|US6345287||Nov 26, 1997||Feb 5, 2002||International Business Machines Corporation||Gang scheduling for resource allocation in a cluster computing environment|
|US6366945||May 23, 1997||Apr 2, 2002||Ibm Corporation||Flexible dynamic partitioning of resources in a cluster computing environment|
|US7082606 *||Apr 30, 2002||Jul 25, 2006||The Regents Of The University Of California||Dedicated heterogeneous node scheduling including backfill scheduling|
|US7143412 *||Jul 25, 2002||Nov 28, 2006||Hewlett-Packard Development Company, L.P.||Method and apparatus for optimizing performance in a multi-processing system|
|US20020083185 *||Aug 21, 2001||Jun 27, 2002||Ruttenberg John C.||System and method for scheduling and executing data transfers over a network|
|US20020087623 *||Dec 30, 2000||Jul 4, 2002||Eatough David A.||Method and apparatus for determining network topology and/or managing network related tasks|
|WO1999061985A1||Mar 26, 1999||Dec 2, 1999||The Dow Chemical Company||Distributed computing environment using real-time scheduling logic and time deterministic architecture|
|1||"A Simulation Model of Backfilling and I/O Scheduling in a Partitionable Parallel System", Helen D. Karatza, Proceedings of the 2000 Winter Simulation Conference, pp. 496-505.|
|2||"Job Scheduling in the Presence of Multiple Resource Requirements", W. Leinberger, G. Karypis, V. Kumar.|
|3||*||"Method of Compute Resource Allocation in a Batch Job Submission environment" IBM Technical bulletin Jul. 1997; p. 1-2.|
|4||"Method of Computer Resource Allocation in a Batch Job Submission Environment", IBM Technical Disclosure Bulletin, Jul. 1997, vol. 40, Pub. No. 7, pp. 7-8.|
|5||"Scheduling Best-Effort and Real-Time Pipelined Applications on Time-Shared Clusters", Yanyong Zhang, Anand Sivasubramaniam, pp. 209-218.|
|6||"The EASY-LoadLeveler API Project", J. Skovira, W. Chan, H. Zhou, D. Kifka, Job Scheduling Strategies for Parallel Processing, edited by Dror G. Feitelson and Larry Rudolph, pp. 41-47, Springer-Verlag, 1996. Lecture Notes on Computer Science, vol. 1162.|
|7||*||IBM Technical Disclosure Bulletin (TDB) "Method of Computer Resource Allocation in a Batch Job Submission environment" Delphion Jul. 1997 pp. 1-2.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7480711 *||Dec 2, 2005||Jan 20, 2009||Packeteer, Inc.||System and method for efficiently forwarding client requests in a TCP/IP computing environment|
|US7596788 *||May 11, 2004||Sep 29, 2009||Platform Computing Corporation||Support of non-trivial scheduling policies along with topological properties|
|US7844967 *||May 19, 2004||Nov 30, 2010||Hewlett-Packard Development Company, L.P.||Method of allocating computing resources|
|US8352868||Jun 27, 2008||Jan 8, 2013||Google Inc.||Computing with local and remote resources including user mode control|
|US8355709||Apr 25, 2007||Jan 15, 2013||Qualcomm Incorporated||Device that determines whether to launch an application locally or remotely as a webapp|
|US8429664||May 12, 2010||Apr 23, 2013||Fujitsu Limited||Job scheduling apparatus and job scheduling method|
|US8601480||Sep 28, 2009||Dec 3, 2013||International Business Machines Corporation||Support of non-trivial scheduling policies along with topological properties|
|US8812578||Nov 7, 2008||Aug 19, 2014||International Business Machines Corporation||Establishing future start times for jobs to be executed in a multi-cluster environment|
|US9424086||Sep 18, 2013||Aug 23, 2016||International Business Machines Corporation||Support of non-trivial scheduling policies along with topological properties|
|US20050262509 *||May 19, 2004||Nov 24, 2005||Kelly Terence P||Method of allocating computing resources|
|US20060085559 *||Dec 2, 2005||Apr 20, 2006||Lownsbrough Derek L||System and method for efficiently forwarding client requests in a TCP/IP computing environment|
|US20060101465 *||Mar 2, 2005||May 11, 2006||Hitachi, Ltd.||Distributed control system|
|US20080268828 *||Apr 25, 2007||Oct 30, 2008||Nagendra Nagaraja||Device that determines whether to launch an application locally or remotely as a webapp|
|US20090327495 *||Jun 27, 2008||Dec 31, 2009||Oqo, Inc.||Computing with local and remote resources using automated optimization|
|US20090327962 *||Jun 27, 2008||Dec 31, 2009||Oqo, Inc.||Computing with local and remote resources including user mode control|
|US20090328036 *||Jun 27, 2008||Dec 31, 2009||Oqo, Inc.||Selection of virtual computing resources using hardware model presentations|
|US20100122255 *||Nov 7, 2008||May 13, 2010||International Business Machines Corporation||Establishing future start times for jobs to be executed in a multi-cluster environment|
|US20100146515 *||Sep 28, 2009||Jun 10, 2010||Platform Computing Corporation||Support of Non-Trivial Scheduling Policies Along with Topological Properties|
|US20100293551 *||May 12, 2010||Nov 18, 2010||Fujitsu Limited||Job scheduling apparatus and job scheduling method|
|U.S. Classification||718/102, 709/233, 718/104|
|International Classification||G06F15/16, G06F9/46, G06F9/00, H04L29/08, G06F9/50|
|Cooperative Classification||H04L67/325, G06F9/5038, G06F2209/509, G06F2209/506|
|European Classification||H04L29/08N31T, G06F9/50A6E|
|Apr 4, 2003||AS||Assignment|
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SKOVIRA, JOSEPH F.;REEL/FRAME:013952/0385
Effective date: 20030317
|Jul 15, 2011||FPAY||Fee payment|
Year of fee payment: 4
|Sep 25, 2015||REMI||Maintenance fee reminder mailed|
|Feb 12, 2016||LAPS||Lapse for failure to pay maintenance fees|
|Apr 5, 2016||FP||Expired due to failure to pay maintenance fee|
Effective date: 20160212