|Publication number||US7035771 B2|
|Application number||US 11/181,285|
|Publication date||Apr 25, 2006|
|Filing date||Jul 14, 2005|
|Priority date||Mar 29, 2004|
|Also published as||US7010465, US7174275, US20050216235, US20050267711, US20060161399|
|Publication number||11181285, 181285, US 7035771 B2, US 7035771B2, US-B2-7035771, US7035771 B2, US7035771B2|
|Inventors||Hammad Butt, Costin Hagiu|
|Original Assignee||Microsoft Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (2), Non-Patent Citations (8), Referenced by (8), Classifications (15), Legal Events (3)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This is a continuation of U.S. patent application Ser. No. 10/811,629 entitled “Scalability Test and Analysis” filed Mar. 29, 2004. The parent application is incorporated herein by reference in its entirety.
An application server, which may function as a multi-user computer system, often supports many users for executing one or more application programs from remote terminals such as workstations and personal computers. An application server can assume different architectures including that of a terminal server. When deploying an application server, it is important that the application server be properly configured to provide satisfactory performance for the expected number of users.
With the prior art, the performance of a multi-user computer system is often gauged by measuring some global system metric such as the processor utilization rate or the available system memory. However, an important performance indicator for an application server is the server's responsiveness as perceived by a user. Each user may be autonomously generating an action (e.g., entering a keystroke) through the user's remote terminal. A corresponding application program processes the action and provides a response to the user. If the time to respond exceeds a performance expectation, the user may perceive that the system is performing unsatisfactorily. Moreover, a multi-user computer system may support different types of users, where each type of user is characterized by different usage profiles. One type of user may be using application programs that place a large demand on the processor resources of the multi-user computer system. A second type of user may use application programs that place a relatively small demand on the processor resources but may enter keystrokes at a relatively high rate. Another type of user may use application programs that place a relatively large demand on the memory resources of the multi-user system. Moreover, different types of users may share application programs with each other. However, the different types of users may enter inputs at much different speeds (for example, a word processor expert versus an engineer who uses one finger to type).
Even though a global system metric may indicate that the multi-user compute system, as configured, provides satisfactory performance, a user may perceive the performance as not being satisfactory. On the flip side, the global system metric may indicate unsatisfactory performance although a user may perceive satisfactory performance.
There is a real need to provide a methodology and test tools that enable a system administrator to test a configured multi-user system to determine whether the multi-user system will provide satisfactory performance, with respect to a user's experience, even though the multi-user system is supporting different types of users and different application programs.
Aspects of the present invention provide system and methods to test whether a multi-user system, e.g., a terminal server, will provide satisfactory performance as perceived by a user. The multi-system user may support different types of users that place different demands on the system resources and that utilize different application programs which stress the system resources in different ways.
One aspect of the invention is to aggregate performance results, where each performance measurement indicates the response time for the multi-user computer system to provide a response to a user input. The response times are logged for each individual user, and the measurements are aggregated together in a single file at the end of the test. The aggregated data are subsequently split into different sets of performance results, in which each set of performance results is associated with a different user actions type. For each action type, a graph is built that correlates the distribution of the response times as a function of the user load. A break point is determined for each action type in which a response time exceeds a predetermined threshold. By analyzing the different break points, the number of users that can be supported by the multi-user computer system is determined.
With another aspect of the invention, an optimal amount of memory is determined to support a user load. During a test run, several indicators of memory management activity are collected. A progression of working set trimming is correlated to paging activity. With a variation of the invention, the amount of memory required per user is computed based on the user load at the projected point where a line that is determined from page output peaks intersects the page input line.
The foregoing summary of the invention, as well as the following detailed description of preferred embodiments, is better understood when read in conjunction with the accompanying drawings, which are included by way of example, and not by way of limitation with regard to the claimed invention.
A basic input/output system 160 (BIOS), containing the basic routines that help to transfer information between elements within the computer 100, such as during start-up, is stored in the ROM 140. The computer 100 also includes a hard disk drive 170 for reading from and writing to a hard disk (not shown), a magnetic disk drive 180 for reading from or writing to a removable magnetic disk 190, and an optical disk drive 191 for reading from or writing to a removable optical disk 192 such as a CD ROM or other optical media. The hard disk drive 170, magnetic disk drive 180, and optical disk drive 191 are connected to the system bus 130 by a hard disk drive interface 192, a magnetic disk drive interface 193, and an optical disk drive interface 194, respectively. The drives and their associated computer-readable media provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for the personal computer 100. It—will be appreciated by those skilled in the art that other types of computer readable media that can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read only memories (ROMs), and the like, may also be used in the example operating environment.
A number of program modules can be stored on the hard disk drive 170, magnetic disk 190, optical disk 192, ROM 140 or RAM 150, including an operating system 195, one or more application programs 196, other program modules 197, and program data 198. A user can enter commands and information into the computer 100 through input devices such as a keyboard 101 and pointing device 102. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner or the like. These and other input devices are often connected to the processing unit 110 through a serial port interface 106 that is coupled to the system bus, but may be connected by other interfaces, such as a parallel port, game port or a universal serial bus (USB). Further still, these devices may be coupled directly to the system bus 130 via an appropriate interface (not shown). A monitor 107 or other type of display device is also connected to the system bus 130 via an interface, such as a video adapter 108. In addition to the monitor, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. A pen digitizer 165 and accompanying pen or stylus 166 are provided in order to digitally capture freehand input. Although a direct connection between the pen digitizer 165 and the serial port is shown, in practice, the pen digitizer 165 may be coupled to the processing unit 110 directly, via a parallel port or other interface and the system bus 130 as known in the art. Furthermore, although the digitizer 165 is shown apart from the monitor 107, it is preferred that the usable input area of the digitizer 165 be co-extensive with the display area of the monitor 107. Further still, the digitizer 165 may be integrated in the monitor 107, or may exist as a separate device overlaying or otherwise appended to the monitor 107.
The computer 100 can operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 109. The remote computer 109 can be a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 100, although only a memory storage device 111 has been illustrated in
When used in a LAN networking environment, the computer 100 is connected to the local network 112 through a network interface or adapter 114. When used in a WAN networking environment, the personal computer 100 typically includes a modem 115 or other means for establishing a communications over the wide area network 113, such as the Internet. The modem 115, which may be internal or external, is connected to the system bus 130 via the serial port interface 106. In a networked environment, program modules depicted relative to the personal computer 100, or portions thereof, may be stored in the remote memory storage device.
The architecture of computer 100 supports various computer functions including a multi-user computer (e.g., a terminal server) and a client terminal as will be discussed. (However, a terminal server may not be configured with some of the peripheral devices, e.g., digitizer 166, as may be configured with a client terminal.)
Paging performance may be an important consideration when configuring computer 100. Referring to
In the exemplary embodiment, the terminal server may comprise one of the following configurations.
In order to test the number of users that can be supported by terminal server 301, terminal server 301 is configured with processor capability and memory capacity that reflects the configuration of the terminal server that will be actually installed subsequent to testing.
Terminal server 301 executes application programs as invoked from client terminals 303 a–303 i and routed through routers 309 a–309 d. Different user types (e.g., knowledge workers and data entry personnel) may utilize different application programs and may generate a different pattern of user actions.)
Test controller 305 configures laboratory configuration 300. For example, test controller 305 activates a subset of the plurality of client terminals 303 a–303 i with test script that generates user actions that simulate a desired user environment. (The different action types are referred as a set of user actions.) For example, a simulated action type may be file write or cut operations that a user would typically use during an application program session. Exchange server 307 provides network connectivity for Internet services in order to simulate applications (e.g., e-mail applications or Internet browsers) requiring network connectivity.
An automated server and client workstation reset is performed before each test-run in order to revert all components to a clean state. Response times, based on user action types, are used to determine when or if a terminal server is overloaded. Client-side scripts drive the user simulation and record the response times for a set of simulated user actions.
The scripts contain many sequences. A sequence starts with the test script sending a key stroke through a client terminal 303 to one of the applications running in the session on test server 301. As a result of the key stroke, a string is displayed by the application. For example, Ctrl-F opens the File menu that would then display the Open string.
Test scripts are constructed for different user types. For example, in the embodiment, knowledge workers and data entry workers are simulated. For a simulated knowledge worker, simulated keystrokes are generated at a typing speed of approximately 35 words per minute. In the embodiment, a knowledge worker is a worker who gathers, adds value to, and communicates information in a decision support process. The cost of downtime is variable but highly visible. Projects and ad-hoc needs towards flexible tasks drive these resources. Knowledge workers make their own decisions about what to work on and how to accomplish the task. The usual tasks knowledge workers perform are marketing, project management, sales, desktop publishing, decision support, data mining, financial analysis, executive and supervisory management, design, and authoring.
The exemplary embodiment shown in
In the embodiment, each client terminal is configured for specific user types. Specific test script is loaded in each client terminal by test controller 305 according to the user type being simulated by the client terminal. Also, other embodiments may simulate different user types according to the operating environment being simulated.
The response time is the time from the key stroke to the display of the string at client terminal 303. To accurately measure the response time, the response measurement is calculated by taking two initial time readings, ti1 and ti2, from a reference time source before and after sending the key stroke. Ti1 is the time when the test manager 305 sends the instructions to client terminal 303, and ti2 represents client terminal 303 sending the keystroke to test server 301. A third reading, tf3, is made after the corresponding text string is received by client terminal 303. Time is measured by milliseconds. Based on these values the response time is estimated as belonging to the interval (tf3−ti2, tf3−ti1). Typically, the measurement error (the time between ti1 and ti2) is less than 1 millisecond (ms), and the response values are approximated by tf3−ti1.
For each scenario, the Test Manager workstation 305 starts groups of ten client sessions on the client terminals 303 with a 30-second interval between each session. After the group of ten client sessions is started, a 5-minute stabilization period is observed in which no additional sessions are started. After the stabilization period, the Knowledge Worker script starts the four applications it will use in the test within five minutes. This prevents login activity interference between each group of ten client sessions. For each action type, as the number of users log on, a degradation point is determined when the response times increase to a value that is deemed significant.
This criteria is based on the assumption that a user will not notice degradation in a response time while this is lower then 200 ms. There are several reasons for response-time degradation. When degradation in response time is attained for approximately the same number of users, CPU saturation is the main reason. Response-time degradation that can be attributed to actions relating to file IO, such as opening a dialog box to select a file to open or save, is due to IO limitations. Also, some actions might exhibit a “noisy” degradation where a small number of response times randomly having values noticeable higher than average, without influencing the average value noticeably.
The test harness supports a series of tests that determine system load threshold based on a canary script that is running between the logon groups while the system is stable. The canary script is run before any users are logged onto the system, and the time the script takes to complete (elapsed time) is recorded. This elapsed time becomes the baseline and is deemed to be the baseline response rate for a given configuration of server. This method would consider that maximum load is reached when the total time needed for running the canary script is 10% higher than the initial value. The response time method is considered to be more accurate because it measures the key parameter for the actual user experience, it takes into account the login period impact and provides a richer data support for decision making. The canary-script method can still be more efficient for setups that support a small number of users where the response time method does not provide a large enough sample of response time values.
One key parameter that is also monitored is the total cycle time for a work unit performed by a user. The total cycle time should be a constant value especially for comparison tests (cross-platform or when testing for special features). Tests that run with different cycle times are effectively performing a different amount of work in the same unit of time, thus consuming more resources in the time unit and can not be compared directly.
The Process performance object's Working Set counter (used on the _Total instance of the counter to measure all processes in the system) is a qualitative indication of how many bytes have been recently accessed by threads in the process. However, if the free memory in the computer is sufficient, pages are left in the working set of a process even if they are not in use. If free memory falls below a threshold, unused pages are trimmed from working sets. The method used in these tests for determining memory requirements is not typically as simple as observing a performance counter. The method should account for the dynamic behavior of a memory-limited system.
The most accurate method of calculating the amount of memory required per user is to analyze the results of several performance counters (Memory\Pages Input/sec, Memory\Pages Output/sec, Memory\Available Bytes and Process\Working Set (Total_) in a memory-constrained scenario. When a system has abundant physical RAM, the working set initially grows at a high rate, and pages are left in the working set of a process even if the pages are not in use. Eventually, when the total working set tends to exhaust the amount of physical memory, the operating system is forced to trim the unused portions of working set until enough pages are made available to free up the memory pressure. This trimming of unused portions of the working sets occurs when the applications collectively need more physical memory than is available, which is a situation that requires system to constantly page to maintain the working sets of all the processes. In operating systems theory terminology, this constant paging state is referred to as thrashing.
Zone_1 611 represents the abundant memory stage. This is when physical memory is greater than the total amount of memory that applications need. In this zone, the operating system does not page anything to disk, even seldom-used pages.
Zone_2 613 represents the stage when unused portions of the working sets are trimmed. In this stage the operating system periodically trims the unused pages from the processes' working sets whenever the amount of available memory drops to a critical value. This process is illustrated by memory curve 605, where discontinuities (e.g., discontinuity 604) are indicative that the management system is paging out memory from RAM 150 to disk 170 (as illustrated in
Zone_3 615 represents the high pressure zone. The working sets are trimmed to a minimal value and mostly contain pages that are frequented by the greater number of users. Page faults will likely cause the ejection of a page that will need to be referenced in the future, thus increasing the frequency of page faults. The output per second of pages (as shown by curve 603) will increase significantly, and the page output curve 603 follows to some degree the shape of page input curve 609. The system does a very good job of controlling degradation, almost linearly, but the paging activity eventually increases to a level where the response times are not acceptable.
The amount of memory needed can be determined from the number of users at the point 651 where the page-out activity starts increasing significantly (end of Zone_2 613 in
Although a reasonable amount of paging is acceptable, paging naturally consumes a small amount of the CPU and other resources. Because the maximum number of users that can be loaded onto a system is determined on systems that have abundant physical RAM, a minimal amount of paging typically occurs. The working set calculations assume that a reasonable amount of paging has occurred to trim the unused portions of the working set, but this only occurs on a system that is memory-constrained. If one takes the base memory requirement and adds it to the number of users, multiplied by the required working set, the result is a system that is naturally memory-constrained, and acceptable paging occurs. On such a system, one expects a slight decrease in performance due to the overhead of paging. The decrease in performance can reduce the number of users who can be actively working on the system before the response time degrades above the acceptable level.
In the embodiment, the memory required for each user may be determined by the intersection of trend line 707 and fitted paging input curve 705. (There are various methods for fitting line 705 such as determining a running average and using regression analysis.) The required memory per user can be approximated by dividing the available memory size by the number of users corresponding to the intersection. The available memory is determined by subtracting the amount of memory needed for overhead operations from the total amount of memory configured for test server 301.
In step 805, test controller 303 activates M client terminals. Aggregated performance measurements are collected by a collection module. In the embodiment, the functionality of the collection module may be distributed across the client terminals 303 a–303 i and test controller 305. However, other embodiments may implement the collection module with a different architecture.
In step 809, test controller 305 determines if additional client terminals should be activated. If so, step 805 is repeated to activate an additional M client terminals and to collect more aggregated performance measurements in step 807. If no additional client terminals should be activated (i.e., the test session is completed), then step 811 is performed. In step 811, the aggregated performance is processed so that performance data is separated (split) according to the action type. Each action type corresponds to a plurality the data points, where each data point corresponds to a response time with a simulated number of users. In the embodiment, a running average is determined for each action type in order to form a performance curve. A break point is determined for each action type where the performance curve exceeds an associated predetermined threshold as previously discussed.
In step 813, if the simulated performance satisfies a desired performance target, then process 800 is completed with step 815. However, if the simulation indicates that the desired performance target has not been achieved with the configuration of test server 301, then test server 301 may be reconfigured with additional resources and steps 801–813 are repeated.
As can be appreciated by one skilled in the art, a computer system with an associated computer-readable medium containing instructions for controlling the computer system can be utilized to implement the exemplary embodiments that are disclosed herein. The computer system may include at least one computer such as a microprocessor, digital signal processor, and associated peripheral electronic circuitry.
Although the invention has been defined using the appended claims, these claims are illustrative in that the invention is intended to include the elements and steps described herein in any combination or sub combination. Accordingly, there are any number of alternative combinations for defining the invention, which incorporate one or more elements from the specification, including the description, claims, and drawings, in various combinations or sub combinations. It will be apparent to those skilled in the relevant technology, in light of the present specification, that alternate combinations of aspects of the invention, either alone or in combination with one or more elements or steps defined herein, may be utilized as modifications or alterations of the invention or as part of the invention. It may be intended that the written description of the invention contained herein covers all such modifications and alterations.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US6542854 *||Apr 30, 1999||Apr 1, 2003||Oracle Corporation||Method and mechanism for profiling a system|
|US6601020 *||May 3, 2000||Jul 29, 2003||Eureka Software Solutions, Inc.||System load testing coordination over a network|
|1||A.J. Fied, P.G. Harrison, J. Parry, "Response Times in Client-Server Systems", pp. 1-18.|
|2||D. Pegler, D. Hutchison and D. Shepherd, "Scalability Issues for Mass Multimedia Storage Systems", 1996 The Institution of Electrical Engineers, pp. 1-7.|
|3||J. Major, "Client/Server Capacity Planning and Relative Throughput Capacity Estimation", Demand Technology, Inc., Capacity Management Review, Apr. 1996, pp. 3-9.|
|4||K. Kant and P. Mohapatra, "Scalable Internet Servers: Issues and Challenges", pp. 1-8.|
|5||Performance Testing of the Citrix(R) MetaFrame(R) Platform http://www.rttsweb.com/services/citrix/index.cfm.|
|6||S. Nishio, "MASEMS: A Scalable and Extensible Multimedia Server", Proceedings of the 1999 International Symposium on Database Applications in Non-Tradtional Environments.|
|7||Scapa Technologies -New Citrix MetaFram Stressing Tool Released, Press Release, http://scapatech.com/news<SUB>-</SUB>events/press<SUB>-</SUB> releases/2002-18-03.html.|
|8||Windows Server 2003-Terminal Server Capacity and Scaling, Whitepaper, 2003 Microsoft Corporation, pp. 1-39.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7957948 *||Aug 22, 2007||Jun 7, 2011||Hyperformit, Inc.||System and method for capacity planning for systems with multithreaded multicore multiprocessor resources|
|US8627036||Sep 12, 2011||Jan 7, 2014||Microsoft Corporation||Memory management techniques|
|US8924677||Jan 7, 2014||Dec 30, 2014||Microsoft Corporation||Memory management techniques|
|US8935669||Apr 11, 2007||Jan 13, 2015||Microsoft Corporation||Strategies for performing testing in a multi-user environment|
|US9158699||Nov 20, 2014||Oct 13, 2015||Microsoft Technology Licensing, Llc||Memory management techniques|
|US20080256389 *||Apr 11, 2007||Oct 16, 2008||Microsoft Corporation||Strategies for Performing Testing in a Multi-User Environment|
|US20090055805 *||Aug 24, 2007||Feb 26, 2009||International Business Machines Corporation||Method and System for Testing Software|
|US20090055823 *||Aug 22, 2007||Feb 26, 2009||Zink Kenneth C||System and method for capacity planning for systems with multithreaded multicore multiprocessor resources|
|U.S. Classification||702/186, 709/224, 714/E11.195, 714/E11.198|
|Cooperative Classification||G06F11/3457, G06F2201/88, G06F11/3419, G06F11/3447, G06F11/3414, G06F11/3476|
|European Classification||G06F11/34M, G06F11/34C2, G06F11/34C4, G06F11/34S|
|Sep 23, 2009||FPAY||Fee payment|
Year of fee payment: 4
|Sep 25, 2013||FPAY||Fee payment|
Year of fee payment: 8
|Dec 9, 2014||AS||Assignment|
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034543/0001
Effective date: 20141014