US7747988B2 - Software feature usage analysis and reporting - Google Patents
Software feature usage analysis and reporting Download PDFInfo
- Publication number
- US7747988B2 US7747988B2 US11/818,600 US81860007A US7747988B2 US 7747988 B2 US7747988 B2 US 7747988B2 US 81860007 A US81860007 A US 81860007A US 7747988 B2 US7747988 B2 US 7747988B2
- Authority
- US
- United States
- Prior art keywords
- feature
- users
- information corresponding
- usage
- sessions
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related, expires
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 72
- 238000000034 method Methods 0.000 claims description 17
- 230000001186 cumulative effect Effects 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 5
- 238000001914 filtration Methods 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 7
- 238000004891 communication Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 5
- 230000006855 networking Effects 0.000 description 4
- 230000002093 peripheral effect Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000005055 memory storage Effects 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000004141 dimensional analysis Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000013442 quality metrics Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 230000003442 weekly effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/865—Monitoring of software
Definitions
- various aspects of the subject matter described herein are directed towards a technology by which usage of program features may be analyzed, including by determining feature usage metrics.
- Information representative of the feature usage metrics is output, such as in a report or the like.
- Software instrumentation data is collected during actual user program usage sessions. The collected data is then processed to determine various feature usage counts and other information, cross-feature usage (e.g., among users who use a feature, how many use another feature or program), and characteristics of feature users, e.g., how long, how much, how often and how extensive feature users use a program. Session analysis may be performed to provide information about the number of sessions in which a set of features occur. Feature usage trends over time may also be determined via analysis.
- a system including an analyzer that processes information corresponding to the software instrumentation data recorded from user software program usage sessions to produce an output.
- the analyzer includes a feature usage analysis, a cross-feature usage analysis, an analysis of characteristics of feature users, a session analysis, and/or an analysis of feature usage trends.
- the analyzer may be coupled to a user interface for facilitating selection of one or more features, for facilitating selection of a group of users, and/or for outputting results corresponding to the analysis.
- FIG. 1 shows an example representation of recording software instrumentation data for subsequent analysis.
- FIG. 2 shows a representation in a table format of example software instrumentation (e.g., software quality metrics) data saved for various program usage sessions by users of a suite of application programs.
- example software instrumentation e.g., software quality metrics
- FIG. 3 shows an example representation of a user interface and analyzer for analyzing software feature data based on the software instrumentation data.
- FIG. 4 shows a representation of various example concepts related to analyzing feature usage.
- FIG. 5 is a flow diagram representing various example concepts related to analyzing software feature usage and users.
- FIG. 6 is an illustrative example of a general-purpose computing environment into which various aspects of the present invention may be incorporated.
- aspects of the technology described herein are generally directed towards analyzing software data collected from real software users to model and recognize software users' usage of software features. Based on such feature modeling and recognition, further analysis such as analyzing characteristics of feature users, cross-feature usage, session analysis (e.g., where a session is typically from program start to finish) and so forth may be performed to gain a better understanding of feature usage.
- example analyses are described herein that evaluate feature usage, generally comprising information about how many users use a feature, the average length of a feature, and so forth, as well as cross-feature usage, generally an evaluation as to which users who use a feature also use another feature or feature set. Characteristics of feature users also may be analyzed, such as to evaluate how long, how much, how often and how extensive feature users use an application. Other types of analysis include session analysis, generally comprising information about the number of sessions in which a set of features occur, and analysis of feature usage trends over time.
- the technology is described herein by use of examples, including those that operate in various environments, such as internal users (e.g., corresponding to employees of the enterprise that is analyzing the software) and external users.
- the programs exemplified herein are generally a suite of application programs such as those provided as part of the Microsoft® Office software product suite.
- these are only non-limiting examples, and the technology is applicable to different user environments and different software products, including individual application programs and operating system components.
- the present invention is not limited to any particular embodiments, aspects, concepts, structures, functionalities or examples described herein. Rather, any of the embodiments, aspects, concepts, structures, functionalities or examples described herein are non-limiting, and the present invention may be used various ways that provide benefits and advantages in computing in general.
- FIG. 1 there is shown a mechanism for collecting software instrumentation data 102 , including a session recorder 104 that collects various data from one or more application instances 106 corresponding to various users 108 1 - 108 n , where n represents any practical number of users.
- the session recorder 104 may be per application instance/user, or may be a mechanism such as an agent on each computing device of a network that communicates with at least one data collection server component running on a network server or the like.
- a user interface 110 allows an test operator or the like to set collection parameters, such as from which program or programs to collect the instrumentation data, from which users to collect data, how long a collection session should last (if the program is not ended by the user within that time limit) and so forth.
- the instrumentation data 102 comprise data collected from each user session, where a session corresponds to actual usage by a user of an executing program.
- a typical session starts from the application start (e.g., by double clicking on the application executable or a document that launches the application executable, or by choosing the application from a start menu), and ends when the application is closed (e.g., by choosing “Exit” in the application or closing the application window).
- Sessions can also be time limited, e.g., if a session exceeds twenty-four hours, the session is ended and the instrumentation data to that point recorded (the application continues to run).
- Sessions can also end by non-normal termination of a program, e.g., because of program or system crashes.
- FIG. 2 provides an example of one type of software instrumentation data 102 , with some of the data (arranged in columns) collected for some number of sessions (arranged in rows); it is equivalent to have the sessions be in the columns and the rows represent the data.
- each session is associated with some or all of the information shown in FIG. 2 , including a session ID, a user ID, and an application name.
- Other information that is typically recorded includes the application version/build, a start time, an end time, the commands used during the session and still other data, such as the number of files opened and so forth.
- the instrumentation data 102 may be many thousands of dimensions.
- Software features can be modeled and recognized from software instrumentation data. Based on feature modeling and recognition, further analysis such as analyzing characteristics of feature users, cross-feature usage, session analysis and so forth may be performed to gain an understanding of feature usage. From feature recognition, the sessions in which a feature occurred are obtained.
- feature recognition provides rich statistical information on how a feature is used, including for example, feature occurrence, feature command count, average length and so on.
- the information can be filtered and formatted for feature usage analysis and reporting.
- the software instrumentation data 102 is processed, such as to measure the overall usage of a feature by a group of users.
- FIG. 3 shows example analysis components, including an analyzer 332 that generates an output report 334 from some set of feature data 336 .
- the data 336 may be first pre-processed into any suitable data structure or set of data structures, and/or the features recognized therein, such as described in the aforementioned U.S. patent applications entitled “Multidimensional Analysis Tool for High Dimensional Data,” “Efficient Data Infrastructure for High Dimensional Analysis” and “Software Feature Modeling and Recognition.” Further, the data may be accessed via a client/service architecture, such as described in the aforementioned U.S. patent application entitled “Analyzing Software Usage with Instrumentation Data,” in which a data explorer system manages the data for the analysis and provides access to the data via APIs.
- a user interface 336 establishes the parameters, criteria and which metrics are used to analyze the feature data 334 .
- FIG. 3 shows an example UI and implementation of such an analysis.
- filters such as a source of users and whether to exclude outliers can be specified, (outliers are described below).
- the source of users can be specified.
- Some examples include all users from whom instrumentation data have been collected (All), users who are external customers and not employees of the company conducting the analysis (External), users who are employees (Internal), users who are from a particular group that has been set up to collect data (Study ID, for example, the beta participants of a next software release), or a custom group.
- All users from whom instrumentation data have been collected
- External users who are external customers and not employees of the company conducting the analysis
- Users who are employees Internal
- users who are employees users who are from a particular group that has been set up to collect data
- Student ID for example, the beta participants of a next software release
- custom groups are described in the aforementioned U.S. patent application entitled “Analyzing Software Users with Instrumentation Data and User Group Modeling and Analysis.”
- a type of user is a potential outlier if his or her use of a feature is substantially different from those of most other users.
- Various criteria can be used to determine an outlier, such as the entropy of the occurrence distribution of each feature. The smaller the entropy, the more unevenly distributed the occurrence of the feature among the set of all users. For example, if the entropy is less than one-half (0.5), a first criterion is met.
- the average occurrence per session of the feature by this user may be considered, e.g., the total occurrence of the feature divided by application session count of the user. If the average occurrence per session is greater than some number, such as one-hundred, the second criterion is met. In this example, any user who meets the two criteria can be grouped and reported; in this example, the user is an outlier that is likely using automation.
- FIG. 4 shows various example components for processing and analyzing SQM data (or similar data) with respect to feature usage and types of analysis.
- the SQM data 102 may first be filtered and otherwise pre-processed (block 402 ), e.g., so that only the commands used in features are efficiently accessed, as represented by the block labeled 404 .
- some commands in a sequence of commands are not part of feature usage, and are referred to as noise; one or more noise models 406 described below may be applied to modify the command sequence data (block 408 ) such as to eliminate such commands.
- a visual modeling tool 420 allows an operator of the analysis to define aspects of feature analysis.
- a compiler 422 compiles the output of the modeling tool 420 into a finite state machine 424 .
- a recognition process 428 uses the finite state machine to locate feature usage.
- Data 430 corresponding to the recognized feature usage and commands may then be analyzed (block 432 ) and output as information in some way, such as in one or more reports 434 .
- the analyzer 332 and output 334 of FIG. 3 may correspond to blocks 432 and 434 of FIG. 4 , respectively.
- Information from feature recognition and instrumentation data can be used to obtain the following information:
- Percentage Percentage of application users who use the of feature feature users Percentage Percentage of application sessions in which of feature the feature occurred. sessions: Feature Total number of times the feature occurred in occurrence analysis data.
- Analysis data is the data set for analysis, e.g., the set of data given the source of users, application and version/build.
- Feature occurrence includes exact matches and matches with noise, e.g., with unnecessary steps such as menu browsing in the middle of the feature.
- Feature Total number of clicks of necessary commands controls i.e. those defined in the feature in click count occurrences of the feature.
- Noise click Total number of clicks of unnecessary count commands (such as menu browsing) in occurrences of the feature Average Average number of feature control clicks and feature noise clicks per feature occurrence.
- Feature Click count of each command and enum in the clicks feature (the Enum element models a set of details commands; each Enum has a unique name, and it can be called by a Feature, Procedure or another Enum. In execution, only one of the child elements is executed).
- details Feature path Top n (e.g., ten) paths of the feature, details including the total occurrence and commands that make up the path. Each path is a way the feature may be executed.
- Feature Distribution of feature length (e.g., ten) paths of the feature, details
- Cross-feature usage analysis 442 answers questions such as how many users who use a certain feature also use another feature, and for a group of features, how many users use all (or some other specified set) of the features in the group. Based on feature recognition results, cross-feature usage can be measured by users who use all (or some specified set of) the features, usage of features by users who use all (or some specified set of) the features, and percentage of ⁇ feature name 1> users who use ⁇ feature name 2>.
- User count Number of users who use the specified (e.g., all) features Users who use Number of users who use all the all the applications to which the features belong. applications For example, if two features are chosen for this analysis, one is a Word 2003 (SP1) feature and the other is a PowerPoint 2003 (Office 2003 Important Builds) feature, this refers to the number of users who use both Word 2003 (SP1) and any of the important builds (RTM, SP1 and SP2) of PowerPoint 2003. Percentage Ratio of User count to Users who use all the applications.
- SP1 Word 2003
- PowerPoint 2003 Office 2003 Important Builds
- Feature user Number of users who use the feature count ⁇ application Number of users who use the application (of name> the specified builds). (builds) user count Percentage Ratio of feature user count to the ⁇ application name> (builds) user count. Occurrence Total occurrence of the feature by feature users.
- Percentage of ⁇ feature name 1> users who use ⁇ feature name 2> (repeat for each remaining feature, and then repeat for each feature):
- Percentage of Percentage of ⁇ feature name 1> users ⁇ feature name 1> who use ⁇ feature name 2>. Users who use ⁇ feature name 2> Usage of ⁇ feature Total occurrence of ⁇ feature name 1> name 1> (occurrence) by these users. Usage of ⁇ feature Total occurrence of ⁇ feature name 2> name 2> (occurrence) by these users.
- Feature user characteristics analysis 443 is directed towards resolving questions such as how often and how much do feature users use an application, and what types of users they are when using an application.
- Feature users may be users who use all of the features specified, any one of the features specified, or none of the features specified. Based on feature recognition results, characteristics of these users can be measured by:
- Average Average number of application sessions by these session users This indicates on average, how much the count users have been using the application during a time period.
- One suitable calculation is: get the session count (total number of application sessions) of each user, and average across all the users in the group. Average Average time between consecutive sessions by session users. This indicates on average, how frequent frequency the users use the application.
- One suitable calculation is: get the session elapse time (the time between the end of the last session and the end of the first session) of each user, get the session frequency (session elapse time divided by session count) of each user, average across some set of (e.g., all) users. Average Average total session time by users. This is total another indication of on average, how much the running analysis users have been using the application.
- One suitable calculation is: get the total running time (sum of all session time) of each user, average across the users. Average Average session time by users. This indicates session on average, how much time users spend in each length session using the application.
- One suitable calculation is: get the average session length (total running time divided by session count) of each user, average across the users. Average Average percentage of total commands of the depth of application used by users. This indicates how usage deep users use the application.
- One suitable calculation is: get the depth of usage (percentage of total application commands used by the user, where total application commands is the total number of distinct commands found in the command stream of the application so far, and used by the user is defined as the command found at least once in the command stream of the user) of each user, average across the users. For example, users can be characterized as beginner, intermediate, advanced and expert users depending on their depth of usage.
- the distribution of the above measures can also be obtained by counting how many or percentage of users whose values fall within an interval.
- the application and version/build for which the analysis is being conducted can also be specified.
- Session analysis 445 is directed to answering questions such as how many users use a set of features in the same session, and how frequent these sessions are. Based on feature recognition results, session analysis may include:
- ⁇ application Total number of sessions of the application name> session count: Sessions in Total number of sessions in which some which specified set of (e.g., all) the features features occurred in the session. occurred Percentage Percentage of application sessions in which of sessions some specified set of (e.g., all) the features occurred, i.e. the ratio of the two numbers above.
- Session analysis also may include basic information of each feature such as the session count and user count of each feature.
- Feature ⁇ feature name> (repeat for each feature):
- Session Total number of sessions in which the feature count occurred.
- User count Total number of users who used the feature.
- Trend analysis 446 of feature usage is directed towards resolving questions such as whether the percentage of users who use a feature changed since the launch of the newest version/build of a software product. Based on feature recognition results, the trend of using a feature, e.g., usage of a feature over time, can be measured.
- the feature and the source of users can be specified, as can the period to analyze and reporting interval can also be specified.
- the period to analyze can be an absolute period such as the time period from a start date to an end date, or a relative period, for example, each user's enrollment length which refers to the time period from a user's first session to the last session.
- a reporting interval is the interval to report the measures. This may be monthly, weekly, daily, or another interval.
- Feature User Count Total number of users using the feature during a reporting interval.
- Feature Session Total number of sessions in which the Count feature occurred during a reporting interval.
- Feature Occurrence Total number of times the feature occurred during a reporting interval.
- Application Count Total number of users using the application during a reporting interval. Percentage of Feature user count divided by Users application user count during a reporting interval.
- Application Total number of sessions of the Session Count application during a reporting interval. Percentage of Feature session count divided by Sessions application session count during a reporting interval.
- FIG. 5 summarizes an overall example process, beginning at step 502 which represents collecting the software instrumentation data.
- the software instrumentation data may be collected at any previous time, not necessarily just prior to analysis.
- Step 504 represents obtaining the analysis criteria (e.g., feature usage, trend analysis and/or others), and obtaining the user set, which may be all, external, internal, a user group and so forth as set above.
- Step 506 generates a query from the operator-input analysis and/or user filtering criteria.
- Step 508 represents submitting the query against the software feature data (in any appropriate format), with step 510 representing receiving the query results.
- Step 512 represents generating the report, which may include performing calculations on the results as needed to match the operator's requirements. For example, as described above, some of the report can include information that is not directly measured but is computed from a combination of two or more measured sets of data.
- FIG. 6 illustrates an example of a suitable computing system environment 600 on which some or all of the various components (e.g., represented in FIG. 4 ) may be implemented.
- the computing system environment 600 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing environment 600 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment 600 .
- the invention is operational with numerous other general purpose or special purpose computing system environments or configurations.
- Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to: personal computers, server computers, hand-held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
- the invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer.
- program modules include routines, programs, objects, components, data structures, and so forth, which perform particular tasks or implement particular abstract data types.
- the invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.
- program modules may be located in local and/or remote computer storage media including memory storage devices.
- an exemplary system for implementing various aspects of the invention may include a general purpose computing device in the form of a computer 610 .
- Components of the computer 610 may include, but are not limited to, a processing unit 620 , a system memory 630 , and a system bus 621 that couples various system components including the system memory to the processing unit 620 .
- the system bus 621 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures.
- such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.
- ISA Industry Standard Architecture
- MCA Micro Channel Architecture
- EISA Enhanced ISA
- VESA Video Electronics Standards Association
- PCI Peripheral Component Interconnect
- the computer 610 typically includes a variety of computer-readable media.
- Computer-readable media can be any available media that can be accessed by the computer 610 and includes both volatile and nonvolatile media, and removable and non-removable media.
- Computer-readable media may comprise computer storage media and communication media.
- Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.
- Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by the computer 610 .
- Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
- modulated data signal means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
- communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer-readable media.
- the system memory 630 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 631 and random access memory (RAM) 632 .
- ROM read only memory
- RAM random access memory
- BIOS basic input/output system
- RAM 632 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 620 .
- FIG. 6 illustrates operating system 634 , application programs 635 , other program modules 636 and program data 637 .
- the computer 610 may also include other removable/non-removable, volatile/nonvolatile computer storage media.
- FIG. 6 illustrates a hard disk drive 641 that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive 651 that reads from or writes to a removable, nonvolatile magnetic disk 652 , and an optical disk drive 655 that reads from or writes to a removable, nonvolatile optical disk 656 such as a CD ROM or other optical media.
- removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like.
- the hard disk drive 641 is typically connected to the system bus 621 through a non-removable memory interface such as interface 640
- magnetic disk drive 651 and optical disk drive 655 are typically connected to the system bus 621 by a removable memory interface, such as interface 650 .
- the drives and their associated computer storage media provide storage of computer-readable instructions, data structures, program modules and other data for the computer 610 .
- hard disk drive 641 is illustrated as storing operating system 644 , application programs 645 , other program modules 646 and program data 647 .
- operating system 644 application programs 645 , other program modules 646 and program data 647 are given different numbers herein to illustrate that, at a minimum, they are different copies.
- a user may enter commands and information into the computer 610 through input devices such as a tablet, or electronic digitizer, 664 , a microphone 663 , a keyboard 662 and pointing device 661 , commonly referred to as mouse, trackball or touch pad.
- Other input devices not shown in FIG. 6 may include a joystick, game pad, satellite dish, scanner, or the like.
- These and other input devices are often connected to the processing unit 620 through a user input interface 660 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB).
- a monitor 691 or other type of display device is also connected to the system bus 621 via an interface, such as a video interface 690 .
- the monitor 691 may also be integrated with a touch-screen panel or the like. Note that the monitor and/or touch screen panel can be physically coupled to a housing in which the computing device 610 is incorporated, such as in a tablet-type personal computer. In addition, computers such as the computing device 610 may also include other peripheral output devices such as speakers 695 and printer 696 , which may be connected through an output peripheral interface 694 or the like.
- the computer 610 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 680 .
- the remote computer 680 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 610 , although only a memory storage device 681 has been illustrated in FIG. 6 .
- the logical connections depicted in FIG. 6 include one or more local area networks (LAN) 671 and one or more wide area networks (WAN) 673 , but may also include other networks.
- LAN local area network
- WAN wide area network
- Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.
- the computer 610 When used in a LAN networking environment, the computer 610 is connected to the LAN 671 through a network interface or adapter 670 .
- the computer 610 When used in a WAN networking environment, the computer 610 typically includes a modem 672 or other means for establishing communications over the WAN 673 , such as the Internet.
- the modem 672 which may be internal or external, may be connected to the system bus 621 via the user input interface 660 or other appropriate mechanism.
- a wireless networking component 674 such as comprising an interface and antenna may be coupled through a suitable device such as an access point or peer computer to a WAN or LAN.
- program modules depicted relative to the computer 610 may be stored in the remote memory storage device.
- FIG. 6 illustrates remote application programs 685 as residing on memory device 681 . It may be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
- An auxiliary subsystem 699 (e.g., for auxiliary display of content) may be connected via the user interface 660 to allow data such as program content, system status and event notifications to be provided to the user, even if the main portions of the computer system are in a low power state.
- the auxiliary subsystem 699 may be connected to the modem 672 and/or network interface 670 to allow communication between these systems while the main processing unit 620 is in a low power state.
Abstract
Description
Percentage | Percentage of application users who use the |
of feature | feature. |
users | |
Percentage | Percentage of application sessions in which |
of feature | the feature occurred. |
sessions: | |
Feature | Total number of times the feature occurred in |
occurrence | analysis data. (Analysis data is the data set |
for analysis, e.g., the set of data given the | |
source of users, application and | |
version/build. Feature occurrence includes | |
exact matches and matches with noise, e.g., | |
with unnecessary steps such as menu browsing | |
in the middle of the feature.) | |
Feature | Total number of clicks of necessary commands |
controls | (i.e. those defined in the feature) in |
click count | occurrences of the feature. |
Noise click | Total number of clicks of unnecessary |
count | commands (such as menu browsing) in |
occurrences of the feature | |
Average | Average number of feature control clicks and |
feature | noise clicks per feature occurrence. |
length | |
Feature | Ratio of the sum of feature controls click |
clicks-total | count and noise click count to the total |
clicks ratio | number of clicks for the application. |
Feature | Click count of each command and enum in the |
clicks | feature (the Enum element models a set of |
details | commands; each Enum has a unique name, and it |
can be called by a Feature, Procedure or | |
another Enum. In execution, only one of the | |
child elements is executed). | |
Enum/command | Click count of each command and enum broken |
details | down by command properties. |
Noise | Exact match occurrence and noise click count. |
details | |
Feature path | Top n (e.g., ten) paths of the feature, |
details | including the total occurrence and commands |
that make up the path. Each path is a way the | |
feature may be executed. | |
Feature | Distribution of feature length. |
length | |
distribution | |
User count | Number of users who use the specified | ||
(e.g., all) features. | |||
Users who use | Number of users who use all the | ||
all the | applications to which the features belong. | ||
applications | For example, if two features are chosen for | ||
this analysis, one is a Word 2003 (SP1) | |||
feature and the other is a PowerPoint 2003 | |||
(Office 2003 Important Builds) feature, | |||
this refers to the number of users who use | |||
both Word 2003 (SP1) and any of the | |||
important builds (RTM, SP1 and SP2) of | |||
PowerPoint 2003. | |||
Percentage | Ratio of User count to Users who use all | ||
the applications. | |||
<feature | Total occurrence of the feature by users | ||
name> total | who use the specified (e.g., all) features | ||
occurrence | |||
Feature user | Number of users who use the feature. | ||
count | |||
<application | Number of users who use the application (of | ||
name> | the specified builds). | ||
(builds) | |||
user count | |||
Percentage | Ratio of feature user count to the | ||
<application name> (builds) user count. | |||
Occurrence | Total occurrence of the feature by feature | ||
users. | |||
Percentage of | Percentage of <feature |
||
<feature |
who use <feature |
||
users who use | |||
<feature |
|||
Usage of <feature | Total occurrence of <feature |
||
|
by these users. | ||
Usage of <feature | Total occurrence of <feature |
||
|
by these users. | ||
Average | Average number of application sessions by these | ||
session | users. This indicates on average, how much the | ||
count | users have been using the application during a | ||
time period. One suitable calculation is: get | |||
the session count (total number of application | |||
sessions) of each user, and average across all | |||
the users in the group. | |||
Average | Average time between consecutive sessions by | ||
session | users. This indicates on average, how frequent | ||
frequency | the users use the application. One suitable | ||
calculation is: get the session elapse time | |||
(the time between the end of the last session | |||
and the end of the first session) of each user, | |||
get the session frequency (session elapse time | |||
divided by session count) of each user, average | |||
across some set of (e.g., all) users. | |||
Average | Average total session time by users. This is | ||
total | another indication of on average, how much the | ||
running | analysis users have been using the application. | ||
time | One suitable calculation is: get the total | ||
running time (sum of all session time) of each | |||
user, average across the users. | |||
Average | Average session time by users. This indicates | ||
session | on average, how much time users spend in each | ||
length | session using the application. One suitable | ||
calculation is: get the average session length | |||
(total running time divided by session count) | |||
of each user, average across the users. | |||
Average | Average percentage of total commands of the | ||
depth of | application used by users. This indicates how | ||
usage | deep users use the application. One suitable | ||
calculation is: get the depth of usage | |||
(percentage of total application commands used | |||
by the user, where total application commands | |||
is the total number of distinct commands found | |||
in the command stream of the application so | |||
far, and used by the user is defined as the | |||
command found at least once in the command | |||
stream of the user) of each user, average | |||
across the users. For example, users can be | |||
characterized as beginner, intermediate, | |||
advanced and expert users depending on their | |||
depth of usage. | |||
<application | Total number of sessions of the application. | ||
name> | |||
session | |||
count: | |||
Sessions in | Total number of sessions in which some | ||
which | specified set of (e.g., all) the features | ||
features | occurred in the session. | ||
occurred | |||
Percentage | Percentage of application sessions in which | ||
of sessions | some specified set of (e.g., all) the | ||
features occurred, i.e. the ratio of the two | |||
numbers above. | |||
<application | Total number of application users. | ||
name > user | |||
count | |||
Users who | Total number of users who use some specified | ||
use the | set of (e.g., all) the features in the same | ||
features in | session. | ||
the same | |||
session | |||
Percentage | Percentage of application users who use some | ||
of users | specified set of (e.g., all) the features in | ||
the same session, i.e. the ratio of the two | |||
numbers above. | |||
Frequency | Frequency of sessions by users. Shows how | ||
distribution | many and the percentage of users who use some | ||
specified set of (e.g., all) the features in | |||
the same session whose frequency of such | |||
sessions (i.e. ratio of the number of | |||
sessions in which some specified set of | |||
(e.g., all) the features occurred in the same | |||
session and total session count of the user) | |||
falls within an interval. | |||
Session | Total number of sessions in which the feature | ||
count | occurred. | ||
User count | Total number of users who used the feature. | ||
Percentage | Percentage of application sessions in which | ||
of sessions | the feature occurred. | ||
Percentage | Percentage of application users who used the | ||
of users | feature. | ||
Feature User Count | Total number of users using the feature | ||
during a reporting interval. | |||
Feature Session | Total number of sessions in which the | ||
Count | feature occurred during a reporting | ||
interval. | |||
Feature Occurrence | Total number of times the feature | ||
occurred during a reporting interval. | |||
Feature | Feature occurrence divided by feature | ||
Occurrence/User | user count during a reporting interval. | ||
Feature | Feature occurrence divided by feature | ||
Occurrence/Session | session count during a reporting | ||
interval. | |||
Application Count | Total number of users using the | ||
application during a reporting | |||
interval. | |||
Percentage of | Feature user count divided by | ||
Users | application user count during a | ||
reporting interval. | |||
Application | Total number of sessions of the | ||
Session Count: | application during a reporting | ||
interval. | |||
Percentage of | Feature session count divided by | ||
Sessions | application session count during a | ||
reporting interval. | |||
Cumulative Feature | Total number of times the feature | ||
Occurrence | occurred from the start time of the | ||
period to analyze to the end of each | |||
reporting interval. | |||
Cumulative | Percentage of users from the start time | ||
Percentage of | of the period to analyze to the end of | ||
Users | each reporting interval. | ||
Cumulative | Percentage of sessions from the start | ||
Percentage of | time of the period to analyze to the | ||
Sessions | end of each reporting interval. | ||
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/818,600 US7747988B2 (en) | 2007-06-15 | 2007-06-15 | Software feature usage analysis and reporting |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/818,600 US7747988B2 (en) | 2007-06-15 | 2007-06-15 | Software feature usage analysis and reporting |
Publications (2)
Publication Number | Publication Date |
---|---|
US20080313633A1 US20080313633A1 (en) | 2008-12-18 |
US7747988B2 true US7747988B2 (en) | 2010-06-29 |
Family
ID=40133551
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/818,600 Expired - Fee Related US7747988B2 (en) | 2007-06-15 | 2007-06-15 | Software feature usage analysis and reporting |
Country Status (1)
Country | Link |
---|---|
US (1) | US7747988B2 (en) |
Cited By (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070162896A1 (en) * | 2006-01-10 | 2007-07-12 | Intel Corporation | Method and apparatus for generating run time profiles for program compilation |
US20090138292A1 (en) * | 2007-11-26 | 2009-05-28 | International Business Machines Corporation | Driving software product changes based on usage patterns gathered from users of previous product releases |
US20100107141A1 (en) * | 2008-10-28 | 2010-04-29 | George Fitzmaurice | System and method for recommending next commands when using a software application |
US20100287204A1 (en) * | 2007-11-13 | 2010-11-11 | Lisa Amini | Systems and methods for using provenance information for data retention in stream-processing |
US20100293373A1 (en) * | 2009-05-15 | 2010-11-18 | International Business Machines Corporation | Integrity service using regenerated trust integrity gather program |
US20110061041A1 (en) * | 2009-09-04 | 2011-03-10 | International Business Machines Corporation | Reliability and availability modeling of a software application |
US20120246619A1 (en) * | 2011-03-23 | 2012-09-27 | Oracle International Corporation | Automated testing of browser based rich internet applications (ria) delivered through proprietary frameworks |
US20130042149A1 (en) * | 2011-08-09 | 2013-02-14 | International Business Machines Corporation | Analyzing a process of software defects handling using percentile-based metrics |
US20130111016A1 (en) * | 2005-07-12 | 2013-05-02 | Visible Measures Corp. | Distributed capture and aggregation of dynamic application usage information |
US9235390B1 (en) * | 2008-03-31 | 2016-01-12 | Symantec Corporation | Application optimization for use based on feature popularity |
US9588777B2 (en) | 2012-10-10 | 2017-03-07 | Landmark Graphics Corporation | Method and system of knowledge transfer between users of a software application |
US9916158B2 (en) | 2016-07-20 | 2018-03-13 | International Business Machines Corporation | Cognitive feature analytics |
US10042739B2 (en) * | 2016-09-29 | 2018-08-07 | International Business Machines Corporation | Real-time analytics of machine generated instrumentation data |
US11755429B2 (en) | 2021-04-23 | 2023-09-12 | EMC IP Holding Company LLC | Method, electronic device, and program product for presenting I/O write records |
Families Citing this family (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7870114B2 (en) | 2007-06-15 | 2011-01-11 | Microsoft Corporation | Efficient data infrastructure for high dimensional data analysis |
US7739666B2 (en) * | 2007-06-15 | 2010-06-15 | Microsoft Corporation | Analyzing software users with instrumentation data and user group modeling and analysis |
US8271543B2 (en) * | 2008-01-28 | 2012-09-18 | Cisco Technology, Inc. | Creation of context for communications based on previous interactions |
FR2952200A1 (en) * | 2009-10-29 | 2011-05-06 | Alcatel Lucent | DEVICE AND METHOD FOR AUTOMATICALLY ANALYZING THE USE OF THE USER INTERFACE OF AN APPLICATION |
US11562013B2 (en) | 2010-05-26 | 2023-01-24 | Userzoom Technologies, Inc. | Systems and methods for improvements to user experience testing |
US20140052853A1 (en) * | 2010-05-26 | 2014-02-20 | Xavier Mestres | Unmoderated Remote User Testing and Card Sorting |
US11494793B2 (en) | 2010-05-26 | 2022-11-08 | Userzoom Technologies, Inc. | Systems and methods for the generation, administration and analysis of click testing |
US11348148B2 (en) | 2010-05-26 | 2022-05-31 | Userzoom Technologies, Inc. | Systems and methods for an intelligent sourcing engine for study participants |
US11544135B2 (en) | 2010-05-26 | 2023-01-03 | Userzoom Technologies, Inc. | Systems and methods for the analysis of user experience testing with AI acceleration |
US11934475B2 (en) | 2010-05-26 | 2024-03-19 | Userzoom Technologies, Inc. | Advanced analysis of online user experience studies |
US11068374B2 (en) | 2010-05-26 | 2021-07-20 | Userzoom Technologies, Inc. | Generation, administration and analysis of user experience testing |
US10691583B2 (en) | 2010-05-26 | 2020-06-23 | Userzoom Technologies, Inc. | System and method for unmoderated remote user testing and card sorting |
US9053307B1 (en) | 2012-07-23 | 2015-06-09 | Amazon Technologies, Inc. | Behavior based identity system |
US9262470B1 (en) | 2013-06-25 | 2016-02-16 | Amazon Technologies, Inc. | Application recommendations based on application and lifestyle fingerprinting |
US9921827B1 (en) | 2013-06-25 | 2018-03-20 | Amazon Technologies, Inc. | Developing versions of applications based on application fingerprinting |
US10269029B1 (en) | 2013-06-25 | 2019-04-23 | Amazon Technologies, Inc. | Application monetization based on application and lifestyle fingerprinting |
US10740361B2 (en) * | 2017-02-21 | 2020-08-11 | Microsoft Technology Licensing, Llc | Clustering and analysis of commands in user interfaces |
US10754762B2 (en) | 2019-01-30 | 2020-08-25 | Walmart Apollo, Llc | Automatic tracking of feature usage of webpages on a website |
US11909100B2 (en) | 2019-01-31 | 2024-02-20 | Userzoom Technologies, Inc. | Systems and methods for the analysis of user experience testing with AI acceleration |
Citations (109)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4068298A (en) | 1975-12-03 | 1978-01-10 | Systems Development Corporation | Information storage and retrieval system |
US5500941A (en) | 1994-07-06 | 1996-03-19 | Ericsson, S.A. | Optimum functional test method to determine the quality of a software system embedded in a large electronic system |
US5542070A (en) | 1993-05-20 | 1996-07-30 | Ag Communication Systems Corporation | Method for rapid development of software systems |
US5548718A (en) | 1994-01-07 | 1996-08-20 | Microsoft Corporation | Method and system for determining software reliability |
US5619709A (en) | 1993-09-20 | 1997-04-08 | Hnc, Inc. | System and method of context vector generation and retrieval |
US5774660A (en) | 1996-08-05 | 1998-06-30 | Resonate, Inc. | World-wide-web server with delayed resource-binding for resource-based load balancing on a distributed resource multi-node network |
US5867144A (en) | 1991-11-19 | 1999-02-02 | Microsoft Corporation | Method and system for the direct manipulation of information, including non-default drag and drop operation |
US5903886A (en) | 1996-04-30 | 1999-05-11 | Smartlynx, Inc. | Hierarchical adaptive state machine for emulating and augmenting software |
US5956720A (en) | 1997-02-06 | 1999-09-21 | At & T Corp | Method and apparatus for web site management |
US6046741A (en) | 1997-11-21 | 2000-04-04 | Hewlett-Packard Company | Visual command sequence desktop agent |
US6079032A (en) * | 1998-05-19 | 2000-06-20 | Lucent Technologies, Inc. | Performance analysis of computer systems |
US6128279A (en) | 1997-10-06 | 2000-10-03 | Web Balance, Inc. | System for balancing loads among network servers |
US6131082A (en) | 1995-06-07 | 2000-10-10 | Int'l.Com, Inc. | Machine assisted translation tools utilizing an inverted index and list of letter n-grams |
US6138159A (en) | 1998-06-11 | 2000-10-24 | Phaal; Peter | Load direction mechanism |
US6144962A (en) | 1996-10-15 | 2000-11-07 | Mercury Interactive Corporation | Visualization of web sites and hierarchical data structures |
US6154746A (en) | 1998-04-22 | 2000-11-28 | At&T Corp. | High-dimensional index structure |
WO2000075814A1 (en) | 1999-06-03 | 2000-12-14 | Keylime Software, Inc. | System and method for monitoring user interaction with web pages |
US6167358A (en) | 1997-12-19 | 2000-12-26 | Nowonder, Inc. | System and method for remotely monitoring a plurality of computer-based systems |
US6182139B1 (en) | 1996-08-05 | 2001-01-30 | Resonate Inc. | Client-side resource-based load-balancing with delayed-resource-binding using TCP state migration to WWW server farm |
EP1083486A1 (en) | 1999-09-10 | 2001-03-14 | Ricoh Company, Ltd. | User interface usage monitoring and reporting system and method |
US6233570B1 (en) | 1996-07-19 | 2001-05-15 | Microsoft Corporation | Intelligent user assistance facility for a software program |
US6237138B1 (en) | 1996-11-12 | 2001-05-22 | International Business Machines Corp. | Buffered screen capturing software tool for usability testing of computer applications |
US6237143B1 (en) | 1998-09-17 | 2001-05-22 | Unisys Corp. | Method and system for monitoring and capturing all file usage of a software tool |
US6260050B1 (en) | 1999-03-23 | 2001-07-10 | Microstrategy, Inc. | System and method of adapting automatic output of service related OLAP reports to disparate output devices |
US6317750B1 (en) | 1998-10-26 | 2001-11-13 | Hyperion Solutions Corporation | Method and apparatus for accessing multidimensional data |
US20010044705A1 (en) | 2000-03-10 | 2001-11-22 | Isogon Corp. | Method of normalizing software usage data from mainframe computers |
US6374369B1 (en) | 1999-05-21 | 2002-04-16 | Philips Electronics North America Corporation | Stochastic performance analysis method and apparatus therefor |
US6385604B1 (en) | 1999-08-04 | 2002-05-07 | Hyperroll, Israel Limited | Relational database management system having integrated non-relational multi-dimensional data store of aggregated data elements |
US20020083003A1 (en) | 2000-09-26 | 2002-06-27 | Halliday David C. | Method and apparatus for the accurate metering of software application usage and the reporting of such usage to a remote site on a public network |
US6418427B1 (en) | 1999-12-30 | 2002-07-09 | Decode Genetics Ehf | Online modifications of dimension structures in multidimensional processing |
US6434544B1 (en) | 1999-08-04 | 2002-08-13 | Hyperroll, Israel Ltd. | Stand-alone cartridge-style data aggregation server providing data aggregation for OLAP analyses |
US20020144124A1 (en) | 2001-03-12 | 2002-10-03 | Remer Eric B. | Method and apparatus to monitor use of a program |
US20030009507A1 (en) * | 2001-06-29 | 2003-01-09 | Annie Shum | System and method for application performance management |
US6564174B1 (en) | 1999-09-29 | 2003-05-13 | Bmc Software, Inc. | Enterprise management system and method which indicates chaotic behavior in system resource usage for more accurate modeling and prediction |
US6567796B1 (en) | 1999-03-23 | 2003-05-20 | Microstrategy, Incorporated | System and method for management of an automatic OLAP report broadcast system |
US20030115207A1 (en) | 2001-09-25 | 2003-06-19 | Bowman David M. | Hierarchical hybrid OLAP analytics generators |
US6587970B1 (en) | 2000-03-22 | 2003-07-01 | Emc Corporation | Method and apparatus for performing site failover |
US6601062B1 (en) | 2000-06-27 | 2003-07-29 | Ncr Corporation | Active caching for multi-dimensional data sets in relational database management system |
US6633782B1 (en) | 1999-02-22 | 2003-10-14 | Fisher-Rosemount Systems, Inc. | Diagnostic expert in a process control system |
US6662362B1 (en) * | 2000-07-06 | 2003-12-09 | International Business Machines Corporation | Method and system for improving performance of applications that employ a cross-language interface |
US6701363B1 (en) | 2000-02-29 | 2004-03-02 | International Business Machines Corporation | Method, computer program product, and system for deriving web transaction performance metrics |
US20040049505A1 (en) | 2002-09-11 | 2004-03-11 | Kelly Pennock | Textual on-line analytical processing method and system |
US6714940B2 (en) | 2001-11-15 | 2004-03-30 | International Business Machines Corporation | Systems, methods, and computer program products to rank and explain dimensions associated with exceptions in multidimensional data |
US20040088699A1 (en) * | 2002-11-06 | 2004-05-06 | Charles Suresh | System and method for measuring code segment performance |
US6748555B1 (en) | 1999-09-09 | 2004-06-08 | Microsoft Corporation | Object-based software management |
US20040117760A1 (en) | 2002-12-11 | 2004-06-17 | Microsoft Corporation | Reality-based optimization |
US6754312B1 (en) | 1998-12-03 | 2004-06-22 | Siemens Aktiengesellschaft | Method for monitoring a subscriber of an intelligent network |
US20040122646A1 (en) | 2002-12-18 | 2004-06-24 | International Business Machines Corporation | System and method for automatically building an OLAP model in a relational database |
US20040133882A1 (en) | 1996-08-27 | 2004-07-08 | Angel David J. | Byte code instrumentation |
US6768986B2 (en) | 2000-04-03 | 2004-07-27 | Business Objects, S.A. | Mapping of an RDBMS schema onto a multidimensional data model |
US20040191743A1 (en) | 2003-03-26 | 2004-09-30 | International Business Machines Corporation | System and method for software development self-assessment |
US6801940B1 (en) * | 2002-01-10 | 2004-10-05 | Networks Associates Technology, Inc. | Application performance monitoring expert |
US6816898B1 (en) | 2000-08-16 | 2004-11-09 | Proactivenet, Inc. | Interfacing external metrics into a performance management system |
US20040230858A1 (en) | 2003-05-14 | 2004-11-18 | Microsoft Corporation | Methods and systems for analyzing software reliability and availability |
US6845474B2 (en) | 2001-09-10 | 2005-01-18 | Hewlett-Packard Development Company, L.P. | Problem detector and method |
US20050015683A1 (en) | 2003-07-16 | 2005-01-20 | Lucent Technologies Inc. | Method, system and computer program product for improving system reliability |
US20050021293A1 (en) | 2002-02-01 | 2005-01-27 | Benedikte Elbel | Early prognosis of the reliability of a technical system |
US6862696B1 (en) | 2000-05-03 | 2005-03-01 | Cigital | System and method for software certification |
US20050065910A1 (en) | 2003-08-08 | 2005-03-24 | Caleb Welton | Method and apparatus for storage and retrieval of information in compressed cubes |
US20050071807A1 (en) | 2003-09-29 | 2005-03-31 | Aura Yanavi | Methods and systems for predicting software defects in an upcoming software release |
US20050081206A1 (en) * | 2003-10-14 | 2005-04-14 | Armstrong Douglas R. | Methods and apparatus for profiling threaded programs |
US6901536B2 (en) | 2001-05-24 | 2005-05-31 | Microsoft Corporation | Service quality monitoring system and method |
US6901347B1 (en) | 2000-02-22 | 2005-05-31 | Sun Microsystems, Inc. | Availability, reliability or maintainability index including outage characterization |
US20050125777A1 (en) * | 2003-12-05 | 2005-06-09 | Brad Calder | System and method of analyzing interpreted programs |
US20050131924A1 (en) | 2003-12-15 | 2005-06-16 | Quantum Matrix Holding, Llc | System and method for multi-dimensional organization, management, and manipulation of data |
US6912692B1 (en) | 1998-04-13 | 2005-06-28 | Adobe Systems Incorporated | Copying a sequence of commands to a macro |
US20050182750A1 (en) | 2004-02-13 | 2005-08-18 | Memento, Inc. | System and method for instrumenting a software application |
US20050183143A1 (en) | 2004-02-13 | 2005-08-18 | Anderholm Eric J. | Methods and systems for monitoring user, application or device activity |
US6963826B2 (en) | 2003-09-22 | 2005-11-08 | C3I, Inc. | Performance optimizer system and method |
US20050278290A1 (en) | 2004-06-14 | 2005-12-15 | International Business Machines Corporation | Systems, methods, and computer program products that automatically discover metadata objects and generate multidimensional models |
US20050278378A1 (en) | 2004-05-19 | 2005-12-15 | Metacarta, Inc. | Systems and methods of geographical text indexing |
US7003766B1 (en) | 2001-06-19 | 2006-02-21 | At&T Corp. | Suite of metrics for software quality assurance and product development |
US20060075399A1 (en) | 2002-12-27 | 2006-04-06 | Loh Choo W | System and method for resource usage prediction in the deployment of software applications |
US7028225B2 (en) | 2001-09-25 | 2006-04-11 | Path Communications, Inc. | Application manager for monitoring and recovery of software based application processes |
US7032214B1 (en) * | 2000-06-29 | 2006-04-18 | Microsoft Corporation | Performance markers to measure performance of features in a program |
US7039166B1 (en) | 2001-03-05 | 2006-05-02 | Verizon Corporate Services Group Inc. | Apparatus and method for visually representing behavior of a user of an automated response system |
US20060106793A1 (en) | 2003-12-29 | 2006-05-18 | Ping Liang | Internet and computer information retrieval and mining with intelligent conceptual filtering, visualization and automation |
US20060116981A1 (en) | 2004-11-30 | 2006-06-01 | Stefan Krimmel | Method and system for automated data collection and analysis of a computer system |
US7062483B2 (en) | 2000-05-18 | 2006-06-13 | Endeca Technologies, Inc. | Hierarchical data-driven search and navigation system and method for information retrieval |
US7080091B2 (en) | 2002-05-09 | 2006-07-18 | Oracle International Corporation | Inverted index system and method for numeric attributes |
US20060174346A1 (en) | 2005-01-31 | 2006-08-03 | Lieberman Software Corporation | Instrumentation for alarming a software product |
US7111282B2 (en) | 2001-06-12 | 2006-09-19 | Hewlett-Packard Development Company, L.P. | Instrumenting a software program and collecting data from the instrumented software program by type |
US20060218138A1 (en) | 2005-03-25 | 2006-09-28 | Microsoft Corporation | System and method for improving search relevance |
US7117208B2 (en) | 2000-09-28 | 2006-10-03 | Oracle Corporation | Enterprise web mining system and method |
US20060242638A1 (en) | 2005-04-22 | 2006-10-26 | Microsoft Corporation | Adaptive systems and methods for making software easy to use via software usage mining |
US20060242636A1 (en) * | 2005-04-26 | 2006-10-26 | Microsoft Corporation | Variational path profiling |
US7131070B1 (en) | 1999-05-13 | 2006-10-31 | Ricoh Company, Ltd. | Application unit monitoring and reporting system and method |
US20060259981A1 (en) | 2005-05-13 | 2006-11-16 | Yaron Ben-Shoshan | System and method of controlling and monitoring computer program usage |
US20060287993A1 (en) | 2005-06-21 | 2006-12-21 | Microsoft Corporation | High scale adaptive search systems and methods |
US20070016672A1 (en) | 2005-07-12 | 2007-01-18 | Visible Measures, Inc. | Distributed capture and aggregation of dynamic application usage information |
US7171406B2 (en) | 2001-05-18 | 2007-01-30 | Hewlett-Packard Development Company, L.P. | OLAP-based web access analysis method and system |
US20070033201A1 (en) | 2005-08-02 | 2007-02-08 | Sap Aktiengesellschaft | Systems and methods of multidimensional software management |
US20070038974A1 (en) | 2005-08-10 | 2007-02-15 | Microsoft Corporation | Software analysis tool |
US20070038983A1 (en) | 2005-08-15 | 2007-02-15 | Sap Aktiengesellschaft | Systems and methods for enterprise software management |
US20070039009A1 (en) | 2000-10-26 | 2007-02-15 | Metilinx | Multi-platform optimization model |
US7185231B2 (en) | 2003-05-14 | 2007-02-27 | Microsoft Corporation | Methods and systems for collecting, analyzing, and reporting software reliability and availability |
US7194386B1 (en) * | 2005-10-17 | 2007-03-20 | Microsoft Corporation | Automated collection of information |
US20070101311A1 (en) * | 2005-10-21 | 2007-05-03 | International Business Machines Corporation | Method and system for generating computer-executable models using augmentations |
US7216341B2 (en) | 2003-05-09 | 2007-05-08 | Microsoft Corporation | Instrumenting software for enhanced diagnosability |
US20070233628A1 (en) | 2006-03-07 | 2007-10-04 | Sherwood Timothy P | Pattern matching technique for high throughput network processing |
US7315849B2 (en) | 2000-02-28 | 2008-01-01 | Hyperroll Israel, Ltd. | Enterprise-wide data-warehouse with integrated data aggregation engine |
US20080127120A1 (en) * | 2006-10-31 | 2008-05-29 | Sun Microsystems, Inc. | Method and apparatus for identifying instructions associated with execution events in a data space profiler |
US7401331B2 (en) | 2001-08-17 | 2008-07-15 | Wu-Hon Francis Leung | Method to add new software features without modifying existing code |
US20080313507A1 (en) | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Software reliability analysis using alerts, asserts and user interface controls |
US20080313213A1 (en) | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Efficient data infrastructure for high dimensional data analysis |
US20080312899A1 (en) | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Software feature modeling and recognition |
US20080313184A1 (en) | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Multidimensional analysis tool for high dimensional data |
US20080313617A1 (en) | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Analyzing software users with instrumentation data and user group modeling and analysis |
US20080313149A1 (en) | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Analyzing software usage with instrumentation data |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US1717406A (en) * | 1927-10-01 | 1929-06-18 | Patent Button Co | Button |
US7028255B1 (en) * | 2000-08-14 | 2006-04-11 | Adobe Systems Incorporated | Self-sizing object placement |
-
2007
- 2007-06-15 US US11/818,600 patent/US7747988B2/en not_active Expired - Fee Related
Patent Citations (113)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4068298A (en) | 1975-12-03 | 1978-01-10 | Systems Development Corporation | Information storage and retrieval system |
US5867144A (en) | 1991-11-19 | 1999-02-02 | Microsoft Corporation | Method and system for the direct manipulation of information, including non-default drag and drop operation |
US5542070A (en) | 1993-05-20 | 1996-07-30 | Ag Communication Systems Corporation | Method for rapid development of software systems |
US5619709A (en) | 1993-09-20 | 1997-04-08 | Hnc, Inc. | System and method of context vector generation and retrieval |
US5548718A (en) | 1994-01-07 | 1996-08-20 | Microsoft Corporation | Method and system for determining software reliability |
US5500941A (en) | 1994-07-06 | 1996-03-19 | Ericsson, S.A. | Optimum functional test method to determine the quality of a software system embedded in a large electronic system |
US6131082A (en) | 1995-06-07 | 2000-10-10 | Int'l.Com, Inc. | Machine assisted translation tools utilizing an inverted index and list of letter n-grams |
US5903886A (en) | 1996-04-30 | 1999-05-11 | Smartlynx, Inc. | Hierarchical adaptive state machine for emulating and augmenting software |
US6233570B1 (en) | 1996-07-19 | 2001-05-15 | Microsoft Corporation | Intelligent user assistance facility for a software program |
US5774660A (en) | 1996-08-05 | 1998-06-30 | Resonate, Inc. | World-wide-web server with delayed resource-binding for resource-based load balancing on a distributed resource multi-node network |
US6182139B1 (en) | 1996-08-05 | 2001-01-30 | Resonate Inc. | Client-side resource-based load-balancing with delayed-resource-binding using TCP state migration to WWW server farm |
US20040133882A1 (en) | 1996-08-27 | 2004-07-08 | Angel David J. | Byte code instrumentation |
US6144962A (en) | 1996-10-15 | 2000-11-07 | Mercury Interactive Corporation | Visualization of web sites and hierarchical data structures |
US6237138B1 (en) | 1996-11-12 | 2001-05-22 | International Business Machines Corp. | Buffered screen capturing software tool for usability testing of computer applications |
US5956720A (en) | 1997-02-06 | 1999-09-21 | At & T Corp | Method and apparatus for web site management |
US6128279A (en) | 1997-10-06 | 2000-10-03 | Web Balance, Inc. | System for balancing loads among network servers |
US6046741A (en) | 1997-11-21 | 2000-04-04 | Hewlett-Packard Company | Visual command sequence desktop agent |
US6167358A (en) | 1997-12-19 | 2000-12-26 | Nowonder, Inc. | System and method for remotely monitoring a plurality of computer-based systems |
US6912692B1 (en) | 1998-04-13 | 2005-06-28 | Adobe Systems Incorporated | Copying a sequence of commands to a macro |
US6154746A (en) | 1998-04-22 | 2000-11-28 | At&T Corp. | High-dimensional index structure |
US6079032A (en) * | 1998-05-19 | 2000-06-20 | Lucent Technologies, Inc. | Performance analysis of computer systems |
US6138159A (en) | 1998-06-11 | 2000-10-24 | Phaal; Peter | Load direction mechanism |
US6237143B1 (en) | 1998-09-17 | 2001-05-22 | Unisys Corp. | Method and system for monitoring and capturing all file usage of a software tool |
US6317750B1 (en) | 1998-10-26 | 2001-11-13 | Hyperion Solutions Corporation | Method and apparatus for accessing multidimensional data |
US6754312B1 (en) | 1998-12-03 | 2004-06-22 | Siemens Aktiengesellschaft | Method for monitoring a subscriber of an intelligent network |
US6633782B1 (en) | 1999-02-22 | 2003-10-14 | Fisher-Rosemount Systems, Inc. | Diagnostic expert in a process control system |
US6260050B1 (en) | 1999-03-23 | 2001-07-10 | Microstrategy, Inc. | System and method of adapting automatic output of service related OLAP reports to disparate output devices |
US6567796B1 (en) | 1999-03-23 | 2003-05-20 | Microstrategy, Incorporated | System and method for management of an automatic OLAP report broadcast system |
US7131070B1 (en) | 1999-05-13 | 2006-10-31 | Ricoh Company, Ltd. | Application unit monitoring and reporting system and method |
US6374369B1 (en) | 1999-05-21 | 2002-04-16 | Philips Electronics North America Corporation | Stochastic performance analysis method and apparatus therefor |
WO2000075814A1 (en) | 1999-06-03 | 2000-12-14 | Keylime Software, Inc. | System and method for monitoring user interaction with web pages |
US6434544B1 (en) | 1999-08-04 | 2002-08-13 | Hyperroll, Israel Ltd. | Stand-alone cartridge-style data aggregation server providing data aggregation for OLAP analyses |
US6385604B1 (en) | 1999-08-04 | 2002-05-07 | Hyperroll, Israel Limited | Relational database management system having integrated non-relational multi-dimensional data store of aggregated data elements |
US7392248B2 (en) | 1999-08-04 | 2008-06-24 | Hyperroll Israel, Ltd. | Data aggregation server supporting rapid query response with sparse multi-dimensional data |
US6748555B1 (en) | 1999-09-09 | 2004-06-08 | Microsoft Corporation | Object-based software management |
EP1083486A1 (en) | 1999-09-10 | 2001-03-14 | Ricoh Company, Ltd. | User interface usage monitoring and reporting system and method |
US6564174B1 (en) | 1999-09-29 | 2003-05-13 | Bmc Software, Inc. | Enterprise management system and method which indicates chaotic behavior in system resource usage for more accurate modeling and prediction |
US6418427B1 (en) | 1999-12-30 | 2002-07-09 | Decode Genetics Ehf | Online modifications of dimension structures in multidimensional processing |
US6901347B1 (en) | 2000-02-22 | 2005-05-31 | Sun Microsystems, Inc. | Availability, reliability or maintainability index including outage characterization |
US7315849B2 (en) | 2000-02-28 | 2008-01-01 | Hyperroll Israel, Ltd. | Enterprise-wide data-warehouse with integrated data aggregation engine |
US7333982B2 (en) | 2000-02-28 | 2008-02-19 | Hyperroll Israel, Ltd. | Information system having a mode of operation in which queries form one or more clients are serviced using aggregated data retrieved from a plurality of different types of data storage structures for improved query performance |
US6701363B1 (en) | 2000-02-29 | 2004-03-02 | International Business Machines Corporation | Method, computer program product, and system for deriving web transaction performance metrics |
US20010044705A1 (en) | 2000-03-10 | 2001-11-22 | Isogon Corp. | Method of normalizing software usage data from mainframe computers |
US6587970B1 (en) | 2000-03-22 | 2003-07-01 | Emc Corporation | Method and apparatus for performing site failover |
US6768986B2 (en) | 2000-04-03 | 2004-07-27 | Business Objects, S.A. | Mapping of an RDBMS schema onto a multidimensional data model |
US6862696B1 (en) | 2000-05-03 | 2005-03-01 | Cigital | System and method for software certification |
US7062483B2 (en) | 2000-05-18 | 2006-06-13 | Endeca Technologies, Inc. | Hierarchical data-driven search and navigation system and method for information retrieval |
US6601062B1 (en) | 2000-06-27 | 2003-07-29 | Ncr Corporation | Active caching for multi-dimensional data sets in relational database management system |
US7032214B1 (en) * | 2000-06-29 | 2006-04-18 | Microsoft Corporation | Performance markers to measure performance of features in a program |
US6662362B1 (en) * | 2000-07-06 | 2003-12-09 | International Business Machines Corporation | Method and system for improving performance of applications that employ a cross-language interface |
US6816898B1 (en) | 2000-08-16 | 2004-11-09 | Proactivenet, Inc. | Interfacing external metrics into a performance management system |
US20020083003A1 (en) | 2000-09-26 | 2002-06-27 | Halliday David C. | Method and apparatus for the accurate metering of software application usage and the reporting of such usage to a remote site on a public network |
US7117208B2 (en) | 2000-09-28 | 2006-10-03 | Oracle Corporation | Enterprise web mining system and method |
US20070039009A1 (en) | 2000-10-26 | 2007-02-15 | Metilinx | Multi-platform optimization model |
US7039166B1 (en) | 2001-03-05 | 2006-05-02 | Verizon Corporate Services Group Inc. | Apparatus and method for visually representing behavior of a user of an automated response system |
US20020144124A1 (en) | 2001-03-12 | 2002-10-03 | Remer Eric B. | Method and apparatus to monitor use of a program |
US7171406B2 (en) | 2001-05-18 | 2007-01-30 | Hewlett-Packard Development Company, L.P. | OLAP-based web access analysis method and system |
US6901536B2 (en) | 2001-05-24 | 2005-05-31 | Microsoft Corporation | Service quality monitoring system and method |
US7111282B2 (en) | 2001-06-12 | 2006-09-19 | Hewlett-Packard Development Company, L.P. | Instrumenting a software program and collecting data from the instrumented software program by type |
US7003766B1 (en) | 2001-06-19 | 2006-02-21 | At&T Corp. | Suite of metrics for software quality assurance and product development |
US20030009507A1 (en) * | 2001-06-29 | 2003-01-09 | Annie Shum | System and method for application performance management |
US7401331B2 (en) | 2001-08-17 | 2008-07-15 | Wu-Hon Francis Leung | Method to add new software features without modifying existing code |
US6845474B2 (en) | 2001-09-10 | 2005-01-18 | Hewlett-Packard Development Company, L.P. | Problem detector and method |
US20030115207A1 (en) | 2001-09-25 | 2003-06-19 | Bowman David M. | Hierarchical hybrid OLAP analytics generators |
US7028225B2 (en) | 2001-09-25 | 2006-04-11 | Path Communications, Inc. | Application manager for monitoring and recovery of software based application processes |
US6714940B2 (en) | 2001-11-15 | 2004-03-30 | International Business Machines Corporation | Systems, methods, and computer program products to rank and explain dimensions associated with exceptions in multidimensional data |
US6801940B1 (en) * | 2002-01-10 | 2004-10-05 | Networks Associates Technology, Inc. | Application performance monitoring expert |
US20050021293A1 (en) | 2002-02-01 | 2005-01-27 | Benedikte Elbel | Early prognosis of the reliability of a technical system |
US7080091B2 (en) | 2002-05-09 | 2006-07-18 | Oracle International Corporation | Inverted index system and method for numeric attributes |
US20040049505A1 (en) | 2002-09-11 | 2004-03-11 | Kelly Pennock | Textual on-line analytical processing method and system |
US20040088699A1 (en) * | 2002-11-06 | 2004-05-06 | Charles Suresh | System and method for measuring code segment performance |
US20040117760A1 (en) | 2002-12-11 | 2004-06-17 | Microsoft Corporation | Reality-based optimization |
US20040122646A1 (en) | 2002-12-18 | 2004-06-24 | International Business Machines Corporation | System and method for automatically building an OLAP model in a relational database |
US20060075399A1 (en) | 2002-12-27 | 2006-04-06 | Loh Choo W | System and method for resource usage prediction in the deployment of software applications |
US20040191743A1 (en) | 2003-03-26 | 2004-09-30 | International Business Machines Corporation | System and method for software development self-assessment |
US7216341B2 (en) | 2003-05-09 | 2007-05-08 | Microsoft Corporation | Instrumenting software for enhanced diagnosability |
US7197447B2 (en) | 2003-05-14 | 2007-03-27 | Microsoft Corporation | Methods and systems for analyzing software reliability and availability |
US7185231B2 (en) | 2003-05-14 | 2007-02-27 | Microsoft Corporation | Methods and systems for collecting, analyzing, and reporting software reliability and availability |
US20040230858A1 (en) | 2003-05-14 | 2004-11-18 | Microsoft Corporation | Methods and systems for analyzing software reliability and availability |
US20050015683A1 (en) | 2003-07-16 | 2005-01-20 | Lucent Technologies Inc. | Method, system and computer program product for improving system reliability |
US20050065910A1 (en) | 2003-08-08 | 2005-03-24 | Caleb Welton | Method and apparatus for storage and retrieval of information in compressed cubes |
US6963826B2 (en) | 2003-09-22 | 2005-11-08 | C3I, Inc. | Performance optimizer system and method |
US20050071807A1 (en) | 2003-09-29 | 2005-03-31 | Aura Yanavi | Methods and systems for predicting software defects in an upcoming software release |
US20050081206A1 (en) * | 2003-10-14 | 2005-04-14 | Armstrong Douglas R. | Methods and apparatus for profiling threaded programs |
US20050125777A1 (en) * | 2003-12-05 | 2005-06-09 | Brad Calder | System and method of analyzing interpreted programs |
US20050131924A1 (en) | 2003-12-15 | 2005-06-16 | Quantum Matrix Holding, Llc | System and method for multi-dimensional organization, management, and manipulation of data |
US20060106793A1 (en) | 2003-12-29 | 2006-05-18 | Ping Liang | Internet and computer information retrieval and mining with intelligent conceptual filtering, visualization and automation |
US20050182750A1 (en) | 2004-02-13 | 2005-08-18 | Memento, Inc. | System and method for instrumenting a software application |
US20050183143A1 (en) | 2004-02-13 | 2005-08-18 | Anderholm Eric J. | Methods and systems for monitoring user, application or device activity |
US20050278378A1 (en) | 2004-05-19 | 2005-12-15 | Metacarta, Inc. | Systems and methods of geographical text indexing |
US20050278290A1 (en) | 2004-06-14 | 2005-12-15 | International Business Machines Corporation | Systems, methods, and computer program products that automatically discover metadata objects and generate multidimensional models |
US20060116981A1 (en) | 2004-11-30 | 2006-06-01 | Stefan Krimmel | Method and system for automated data collection and analysis of a computer system |
US20060174346A1 (en) | 2005-01-31 | 2006-08-03 | Lieberman Software Corporation | Instrumentation for alarming a software product |
US20060218138A1 (en) | 2005-03-25 | 2006-09-28 | Microsoft Corporation | System and method for improving search relevance |
US20060242638A1 (en) | 2005-04-22 | 2006-10-26 | Microsoft Corporation | Adaptive systems and methods for making software easy to use via software usage mining |
US20060242636A1 (en) * | 2005-04-26 | 2006-10-26 | Microsoft Corporation | Variational path profiling |
US20060259981A1 (en) | 2005-05-13 | 2006-11-16 | Yaron Ben-Shoshan | System and method of controlling and monitoring computer program usage |
US7627564B2 (en) | 2005-06-21 | 2009-12-01 | Microsoft Corporation | High scale adaptive search systems and methods |
US20060287993A1 (en) | 2005-06-21 | 2006-12-21 | Microsoft Corporation | High scale adaptive search systems and methods |
US20070016672A1 (en) | 2005-07-12 | 2007-01-18 | Visible Measures, Inc. | Distributed capture and aggregation of dynamic application usage information |
US20070033201A1 (en) | 2005-08-02 | 2007-02-08 | Sap Aktiengesellschaft | Systems and methods of multidimensional software management |
US20070038974A1 (en) | 2005-08-10 | 2007-02-15 | Microsoft Corporation | Software analysis tool |
US20070038983A1 (en) | 2005-08-15 | 2007-02-15 | Sap Aktiengesellschaft | Systems and methods for enterprise software management |
US7194386B1 (en) * | 2005-10-17 | 2007-03-20 | Microsoft Corporation | Automated collection of information |
US20070101311A1 (en) * | 2005-10-21 | 2007-05-03 | International Business Machines Corporation | Method and system for generating computer-executable models using augmentations |
US20070233628A1 (en) | 2006-03-07 | 2007-10-04 | Sherwood Timothy P | Pattern matching technique for high throughput network processing |
US20080127120A1 (en) * | 2006-10-31 | 2008-05-29 | Sun Microsystems, Inc. | Method and apparatus for identifying instructions associated with execution events in a data space profiler |
US20080313507A1 (en) | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Software reliability analysis using alerts, asserts and user interface controls |
US20080312899A1 (en) | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Software feature modeling and recognition |
US20080313184A1 (en) | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Multidimensional analysis tool for high dimensional data |
US20080313617A1 (en) | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Analyzing software users with instrumentation data and user group modeling and analysis |
US20080313149A1 (en) | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Analyzing software usage with instrumentation data |
US20080313213A1 (en) | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Efficient data infrastructure for high dimensional data analysis |
Non-Patent Citations (38)
Title |
---|
"Usage Analytics For Rich Internet Applications (RIA)", 2006, Visible Measures Corp, Cambridge, MA, 2 pages. |
Bowring, et al., "Monitoring Deployed Software Using Software Tomography", Date: 2002, pp. 2-9, ACM Press, NY, USA. |
Corbett et al., "Bandera: Extracting Finite state Models from Java Source Code", 2000, Proceedings of the 22nd Intl. Conference on Software Engineering, pp. 439-448. |
Donnell, Jack, Java* Performance Profiling using the VTune(TM) Performance Analyzer White Pater, Intel, 2004. * |
Donnell, Jack, Java* Performance Profiling using the VTune™ Performance Analyzer White Pater, Intel, 2004. * |
Farina et al., "Representing Software Usage Models with Stochaistic Automata Networks", Jul. 15-19, 2002, SEKE '02, pp. 401-407. |
Fischer, "User Modeling in Human-Computer Interaction", 2001, User Modeling and User-Adapted Interaction 11, pp. 65-86. |
Gokhaie et al., "Software Reliability Analysis Incorporating Fault Detection and Debugging Activities", Nov. 1998, The 9th International Symposium on Software Reliability Engineering, 10 pages. |
Intel, VTune(TM) Performance Analyzer 8.1 for Linux* Getting Started Guide, 2006. * |
Intel, VTune™ Performance Analyzer 8.1 for Linux* Getting Started Guide, 2006. * |
Intel® VTune(TM) Performance Analyzer 7.2, 2005. * |
Intel® VTune(TM)Performance Analyzer 8.0 for Windows, 2006. * |
Intel® VTune™ Performance Analyzer 7.2, 2005. * |
Intel® VTune™Performance Analyzer 8.0 for Windows, 2006. * |
Jiawei et al., "Stream Cube: An Architecture for Multi-Dimensional Analysis of Data Streams", 2005, Distributed and Parallel Databases, v. 18, pp. 173-197. |
Kratky, Michal, "Multi-dimensional Approach to Indexing XML Data", 2004, Ph.D Thesis, Department of Computer Science, Technical University of Ostrava, Czech Republic, 127 pages. |
Li, Xiaolei, et al., "High-Dimensional OLAP: A Minimal Cubing Approach", 2004, Proceedings of the 30th VLDB Conference, pp. 528-539. |
Li, Xiaoming, et al., "Analyzing the Uses of a Software Modeling Tool", 2006, LDTA, pp. 1-18. |
Linton et al., "Building User and Expert Models by Long-Term Observation of Application Usage", 1999, Proceedings of the 7th Intl. conference on User Modeling, pp. 129-138. |
Nagappan et al., "Towards a Metric Suite for Early Software Reliability Assessment", 2003, Fast Abstract at the International Symposium on Software Reliability Engineering (ISSRE) 2003, Denver, CO., 2 pages. |
Neal, et al. "Playback: A Method for Evaluating the Usability of Software and Its Documentation", Human Factors Center, International Business Machines Corporation, San Jose, CA. Date: 1983, pp. 78-82, ACM Press, NY, USA. |
Orlandic et al., "Inverted-Space Storage Organization for Persistent Data of Very High Dimensionality", 2001, International Conference on Information Technology: Coding and Computing (ITCC '01), 6 pages. |
Puerta et al., "XIML: A Common Representation for interaction Data", Jan. 13-16, 2002, IUP02, pp. 214-215. |
Rosenberg, et al., "Software metrics and reliability", Nov. 1998, 9th International Symposium on Software Reliability, Germany, 8 pages. |
Shah, Biren, "On Improving Information Retrieval Performance from Structured, Semistructured and Un-structured Information Sources", Fall 2005, Dissertation, University of Louisiana at Lafayette, 198 pages. |
Titzer et al., "Nonintrusive Precision Instrumentation of Microcontroller Software", Jul. 2005, ACM SIGPLAN Notices, vol. 40, Issue 7, pp. 59-68. |
U.S. Notice of Allowance / Allowability dated Apr. 19, 2010 in U.S. Appl. No. 11/818,879. |
U.S. Notice of Allowance / Allowability dated Apr. 6, 2010 in U.S. Appl. No. 11/818,607. |
U.S. Notice of Allowance / Allowability dated Dec. 17, 2009 in U.S. Appl. No. 11/818,596. |
U.S. Notice of Allowance / Allowability dated Dec. 22, 2009 in U.S. Appl. No. 11/818,612. |
U.S. Notice of Allowance / Allowability dated Jan. 13, 2010 in U.S. Appl. No. 11/818,610. |
U.S. Notice of Allowance / Allowability dated Jan. 6, 2010 in U.S. Appl. No. 11/818,879. |
U.S. Official Action dated Aug. 6, 2009 in U.S. Appl. No. 11/818,610. |
U.S. Official Action dated Jul. 24, 2009 in U.S. Appl. No. 11/818,596. |
U.S. Official Action dated Oct. 5, 2009 in U.S. Appl. No. 11/818,612. |
U.S. Official Action dated Oct. 7, 2009 in U.S. Appl. No. 11/818,607. |
U.S. Official Action dated Sep. 4, 2009 in U.S. Appl. No. 11/818,879. |
Voas, "Third-Party Usage Profiling: A Model for Optimizing the Mass-Marketed Software Industry", 2000, Reliable Software Technologies. Sterling. VA, 9 pages. |
Cited By (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8626908B2 (en) * | 2005-07-12 | 2014-01-07 | Visible Measures Corp. | Distributed capture and aggregation of dynamic application usage information |
US20130111016A1 (en) * | 2005-07-12 | 2013-05-02 | Visible Measures Corp. | Distributed capture and aggregation of dynamic application usage information |
US20070162896A1 (en) * | 2006-01-10 | 2007-07-12 | Intel Corporation | Method and apparatus for generating run time profiles for program compilation |
US7975263B2 (en) * | 2006-01-10 | 2011-07-05 | Intel Corporation | Method and apparatus for generating run time profiles for program compilation |
US20100287204A1 (en) * | 2007-11-13 | 2010-11-11 | Lisa Amini | Systems and methods for using provenance information for data retention in stream-processing |
US8856313B2 (en) * | 2007-11-13 | 2014-10-07 | International Business Machines Corporation | Systems and methods for using provenance information for data retention in stream-processing |
US20090138292A1 (en) * | 2007-11-26 | 2009-05-28 | International Business Machines Corporation | Driving software product changes based on usage patterns gathered from users of previous product releases |
US9235390B1 (en) * | 2008-03-31 | 2016-01-12 | Symantec Corporation | Application optimization for use based on feature popularity |
US20100107141A1 (en) * | 2008-10-28 | 2010-04-29 | George Fitzmaurice | System and method for recommending next commands when using a software application |
US9063925B2 (en) * | 2008-10-28 | 2015-06-23 | Autodesk, Inc. | System and method for recommending next commands when using a software application |
US8589698B2 (en) * | 2009-05-15 | 2013-11-19 | International Business Machines Corporation | Integrity service using regenerated trust integrity gather program |
US20100293373A1 (en) * | 2009-05-15 | 2010-11-18 | International Business Machines Corporation | Integrity service using regenerated trust integrity gather program |
US20110061041A1 (en) * | 2009-09-04 | 2011-03-10 | International Business Machines Corporation | Reliability and availability modeling of a software application |
US20120246619A1 (en) * | 2011-03-23 | 2012-09-27 | Oracle International Corporation | Automated testing of browser based rich internet applications (ria) delivered through proprietary frameworks |
US20130042149A1 (en) * | 2011-08-09 | 2013-02-14 | International Business Machines Corporation | Analyzing a process of software defects handling using percentile-based metrics |
US9021308B2 (en) * | 2011-08-09 | 2015-04-28 | International Business Machines Corporation | Analyzing a process of software defects handling using percentile-based metrics |
US9588777B2 (en) | 2012-10-10 | 2017-03-07 | Landmark Graphics Corporation | Method and system of knowledge transfer between users of a software application |
US9916158B2 (en) | 2016-07-20 | 2018-03-13 | International Business Machines Corporation | Cognitive feature analytics |
US10083031B2 (en) | 2016-07-20 | 2018-09-25 | International Business Machines Corporation | Cognitive feature analytics |
US10042739B2 (en) * | 2016-09-29 | 2018-08-07 | International Business Machines Corporation | Real-time analytics of machine generated instrumentation data |
US10579506B2 (en) | 2016-09-29 | 2020-03-03 | International Business Machines Corporation | Real-time analytics of machine generated instrumentation data |
US11755429B2 (en) | 2021-04-23 | 2023-09-12 | EMC IP Holding Company LLC | Method, electronic device, and program product for presenting I/O write records |
Also Published As
Publication number | Publication date |
---|---|
US20080313633A1 (en) | 2008-12-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7747988B2 (en) | Software feature usage analysis and reporting | |
US7739666B2 (en) | Analyzing software users with instrumentation data and user group modeling and analysis | |
US8176476B2 (en) | Analyzing software usage with instrumentation data | |
US7681085B2 (en) | Software reliability analysis using alerts, asserts and user interface controls | |
US7680645B2 (en) | Software feature modeling and recognition | |
US7827486B2 (en) | Evaluation of visual content usage | |
EP2572294B1 (en) | System and method for sql performance assurance services | |
US20130121580A1 (en) | Analysis of service delivery processes based on interrogation of work assisted devices | |
US7908239B2 (en) | System for storing event data using a sum calculator that sums the cubes and squares of events | |
US9064037B2 (en) | Automated correlation and analysis of callstack and context data | |
Linn et al. | Desktop activity mining-a new level of detail in mining business processes | |
US20160092333A1 (en) | Telemetry for Data | |
CN107678932A (en) | Application performance analysis method and device | |
CN110928930A (en) | Software development behavior monitoring system | |
KR101830936B1 (en) | Performance Improving System Based Web for Database and Application | |
Nagappan | Analysis of execution log files | |
JP3737713B2 (en) | Traffic fluctuation factor analyzer and traffic fluctuation factor analysis program | |
US20220237160A1 (en) | Data-analysis-based facility for adjusting trace fields of a database trace | |
Serrano et al. | Webtool for the analysis and visualization of the evolution of wiki online communities | |
Eick et al. | Displaying trace files | |
CN113656391A (en) | Data detection method and device, storage medium and electronic equipment | |
Miranskyy et al. | An iterative, multi-level, and scalable approach to comparing execution traces | |
CN112416727A (en) | Batch processing operation checking method, device, equipment and medium | |
US20080155539A1 (en) | Automated Data Processing Reconstruction System And Method | |
Murphy et al. | Using activity traces to characterize programming behaviour beyond the lab |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: MICROSOFT CORPORATION, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZHU, WENLI;LI, YANTAO;WANG, QIANG;AND OTHERS;REEL/FRAME:019746/0050;SIGNING DATES FROM 20070613 TO 20070614 Owner name: MICROSOFT CORPORATION,WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZHU, WENLI;LI, YANTAO;WANG, QIANG;AND OTHERS;SIGNING DATES FROM 20070613 TO 20070614;REEL/FRAME:019746/0050 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
CC | Certificate of correction | ||
FPAY | Fee payment |
Year of fee payment: 4 |
|
AS | Assignment |
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034542/0001 Effective date: 20141014 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.) |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20220629 |