CA1318030C - Expert system for identifying failure points in a digital data processing system - Google Patents
Expert system for identifying failure points in a digital data processing systemInfo
- Publication number
- CA1318030C CA1318030C CA000592442A CA592442A CA1318030C CA 1318030 C CA1318030 C CA 1318030C CA 000592442 A CA000592442 A CA 000592442A CA 592442 A CA592442 A CA 592442A CA 1318030 C CA1318030 C CA 1318030C
- Authority
- CA
- Canada
- Prior art keywords
- failure
- error
- accordance
- entries
- error entries
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/22—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
- G06F11/2257—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing using expert systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/22—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
- G06F11/2205—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing using arrangements specific to the hardware being tested
Abstract
Abstract of the Disclosure An expert system for determining the likelihood of failure of a unit in a computer system. The operating system of the computer system maintains a log of the errors occurring for each unit in the computer system. If a predetermine number of errors have been entered in the log for a specific unit, the expert system retrieves the error entries relating to that unit and processes them to determine whether a failure is likely to occur. In this, the processing performed by the expert system is arranged so that tests relating to components of increasing particularity, and decreasing generality, are performed after the tests relating to more general components.
Description
EXPERT SYSTEM FOR IDENTIFYING
LIKELY FAILURE POINTS IN A
DIGITAL DATA PROCESSING S'~STEM
Backqround of the Invention 5 1. Field of the Invention --~
The invention relates generally to the field of digital data processing systems, and more specifically to mechanisms for diagnosing faults and errors in such systems.
10 2. Description of the Prior Art - Over the past several years, the importance of the availability of computers to government and industry has increased markedly. Computers have been developed and adapted for use not only in fairly conventional 15 activities such as bookkeeping, accounting and inventory control activities but also in more esoteric areas as design, engineering; and manufacturing. Computers have also been adapted for, use in many office activities such as document generation using word processing and graphic 20 design. Access to computerized databases, either on-site or over telephone lines, is also important in knowledge-intensive activities.
In attempting to fulfill the ever-increasing demand for both processing power and sharing of 25 information among users, individual computers are being made faster and more complex, and multiple computers are being linked in clusters or networks to facilitate sharing of data and resources, such as telecommunications links, printers, and so forth, among 30 users. As such computer systems incre?se in complexity, ;~ the likelihood of a failure, either in hardware or in J software, also increases. A number of strategies have been devised to minimize disruption, as well as service ~L
costs, which may occur as a result of a failure.
Typically, however, such strategies rely on the ~ccurrence of a failure, which may be manifested by a user noticing a disruption in service or, potentially more disastrous, the loss of data. At that point, a service technician normally attempts to identify the - -failed component and repair it, which can require several service trips to the computer site and extensive running of diagnostic programs to identify the failed component. Thus, the repair process may be somewhat costly, not only in service charges but also in the fact that the computer is either not available or provides only a degraded level of performance.
SummarY of the Invention The invention provides a new and improved expert system for use in conjunction with a digital computer system that monitors the operations of the various components of the computer system and identifies when a component is likely to fail, permitting it to be replaced. The expert system thus identifies likely points of failure in the computer system before a failure is noticeable by a user or operator. In addition, the expert system may notify the computer system's operating system of the component likely to fail so that the operating system may take measures to minimize loss of data or degradation of system performance.
In brief summary, the expert system determines the likelihood of failure of a unit in a computer system. The operating system of the computer system maintains a log of the errors occurring for each unit in -~- the computer system. If a predeterminednumber of errors have been entered in the log for a specific unit, the expert system retrieves the error entries relating to that unit and processes them to determine whether a failure is likely to occur. In this, the processing performed by tbe e~pert system is arranged so that tests relating to components of increasing particularity, and decreasing generality, are performed after the tests relating to more general components.
According to a broad aspect of the invention there is provided a method of detecting one of a plurality of likely failures of components in a digital data processing system, comprising the steps of storing a plurality of error entries, each error entry containing a plurality of differing indicia identifying components of differing types associated with a single error event in said digital data processing system, analyzing, through use of a digital expert system, said plurality of differing indicla contained within said error entries, by determinlng whether there i8 a substantlally random distribution of indicia with respect to a plurality of components of a given type or a concentration of indicia with respect to one or more components of a given type as at least one step in identifylng a pattern of indicia identifylng components of different types that corresponds with one of a plurality of failure theories, and, based on said failure theory, identifying a said likely failure of a said component.
According to another broad aspect of the invention there is provided an expert system for detecting one of a plurality of likely failures of components in a digital data processing system, :
comprising a collector module for collecting a plurality of stored error entries, each error entry containing a plurality of differing indicia identlfying components of differing types associated with a single error event in said digital data processing system, an analyzer module for analyzing said plurality of differing indicia contained within said error entries, by determining whether there is a substantially random distribution of indicia with respect to a plurality of components of a given type or a concentration of indicia with respect to one or more components of a given type as at least one step in identifying a pattern of indicia identifying components of differing types that corresponds with one of a plurality of failure theories, said analyzer module identifying a said likely failure of a said component based on said failure theory, said collector module and said analyzer module being adapted for implementation by a digital data processing system.
According to another broad aspect of the invention there is provided an expert system for detecting one of a plurality of likely failures of components in a digital data processing system, said digital data processing system comprising a plurality of units each comprising a plurality of said components, said expert system comprising an operating system for storing in an error log a plurality of error entries, each error entry containing a plurality of differing indicia identifying components of differing types associated with a single error event in said digital data 3a ~., ~3lsn3~ 60~l2-l909 processing system, a monitor module for monitoring said error entries to determine whether the number of error entries associated with a particular unit exceeds a threshold, generating a fault entry for each unit having error entries that exceed said threshold, each said fault entry identifying a unit and identifying said error entries associated with said unit, and insertlng said fault entries into a fault queue, a collector module for retrieving a fault entry from said fault queue, retrieving, from said error log, stored error entries associated with a unit identified in sald fault entry, and inserting said error entries into an error log subset, an analyzer module for analyzing said plurality of differing indlcla contained within said error entries in said error log subset, by determlning whether there is a substantially random distribution of indicia with respect to a plurality of components of a given type or a concentration of lndicia with respect to one or more components of a given type as at least one step in identifying a pattern of indicia identifying components of differing types that corresponds with one of a plurality of failure theories, said analyzer module storing sald failure theory ln a theory flle, a notification module for querying said theory file, and, based on a failure theory in said theory file, notlfylng a user of likely failure of said component, and a recovery module for querying said theory file, and, based on a failure theory in said theory file, initiating recovery 3b operations, to avoid data loss, said operating system, said monitor module, said collector module, said analyzer module, said notification module, and said recovery module being adapted for implementation by a digital data processing system.
Brief DescriPtion of_the Drawinqs This invention is pointed out with particularity in the appended claims. The above and further advantages of thls invention may be better understood by referring to the following description taken in con~unction with the accompanying drawings, in which:
Figure 1 is a functional block diagram depicting the major components of an expert system constructed in accordance with the invention; and Flgures 2A through 2D deplct flow dlagrams detailing failure analyses performed by the expert system depicted in Figure 1.
Detalled DescriDtion of an Illustrative Fmbodi~ent Figure 1 is a functional block diagram of an expert system for identifying likely failure points in a digital data processing system, that is, a computer system. Preliminarily, it will be appreciated that the expert system may be in the form of a computer program processed by the computer system. Typically, a computer system includes one or more computers. If the computer system includes a plurality of computers, the computers are interconnected by communications links to form clusters or networks to permit the sharlng of data and programs among the 3c F
60412-lgOg diverse computers.
Each computer in a computer system comprises a number of units, including one or more processors and memories, and may also include mass storage subsystems~
3d 1 31 8~30 - 4 - ` 60412-1909 such as disk and/or tape storage systems, as backup and auxiliary storage, and input/output systems such as video display terminals, printers, telecommunications links, and so forth, w~th all of the units being selectively ~nterconnected by buses. The expert system, which may be run in aonnection with any of the computers in the network or cluster, e~ detects likely failures of any of the units included therein. In one specific embodiment, the expert system detects likely ~ilures of one specific type of unit, in particular, a disk storagè
subsystem, but it will be appreciated that similar expert systems, embodying the invention, may be used to detect likely failures in any type of unit which may be included in the computer system.
lS With reference to Fig. l, the expert system includes a plurality of operational elements which communicate fault information, which relates to intermittent or permanent failures in the various units comprising the system, through a plurality of data structures. As i8 conventional, the computer system's operating system lO, which manages the various hardware and software resources included in the computer system, maintains an error log ll in which it records indicia pertaining to the various faults, errors, and so orth, arising from the operation of the units in the computer system.
When the operating system lO stores an item in the error log ll, it also enables a monitor module 12, which forms part of the expert system. Upon being enabled by the operating system lO, the monitor module categorizes the various entries in the error log ll to determine whether there are enough entries relating to possible failure of any specific unit to justify further analysi~j and, if so, generates a record for entry in a fault queue 13. It will be appreciated that the likely failure of a unit, or a component thereof, may be presaged by multiple types of errors, which may or may not be directly attributed to or associated with the unit. For example, an impending failure of a bus interface component of a unit, or of a bus wire itself, may give rise to errors indicated for various un~ts attached to the bus. The attribution of errors to the likelihood of failure of particular units depends upon the particular computer system in which the expert system is being run.
In any event, the monitor module 12, upon being enabled by the operating system 10, categorizes the various entries in the error log 11 to determine whether the errors noted therein are random or transient, or whether they indicate that any particular unit in the computer system is likely to fail. In that operation, the monitor module 12 may associate entries in the error log with particular units thereof and determine whether the number of entries associated with any particular unit exceed a predetermined threshold. If not, the monitor module 12 exits, waiting for the operating system 10 to enable it again.
On the other hand, if the monitor module.l2 determines that the number of entries associated with any particular unit exceed a predetermined threshold, the monitor module 12 generates a fault entry and inserts it into a fault queue 13. The fault entry identifies the particular unit and the entries in the error log 11 relating to the unit which prompted generation of the fault entry. In addition, if the fault entry is the first entry in the fault queue 13, the monitor module activiates a fault manager 14, which processes fault entries in the fault queue 13.
131803~
The fault manager 14 includes two modules, namely, a collector module 15 and an analyzer module 16. When initially activiated by the monitor module, the collector module retrieves a fault entry from ~he fault queue 13, identifies the unit in the computer system which prompted generation of the fault entry and retrieves, from the error log 11, the error entries in the error log 11 associated with the unit. The collector module 15 then creates an error log subset 17, which contains the error entries from the error log 11 associated with the unit, and activates the analyzer module 16.
The analyzer module 16 analyzes the error entries in the error log subset 17 provided by the collector module 15 to determine whether the unit is likely to fail. The analyzer module 16 performs the analysis in connection with various failure theories relating to the unit, as described below in detail in connection with Figs, 2A through 2D, and determines the likelihood of the unit failing. Briefly, each type of error constitutes evidence of the likelihood of failure of the unit according to one or more failure theories.
Each failure theory, in turn, relates to a specific one of a plurality of modes in which the unit may fail, with each failure mode being directed to a particular component in the unit that may fail. Each failure theory requires that a predetermined number of errors relating to the component occur in order for the expert system, and particularly the analyzer module 16, to deem the failure likely to occur.
The analyzer module 16 determines whether the error log subset 17 c¢ntains sufficient error entries - which are related to any of the various failure .: ., .
~ 3l sa3~
theories, and, if so, generates a fault theory entry identifying the unit and the fault theory and s~ores them in a theory file 20.
After the analyzer module 16 has generated a fault theory entry and stored it in the theory file 20, it activates a notification module queries the theory file 20 and, using the fault theory entries stored therein, generates an operator notification message to notify the system operator of the likelihood of a failure. Th~ operator notification message permits the operator to initiate recovery procedures in connection with the failure indicated thereby. The notification module 21 also generates a service technician notification message that includes indications of the types of errors which gave rise to the satisfaction of a failure theory and the determination that a failure is likely to occur, which may be used by a service technician during a repair operation.
Upon activiation by the analyzer module 16, the recover module 22 also queries the theory file 20 and initiates selected recovery operations which may be indicated in response to the various fault theory entries stored therein. Specifically, in one embodiment in which the expert system is used to determine the likelihood of failures of various disk storage units in a computer system, if a fault theory entry in the theory file 20 indicates that a disk storage unit may be failing, the recovery module 22 may enable the operating system to use another disk storage unit as a shadow or backup. In that case, the operating system enables data stored on the failing disk storage unit to also be stored on the other disk storage unit, thereby reducing . . . . . . . . ...
the likelihood that data will be lost. Furthermore, the data so stored is thus available on the other disk -storage unit during the repair operation.
As noted above, the analyzer module 16 analyzes the error entries in the error log subset 17 in connection with a plurality of failure theories to determine whether the unit is likely to fail. If the analyzer module 17 determines that the error entries do not satisfy any of the failure theories, it returns control to the collector module 15, which then processes the next entry in the fault queue 13. The fault manager 14 iteratively processes the entries in the fault ~ueue 13 provided by monitor module 12, until all of the entries have been processed.. It will be appreciated that the computer system may run monitor module 12 and the modules comprising the fault manager 14 concurrently, and so the monitor module 12 may load entries in the fault queue 13 concurrently with removal by the fault manager 14, and specifically by the collector module 15, for processing.
As described above, the analyzer module 16 performs analyses using the error entries in the error log subset 17 in connection with a plurality of failure theories. The failure theories are applied in a hierarchical fashion, since errors which may be attributed by the operating system lO to a plurality of components of the same type in a unit may actually provide evidence of the likely failure of another component.
For example, in a disk storage unit, while a number of errors attributed to a single read/write head may indicate likely failure of that head, random errors attributed to a plurality of read/write heads may indicate likely failure of another component which may `1 31 8030 be connected to all of those heads, such as~circuitry t~at energizes the heads for writing or that receives the read signals from the heads. To ensure that the other component is the one identified as being likely to S fail, rather than all of the heads, the analyzer module 16 performs the analysis in connection with that other component before performing the analyses in connection with the heads.
In one embodiment, the failure theori~s are divided into three groups, which are identified generally as communications failure theories, drive-detected non-media failure theories, and media failure theories. Generally, the communications failure theories are directed to errors in connection with information received from the drive storage unit, including such errors as indications of failure of the disk storage unit to execute a command within a selected timeout interval, of dropouts of certain signals from the disk storage unit~, of parity errors in signals from the disk storage unit, and so forth. The controller in the mass storage subsystem including the disk storage unit may notify the operating system 12 of any such errors in connection with requests for error recovery by means of, for example, interrupt service.
The drive-detected non-media failure theories are generally directed to errors in connection with transmissions of information to the drive storage unit by other units in the computer system. The disk storage unit may notify the operating system 12 of any such errors in connection with requests for error recovery by, for example, interrupt service.
Finally, the media failure theories are generally directed to such errors as may be indicated by timing, head/disk movement or error detection/correction - lO ~ 1 3 1 8030 circuitry, which may indicate likelihood of failure of ,-~ one or more read/write head~ the circuitry for ,~ .. ~
energizing the heads, servo circuitry, failures, such as scratches, in the media, and so forth within the disk storage unit, and may also indicate the likelihood of failure of the error detection/correction circuitry itself.
With this background, the general operations performed by one embodiment of the analysis module 16 in connection with the error entries in the error log subset 17 are depicted in Fig.s. ~A-l and 2A-2. It will be appreciated that the specific operations performed will be based on the specific units, and their components, with which the expert system (Fig. 1) is to be used. With reference to Fig. 2A-l, the analysis module 16 first performs an analysis operation in connection with the communications failure theories ~step 30). The analysis operation performed in step 30 is described below in connection with Fig. 2B. If the analysis module 16 determines that a comunications failure theory is satisfied ~step 31) it generates a fault theory entry and stores it in the theory file 20 (step 32). The analysis module 16 thereafter activates the notification module 21 to generate an appropriate message for the operator and the recover module 22 to take suitable recovery operations (step 33).
If, in step 31, the analysis module 16 determines that the error indications do not satisfy a communications failure theory, the analysis module sequence to step 34, in which it performs an analysis operation in connection with the drive-detected non-media failure theories. The analysis operation performed in step 34 is described below in connection with Fig. 2C If the analysis module 16 determines that /
1 3 1 ~030 ~ 60412-1909 a drive-detected non-media failure theory i9 satisfied (step 35) it generates a fault theory entry and stores.:.
it in the theory file 20 (step 36). The analysis;`module`.
16 thereafter activates the notification module 21 to generate an appropriate message for the operator and the recover module 22 to take suitable recovery operations (step 37).
On the other hand, if, in step 35, the analysis module 16 determines that the error ind~cations do not satisfy a drive-detected non-media failure theory, the analysis module sequences to step 40, in which it performs an analysis operation in connection with the media failure theories. The analysis operation performed in step ~ is described below in connection with Fig. 2D. If the analysis module 16 determines that a media failure theory is satisfied (step 41) it generates a fault theory entry and stores it in the theory file 20 ~step 42). The analysis module 16 thereafter activates~the notification module 21 to generate an appropriate message for the operator and the recover module 22 to take suitable recovery operations (step 43).
Finally, if, in step 41, the analysis module 16 determines that the error indications do not satisfy a media failure theory, the analysis model sequences to step 44, in which it exits and returns control to the collector module 15. The collector module 15 may then process the next fault entry from the monitor module 12 in the fault queue 13. As described above, if the fault queue 13 is empty, the collector module 15 terminates operation, pending reactivation by the monitor module 12.
As noted above, the operations performed by the analysis module 16 in connection with the communications failure theories are described in connection with Fig.
2B. Such theories relate to errors detected by the host computer or the controller controlling the disk storage unit, such as command timeouts, errors in transmission of certain signals, parity or protocol errors in S transmissions, and the like. With reference to Fig. 2B, the analyzer module 16 determines whether at least a predetermined number of the error entries in the error log subset 17 relate to communications errors as detected by controller controlling the disk st~rage unit or by the host computer (step 50). If there are insufficient numbers of such error entries, the analyzer module 16 exits the communications failure analysis (step 51).
If, on the other hand, the analyzer module 16 determines that the error log subset 17 contains sufficient numbers of such error entries, it sequences to step 52 to provide various ratios relating to error entries relating to thé communications errors and non-media drive detected errors (step 52). Since errors giving rise to error entries identifying communications entries may also give rise to error entries identifying non-media drive detected errors, the ratios provided in step 52 assist in determining which is the cause of the errors. If the ratios do not indicate that the errors are due to communications failure (step 53), the analyzer module 16 exits (step 54).
If, on the other hand, the analyzer module 16 in step 53 determines that the ratios do indicate that the errors are due to communications failure, it generates a fault theory entry so indicating for insertion into the theory file 2~ (step 55) and exits (step 56).
- 13 - 1318~3~ 6041~-1909 If the analyzer module 16 exits in either steps 51 or S3, it sequences to perform the analysis relating to drive detected non-media failure theories (step 34, Fig. 2A-l), which is depicted in Fig. 2C.
Such theories relate to errors detected by the disk storage unit, such as errors in transmission of certain signals, parity or protocol errors in transmissions, and the like. With reference to Fig. 2C, the analyzer module initially determines whether the error l;ag subset 17 contains a threshold number of error entries which relate to non-media drive-detected errors. If the error log su~set 17 does not contain the threshold number of such error entries, the analyzer module 16 sequences to setp 61, in which it exits.
On the other hand, if, in step 60, the analyzer module 16 determines that the error log subset 17 does contain the threshold number of such error entries, it then determines whether most relate to a particular type of error (step 62), and, if so, it generates a fault theory entry identifying the error type for insertion in the theory file 20 (step 63) and then exits (step 64).
If no single error type predominates among the error entries identifying the non-media drive detected errors, ~ the analyzer module 16 generates one or more fault-theory entries identifying the error types identified bythe largest number of error entries for storage in the theory file 20 (step 65). Following step 65, the analyzer module 16 exits (step 66).
If the analyzer module 16 exits in step 61, it sequences to perform analysis relating to media-related failures theories (step 40, Fig. 2A-2), which are exemplified in Figs. 2D-l through 2D-8. In brief, the media-related failure theories relate to two general classes of errors. One class, namely, random errors, - 14 - 13~8030 generally includes detection of invalid headers, loss of data synchronization, correctable or uncorrectable data errors detected by error detection and correction circuitry, and so forth. The section class of media related failure theories, namely, drive-detected errors generally related to the drive circuitry or other hardware problems, includes seek errors, off track errors, and errors due to problems with read and write circuitry, including the heads, the servo system, and so forth.
Figs. 2D-1 through 2D-8 depict a series of eight illustrative tests which the analyzer module 16 performs seriatim. The tests are ordered hierarchically so that later tests are addressed to likelihood of failure of components of increasing particularity, since, as noted above, errors which may be attributed by the operating system 10 to a plurality of more particular components of the same typé in a unit may actually provide evidence of the likely failure of another component of more general applicability. Thus, the analyzer module 16 performs a test to determine likelihood of failure of a head matrix, as shown in Fig.
2D-l, before it performs a test to determine likelihood of failure of a single head, as shown in Fig. 2D-8~
since a head matrix is a component that relates to a plurality of heads. If the analyzer module 16 were to perform the head failure test prior to performing the head matrix test, it would likely terminate testing on determining that the single head was likely to fail, and never reach the head matrix test. It will be appreciated that, if one test is satisfied, that is, if the analyzer module 16 determines from a test in one of Figs. 2D-1 through 2D-8 that a failure is likely, it does not progress to the subsequent tests.
1S -- 1 3 t 8030 60412-1909 The sequences of operations depicted in Figs.
2D-1 through 2D-8 are generally self-explanatory, and will not be described in detail. In each test, the analyzer module 16 performs a predetermined series of operations in connection with error entries in the error log subset 17. If criteria set forth in a test are satisfied, it generates a fault theory entry identifying the likely failure and inserts it into the theory file 20. Otherwise, the analyzer module 16 steps to the next test, or, in the case of the last test, exits and returns control to the collector module 15.
Generally, in the head matrix failure test depicted in Pig. 2D-l, since a head matrix relates to the operation of four read/write heads, the analyzer module 16 performs a series of steps, first, to determine whether the error log subset 17 contains error entries that relate to more than one head (step 92), and, second, to determine whether most of the error entries relate to heads related to a specific head matrix (steps 94, 96 and 100). If so, it generates a fault theory entry for storage in the theory file 20 that identifies the head matrix as being likely to fail (step 102).
If the analyzer module 16 determines, in the sequence depicted in Fig. 2D-l, that the head matrix failure is unlikely, it initiates a test, as depicted in Fig. 2D-2, to determine the likelihood of a bad disk surface, that is, a generally bad storage medium. This test requires errors generally evenly distributed among read/write heads that operate in connection with the disk surface.
If the analyzer module 16 determines that a bad disk surface is not indicated, in the sequence depicted in Fig. 2D-2, it steps to the sequence depicted in Fig.
~ 31 8030 2D-3 to perform a test to determine the likalihood of one sector of the disk being bad, which generally occurs as a result of a "head-slap", that is, a head striking the disk generally resulting in a defect in one sector.
In a disk storage unit having mulitple read/write heads for each disk surface, the analyzer module 16 determines whether at least a predetermined threshold number of error entries in the error log subsat 17 identify at least some of the heads on the same disk surface and identify an error occurring in the same sector.
If the analyzer module 17 determines that a "head slap" is not indicated, in the sequence depicted in Fig. 2D-3, it steps to the sequence depicted in Figs.
2D-4(a) and 2D-4(b) to determine the likelihood of failure of the disk storage unit's servo system. In that sequence, the analyzer module 16 first determines that error entries identify servo errors and that they relate to multiple heads in the disk storage unit. The analyzer module 16 then determines the likelihood of a circumferential scratch (step 156) or radial scratch (step 161) on the servo surface of the disk. If the analyzer module 16 determines that neither a circumferentail nor a radial scratch is likely, it determines that a general servo failure is likely.
Thereafter, the analyzer module 16 performs a read path failure test (Fig. 2D-s) to determine the likelihood of a general read path failure. If a predetermined number of error entries identify random errors, indicating detection of invalid headers, loss of data synchronization, correctable or uncorrectable data errors detected by error detection and correction circuitry, and so forth, the analyzer module 16 determines that a general read path failure is likely.
- 17 - 1 3 ~ ~ 0 3 a Three additional tests are depicted in the Figs. Following the read path failure test (Fig. 2D-5), the analyzer module 16 performs a test to determine whether heads on opposing, that is, facing, disk surfaces are li~ely to fail (Fig. 2D-6), to determine whether a radial scratch is present on a disk surface (Fig. 2D-7), and a test to indicate whether a specific read/write head is likely to fail (Fig. 2D-8). In the test depicted in Fig. 2D-7, the test is iteratively performed in connection with each of the heads in the disk storage unit.
It will be appreciated that the specific tests, and the order in which they are performed, in connection with the expert system are determined by the specific units in connection with which the expert system (Fig.
1) is to be used. If the expert system is to be used in connection with the controller of the disk storage unit, as well as the disk storage unit itself, the expert system will be required to perform additional tests directed to the controller prior to ~s performing the tests directed to the disk storage unit. Similarly, if the expert system is used in connection with a clustered or networked computer system comprising a plurality of computers which communicate over one or more communications link, the expert system may be run on one computer in the digital data processing system and may, with suitable additions to detect errors in a processor, memory, and other types of components of a computer, be used to detect the likelihood of failures occurring in other computers in the computer system.
The foregoing description has been limited to a specific embodiment of this invention. It will be apparent, however, that variations and modifications may be made to the invention, with the attainment of some or - 18 - t 31 8030 all of the advantages of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.
~
.. . . . .
LIKELY FAILURE POINTS IN A
DIGITAL DATA PROCESSING S'~STEM
Backqround of the Invention 5 1. Field of the Invention --~
The invention relates generally to the field of digital data processing systems, and more specifically to mechanisms for diagnosing faults and errors in such systems.
10 2. Description of the Prior Art - Over the past several years, the importance of the availability of computers to government and industry has increased markedly. Computers have been developed and adapted for use not only in fairly conventional 15 activities such as bookkeeping, accounting and inventory control activities but also in more esoteric areas as design, engineering; and manufacturing. Computers have also been adapted for, use in many office activities such as document generation using word processing and graphic 20 design. Access to computerized databases, either on-site or over telephone lines, is also important in knowledge-intensive activities.
In attempting to fulfill the ever-increasing demand for both processing power and sharing of 25 information among users, individual computers are being made faster and more complex, and multiple computers are being linked in clusters or networks to facilitate sharing of data and resources, such as telecommunications links, printers, and so forth, among 30 users. As such computer systems incre?se in complexity, ;~ the likelihood of a failure, either in hardware or in J software, also increases. A number of strategies have been devised to minimize disruption, as well as service ~L
costs, which may occur as a result of a failure.
Typically, however, such strategies rely on the ~ccurrence of a failure, which may be manifested by a user noticing a disruption in service or, potentially more disastrous, the loss of data. At that point, a service technician normally attempts to identify the - -failed component and repair it, which can require several service trips to the computer site and extensive running of diagnostic programs to identify the failed component. Thus, the repair process may be somewhat costly, not only in service charges but also in the fact that the computer is either not available or provides only a degraded level of performance.
SummarY of the Invention The invention provides a new and improved expert system for use in conjunction with a digital computer system that monitors the operations of the various components of the computer system and identifies when a component is likely to fail, permitting it to be replaced. The expert system thus identifies likely points of failure in the computer system before a failure is noticeable by a user or operator. In addition, the expert system may notify the computer system's operating system of the component likely to fail so that the operating system may take measures to minimize loss of data or degradation of system performance.
In brief summary, the expert system determines the likelihood of failure of a unit in a computer system. The operating system of the computer system maintains a log of the errors occurring for each unit in -~- the computer system. If a predeterminednumber of errors have been entered in the log for a specific unit, the expert system retrieves the error entries relating to that unit and processes them to determine whether a failure is likely to occur. In this, the processing performed by tbe e~pert system is arranged so that tests relating to components of increasing particularity, and decreasing generality, are performed after the tests relating to more general components.
According to a broad aspect of the invention there is provided a method of detecting one of a plurality of likely failures of components in a digital data processing system, comprising the steps of storing a plurality of error entries, each error entry containing a plurality of differing indicia identifying components of differing types associated with a single error event in said digital data processing system, analyzing, through use of a digital expert system, said plurality of differing indicla contained within said error entries, by determinlng whether there i8 a substantlally random distribution of indicia with respect to a plurality of components of a given type or a concentration of indicia with respect to one or more components of a given type as at least one step in identifylng a pattern of indicia identifylng components of different types that corresponds with one of a plurality of failure theories, and, based on said failure theory, identifying a said likely failure of a said component.
According to another broad aspect of the invention there is provided an expert system for detecting one of a plurality of likely failures of components in a digital data processing system, :
comprising a collector module for collecting a plurality of stored error entries, each error entry containing a plurality of differing indicia identlfying components of differing types associated with a single error event in said digital data processing system, an analyzer module for analyzing said plurality of differing indicia contained within said error entries, by determining whether there is a substantially random distribution of indicia with respect to a plurality of components of a given type or a concentration of indicia with respect to one or more components of a given type as at least one step in identifying a pattern of indicia identifying components of differing types that corresponds with one of a plurality of failure theories, said analyzer module identifying a said likely failure of a said component based on said failure theory, said collector module and said analyzer module being adapted for implementation by a digital data processing system.
According to another broad aspect of the invention there is provided an expert system for detecting one of a plurality of likely failures of components in a digital data processing system, said digital data processing system comprising a plurality of units each comprising a plurality of said components, said expert system comprising an operating system for storing in an error log a plurality of error entries, each error entry containing a plurality of differing indicia identifying components of differing types associated with a single error event in said digital data 3a ~., ~3lsn3~ 60~l2-l909 processing system, a monitor module for monitoring said error entries to determine whether the number of error entries associated with a particular unit exceeds a threshold, generating a fault entry for each unit having error entries that exceed said threshold, each said fault entry identifying a unit and identifying said error entries associated with said unit, and insertlng said fault entries into a fault queue, a collector module for retrieving a fault entry from said fault queue, retrieving, from said error log, stored error entries associated with a unit identified in sald fault entry, and inserting said error entries into an error log subset, an analyzer module for analyzing said plurality of differing indlcla contained within said error entries in said error log subset, by determlning whether there is a substantially random distribution of indicia with respect to a plurality of components of a given type or a concentration of lndicia with respect to one or more components of a given type as at least one step in identifying a pattern of indicia identifying components of differing types that corresponds with one of a plurality of failure theories, said analyzer module storing sald failure theory ln a theory flle, a notification module for querying said theory file, and, based on a failure theory in said theory file, notlfylng a user of likely failure of said component, and a recovery module for querying said theory file, and, based on a failure theory in said theory file, initiating recovery 3b operations, to avoid data loss, said operating system, said monitor module, said collector module, said analyzer module, said notification module, and said recovery module being adapted for implementation by a digital data processing system.
Brief DescriPtion of_the Drawinqs This invention is pointed out with particularity in the appended claims. The above and further advantages of thls invention may be better understood by referring to the following description taken in con~unction with the accompanying drawings, in which:
Figure 1 is a functional block diagram depicting the major components of an expert system constructed in accordance with the invention; and Flgures 2A through 2D deplct flow dlagrams detailing failure analyses performed by the expert system depicted in Figure 1.
Detalled DescriDtion of an Illustrative Fmbodi~ent Figure 1 is a functional block diagram of an expert system for identifying likely failure points in a digital data processing system, that is, a computer system. Preliminarily, it will be appreciated that the expert system may be in the form of a computer program processed by the computer system. Typically, a computer system includes one or more computers. If the computer system includes a plurality of computers, the computers are interconnected by communications links to form clusters or networks to permit the sharlng of data and programs among the 3c F
60412-lgOg diverse computers.
Each computer in a computer system comprises a number of units, including one or more processors and memories, and may also include mass storage subsystems~
3d 1 31 8~30 - 4 - ` 60412-1909 such as disk and/or tape storage systems, as backup and auxiliary storage, and input/output systems such as video display terminals, printers, telecommunications links, and so forth, w~th all of the units being selectively ~nterconnected by buses. The expert system, which may be run in aonnection with any of the computers in the network or cluster, e~ detects likely failures of any of the units included therein. In one specific embodiment, the expert system detects likely ~ilures of one specific type of unit, in particular, a disk storagè
subsystem, but it will be appreciated that similar expert systems, embodying the invention, may be used to detect likely failures in any type of unit which may be included in the computer system.
lS With reference to Fig. l, the expert system includes a plurality of operational elements which communicate fault information, which relates to intermittent or permanent failures in the various units comprising the system, through a plurality of data structures. As i8 conventional, the computer system's operating system lO, which manages the various hardware and software resources included in the computer system, maintains an error log ll in which it records indicia pertaining to the various faults, errors, and so orth, arising from the operation of the units in the computer system.
When the operating system lO stores an item in the error log ll, it also enables a monitor module 12, which forms part of the expert system. Upon being enabled by the operating system lO, the monitor module categorizes the various entries in the error log ll to determine whether there are enough entries relating to possible failure of any specific unit to justify further analysi~j and, if so, generates a record for entry in a fault queue 13. It will be appreciated that the likely failure of a unit, or a component thereof, may be presaged by multiple types of errors, which may or may not be directly attributed to or associated with the unit. For example, an impending failure of a bus interface component of a unit, or of a bus wire itself, may give rise to errors indicated for various un~ts attached to the bus. The attribution of errors to the likelihood of failure of particular units depends upon the particular computer system in which the expert system is being run.
In any event, the monitor module 12, upon being enabled by the operating system 10, categorizes the various entries in the error log 11 to determine whether the errors noted therein are random or transient, or whether they indicate that any particular unit in the computer system is likely to fail. In that operation, the monitor module 12 may associate entries in the error log with particular units thereof and determine whether the number of entries associated with any particular unit exceed a predetermined threshold. If not, the monitor module 12 exits, waiting for the operating system 10 to enable it again.
On the other hand, if the monitor module.l2 determines that the number of entries associated with any particular unit exceed a predetermined threshold, the monitor module 12 generates a fault entry and inserts it into a fault queue 13. The fault entry identifies the particular unit and the entries in the error log 11 relating to the unit which prompted generation of the fault entry. In addition, if the fault entry is the first entry in the fault queue 13, the monitor module activiates a fault manager 14, which processes fault entries in the fault queue 13.
131803~
The fault manager 14 includes two modules, namely, a collector module 15 and an analyzer module 16. When initially activiated by the monitor module, the collector module retrieves a fault entry from ~he fault queue 13, identifies the unit in the computer system which prompted generation of the fault entry and retrieves, from the error log 11, the error entries in the error log 11 associated with the unit. The collector module 15 then creates an error log subset 17, which contains the error entries from the error log 11 associated with the unit, and activates the analyzer module 16.
The analyzer module 16 analyzes the error entries in the error log subset 17 provided by the collector module 15 to determine whether the unit is likely to fail. The analyzer module 16 performs the analysis in connection with various failure theories relating to the unit, as described below in detail in connection with Figs, 2A through 2D, and determines the likelihood of the unit failing. Briefly, each type of error constitutes evidence of the likelihood of failure of the unit according to one or more failure theories.
Each failure theory, in turn, relates to a specific one of a plurality of modes in which the unit may fail, with each failure mode being directed to a particular component in the unit that may fail. Each failure theory requires that a predetermined number of errors relating to the component occur in order for the expert system, and particularly the analyzer module 16, to deem the failure likely to occur.
The analyzer module 16 determines whether the error log subset 17 c¢ntains sufficient error entries - which are related to any of the various failure .: ., .
~ 3l sa3~
theories, and, if so, generates a fault theory entry identifying the unit and the fault theory and s~ores them in a theory file 20.
After the analyzer module 16 has generated a fault theory entry and stored it in the theory file 20, it activates a notification module queries the theory file 20 and, using the fault theory entries stored therein, generates an operator notification message to notify the system operator of the likelihood of a failure. Th~ operator notification message permits the operator to initiate recovery procedures in connection with the failure indicated thereby. The notification module 21 also generates a service technician notification message that includes indications of the types of errors which gave rise to the satisfaction of a failure theory and the determination that a failure is likely to occur, which may be used by a service technician during a repair operation.
Upon activiation by the analyzer module 16, the recover module 22 also queries the theory file 20 and initiates selected recovery operations which may be indicated in response to the various fault theory entries stored therein. Specifically, in one embodiment in which the expert system is used to determine the likelihood of failures of various disk storage units in a computer system, if a fault theory entry in the theory file 20 indicates that a disk storage unit may be failing, the recovery module 22 may enable the operating system to use another disk storage unit as a shadow or backup. In that case, the operating system enables data stored on the failing disk storage unit to also be stored on the other disk storage unit, thereby reducing . . . . . . . . ...
the likelihood that data will be lost. Furthermore, the data so stored is thus available on the other disk -storage unit during the repair operation.
As noted above, the analyzer module 16 analyzes the error entries in the error log subset 17 in connection with a plurality of failure theories to determine whether the unit is likely to fail. If the analyzer module 17 determines that the error entries do not satisfy any of the failure theories, it returns control to the collector module 15, which then processes the next entry in the fault queue 13. The fault manager 14 iteratively processes the entries in the fault ~ueue 13 provided by monitor module 12, until all of the entries have been processed.. It will be appreciated that the computer system may run monitor module 12 and the modules comprising the fault manager 14 concurrently, and so the monitor module 12 may load entries in the fault queue 13 concurrently with removal by the fault manager 14, and specifically by the collector module 15, for processing.
As described above, the analyzer module 16 performs analyses using the error entries in the error log subset 17 in connection with a plurality of failure theories. The failure theories are applied in a hierarchical fashion, since errors which may be attributed by the operating system lO to a plurality of components of the same type in a unit may actually provide evidence of the likely failure of another component.
For example, in a disk storage unit, while a number of errors attributed to a single read/write head may indicate likely failure of that head, random errors attributed to a plurality of read/write heads may indicate likely failure of another component which may `1 31 8030 be connected to all of those heads, such as~circuitry t~at energizes the heads for writing or that receives the read signals from the heads. To ensure that the other component is the one identified as being likely to S fail, rather than all of the heads, the analyzer module 16 performs the analysis in connection with that other component before performing the analyses in connection with the heads.
In one embodiment, the failure theori~s are divided into three groups, which are identified generally as communications failure theories, drive-detected non-media failure theories, and media failure theories. Generally, the communications failure theories are directed to errors in connection with information received from the drive storage unit, including such errors as indications of failure of the disk storage unit to execute a command within a selected timeout interval, of dropouts of certain signals from the disk storage unit~, of parity errors in signals from the disk storage unit, and so forth. The controller in the mass storage subsystem including the disk storage unit may notify the operating system 12 of any such errors in connection with requests for error recovery by means of, for example, interrupt service.
The drive-detected non-media failure theories are generally directed to errors in connection with transmissions of information to the drive storage unit by other units in the computer system. The disk storage unit may notify the operating system 12 of any such errors in connection with requests for error recovery by, for example, interrupt service.
Finally, the media failure theories are generally directed to such errors as may be indicated by timing, head/disk movement or error detection/correction - lO ~ 1 3 1 8030 circuitry, which may indicate likelihood of failure of ,-~ one or more read/write head~ the circuitry for ,~ .. ~
energizing the heads, servo circuitry, failures, such as scratches, in the media, and so forth within the disk storage unit, and may also indicate the likelihood of failure of the error detection/correction circuitry itself.
With this background, the general operations performed by one embodiment of the analysis module 16 in connection with the error entries in the error log subset 17 are depicted in Fig.s. ~A-l and 2A-2. It will be appreciated that the specific operations performed will be based on the specific units, and their components, with which the expert system (Fig. 1) is to be used. With reference to Fig. 2A-l, the analysis module 16 first performs an analysis operation in connection with the communications failure theories ~step 30). The analysis operation performed in step 30 is described below in connection with Fig. 2B. If the analysis module 16 determines that a comunications failure theory is satisfied ~step 31) it generates a fault theory entry and stores it in the theory file 20 (step 32). The analysis module 16 thereafter activates the notification module 21 to generate an appropriate message for the operator and the recover module 22 to take suitable recovery operations (step 33).
If, in step 31, the analysis module 16 determines that the error indications do not satisfy a communications failure theory, the analysis module sequence to step 34, in which it performs an analysis operation in connection with the drive-detected non-media failure theories. The analysis operation performed in step 34 is described below in connection with Fig. 2C If the analysis module 16 determines that /
1 3 1 ~030 ~ 60412-1909 a drive-detected non-media failure theory i9 satisfied (step 35) it generates a fault theory entry and stores.:.
it in the theory file 20 (step 36). The analysis;`module`.
16 thereafter activates the notification module 21 to generate an appropriate message for the operator and the recover module 22 to take suitable recovery operations (step 37).
On the other hand, if, in step 35, the analysis module 16 determines that the error ind~cations do not satisfy a drive-detected non-media failure theory, the analysis module sequences to step 40, in which it performs an analysis operation in connection with the media failure theories. The analysis operation performed in step ~ is described below in connection with Fig. 2D. If the analysis module 16 determines that a media failure theory is satisfied (step 41) it generates a fault theory entry and stores it in the theory file 20 ~step 42). The analysis module 16 thereafter activates~the notification module 21 to generate an appropriate message for the operator and the recover module 22 to take suitable recovery operations (step 43).
Finally, if, in step 41, the analysis module 16 determines that the error indications do not satisfy a media failure theory, the analysis model sequences to step 44, in which it exits and returns control to the collector module 15. The collector module 15 may then process the next fault entry from the monitor module 12 in the fault queue 13. As described above, if the fault queue 13 is empty, the collector module 15 terminates operation, pending reactivation by the monitor module 12.
As noted above, the operations performed by the analysis module 16 in connection with the communications failure theories are described in connection with Fig.
2B. Such theories relate to errors detected by the host computer or the controller controlling the disk storage unit, such as command timeouts, errors in transmission of certain signals, parity or protocol errors in S transmissions, and the like. With reference to Fig. 2B, the analyzer module 16 determines whether at least a predetermined number of the error entries in the error log subset 17 relate to communications errors as detected by controller controlling the disk st~rage unit or by the host computer (step 50). If there are insufficient numbers of such error entries, the analyzer module 16 exits the communications failure analysis (step 51).
If, on the other hand, the analyzer module 16 determines that the error log subset 17 contains sufficient numbers of such error entries, it sequences to step 52 to provide various ratios relating to error entries relating to thé communications errors and non-media drive detected errors (step 52). Since errors giving rise to error entries identifying communications entries may also give rise to error entries identifying non-media drive detected errors, the ratios provided in step 52 assist in determining which is the cause of the errors. If the ratios do not indicate that the errors are due to communications failure (step 53), the analyzer module 16 exits (step 54).
If, on the other hand, the analyzer module 16 in step 53 determines that the ratios do indicate that the errors are due to communications failure, it generates a fault theory entry so indicating for insertion into the theory file 2~ (step 55) and exits (step 56).
- 13 - 1318~3~ 6041~-1909 If the analyzer module 16 exits in either steps 51 or S3, it sequences to perform the analysis relating to drive detected non-media failure theories (step 34, Fig. 2A-l), which is depicted in Fig. 2C.
Such theories relate to errors detected by the disk storage unit, such as errors in transmission of certain signals, parity or protocol errors in transmissions, and the like. With reference to Fig. 2C, the analyzer module initially determines whether the error l;ag subset 17 contains a threshold number of error entries which relate to non-media drive-detected errors. If the error log su~set 17 does not contain the threshold number of such error entries, the analyzer module 16 sequences to setp 61, in which it exits.
On the other hand, if, in step 60, the analyzer module 16 determines that the error log subset 17 does contain the threshold number of such error entries, it then determines whether most relate to a particular type of error (step 62), and, if so, it generates a fault theory entry identifying the error type for insertion in the theory file 20 (step 63) and then exits (step 64).
If no single error type predominates among the error entries identifying the non-media drive detected errors, ~ the analyzer module 16 generates one or more fault-theory entries identifying the error types identified bythe largest number of error entries for storage in the theory file 20 (step 65). Following step 65, the analyzer module 16 exits (step 66).
If the analyzer module 16 exits in step 61, it sequences to perform analysis relating to media-related failures theories (step 40, Fig. 2A-2), which are exemplified in Figs. 2D-l through 2D-8. In brief, the media-related failure theories relate to two general classes of errors. One class, namely, random errors, - 14 - 13~8030 generally includes detection of invalid headers, loss of data synchronization, correctable or uncorrectable data errors detected by error detection and correction circuitry, and so forth. The section class of media related failure theories, namely, drive-detected errors generally related to the drive circuitry or other hardware problems, includes seek errors, off track errors, and errors due to problems with read and write circuitry, including the heads, the servo system, and so forth.
Figs. 2D-1 through 2D-8 depict a series of eight illustrative tests which the analyzer module 16 performs seriatim. The tests are ordered hierarchically so that later tests are addressed to likelihood of failure of components of increasing particularity, since, as noted above, errors which may be attributed by the operating system 10 to a plurality of more particular components of the same typé in a unit may actually provide evidence of the likely failure of another component of more general applicability. Thus, the analyzer module 16 performs a test to determine likelihood of failure of a head matrix, as shown in Fig.
2D-l, before it performs a test to determine likelihood of failure of a single head, as shown in Fig. 2D-8~
since a head matrix is a component that relates to a plurality of heads. If the analyzer module 16 were to perform the head failure test prior to performing the head matrix test, it would likely terminate testing on determining that the single head was likely to fail, and never reach the head matrix test. It will be appreciated that, if one test is satisfied, that is, if the analyzer module 16 determines from a test in one of Figs. 2D-1 through 2D-8 that a failure is likely, it does not progress to the subsequent tests.
1S -- 1 3 t 8030 60412-1909 The sequences of operations depicted in Figs.
2D-1 through 2D-8 are generally self-explanatory, and will not be described in detail. In each test, the analyzer module 16 performs a predetermined series of operations in connection with error entries in the error log subset 17. If criteria set forth in a test are satisfied, it generates a fault theory entry identifying the likely failure and inserts it into the theory file 20. Otherwise, the analyzer module 16 steps to the next test, or, in the case of the last test, exits and returns control to the collector module 15.
Generally, in the head matrix failure test depicted in Pig. 2D-l, since a head matrix relates to the operation of four read/write heads, the analyzer module 16 performs a series of steps, first, to determine whether the error log subset 17 contains error entries that relate to more than one head (step 92), and, second, to determine whether most of the error entries relate to heads related to a specific head matrix (steps 94, 96 and 100). If so, it generates a fault theory entry for storage in the theory file 20 that identifies the head matrix as being likely to fail (step 102).
If the analyzer module 16 determines, in the sequence depicted in Fig. 2D-l, that the head matrix failure is unlikely, it initiates a test, as depicted in Fig. 2D-2, to determine the likelihood of a bad disk surface, that is, a generally bad storage medium. This test requires errors generally evenly distributed among read/write heads that operate in connection with the disk surface.
If the analyzer module 16 determines that a bad disk surface is not indicated, in the sequence depicted in Fig. 2D-2, it steps to the sequence depicted in Fig.
~ 31 8030 2D-3 to perform a test to determine the likalihood of one sector of the disk being bad, which generally occurs as a result of a "head-slap", that is, a head striking the disk generally resulting in a defect in one sector.
In a disk storage unit having mulitple read/write heads for each disk surface, the analyzer module 16 determines whether at least a predetermined threshold number of error entries in the error log subsat 17 identify at least some of the heads on the same disk surface and identify an error occurring in the same sector.
If the analyzer module 17 determines that a "head slap" is not indicated, in the sequence depicted in Fig. 2D-3, it steps to the sequence depicted in Figs.
2D-4(a) and 2D-4(b) to determine the likelihood of failure of the disk storage unit's servo system. In that sequence, the analyzer module 16 first determines that error entries identify servo errors and that they relate to multiple heads in the disk storage unit. The analyzer module 16 then determines the likelihood of a circumferential scratch (step 156) or radial scratch (step 161) on the servo surface of the disk. If the analyzer module 16 determines that neither a circumferentail nor a radial scratch is likely, it determines that a general servo failure is likely.
Thereafter, the analyzer module 16 performs a read path failure test (Fig. 2D-s) to determine the likelihood of a general read path failure. If a predetermined number of error entries identify random errors, indicating detection of invalid headers, loss of data synchronization, correctable or uncorrectable data errors detected by error detection and correction circuitry, and so forth, the analyzer module 16 determines that a general read path failure is likely.
- 17 - 1 3 ~ ~ 0 3 a Three additional tests are depicted in the Figs. Following the read path failure test (Fig. 2D-5), the analyzer module 16 performs a test to determine whether heads on opposing, that is, facing, disk surfaces are li~ely to fail (Fig. 2D-6), to determine whether a radial scratch is present on a disk surface (Fig. 2D-7), and a test to indicate whether a specific read/write head is likely to fail (Fig. 2D-8). In the test depicted in Fig. 2D-7, the test is iteratively performed in connection with each of the heads in the disk storage unit.
It will be appreciated that the specific tests, and the order in which they are performed, in connection with the expert system are determined by the specific units in connection with which the expert system (Fig.
1) is to be used. If the expert system is to be used in connection with the controller of the disk storage unit, as well as the disk storage unit itself, the expert system will be required to perform additional tests directed to the controller prior to ~s performing the tests directed to the disk storage unit. Similarly, if the expert system is used in connection with a clustered or networked computer system comprising a plurality of computers which communicate over one or more communications link, the expert system may be run on one computer in the digital data processing system and may, with suitable additions to detect errors in a processor, memory, and other types of components of a computer, be used to detect the likelihood of failures occurring in other computers in the computer system.
The foregoing description has been limited to a specific embodiment of this invention. It will be apparent, however, that variations and modifications may be made to the invention, with the attainment of some or - 18 - t 31 8030 all of the advantages of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.
~
.. . . . .
Claims (51)
1. A method of detecting one of a plurality of likely failures of components in a digital data processing system, comprising the steps of storing a plurality of error entries, each error entry containing a plurality of differing indicia identifying components of differing types associated with a single error event in said digital data processing system, analyzing, through use of a digital expert system, said plurality of differing indicia contained within said error entries, by determining whether there is a substantially random distribution of indicia with respect to a plurality of components of a given type or a concentration of indicia with respect to one or more components of a given type as at least one step in identifying a pattern of indicia identifying components of different types that corresponds with one of a plurality of failure theories, and, based on said failure theory, identifying a said likely failure of a said component.
2. A method in accordance with claim 1, wherein said digital data processing system comprises a plurality of units each comprising a plurality of components, said method further comprises the steps of monitoring said stored error entries to determine whether the number of stored error entries associated with a particular unit exceeds a threshold, and collecting said stored error entries associated with said particular unit when said number of said stored entries associated with said particular unit exceeds said threshold, and said analysis step comprises analyzing said collected error entries associated with said particular unit, to identify said pattern of said different types of error events that corresponds with said one of said plurality of failure theories.
3. A method in accordance with claim 2, wherein said step of monitoring error entries associated with a particular unit comprises generating a fault entry for each unit having error entries that exceed said threshold, and inserting said fault entries into a fault queue.
4. A method in accordance with claim 3, wherein each said fault entry identifies a unit and identifies error entries associated with said unit.
5. A method in accordance with claim 4, wherein said step of collecting said error entries associated with said particular unit comprises retrieving a fault entry from said fault queue, retrieving, from an error log, said error entries associated with said unit identified in said fault entry, and inserting said error entries into an error log subset.
6. A method in accordance with claim 1, wherein said failure theories comprise communication failure theories, drive-detected non-media failure theories, and media failure theories.
7. A method in accordance with claim 1, wherein said analyzing step comprises a hierarchical sequence of pattern analysis steps in which said error entries are tested for patterns relating to failures of relative generality before said error entries are tested for patterns relating to failures of relative particularity.
8. A method in accordance with claim 1, wherein said step of storing said error entries is performed by an operating system that stores said error entries in an error log.
9. A method in accordance with claim 1 , wherein said analyzing step comprises storing said failure theories in a theory file.
10. A method in accordance with claim 9, wherein said step of identifying a likely failure of a component comprises querying said theory file, and, based on a failure theory in said theory file, notifying a user of a likely failure of said component.
11. A method in accordance with claim 9, wherein said step of identifying a likely failure of a component comprises querying said theory file, and, based on a failure theory in said theory file, initiating recovery operations, to avoid data loss.
12 A method in accordance with claim 1, wherein said step of identifying a likely failure of a component comprises predicting failure of a component before said component actually fails.
13. A method in accordance with claim 1, wherein said step of identifying a likely failure of a component comprises detecting an actual failure of a component.
14. A method in accordance with claim 1, wherein said step of analyzing said error entries comprises the steps of performing analysis in connection with communications failure theories, if no communications failure theories are satisfied, performing analysis in connection with drive-detected non-media failure theories, and if no drive-detected non-media failure theories are satisfied, performing analysis in connection with media failure theories.
15. A method in accordance with claim 1, wherein said step of analyzing said error entries comprises the steps of determining whether a sufficient number of error entries that identify communications errors have been stored to justify generating a fault theory entry indicating a likely communication failure, if said sufficient number of error entries that identify communications errors have been stored, comparing at least one number representing occurrences of said error entries that identify communications errors with at least one number representing occurrences of error entries that identify non-media drive-detected errors, and determining whether a ratio of said number representing occurrences of error entries that identify communications errors to said number representing occurrences of error entries that identify non-media drive-detected errors is sufficient to justify generating a fault theory indicating a likely communications failure
16. A method in accordance with claim 1, wherein said step of analyzing said error entries comprises the steps of determining whether a sufficient number of error entries that identify non-media drive-detected errors have been stored to justify generating a fault theory entry indicating a likely drive-detected non-media failure, if said sufficient number of said error entries that identify non-media drive-detected errors have been stored, and if most of said error entries that identify non-media drive-detected errors identify a common error type, generating a fault theory entry indicating a likelihood of said common error type, and if said sufficient number of said error entries that identify non-media drive-detected errors have been stored, and if moot of said error entries that identify non-media drive-detected errors do no identify a common error type, generating fault theory entries identifying error types most frequently identified by said error entries that identify non-media drive-detected errors.
17. A method in accordance with claim 1, wherein said step of analyzing said error entries comprises the steps of performing analysis in connection with head matrix failure, if no head matrix failure is likely, performing analysis in connection with bad surfaces, if there are no likely bad surfaces, performing analysis in connection with head slaps, if no head slap failure is likely, performing analysis in connection with errors directed to a servo system, if no servo system failure is likely, performing analysis in connection with read path failure, if no read path failure is likely, performing analysis in connection with bad heads on opposing media surfaces, if there are no likely bad heads on opposing media surfaces, performing analysis in connection with radial scratches, and, if there are no likely radial scratches, performing analysis in connection with bad heads.
18 A method in accordance with claim 1, further comprising the step of initiating recovery operations to avoid loss of data.
19. A method in accordance with claim 18, wherein said step of initiating recovery operations comprises enabling an operating system to use a substitute disk storage unit as a backup for a likely-to-fail disk storage unit.
20. A method in accordance with claim 18, wherein said step of initiating recovery operations comprises substituting a redundant component for a likely-to-fail component.
21. A method in accordance with claim 18, wherein said step of initiating recovery operations comprises transferring data from a likely-to-fail component to a substitute component.
22. A method in accordance with claim 18, wherein said step of initiating recovery operations comprises reconfiguring of redundant components in a manner such that said digital data processing system is serviced without interruption of operations and without loss of data.
23. A method in accordance with claim 22, wherein said step of initiating recovery operations is embedded in software that automatically notifies a maintenance utility that service is required.
24. A method in accordance with claim 23, wherein said software identifies a likely-to-fail component to said maintenance utility.
25. An expert system for detecting one of a plurality of likely failures of components in a digital data processing system, comprising a collector module for collecting a plurality of stored error entries, each error entry containing a plurality of differing indicia identifying components of differing types associated with a single error event in said digital data processing system, an analyzer module for analyzing said plurality of differing indicia contained within said error entries, by determining whether there is a substantially random distribution of indicia with respect to a plurality of components of a given type or a concentration of indicia with respect to one or more components of a given type as at least one step in identifying a pattern of indicia identifying components of differing types that corresponds with one of a plurality of failure theories, said analyzer module identifying a said likely failure of a said component based on said failure theory, said collector module and said analyzer module being adapted for implementation by a digital data processing system.
26. An expert system in accordance with claim 25, wherein said digital data processing system comprises a plurality of units each comprising a plurality of components, said digital data processing system further comprises a monitor module for monitoring said error entries to determine whether the number of error entries associated with a particular unit exceeds a threshold, and said collector module collects said error entries associated with said particular unit for purposes of pattern analysis.
27. An expert system in accordance with claim 26, wherein said monitor module generates a fault entry for each unit having error entries that exceed said threshold, and inserts said fault entries into a fault queue.
28. An expert system in accordance with claim 27, wherein each said fault entry identifies a unit and identifies error entries associated with said unit.
29. An expert system in accordance with claim 28, wherein said collector module retrieves a fault entry from said fault queue, retrieves, from an error log, said stored error entries associated with said unit identified in said fault entry, and inserts said error entries into an error log subset.
30. An expert system in accordance with claim 25, wherein said failure theories comprise communication failure theories, drive-detected non-media failure theories, and media failure theories.
31. An expert system in accordance with claim 25, wherein said analyzer module analyzes said error entries in a hierarchical sequence of pattern analysis steps in which said error entries are tested for patterns relating to failures of relative generality before said error entries are tested for patterns relating to failures of relative particularity.
32. An expert system in accordance with claim 25, further comprising an operating system for storing said error entries in an error log.
33. An expert system in accordance with claim 25, wherein said analyzer module stores said failure theories in a theory file.
34. An expert system in accordance with claim 33, further comprising a notification module for querying said theory file, and, based on a failure theory in said theory file, notifying a user of likely failure of said component.
35. An expert system in accordance with claim 33, further comprising a recovery module for querying said theory file, and, based on a failure theory in said theory file, initiating recovery operations, to avoid data loss.
36. An expert system in accordance with claim 25, wherein said analyzer module predicts failure of a component before said component actually fails.
37. An expert system in accordance with claim 25, wherein said analyzer module detects an actual failure of a component.
38. An expert system in accordance with claim 25, wherein said analyzer module performs analysis of said error entries in connection with communications failure theories, if no communications failure theories are satisfied, performs analysis of said error entries in connection with drive-detected non-media failure theories, and if no drive-detected non-media failure theories are satisfied, performs analysis of said error entries in connection with media failure theories.
39. An expert system in accordance with claim 25 , wherein said analyzer module determines whether a sufficient number of error entries that identify communications errors have been stored to justify generating a fault theory entry indicating a likely communications failure, if said sufficient number of error entries that identify communications errors have been stored, said analyzer module compares at least one number representing occurrences of said error entries that identify communications errors with at least one number representing occurrences of error entries that identify non-media drive-detected errors, and said analyzer module determines whether a ratio of said number representing occurrences of error entries that identify communications errors to said number representing occurrences of error entries that identify non-media drive-detected errors is sufficient to justify generating a fault theory indicating a likely communications failure.
40. An expert system in accordance with claim 25, wherein said analyzer module determines whether a sufficient number of error entries that identify non-media drive-detected errors have been stored to justify generating a fault theory entry indicating a likely drive-detected non-media failure, if said sufficient number of said error entries that identify non-media drive-detected errors have been stored, and if most of said error entries that identify non-media drive-detected errors identify a common error type, said analyzer module generates a fault theory entry indicating a likelihood of said common error type, and if said sufficient number of said error entries that identify non-media drive-detected errors have been stored, and if most of said error entries that identify non-media drive-detected errors do no identify a common error type, said analyzer module generates fault theory entries identifying error types most frequently identified by said error entries that identify non-media drive-detected errors.
41. An expert system in accordance with claim 25, wherein said analyzer module performs analysis in connection with head matrix failure, if no head matrix failure is likely, said analyzer module performs analysis in connection with bad surfaces, if there are no likely bad surfaces, said analyzer module performs analysis in connection with head slaps, if no head slap failure is likely, said analyzer module performs analysis in connection with errors directed to a servo system, if no servo system failure is likely, said analyzer module performs analysis in connection with read path failure, if no read path failure is likely, said analyzer module performs analysis in connection with bad heads on opposing media surfaces, if there are no likely bad heads on opposing media surfaces, said analyzer module performs analysis in connection with radial scratches, and if there are no likely radial scratches, said analyzer module performs analysis in connection with bad heads.
42. An expert system in accordance with claim 25, further comprising a recovery module for initiating recovery operations, based on said failure theory, to avoid loss of data.
43. An expert system in accordance with claim 42, wherein said recovery module enables an operating system to use a substitute disk storage unit as a backup for a likely-to-fail disk storage unit.
44. An expert system in accordance with claim 42, wherein said recovery module substitutes a redundant component for a likely-to-fail component.
45. An expert system in accordance with clalm 42 , wherein said recovery module transfers data from a likely-to-fail component to a substitute component.
46. An expert system in accordance with claim 42, wherein said recovery module reconfigures redundant components in a manner such that said digital data processing system is serviced without interruption of operations and without loss of data.
47. An expert system in accordance with claim 46, wherein said recovery module comprises software that automatically notifies a maintenance utility that service is required.
48. An expert system in accordance with claim 47, wherein said software identifies a likely-to-fail component to said maintenance utility.
49. An expert system for detecting one of a plurality of likely failures of components in a digital data processing system, said digital data processing system comprising a plurality of units each comprising a plurality of said components, said expert system comprising an operating system for storing in an error log a plurality of error entries, each error entry containing a plurality of differing indicia identifying components of differing types associated with a single error event in said digital data processing system, a monitor module for monitoring said error entries to determine whether the number of error entries associated with a particular unit exceeds a threshold, generating a fault entry for each unit having error entries that exceed said threshold, each said fault entry identifying a unit and identifying said error entries associated with said unit, and inserting said fault entries into a fault queue, a collector module for retrieving a fault entry from said fault queue, retrieving, from said error log, stored error entries associated with a unit identified in said fault entry, and inserting said error entries into an error log subset, an analyzer module for analyzing said plurality of differing indicia contained within said error entries in said error log subset, by determining whether there is a substantially random distribution of indicia with respect to a plurality of components of a given type or a concentration of indicia with respect to one or more components of a given type as at least one step in identifying a pattern of indicia identifying components of differing types that corresponds with one of a plurality of failure theories, said analyzer module storing said failure theory in a theory file, a notification module for querying said theory file, and, based on a failure theory in said theory file, notifying a user of likely failure of said component, and a recovery module for querying said theory file, and, based on a failure theory in said theory file, initiating recovery operations, to avoid data loss, said operating system, said monitor module, said collector module, said analyzer module, said notification module, and said recovery module being adapted for implementation by a digital data processing system.
50. An expert system in accordance with claim 25, wherein said digital data processing system that implements said collector module and said analyzer module is said digital data processing system in which said expert system detects likely failures.
51. An expert system in accordance with claim 49, wherein said digital data processing system that implements said operating system, said monitor module, said collector module, said analyzer module, said notification module, and said recovery module is said digital data processing system in which said expert system detects likely failures.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US17548588A | 1988-03-30 | 1988-03-30 | |
| US175,485 | 1988-03-30 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA1318030C true CA1318030C (en) | 1993-05-18 |
Family
ID=22640402
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA000592442A Expired - Fee Related CA1318030C (en) | 1988-03-30 | 1989-03-01 | Expert system for identifying failure points in a digital data processing system |
Country Status (5)
| Country | Link |
|---|---|
| US (1) | US5469463A (en) |
| EP (1) | EP0335507B1 (en) |
| JP (1) | JPH0758474B2 (en) |
| CA (1) | CA1318030C (en) |
| DE (1) | DE68924923T2 (en) |
Families Citing this family (44)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE69228986T2 (en) * | 1991-02-05 | 1999-08-12 | Storage Technology Corp | MAINTENANCE DEVICE AND METHODS TRIGGERED BY KNOWLEDGE-BASED KNOWLEDGE-BASED MACHINE |
| US5828583A (en) * | 1992-08-21 | 1998-10-27 | Compaq Computer Corporation | Drive failure prediction techniques for disk drives |
| US5535335A (en) * | 1992-12-22 | 1996-07-09 | International Business Machines Corporation | Method and system for reporting the status of an aggregate resource residing in a network of interconnected real resources |
| US6446224B1 (en) * | 1995-03-03 | 2002-09-03 | Fujitsu Limited | Method and apparatus for prioritizing and handling errors in a computer system |
| US5761411A (en) * | 1995-03-13 | 1998-06-02 | Compaq Computer Corporation | Method for performing disk fault prediction operations |
| US6467054B1 (en) | 1995-03-13 | 2002-10-15 | Compaq Computer Corporation | Self test for storage device |
| US5758057A (en) * | 1995-06-21 | 1998-05-26 | Mitsubishi Denki Kabushiki Kaisha | Multi-media storage system |
| US5923876A (en) * | 1995-08-24 | 1999-07-13 | Compaq Computer Corp. | Disk fault prediction system |
| US5845064A (en) * | 1995-09-11 | 1998-12-01 | Digital Equipment Corporation | Method for testing and verification of a CPU using a reference model |
| US5850388A (en) * | 1996-08-02 | 1998-12-15 | Wandel & Goltermann Technologies, Inc. | Protocol analyzer for monitoring digital transmission networks |
| US5850386A (en) * | 1996-11-01 | 1998-12-15 | Wandel & Goltermann Technologies, Inc. | Protocol analyzer for monitoring digital transmission networks |
| US6125393A (en) * | 1997-03-28 | 2000-09-26 | International Business Machines Corporation | System of compressing the tail of a sparse log stream of a multisystem environment |
| US5920875A (en) * | 1997-03-28 | 1999-07-06 | International Business Machines Corporation | Tail compression of a sparse log stream of a computer system |
| US5999935A (en) * | 1997-03-28 | 1999-12-07 | International Business Machines Corporation | Tail compression of a sparse log stream of a multisystem environment |
| US5956735A (en) * | 1997-03-28 | 1999-09-21 | International Business Machines Corporation | System of compressing the tail of a sparse log stream of a computer system |
| US5983364A (en) * | 1997-05-12 | 1999-11-09 | System Soft Corporation | System and method for diagnosing computer faults |
| US6311175B1 (en) | 1998-03-06 | 2001-10-30 | Perot Systems Corp. | System and method for generating performance models of complex information technology systems |
| US6393387B1 (en) | 1998-03-06 | 2002-05-21 | Perot Systems Corporation | System and method for model mining complex information technology systems |
| US6611877B2 (en) * | 1998-06-30 | 2003-08-26 | Sun Microsystems, Inc. | System and method for aggregating registration of entities for notifications of events |
| US6529893B1 (en) | 1999-02-26 | 2003-03-04 | Mandel Foner | Expert diagnostic system with inference generator |
| US6412089B1 (en) | 1999-02-26 | 2002-06-25 | Compaq Computer Corporation | Background read scanning with defect reallocation |
| US6493656B1 (en) | 1999-02-26 | 2002-12-10 | Compaq Computer Corporation, Inc. | Drive error logging |
| US6598179B1 (en) * | 2000-03-31 | 2003-07-22 | International Business Machines Corporation | Table-based error log analysis |
| US6538453B1 (en) * | 2000-03-31 | 2003-03-25 | Nexpress Solutions Llc | Detecting erratic resistance in temperature sensors |
| US6738928B1 (en) | 2000-06-19 | 2004-05-18 | Hewlett-Packard Development Company, L.P. | Method and expert system for analysis of crash dumps |
| US6842806B2 (en) | 2001-05-29 | 2005-01-11 | Sun Microsystems, Inc. | Method and apparatus for interconnecting wired-AND buses |
| US20040225814A1 (en) * | 2001-05-29 | 2004-11-11 | Ervin Joseph J. | Method and apparatus for constructing wired-AND bus systems |
| US7149838B2 (en) * | 2001-05-29 | 2006-12-12 | Sun Microsystems, Inc. | Method and apparatus for configuring multiple segment wired-AND bus systems |
| EP1378827A1 (en) * | 2002-07-05 | 2004-01-07 | Alcatel | Process for changing the language of a GUI application without exiting and re-entering the application |
| US7484125B2 (en) * | 2003-07-07 | 2009-01-27 | Hewlett-Packard Development Company, L.P. | Method and apparatus for providing updated processor polling information |
| US7437704B2 (en) * | 2003-08-28 | 2008-10-14 | Ines Antje Dahne-Steuber | Real-time generation of software translation |
| US7213176B2 (en) * | 2003-12-10 | 2007-05-01 | Electronic Data Systems Corporation | Adaptive log file scanning utility |
| US20060026466A1 (en) * | 2004-08-02 | 2006-02-02 | Bea Systems, Inc. | Support methodology for diagnostic patterns |
| JP4125274B2 (en) * | 2004-08-26 | 2008-07-30 | キヤノン株式会社 | Image input / output device, information processing method, computer-readable storage medium storing program, and program |
| US20060100806A1 (en) * | 2004-11-10 | 2006-05-11 | Matsushita Electric Industrial Co., Ltd. | Enhanced system for electronic storage device calibrations |
| US7788205B2 (en) | 2006-05-12 | 2010-08-31 | International Business Machines Corporation | Using stochastic models to diagnose and predict complex system problems |
| US7349826B2 (en) | 2006-05-23 | 2008-03-25 | International Business Machines Corporation | Causal ladder mechanism for proactive problem determination, avoidance and recovery |
| US20090259890A1 (en) * | 2008-04-14 | 2009-10-15 | Turin Networks | Method & apparatus for hardware fault management |
| DE102008019983B4 (en) | 2008-04-21 | 2014-12-18 | Maschinenbau Heinrich Hajek Gmbh & Co. | Method for operating a cutting machine with housing overpressure and device |
| CN103761173A (en) * | 2013-12-28 | 2014-04-30 | 华中科技大学 | Log based computer system fault diagnosis method and device |
| US10044556B2 (en) | 2015-06-23 | 2018-08-07 | International Business Machines Corporation | Identifying performance-degrading hardware components in computer storage systems |
| CN105577440B (en) * | 2015-12-24 | 2019-06-11 | 华为技术有限公司 | A kind of network downtime localization method and analytical equipment |
| US10810103B2 (en) * | 2016-12-14 | 2020-10-20 | Vmware, Inc. | Method and system for identifying event-message transactions |
| US11074121B2 (en) | 2019-03-20 | 2021-07-27 | International Business Machines Corporation | Predicting failure of a magnetic tape head element |
Family Cites Families (28)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US3704363A (en) * | 1971-06-09 | 1972-11-28 | Ibm | Statistical and environmental data logging system for data processing storage subsystem |
| US3815097A (en) * | 1972-08-20 | 1974-06-04 | Memorex Corp | Disc drive diagnostic display apparatus |
| US3928830A (en) * | 1974-09-19 | 1975-12-23 | Ibm | Diagnostic system for field replaceable units |
| DE2654389C3 (en) * | 1976-12-01 | 1980-07-31 | Ibm Deutschland Gmbh, 7000 Stuttgart | Maintenance control with processor and memory for error analysis and diagnosis for electronic data processing systems and methods for their operation |
| JPS5431212A (en) * | 1977-08-15 | 1979-03-08 | Nec Corp | Monitor circuit for digital signal transmission line |
| US4242751A (en) * | 1978-08-28 | 1980-12-30 | Genrad, Inc. | Automatic fault-probing method and apparatus for checking electrical circuits and the like |
| JPS594054B2 (en) * | 1979-04-17 | 1984-01-27 | 株式会社日立製作所 | Multiprocessor failure detection method |
| US4339657A (en) * | 1980-02-06 | 1982-07-13 | International Business Machines Corporation | Error logging for automatic apparatus |
| JPS56137455A (en) * | 1980-03-31 | 1981-10-27 | Fujitsu Ltd | Maintenance system against failure |
| JPS57109061A (en) * | 1980-12-26 | 1982-07-07 | Mitsubishi Electric Corp | Forecasting method for equipment deterioration of computer system |
| US4554661A (en) * | 1983-10-31 | 1985-11-19 | Burroughs Corporation | Generalized fault reporting system |
| IL74952A0 (en) * | 1984-05-04 | 1985-08-30 | Gould Inc | Method and system for improving the operational reliability of electronic systems formed of subsystems which perform different functions |
| US4633467A (en) * | 1984-07-26 | 1986-12-30 | At&T Bell Laboratories | Computer system fault recovery based on historical analysis |
| US4644479A (en) * | 1984-07-31 | 1987-02-17 | Westinghouse Electric Corp. | Diagnostic apparatus |
| US4754409A (en) * | 1985-06-26 | 1988-06-28 | International Business Machines Corporation | Method for dynamically collecting current data from specified external processes and procedures for use in an expert system |
| US4713810A (en) * | 1985-09-19 | 1987-12-15 | Gte Sprint Communications Corp. | Diagnostic technique for determining fault locations within a digital transmission system |
| US4710924A (en) * | 1985-09-19 | 1987-12-01 | Gte Sprint Communications Corp. | Local and remote bit error rate monitoring for early warning of fault location of digital transmission system |
| WO1987006371A1 (en) * | 1986-04-10 | 1987-10-22 | Hewlett Packard Limited | Expert system using pattern recognition techniques |
| JPS6359638A (en) * | 1986-08-25 | 1988-03-15 | Mitsubishi Electric Corp | Error log system |
| US4841456A (en) * | 1986-09-09 | 1989-06-20 | The Boeing Company | Test system and method using artificial intelligence control |
| US4769761A (en) * | 1986-10-09 | 1988-09-06 | International Business Machines Corporation | Apparatus and method for isolating and predicting errors in a local area network |
| US4847795A (en) * | 1987-08-24 | 1989-07-11 | Hughes Aircraft Company | System for diagnosing defects in electronic assemblies |
| US4817092A (en) * | 1987-10-05 | 1989-03-28 | International Business Machines | Threshold alarms for processing errors in a multiplex communications system |
| JPH01118934A (en) * | 1987-10-31 | 1989-05-11 | Nec Corp | Analysis/editing output system for error information |
| US4866712A (en) * | 1988-02-19 | 1989-09-12 | Bell Communications Research, Inc. | Methods and apparatus for fault recovery |
| US5090014A (en) * | 1988-03-30 | 1992-02-18 | Digital Equipment Corporation | Identifying likely failure points in a digital data processing system |
| US4932028A (en) * | 1988-06-21 | 1990-06-05 | Unisys Corporation | Error log system for self-testing in very large scale integrated circuit (VLSI) units |
| US4922491A (en) * | 1988-08-31 | 1990-05-01 | International Business Machines Corporation | Input/output device service alert function |
-
1989
- 1989-03-01 CA CA000592442A patent/CA1318030C/en not_active Expired - Fee Related
- 1989-03-02 EP EP89302072A patent/EP0335507B1/en not_active Expired - Lifetime
- 1989-03-02 DE DE68924923T patent/DE68924923T2/en not_active Expired - Lifetime
- 1989-03-30 JP JP1080198A patent/JPH0758474B2/en not_active Expired - Fee Related
-
1991
- 1991-05-08 US US07/700,112 patent/US5469463A/en not_active Expired - Lifetime
Also Published As
| Publication number | Publication date |
|---|---|
| EP0335507A2 (en) | 1989-10-04 |
| EP0335507A3 (en) | 1991-06-12 |
| US5469463A (en) | 1995-11-21 |
| DE68924923T2 (en) | 1996-07-25 |
| EP0335507B1 (en) | 1995-11-29 |
| AU3088889A (en) | 1989-10-05 |
| AU602808B2 (en) | 1990-10-25 |
| JPH0216640A (en) | 1990-01-19 |
| JPH0758474B2 (en) | 1995-06-21 |
| DE68924923D1 (en) | 1996-01-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA1318030C (en) | Expert system for identifying failure points in a digital data processing system | |
| US5090014A (en) | Identifying likely failure points in a digital data processing system | |
| US6178452B1 (en) | Method of performing self-diagnosing and self-repairing at a client node in a client/server system | |
| US4922491A (en) | Input/output device service alert function | |
| US8347142B2 (en) | Non-disruptive I/O adapter diagnostic testing | |
| US7487408B2 (en) | Deferring error reporting for a storage device to align with staffing levels at a service center | |
| US20070245170A1 (en) | Computer boot operation utilizing targeted boot diagnostics | |
| CN110807064B (en) | Data recovery device in RAC distributed database cluster system | |
| US6347335B1 (en) | System using a common and local event logs for logging event information generated by plurality of devices for determining problem in storage access operations | |
| CN103415840A (en) | Error management across hardware and software layers | |
| EP0530863B1 (en) | Method of starting up a subsystem in a distributed processing system | |
| CN100394394C (en) | Fault tolerant duplex computer system and its control method | |
| US7376864B1 (en) | Method and system for diagnostic preservation of the state of a computer system | |
| US6643734B2 (en) | Control device and control method for a disk array | |
| CN111400121A (en) | Server hard disk slot positioning and maintaining method | |
| CN114675791A (en) | Disk processing method and system and electronic equipment | |
| EP0632381B1 (en) | Fault-tolerant computer systems | |
| US8543923B2 (en) | Apparatus, system, and method for displaying health spatially and temporally | |
| US8924773B1 (en) | Reducing file system data unavailability window by adapting hierarchical recovery framework | |
| CN107707402B (en) | Management system and management method for service arbitration in distributed system | |
| JPH08147112A (en) | Error recovery device for disk array device | |
| JP2002123372A (en) | Disk array device with cache memory, its error- controlling method and recording medium with its control program recorded thereon | |
| JPH0252290B2 (en) | ||
| JPH07306811A (en) | Memory fault diagnosing method | |
| Hopkins et al. | Automatic intercept system: File subsystem |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| MKLA | Lapsed |