|Publication number||US7627781 B2|
|Application number||US 10/973,077|
|Publication date||Dec 1, 2009|
|Filing date||Oct 25, 2004|
|Priority date||Oct 25, 2004|
|Also published as||US20060107112|
|Publication number||10973077, 973077, US 7627781 B2, US 7627781B2, US-B2-7627781, US7627781 B2, US7627781B2|
|Inventors||Scott L. Michaelis, Anurupa Rajkumari|
|Original Assignee||Hewlett-Packard Development Company, L.P.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (61), Non-Patent Citations (41), Referenced by (26), Classifications (13), Legal Events (3)|
|External Links: USPTO, USPTO Assignment, Espacenet|
The present application is related to the following concurrently filed and commonly assigned U.S. Patent Applications: U.S. patent application Ser. No. 10/972,835 (U.S. Published Patent Application 20060107114) titled “SYSTEM AND METHOD FOR USING INFORMATION RELATING TO A DETECTED LOSS OF LOCKSTEP FOR DETERMINING A RESPONSIVE ACTION”; U.S. patent application Ser. No. 10/973,004 (U.S. Published Patent Application 20060107117, now U.S. Pat. No. 7,308,566) titled “SYSTEM AND METHOD FOR CONFIGURING LOCKSTEP MODE OF A PROCESSOR MODULE”; U.S. patent application Ser. No. 10/973,076 (U.S. Published Patent Application 20060107107) titled “SYSTEM AND METHOD FOR PROVIDING FIRMWARE RECOVERABLE LOCKSTEP PROTECTION”; U.S. patent application Ser. No. 10/972,588 (U.S. Published Patent Application 20060090064) titled “SYSTEM AND METHOD FOR SWITCHING THE ROLE OF BOOT PROCESSOR TO A SPARE PROCESSOR RESPONSIVE TO DETECTTON OF LOSS OF LOCKSTEP IN A BOOT PROCESSOR”; U.S. patent application Ser. No. 10/973,003 (U.S. Published Patent Application 20060107116) titled “SYSTEM AND METHOD FOR REESTABLISHING LOCKSTEP FOR A PROCESSOR MODULE FOR WHICH LOSS OF LOCKSTEP IS DETECTED”; U.S. patent application Ser. No. 10/972,888 (U.S. Published Patent Application 20060107115) titled “SYSTEM AND METHOD FOR SYSTEM FIRMWARE CAUSING AN OPERATING SYSTEM TO IDLE A PROCESSOR”; U.S. patent application Ser. No. 10/973,075 (U.S. Published Patent Application 20060107111) titled “SYSTEM AND METHOD FOR REINTRODUCING A PROCESSOR MODULE TO AN OPERATING SYSTEM AFTER LOCKSTEP RECOVERY”; and U.S. patent application Ser. No. 10/972,796 (U.S. Published Patent Application 20060107106) titled “SYSTEM AND METHOD FOR MAINTAINING IN A MULTI-PROCESSOR SYSTEM A SPARE PROCESSOR THAT IS IN LOCKSTEP FOR USE IN RECOVERING FROM LOSS OF LOCKSTEP FOR ANOTHER PROCESSOR”, the disclosures of which are hereby incorporated herein by reference.
Silent Data Corruption (“SDC”) is a difficult problem in the computing industry. In general, SDC refers to data that is corrupt, but which the system does not detect as being corrupt. SDCs primarily occur due to one of two factors: a) a broken hardware unit or b) a “cosmic” event that causes values to change somewhere in the system. Broken hardware means that a “trusted” piece of hardware is silently giving wrong answers. For example, the arithmetic unit in a processor is instructed to add 1+1 and it returns the incorrect answer 3 instead of the correct answer 2. An example of a cosmic event is when a charged particle (e.g., alpha particle or cosmic ray) strikes a region of a computing system and causes some bits to change value (e.g., from a 0 to a 1 or from a 1 to a 0).
Numerous techniques have been developed for detecting SDC to prevent the SDC from remaining “silent” or “undetected” within a system, as well as preventing such SDC from propagating through the system. Examples of these techniques include parity-based mechanisms and error correcting codes (ECCs) on buses and memory locations, as well as checksums and/or cyclic redundancy checks (CRC) over regions of memory. Parity-based mechanisms are often employed in processors, wherein a parity bit is associated with each block of data when it is stored. The parity bit is set to one or zero according to whether there is an odd or even number of ones in the data block. When the data block is read out of its storage location, the number of ones in the block is compared with the parity bit. A discrepancy between the values indicates that the data block has been corrupted. ECCs are parity-based mechanisms that track additional information for each data block. The additional information allows the corrupted bit(s) to be identified and corrected.
Parity/ECC mechanisms have been employed extensively for caches, memories, and similar data storage arrays. In the remaining circuitry on a processor, such as data paths, control logic, execution logic, and registers (the “execution core”), it is more difficult to apply parity/ECC mechanisms for SDC detection. Thus, there is typically some unprotected area on a processor in which data corruption may occur and the parity/ECC mechanisms do not prevent the corrupted data from actually making it out onto the system bus. One approach to SDC detection in an execution core (or other unprotected area of the processor chip) is to employ “lockstep processing.” Generally, in lockstep processing two processors are paired together, and the two processors perform exactly the same operations and the results are compared (e.g., with an XOR gate). If there is ever a discrepancy between the results of the lockstep processors, an error is signaled. The odds of two processors experiencing the exact same error at the exact same moment (e.g., due to a cosmic event occurring in both processors at exactly the same time or due to a mechanical failure occurring in each processor at exactly the same time) is nearly zero.
A pair of lockstep processors may, from time to time, lose their lockstep. “Loss of lockstep” (or “LOL”) is used broadly herein to refer to any error in the pair of lockstep processors. One example of LOL is detection of data corruption (e.g., data cache error) in one of the processors by a parity-based mechanism and/or ECC mechanism. Another example of LOL is detection of the output of the paired processors not matching, which is referred to herein as a “lockstep mismatch.” It should be recognized that in some cases the data in the cache of a processor may become corrupt (e.g., due to a cosmic event), which once detected (e.g., by a parity-based mechanism or ECC mechanism of the processor) results in LOL. Of course, unless such corrupt data is acted upon by the processor, the output of that processor will not fail to match the output of its paired processor and thus a “lockstep mismatch” will not occur. For example, suppose that a value of “1” is stored to first location of cache in each of a pair of lockstep processors and a value of “1” is also stored to a second location of cache in each of the pair of lockstep processors. Further suppose that a cosmic event occurs for a first one of the processors, resulting in the first location of its cache being changed from “1” to “0”, and thus corrupted. This data corruption in the first processor is a LOL for the pair. An error detection mechanism of this first processor may detect the data corruption, thus detecting the LOL. If the processors are instructed to act on the data of their first cache locations, then a lockstep mismatch will occur as the output of each of the processors will not match. For instance, if the processors each add the data stored to the first location of their respective cache with the data stored to the second location of their respective cache, the first processor (having the corrupt data) will output a result of “1” (0+1=1) while the second processor outputs a result of “2” (1+1=2), and thus their respective outputs will not match.
By employing such techniques as parity-based error detection mechanisms and output comparisons for lockstep paired processors, SDC detection can be enhanced such that practically no SDC occurring in a processor goes undetected (and thus such SDC does not remain “silent”) but instead results in detection of LOL. However, the issue then becomes how best for the system to respond to detected LOL. The traditional response to detected LOL has been to crash the system to ensure that the detected error is not propagated through the system. That is, LOL in one pair of lockstep processors in a system halts processing of the system even if other processors that have not encountered an error are present in the system. However, with the increased desire for many systems to maintain high availability, crashing the system each time LOL is detected is not an attractive proposition. This is particularly unattractive for large systems having many processors because cosmic events typically occur more frequently as the processor count goes up, which would result in much more frequent system crashes in those large systems. High availability is a major desire for many customers having large, multi-processor systems, and thus having their system crash every few weeks is not an attractive option. Of course, permitting corrupt data to propagate through the system is also not a viable option.
Prior solutions attempting to resolve at least some detected SDCs without requiring the system to be crashed have been Operating System (“OS”) centric. That is, in certain solutions the OS has been implemented in a manner to recover from a detected LOL without necessarily crashing the system. This OS-centric type of solution requires a lot of processor and platform specific knowledge to be embedded in the OS, and thus requires that the OS provider maintain the OS up-to-date as changes occur in later versions of the processors and platforms in which the OS is to be used. This is such a large burden that most commonly used OSs do not support lockstep recovery.
Certain solutions have attempted to recover from a LOL without involving the OS in such recovery procedure. For instance, in one technique upon LOL being detected, firmware is used to save the state of one of the processors in a lockstep pair (the processor that is considered “good”) to memory, and then both processors of the pair are reset and reinitialized. Thereafter, the state is copied from the memory to each of the processors in the lockstep pair. This technique makes the processors unavailable for an amount of time without the OS having any knowledge regarding this unavailability, and if the amount of time required for recovery is too long, the system may crash. That is, typically, if a processor is unresponsive for X amount of time, the OS will assume that the processor is hung and will crashdump the system so that the problem can be diagnosed. Further, in the event that a processor in the pair cannot be reset and reinitialized (e.g., the processor has a physical problem and fails to pass its self-test), this technique results in crashing the system.
According to at least one embodiment, a system comprises a plurality of processors, and data storage storing information that assigns a role of boot processor to one of the plurality of processors and assigns a role of spare processor to another of the plurality of processors. The system further comprises logic operable, responsive to detecting loss of lockstep for the boot processor, for transferring, during system runtime, the role of boot processor to the spare processor.
According to at least one embodiment, a method comprises assigning a role of boot processor to one of a plurality of processors in a system; assigning a role of spare processor to another of the plurality of processors; and responsive to detecting loss of lockstep for the boot processor, changing, during system runtime, the assignment of the role of boot processor to the spare processor.
As described further herein and in concurrently filed and commonly assigned U.S. patent application Ser. No. 10/973,076 (U.S. Published Patent Application 20060107107) titled “SYSTEM AND METHOD FOR PROVIDING FIRMWARE RECOVERABLE LOCKSTEP PROTECTION,” the disclosure of which is incorporated herein by reference, certain techniques are provided for recovering from LOL detected for a boot processor in a multi-processor system, in which a spare processor is utilized for such recovery. For instance, upon LOL being detected for the system's boot processor, the spare processor assumes the role of boot processor, and then recovery of lockstep for the original boot processor is attempted. Accordingly, for such lockstep recovery technique, a spare processor is used for attempting to recover from LOL detected for the boot processor. Embodiments are provided herein for establishing a spare processor for the boot processor. That is, embodiments described herein provide techniques for establishing, in a multi-processor system, a spare processor that assumes the role of boot processor upon LOL being detected for the system's boot processor.
In certain embodiments, information (referred to herein as a “device tree”) is stored to the multi-processor system, wherein such information identifies the processor in the system that is the boot processor and identifies the processor that is the spare for the boot processor. The information is stored such that it is accessible by the system's firmware, and the firmware, upon detecting LOL for the boot processor, determines the spare processor from the device tree and assigns the spare processor the role of boot processor. The firmware then attempts to recover lockstep for the processor for which LOL was detected. If the recovery of lockstep is successful, the processor for which lockstep was recovered (i.e., the former boot processor) may be assigned the role of spare for the current boot processor. Thus, the multi-processor system may recover from LOL detected for the boot processor without requiring a crash of the system (at least for certain types of “recoverable” LOLs).
In some implementations described herein, the spare processor for the boot processor is a “hot” spare, in which it is idled and not made available to the OS for processing instructions during normal system operation. That is, the hot spare processor is held in reserve, idling until such time as it assumes the role of boot processor responsive to detected LOL for the boot processor. In other implementations, the spare processor is not held in reserve, but is instead an “active” processor that is made available to the OS for executing instructions during normal system operation. Upon LOL being detected for the boot processor, the designated spare processor is dynamically transformed into a hot spare (i.e., it is idled) such that it can assume the role of the boot processor. For instance, upon the system's firmware detecting a LOL for the boot processor, the firmware accesses the device tree and determines the designated spare processor, and then the firmware instructs the OS to idle the designated spare processor (i.e., stop scheduling instructions for the processor) to make the “active” spare a “hot” spare, which then assumes the role of boot processor.
Concurrently filed and commonly assigned U.S. patent application Ser. No. 10/973,076 (U.S. Published Patent Application 20060107107) titled “SYSTEM AND METHOD FOR PROVIDING FIRMWARE RECOVERABLE LOCKSTEP PROTECTION,” discloses a technique in which lockstep can be recovered for processors (particularly, non-boot processors) without requiring use of a spare processor. Rather, upon LOL being detected for a lockstep pair of processors, the firmware instructs the OS to idle (or “eject”) the lockstep pair of processors for which LOL was detected and then the firmware attempts to reestablish lockstep (e.g., by resetting the processors). If successful in reestablishing lockstep, the firmware triggers the OS to recognire the lockstep pair of processors as again being available for servicing instructions.
However, for various reasons, in certain system architectures problems arise in attempting to idle (or eject) the boot processor from the system. Thus, in certain embodiments, a spare processor is used for recovering from LOL for the system's boot processor. Again, embodiments are provided herein for establishing/defining such a spare processor for the boot processor. In certain embodiments, the spare processor is used for recovering from LOL only in the case in which LOL is detected for the boot processor. For instances in which LOL is detected for non-boot processors, the lockstep may be recovered without the use of a spare by using the recovery technique described in concurrently filed and commonly assigned U.S. patent application Ser. No. 10/973,076 (U.S. Published Patent Application 20060107107) titled “SYSTEM AND METHOD FOR PROVIDING FIRMWARE RECOVERABLE LOCKSTEP PROTECTION,” for example.
Various techniques are provided herein for designating the spare for a boot processor. In certain embodiments, a user interface is provided for receiving input specifying the boot processor and/or its spare. In certain embodiments, the system's firmware selects the boot processor and/or its spare. In accordance with one embodiment provided herein, the multi-processor system includes multiple buses, wherein certain processors are communicatively coupled to a first bus and certain other processors are communicatively coupled to a different bus. In such an architecture, the spare processor is designated as a processor that is coupled to the same bus as the boot processor. Thus, if a user is allowed to select the boot processor and the spare, once the boot processor is selected, the user interface may present the other processor(s) that are coupled to the same bus as the selected boot processor for selection of a spare therefrom. Alternatively, once the boot processor is selected by a user, the system firmware may select a spare as a processor that is coupled to the same bus as the boot processor. In certain embodiments, the system firmware may select the boot processor, as well as a spare for the boot processor.
Master processor 12A includes cache 14A, and slave processor 12B includes cache 14B. OS 11 and lockstep processor pair 12 are communicatively coupled to bus 16. Typically, master processor 12A and slave processor 12B are coupled to bus 16 via an interface that allows each of such processors to receive the same instructions to process, but such interface only communicates the output of master processor 12A back onto bus 16. The output of slave processor 12B is used solely for checking the output of mater processor 12A.
In this example, master processor 12A includes error detect logic 13A, and slave processor 12B includes error detect logic 13B. While shown as included in each of the processors 12A and 12B in this example, in certain embodiments the error detect logic 13A and 13B may be implemented external to processors 12A and 12B. Error detect logic 13A and 13B include logic for detecting errors, such as data cache errors, present in their respective processors 12A and 12B. Examples of error detect logic 13A and 13B include known parity-based mechanisms and ECC mechanisms. Error detect logic 13C is also included, which may include an XOR (exclusive OR) gate, for detecting a lockstep mismatch between master processor 12A and slave processor 12B. As mentioned above, a lockstep mismatch refers to the output of master processor 12A and slave processor 12B failing to match. While shown as external to the lockstep processor pair 12 in this example, in certain embodiments error detect logic 13C may be implemented on a common silicon chip with processors 12A and 12B.
Lockstep mismatch is one way of detecting a LOL between the master processor 12A and slave processor 12B. A detection of an error by either of error detect logic 13A and 13B also provides detection of LOL in the processors 12A and 12B. Because the detection of LOL by error detect logic 13A and 13B may occur before an actual lockstep mismatch occurs, the detection of LOL by error detect logic 13A and 13B may be referred to as a detection of a “precursor to lockstep mismatch”. In other words, once an error (e.g., corrupt data) is detected by error detect logic 13A or 13B, such error may eventually propagate to a lockstep mismatch error that is detectable by error detect logic 13C.
System 10 further includes a second lockstep processor pair 15 that is communicatively coupled to bus 16. Lockstep processor pair 15 includes master processor 15A and slave processor 15B. Master processor 15A includes cache 17A, and slave processor 15B includes cache 17B. As with error detect logic 13A-13C implemented for lockstep processor pair 12, lockstep processor pair 15 has error detect logic 16A-16C. While only two lockstep processor pairs 12 and 15 are shown for simplicity in the example of
In certain embodiments described herein, techniques for designating a spare processor for a boot processor are provided, wherein the spare processor is used for recovery when LOL is detected for the system's boot processor. That is, the designated spare processor assumes the role of boot processor in the event that LOL is detected for the boot processor. In accordance with certain embodiments, upon LOL being detected for a boot processor, a hot spare processor (i.e., an idling processor that is available in the system) is transferred the role of boot processor, and then the old boot processor having LOL is reset to reestablish its lockstep. Thus, in certain embodiments, a “hot” spare is designated for the boot processor.
In other embodiments, a processor is designated as a spare processor for the boot processor, but such designated spare processor is not held in reserve (idled) during normal system operation. Rather, the designated spare is utilized as an active processing resource made available to OS 11 for processing instructions that OS 11 schedules for such “active” spare processor. Upon LOL being detected for the boot processor, the designated spare processor is idled (thereby, transforming such active processor into a hot spare), and is transferred the role of boot processor. Then, the former boot processor for which the LOL was detected is reset to reestablish its lockstep. Thus, in certain embodiments, a designated spare processor is not required to be held in reserve, but is instead an available resource for use within the system and is only transformed into an actual spare processor when needed for recovering from LOL for the boot processor.
Firmware 18 is also included in system 10, which in this embodiment is invoked upon an error being detected by any of the error detect logics 13A-13C and 16A-16C. In certain embodiments, processors 12A, 12B, 15A, and 15B are processors from the Itanium Processor Family (IPF). IPF is a 64-bit processor architecture co-developed by Hewlett-Packard Company and Intel Corporation, which is based on Explicitly Parallel Instruction Computing (EPIC). IPF is a well-known family of processors. IPF includes processors such as those having the code names of MERCED, MCKINLEY, and MADISON. In addition to supporting a 64-bit processor bus and a set of 128 registers, the 64-bit design of IPF allows access to a very large memory (VLM) and exploits features in EPIC. While a specific example implementation of one embodiment is described below for the IPF architecture, embodiments described herein for establishing a spare processor for recovering from LOL detected for a boot processor are not limited in application to an IPF architecture, but may be applied as well to other architectures (e.g., 32-bit processor architectures, etc.).
Processor architecture generally comprises corresponding supporting firmware, such as firmware 18 of system 10. For example, as described further below in conjunction with the specific example of
In the example system 10 of
Suppose that during the system boot-up procedure, master processor 12A assumes the role of boot processor, as shown in the example of
Firmware 18 is included in this example, and upon detection of LOL by any of error detect logics 13A-13C, firmware 18 determines, in operational block 101, whether the detected LOL is a recoverable LOL. That is, firmware 18 determines in block 101 whether the detected LOL is of a type from which the firmware can recover lockstep for the lockstep processor pair 12 without crashing the system. As described further herein, lockstep is recoverable for certain detected LOLs (which may be referred to as “recoverable LOLs”), while lockstep is not recoverable for other detected LOLs (which may be referred to as “non-recoverable LOLs”). If the lockstep is not recoverable from the detected LOL, then in the example of
If the lockstep is recoverable, then operation advances to block 103 whereat the firmware determines whether the processor for which the LOL was detected is the system's boot processor. Because the LOL is detected for the lockstep processor pair 12 in this example, firmware 18 determines whether master processor 12A is the system's boot processor. This can be determined, for example, by accessing the device tree 203 of
In the specific example shown in
In the example embodiment of
Firmware 18 then attempts to recover lockstep for the lockstep processor pair 12 in block 106. For instance, firmware 18 resets the processor pair 12. During such reset of processor pair 12, system 10 can continue to operate on its remaining available processors. Once the processor pair 12 is reset and lockstep is recovered, firmware 18 reintroduces master processor 12A to OS 11 in operational block 107. In this embodiment, firmware 18 updates the ACPI device table information for master processor 12A to indicate that such master processor 12A is “present, functioning and enabled.” As discussed in the ACPI 2.0 specification for the _STA status method of a device, the _STA (status) object returns the status of a device, which can be one of the following: enabled, disabled, or removed. In this respect, in the result code (bitmap) bit 0 is set if the device is present; bit 1 is set if the device is enabled and decoding its resources; bit 2 is set if the device should be shown in the UI; bit 3 is set if the device is functioning properly (cleared if the device failed its diagnostics); bit 4 is set if the battery is present; and bits 5-31 are reserved. A device can only decode its hardware resources if both bits 0 and 1 are set. If the device is not present (bit 0 cleared) or not enabled (bit 1 cleared), then the device must not decode its resources. Bits 0, 1 and 3 are the “present, enabled and functioning” bits mentioned above. Firmware 18 utilizes an ACPI method 108 to trigger OS 11 to “check for” master processor 12A, thereby reintroducing the master processor 12A to OS 11. As a result of checking for master processor 12A, OS 11 will recognize that such master processor 12A is again available and will thus begin scheduling tasks for master processor 12A once again.
In view of the above, a recovery technique is provided that does not require utilizing a spare processor for recovering from LOL detected for non-boot processors. For various reasons, in certain system architectures problems arise in attempting to idle (or eject) the boot processor from the system. Thus, in certain embodiments, a hot spare processor is used for recovering from LOL for the system's boot processor. For instance, in the example of
Thus, operational block 109 essentially makes the spare processor pair 15 the system's boot processor, and then in block 110 firmware 18 resets the lockstep processor pair 12 and reestablishes its lockstep. Once lockstep is reestablished for lockstep processor pair 12, that pair is held in operational block 111 as a hot spare for the new boot processor pair 15. That is, tirrnware 18 updates the device tree 203 of
Embodiments described herein enable a spare processor to be designated for a boot processor, such as spare processor 15 in the above example. In certain implementations, a system administrator can designate through a user interface the boot processor, and, in some cases, further designate the corresponding spare for such boot processor. In other implementations, a system administrator may designate the boot processor and firmware selects the appropriate spare processor. In other embodiments, the firmware may select a boot processor and the corresponding spare processor for such boot processor. The information identifying the boot processor and the spare processor to be used for recovery from LOL detected for the boot processor is stored to the system in a data storage device (e.g., SRAM) that is accessible by the system's firmware to enable the firmware to identify those processors.
The boot-up process of a traditional IPF system, for example, proceeds as follows: When the system is first powered on, there are some sanity checks (e.g., power on self-test) that are performed by microprocessors included in the system platform, which are not the main system processors that run applications. After those checks have passed, power and clocks are given to a boot processor (which may, for example, be master processor 12A). The boot processor begins executing code out of the system's Read-Only Memory (ROM) (not specifically shown in
After it has performed its duty of initializing the processor(s), PAL 201 passes control of system 10 A to SAL 202. It is the responsibility of SAL 202 to discover what hardware is present on the system platform, and initialize it to make it available for the OS 11. When main memory is initialized and functional, the firmware 18 A is copied into the main memory. Then, control is passed to EFI (not shown), which is responsible for activating boot devices, which typically includes the disk. The EFI reads the disk to load a program into memory, typically referred to as an operating system loader. The EFI loads the OS loader into memory, and then passes it control of system 10 A by branching the boot processor into the entry point of such OS loader program.
The OS loader program then uses the standard firmware interfaces to discover and initialize system 10 A further for control. One of the things that the OS loader typically has to do in a multi-processor system is to retrieve control of the other processors (those processors other than the boot processor). For instance, at this point in a multi-processor system, the other processors may be executing in do-nothing loops. In an ACPI-compatible system, OS 11 makes ACPI calls to parse the ACPI tables to discover the other processors of a multi-processor system in a manner as is well-known in the art. Then OS 11 uses the firmware interfaces to cause those discovered processors to branch into the operating system code. At that point, OS 11 controls all of the processors and the firmware 18 A is no longer in control of system 10 A.
As OS 11 is initializing, it has to discover from the firmware 18 A what hardware is present at boot time. And in the ACPI standards, it also discovers what hardware is present or added or removed at run-time. Further, the supporting firmware (PAL, SAL, and EFI) are also used during system runtime to support the processor. For example, OS 11 may access a particular function of master processor 12A via the supporting firmware 18 A, such as querying PAL 201 for the number, size, etc., of the processor's cache 14A. Some other well-known firmware functions that OS 11 may employ during runtime include: (a) PAL 201 may be invoked to configure or change processor features such as disabling transaction queuing (PAL_BUS_SET_FEATURES); (b) PAL 201 may be invoked to flush processor caches (PAL_CACHE_FLUSH); (c) SAL 202 may be invoked to retrieve error logs following a system error (SAL_GET_STATE_INFO, SAL_CLEAR_STATE_INFO); (d) SAL 202 may be invoked as part of hot-plug sequences in which new I/O cards are installed into the hardware (SAL_PCI_CONFIG_READ, SAL_PCI_CONFIG_WRIT); (e) EFI may be invoked to change the boot device path for the next time the system reboots (SetVariable); (f) EFI may be invoked to change the clock/calendar hardware settings; and (g) EFI may be invoked to shutdown the system (ResetSystem).
A “device tree” is provided, which is shown as device tree 203 in this example. Device tree 203 is stored in SRAM (Scratch RAM) on the cell, which is RAM that is reinitialized. Firmware 18A builds the device tree 203 as it discovers what hardware is installed in the system. Firmware then converts this information to the ACPI tables format and presents it to OS 11 so that OS 11 can know what is installed in the system. The ACPI device tables (not shown) are only consumed by OS 11 at boot time, so they are never updated as things change. For OS 11 to find the current status, it calls an ACPI “method” to discover the “current status”. The _STA method described above is an example of such an ACPI method. When _STA is called, the AML can look for properties on the device specified in the firmware device tree and convert that into the Result Code bitmap described above. So, if lockstep has been lost on a processor, firmware 15A will set the device tree property that indicates loss of lockstep, then when OS 11 calls _STA for that device, the “lockstep lost” property directs the AML code to return to “0” in the “functioning properly” bit so that OS 11 can know there is a problem with that processor.
If a lost lockstep is recovered in accordance with the recovery technique described herein, firmware 18A can indicate that lockstep has been recovered in the device tree 203. Then when _STA is called on that device responsive to the OS receiving the “check for device” ACPI method, the present, enabled and functioning bits will all be set and OS 11 will know the CPU is safe to use.
Further, device tree 203 includes information designating which processor is the boot processor, as well as a spare processor for the boot processor. A simple example of device tree 203 is shown below in Table 1:
. . .
. . .
. . .
. . .
. . .
In the above example of Table 1, 3 processors “Processor A,” “Processor B,” and “Processor C” are identified in the device tree as being included in the multi-processor system. The status of each of processors Processor A and Processor B is identified as “Present, Enabled, and Functioning.” In this example Processor B is a hot spare, so it is held in reserve and does not show up in the static ACPI tables that are built when the system boots. Accordingly the OS does not know about Processor B and does not have a device handle to call the _STA method on. As described above, the _STA method is a known ACPI method for checking the status of the corresponding device. Therefore, the status of Processor B is designated N/A (not applicable) in this example because the _STA method will not be called for this processor as long as it is the hot spare. Of course, upon such spare assuming the role of boot processor, its status is updated to “Present, Enabled, and Functioning.” In other embodiments in which Processor B is an active spare, the OS will be aware of such processor and accordingly its status may be also indicated as “Present, Enabled, and Functioning.”
Further, lockstep is identified in this example as enabled for these processors. In certain embodiments, the system administrator can designate, via user interface 20, whether lockstep mode is to be enabled for the system's processors. Further, Processor A is designated as the system's boot processor in the example device tree shown in Table 1, and Processor B is designated as the spare processor. Thus, upon detection of LOL for one of the processors in the system, the firmware can access this device tree to determine if the processor for which LOL is detected is the system's boot processor; and if determined that the processor for which LOL is detected is the system's boot processor, the firmware can further determine from this device tree the designated spare processor to use in the lockstep recovery process.
In the example of
One example system in which firmware 18 A of
Coherency controller 304 decodes the address an interrupt is targeted towards and determines which bus (305 or 306) the interrupt packet should be delivered to. Each CPU on the destination bus sees the interrupt and compares the interrupt target address with its internal “LID” register to determine if it is the targeted CPU. If the match is correct, the CPU responds to the interrupt. If the match is not correct, the CPU ignores the interrupt packet.
As shown, the example firmware 18 A described above with
It should be recognized that in the example system of
While in the above example embodiment, a hot spare CPU is maintained for recovery from LOL for the system's boot processor, in an alternative embodiment a hot spare need not be held as such, but rather a processor can be dynamically made a “spare” when needed for recovery from LOL encountered for the system boot processor. That is, assuming as in the above example of
Thus, the resource of this spare CPU need not be wasted during normal runtime, but can be dynamically turned into a spare processor that is available for recovering from LOL encountered on CPU 2 in the manner described above. As an example of this embodiment, the firmware may send an “eject request” on a healthy processor pair (e.g., CPU 3), and lie to the OS by indicating (e.g., in response to an ACPI_STA method for the healthy processor pair) that such processor pair is not functioning. This would cause the OS to eject that processor pair as though it had encountered a LOL. This processor would become the “hot spare” and be used to replace the boot processor. The system boot processor role would be assumed by the idled healthy processor pair, just as described above in the example in which the healthy CPU 3 is held idle for the system boot processor, CPU 2. The only difference being that now the firmware had to take some initial action to turn the healthy processor pair into a “hot spare” by idling it and making it appear to the OS as though it was not functioning. Thus, the LIDs would be swapped. That is, the LID register of the boot processor that lost lockstep is copied into the LID register of the now spare processor pair (CPU 3 in this example), and the LID register of this spare processor pair would be copied into the LID register of the original boot processor. Lockstep is then reestablished on the original boot processor, and it can be reintroduced to the OS with the ID of the original non-boot processor that replaced it as the system boot processor. The processor that was formerly the boot processor may be designated, in device tree 203, as the spare for the now boot processor.
In operational block 405, the hot spare processor executes in an idle loop waiting for a PAL call to set state. Such PAL call to “set state” copies the saved state from the processor that experienced LOL (i.e., the state of the “good” one of the lockstep pair of processors in the processor module that experienced LOL) into the hot spare processor. Specifically, PAL copies all the needed information from memory into the internal registers of the hot spare processor so that the hot spare processor becomes a new active processor, transforming the role of the hot spare to boot processor. This call does not change the device tree settings. Accordingly, with the spare processor hidden from the OS (i.e., the OS is unaware of the hot spare), the hot spare sits in an idle loop until it is needed for recovering from LOL detected for the boot processor, at which time the system firmware interrupts its loop and assigns it the role of boot processor, as described above.
Operational block 505 represents the system's normal operation until such time as a LOL is detected for the boot processor. Upon error detect logic detecting a LOL for a processor, the error detect logic invokes the system's firmware. For instance, the LOL may invoke the PAL, which in turn may report the LOL to the SAL. The firmware (e.g., SAL) determines if the processor for which the LOL is detected is designated (e.g., in the device tree) as the boot processor. If so, then the firmware determines in block 505 that LOL is detected for the boot processor. Thus, in operational block 506, the firmware determines (e.g., from the device tree) the processor designated as the active spare, and the firmware instructs the OS to “eject” the active spare processor. Thus, the active spare is idled and effectively transformed into a hot spare. In operational block 507, the firmware assigns the active spare processor the role of boot processor. The state of the “good” one of the lockstep processor pair of the boot processor for which the LOL is detected is copied into both of the lockstep processors of the active spare processor. Further, the firmware updates the device tree to identify the active spare as the boot processor, and performs any other configuration changes (e.g., changing the LID of the active spare, etc.) to cause the OS to recognize the active spare as the boot processor.
In block 508, the firmware attempts to recover lockstep for the former boot processor for which LOL was detected in block 505. In operational block 509, the firmware determines whether the lockstep recovery for the former boot processor is successful. In certain embodiments, if the lockstep cannot be recovered in operational block 508 (e.g., the processor fails its self-test when being reset), then that processor is simply not reintroduced back to the OS and in operational block 510 a message is generated by the system to provide notice of the loss of this processor and that the boot processor no longer has a spare. If the lockstep cannot be reestablished in block 508, then if a LOL occurs in the new boot processor there will not be a spare processor for taking over its role (and thus the system may crash). If desired, the system administrator can schedule an orderly shutdown and re-boot in order to establish a new boot processor that has a hot spare available and/or schedule service of the system (e.g., for replacing the failed processor).
If the lockstep recovery is successful, then the former boot processor is assigned the role of active spare (e.g., by designating it as such in the device tree) in block 511, and the firmware triggers the OS to recognize the active spare as available for executing instructions. Operation then returns to block 505, and if a LOL is detected for the current boot processor, the procedure of blocks 506-512 may be repeated.
In block 602, the role of spare processor is assigned to another one of the processors. As described above, in certain embodiments the spare processor may be a hot spare, while in other embodiments the spare processor may be an active spare. As also described above, the role of spare processor may be so assigned in a device tree. Such role of spare processor may be assigned by a user (e.g., via a user interface) in certain embodiments, and assigned by the system's firmware in other embodiments.
In operational block 603, the system firmware, responsive to detecting LOL for the boot processor, changes the assignment of the role of boot processor to the spare processor. This change in the role of boot processor occurs dynamically during system runtime, rather than requiring that the system be interrupted (e.g., shutdown and restarted, or crashed).
Certain systems include a plurality of partitions, wherein each partition includes a boot processor and OS. It should be understood that the above techniques for establishing a spare processor for use in assuming the role of boot processor in the event that a LOL is detected for the boot processor may be employed on each partition. Thus, a device tree may designate a boot processor and spare (e.g., active or hot) for each partition.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4358823||Apr 12, 1979||Nov 9, 1982||Trw, Inc.||Double redundant processor|
|US5226152||Dec 7, 1990||Jul 6, 1993||Motorola, Inc.||Functional lockstep arrangement for redundant processors|
|US5249188||Aug 26, 1991||Sep 28, 1993||Ag Communication Systems Corporation||Synchronizing two processors as an integral part of fault detection|
|US5537655||Nov 21, 1994||Jul 16, 1996||The Boeing Company||Synchronized fault tolerant reset|
|US5675794||Mar 20, 1996||Oct 7, 1997||Intel Corporation||Method and apparatus for configuring multiple agents in a computer system|
|US5751932||Jun 7, 1995||May 12, 1998||Tandem Computers Incorporated||Fail-fast, fail-functional, fault-tolerant multiprocessor system|
|US5758058||Feb 27, 1996||May 26, 1998||Intel Corporation||Apparatus and method for initializing a master/checker fault detecting microprocessor|
|US5764660||Jul 9, 1997||Jun 9, 1998||Elsag International N.V.||Processor independent error checking arrangement|
|US5896523||Jun 4, 1997||Apr 20, 1999||Marathon Technologies Corporation||Loosely-coupled, synchronized execution|
|US5915082 *||Jun 7, 1996||Jun 22, 1999||Lockheed Martin Corporation||Error detection and fault isolation for lockstep processor systems|
|US5991900||Jun 15, 1998||Nov 23, 1999||Sun Microsystems, Inc.||Bus controller|
|US6065135||Jun 4, 1999||May 16, 2000||Lockhead Martin Corporation||Error detection and fault isolation for lockstep processor systems|
|US6141770||Jun 11, 1999||Oct 31, 2000||General Dynamics Information Systems, Inc.||Fault tolerant computer system|
|US6148348||Jun 15, 1998||Nov 14, 2000||Sun Microsystems, Inc.||Bridge interfacing two processing sets operating in a lockstep mode and having a posted write buffer storing write operations upon detection of a lockstep error|
|US6220581||Oct 6, 1995||Apr 24, 2001||Thomas L. Mueller||Bicycle seat gas spring adjustment system|
|US6263452||Jan 8, 1999||Jul 17, 2001||Compaq Computer Corporation||Fault-tolerant computer system with online recovery and reintegration of redundant components|
|US6330356||Sep 29, 1999||Dec 11, 2001||Rockwell Science Center Llc||Dynamic visual registration of a 3-D object with a graphical model|
|US6438687 *||Aug 17, 2001||Aug 20, 2002||Micron Technology, Inc.||Method and apparatus for improved storage of computer system configuration information|
|US6473869||Aug 10, 2001||Oct 29, 2002||Marathon Technologies Corporation||Fault resilient/fault tolerant computing|
|US6560682||Oct 3, 1997||May 6, 2003||Intel Corporation||System and method for terminating lock-step sequences in a multiprocessor system|
|US6604177||Sep 29, 2000||Aug 5, 2003||Hewlett-Packard Development Company, L.P.||Communication of dissimilar data between lock-stepped processors|
|US6625749 *||Dec 21, 1999||Sep 23, 2003||Intel Corporation||Firmware mechanism for correcting soft errors|
|US6643387||Jan 24, 2000||Nov 4, 2003||Sarnoff Corporation||Apparatus and method for context-based indexing and retrieval of image sequences|
|US6675324||Sep 27, 1999||Jan 6, 2004||Intel Corporation||Rendezvous of processors with OS coordination|
|US6687851||Apr 13, 2000||Feb 3, 2004||Stratus Technologies Bermuda Ltd.||Method and system for upgrading fault-tolerant systems|
|US6735253||May 16, 1997||May 11, 2004||The Trustees Of Columbia University In The City Of New York||Methods and architecture for indexing and editing compressed video over the world wide web|
|US6754787||Nov 22, 2002||Jun 22, 2004||Intel Corporation||System and method for terminating lock-step sequences in a multiprocessor system|
|US6920581 *||Jan 2, 2002||Jul 19, 2005||Intel Corporation||Method and apparatus for functional redundancy check mode recovery|
|US6948092||Jan 31, 2002||Sep 20, 2005||Hewlett-Packard Development Company, L.P.||System recovery from errors for processor and associated components|
|US6983018||Nov 30, 1998||Jan 3, 2006||Microsoft Corporation||Efficient motion vector coding for video compression|
|US7003691||Jun 28, 2002||Feb 21, 2006||Hewlett-Packard Development Company, L.P.||Method and apparatus for seeding differences in lock-stepped processors|
|US7020800||Jan 24, 2002||Mar 28, 2006||Hewlett-Packard Development Company L.P.||System and method for memory failure recovery using lockstep processes|
|US7042943||Jul 7, 2003||May 9, 2006||Apple Computer, Inc.||Method and apparatus for control of rate-distortion tradeoff by mode selection in video encoders|
|US7085959 *||Jul 3, 2002||Aug 1, 2006||Hewlett-Packard Development Company, L.P.||Method and apparatus for recovery from loss of lock step|
|US7134047||Jul 18, 2003||Nov 7, 2006||Intel Corporation||Firmwave mechanism for correcting soft errors|
|US7155721||Jun 28, 2002||Dec 26, 2006||Hewlett-Packard Development Company, L.P.||Method and apparatus for communicating information between lock stepped processors|
|US7191292||Jun 2, 2005||Mar 13, 2007||Sun Microsystems, Inc.||Logging of level-two cache transactions into banks of the level-two cache for system rollback|
|US7225355||Jul 8, 2003||May 29, 2007||Nec Corporation||Fault-tolerant computer system, re-synchronization method thereof and re-synchronization program thereof|
|US7237144||Apr 6, 2004||Jun 26, 2007||Hewlett-Packard Development Company, L.P.||Off-chip lockstep checking|
|US7251746 *||Jan 21, 2004||Jul 31, 2007||International Business Machines Corporation||Autonomous fail-over to hot-spare processor using SMI|
|US20020144177||Jan 31, 2002||Oct 3, 2002||Kondo Thomas J.||System recovery from errors for processor and associated components|
|US20020152420||Jan 31, 2002||Oct 17, 2002||Shailender Chaudhry||Providing fault-tolerance by comparing addresses and data from redundant processors running in lock-step|
|US20030051190||Sep 27, 1999||Mar 13, 2003||Suresh Marisetty||Rendezvous of processors with os coordination|
|US20030070050||Nov 22, 2002||Apr 10, 2003||Miller Robert J.||System and method for terminating lock-step sequences in a multiprocessor system|
|US20030126498||Jan 2, 2002||Jul 3, 2003||Bigbee Bryant E.||Method and apparatus for functional redundancy check mode recovery|
|US20030135711||Jan 15, 2002||Jul 17, 2003||Intel Corporation||Apparatus and method for scheduling threads in multi-threading processors|
|US20030140281||Jan 24, 2002||Jul 24, 2003||Guangrui Fu||System and method for memory failure recovery using lockstep processes|
|US20040006722||Jul 3, 2002||Jan 8, 2004||Safford Kevin David||Method and apparatus for recovery from loss of lock step|
|US20040019771||Jul 18, 2003||Jan 29, 2004||Nhon Quach||Firmwave mechanism for correcting soft errors|
|US20040078650||Jun 28, 2002||Apr 22, 2004||Safford Kevin David||Method and apparatus for testing errors in microprocessors|
|US20040078651 *||Jun 28, 2002||Apr 22, 2004||Safford Kevin David||Method and apparatus for seeding differences in lock-stepped processors|
|US20040153857||Jul 8, 2003||Aug 5, 2004||Nec Corporation||Fault-tolerant computer system, re-synchronization method thereof and re-synchronization program thereof|
|US20050172164 *||Jan 21, 2004||Aug 4, 2005||International Business Machines Corporation||Autonomous fail-over to hot-spare processor using SMI|
|US20050240806||Sep 28, 2004||Oct 27, 2005||Hewlett-Packard Development Company, L.P.||Diagnostic memory dump method in a redundant processor|
|US20050240811 *||Apr 6, 2004||Oct 27, 2005||Safford Kevin D||Core-level processor lockstepping|
|US20050240829||Apr 6, 2004||Oct 27, 2005||Safford Kevin D||Lockstep error signaling|
|US20060085677||Dec 1, 2005||Apr 20, 2006||Safford Kevin D||Method and apparatus for seeding differences in lock-stepped processors|
|US20060136672||Jun 2, 2005||Jun 22, 2006||Shailender Chaudhry||Logging of level-two cache transactions into banks of the level-two cache for system rollback|
|US20060248384 *||Jun 16, 2006||Nov 2, 2006||Safford Kevin D||Method and apparatus for recovery from loss of lock step|
|US20060248684||Jun 22, 2006||Nov 9, 2006||Vosbikian Peter S||Implements With Handles and Working Ends and Method of Use Thereof|
|WO2001045421A1||Nov 23, 2000||Jun 21, 2001||Koninkl Philips Electronics Nv||Motion estimation in video images|
|1||"System and Method for Configuring Lockstep Mode of a Prcessor Module".|
|2||"System and Method for Maintaining in a Multi-Processor System a Spare Processor That Is in Lockstep for Use in Recovering From Loss of Lockstep for Another Processor".|
|3||"System and Method for Providing Firmware Recoverable Lockstep Protection".|
|4||"System and Method for Reestablishing Lockstep for a Processor Module for Which Loss of Lockstep Is Detected".|
|5||"System and Method for Reintroducing a Processor Module to an Operating System After Lockstep Recovery".|
|6||"System and Method for Switching the Role of Boot Processor to a Spare Processor Responsive to Detection of Loss of Lockstep in a Boot Processor".|
|7||"System and Method for System Firmware Causing an Operating System to Idle a Processor".|
|8||"System and Method for Using Information Relating to a detected Loss of Lockstep for Determining a Responsive Action".|
|9||Dong Kyu Kim, Active Object Tracking Using Image Mosaic Background, 4 pages.|
|10||Final Office Action for U.S. Appl. No. 10/972,835, dated Nov. 21, 2007 in the United States.|
|11||Final Office Action for U.S. Appl. No. 10/973,075, dated May 12, 2009 in the United States.|
|12||Final Office Action for U.S. Appl. No. 10/973,076, dated Feb. 27, 2008 in the United States.|
|13||Jiang-Bin Zheng, Proceedings of the First International Conference on Machine Learning and Cybernetics, Beijing, Nov. 4-5, 2002, 4 pages.|
|14||Jianhoa Meng, Cveps-A Compressed Video Editing and Parsing System, Nov. 18, 1996, pp. 1-11.|
|15||Jong Ryul Kim, Automatic Localization and Trackign of Moving Objects Using Adaptive Snake Algorithm, ICICS-PCM 2003, Dec. 15-18, 2003, 5 pages, Singapore.|
|16||Notice of Allowance dated Feb. 12, 2009 for U.S. Appl. No. 10/972,588 in the United States.|
|17||Notice of Allowance dated Sep. 23, 2008 for U.S. Appl. No. 10/972,588 in the United States.|
|18||Notice of Allowance for U.S. Appl. No. 10/972,796, dated Dec. 19, 2007 in the United States.|
|19||Notice of Allowance for U.S. Appl. No. 10/972,835, dated Aug. 1, 2008 in the United States.|
|20||Notice of Allowance for U.S. Appl. No. 10/972,835, dated Nov. 20, 2008 in the United States.|
|21||Notice of Allowance for U.S. Appl. No. 10/972,888, dated Nov. 14, 2007 in the United States.|
|22||Notice of Allowance for U.S. Appl. No. 10/973,004, dated Aug. 1, 2007 in the United States.|
|23||Notice of Allowance for U.S. Appl. No. 10/973,076, dated Jul. 16, 2008 in the United States.|
|24||Notice of Allowance for U.S. Appl. No. 10/973,076, dated Jun. 3, 2008 in the United States.|
|25||Notice of Allowance for U.S. Appl. No. 10/973,076, dated Nov. 28, 2008 in the United States.|
|26||Notice of Allowance, dated Jul. 14, 2009, U.S. Appl. No. 10/972,588, 12 pages.|
|27||Office Action dated Mar. 10, 2008 for U.S. Appl. No. 10/972,588 in the United States.|
|28||Office Action dated Sep. 12, 2007 for U.S. Appl. No. 10/972,588 in the United States.|
|29||Office Action for U.S. Appl. No. 10/972,796, dated Jul. 12, 2007 in the United States.|
|30||Office Action for U.S. Appl. No. 10/972,835, dated Jul. 12, 2007 in the United States.|
|31||Office Action for U.S. Appl. No. 10/972,835, dated Mar. 3, 2008 in the United States.|
|32||Office Action for U.S. Appl. No. 10/972,888, dated Jul. 18, 2007 in the United States.|
|33||Office Action for U.S. Appl. No. 10/973,003, dated Mar. 5, 2008 in the United States.|
|34||Office Action for U.S. Appl. No. 10/973,003, dated Sep. 12, 2007 in the United States.|
|35||Office Action for U.S. Appl. No. 10/973,004, dated Apr. 10, 2007 in the United States.|
|36||Office Action for U.S. Appl. No. 10/973,075, dated Dec. 13, 2007 in the United States.|
|37||Office Action for U.S. Appl. No. 10/973,075, dated Jul. 2, 2007 in the United States.|
|38||Office Action for U.S. Appl. No. 10/973,075, dated Mar. 12, 2008 in the United States.|
|39||Office Action for U.S. Appl. No. 10/973,075, dated Oct. 29, 2008 in the United State.|
|40||Office Action for U.S. Appl. No. 10/973,076, dated Sep. 12, 2007 in the United States.|
|41||Shoichi Araki, Real-Time Tracking of Multiple Moving Object Contours in a Moving Camera Image Sequence, vol. E83-D IEICE Trans. Inf. & Syst, Jul. 7, 2000, 9 pages.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US8037293 *||May 30, 2008||Oct 11, 2011||International Business Machines Corporation||Selecting a random processor to boot on a multiprocessor system|
|US8046573||May 30, 2008||Oct 25, 2011||International Business Machines Corporation||Masking a hardware boot sequence|
|US8046574 *||May 30, 2008||Oct 25, 2011||International Business Machines Corporation||Secure boot across a plurality of processors|
|US8090975 *||May 8, 2009||Jan 3, 2012||Fujitsu Limited||Recovery server for recovering managed server|
|US9021179||Jun 10, 2011||Apr 28, 2015||International Business Machines Corporation||Store storage class memory information command|
|US9021180||Jun 10, 2011||Apr 28, 2015||International Business Machines Corporation||Clearing blocks of storage class memory|
|US9021226||Jun 10, 2011||Apr 28, 2015||International Business Machines Corporation||Moving blocks of data between main memory and storage class memory|
|US9037784||Dec 18, 2012||May 19, 2015||International Business Machines Corporation||Clearing blocks of storage class memory|
|US9037785||Dec 20, 2012||May 19, 2015||International Business Machines Corporation||Store storage class memory information command|
|US9043568||Dec 20, 2012||May 26, 2015||International Business Machines Corporation||Moving blocks of data between main memory and storage class memory|
|US9058243||Jun 10, 2011||Jun 16, 2015||International Business Machines Corporation||Releasing blocks of storage class memory|
|US9058245||Dec 18, 2012||Jun 16, 2015||International Business Machines Corporation||Releasing blocks of storage class memory|
|US9058275||Jun 10, 2011||Jun 16, 2015||International Business Machines Corporation||Data returned responsive to executing a start subchannel instruction|
|US9116634||Jun 10, 2011||Aug 25, 2015||International Business Machines Corporation||Configure storage class memory command|
|US9116635||Dec 19, 2012||Aug 25, 2015||International Business Machines Corporation||Configure storage class memory command|
|US9116788||Jun 10, 2011||Aug 25, 2015||International Business Machines Corporation||Using extended asynchronous data mover indirect data address words|
|US9116789||Jun 10, 2011||Aug 25, 2015||International Business Machines Corporation||Chaining move specification blocks|
|US9116813||Dec 19, 2012||Aug 25, 2015||International Business Machines Corporation||Data returned responsive to executing a Start Subchannel instruction|
|US9122573||Dec 19, 2012||Sep 1, 2015||International Business Machines Corporation||Using extended asynchronous data mover indirect data address words|
|US9164882||Dec 19, 2012||Oct 20, 2015||International Business Machines Corporation||Chaining move specification blocks|
|US20090049318 *||Jun 3, 2008||Feb 19, 2009||Pradip Bose||Method and system for controlling power in a chip through a power-performance monitor and control unit|
|US20090164810 *||Dec 21, 2007||Jun 25, 2009||Nokia Corporation||Methods, apparatuses, and computer program products for managing power usage in mobile devices|
|US20090282284 *||Nov 12, 2009||Fujitsu Limited||Recovery server for recovering managed server|
|US20090327680 *||May 30, 2008||Dec 31, 2009||International Business Machines Corporation||Selecting a Random Processor to Boot on a Multiprocessor System|
|US20120317445 *||Dec 13, 2012||International Business Machines Corporation||Deconfigure storage class memory command|
|US20130111178 *||May 2, 2013||International Business Machines Corporation||Deconfigure storage class memory command|
|U.S. Classification||714/13, 714/11|
|Cooperative Classification||G06F11/1641, G06F11/2028, G06F11/2043, G06F11/2025, G06F11/2038, G06F11/1654|
|European Classification||G06F11/16C12, G06F11/20P10, G06F11/20P2C, G06F11/20P2E|
|Oct 25, 2004||AS||Assignment|
Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY, LP, TEXAS
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MICHAELIS, SCOTT L.;RAJKUMARI, ANURUPA;REEL/FRAME:015941/0381
Effective date: 20041020
|Oct 19, 2010||CC||Certificate of correction|
|Mar 8, 2013||FPAY||Fee payment|
Year of fee payment: 4