DIAGNOSTIC CAGEMODE FOR TESTING REDUNDANT SYSTEM
CONTROLLERS
BACKGROUND OF THE INVENTION Field of the Invention
This invention relates to the field of multiprocessor computer systems with built-in redundancy, and more particularly, to systems and methods for testing redundant functional components during normal system operation
Description of the Related Art Multiprocessor computer systems include two or more processors which may be employed to perform computmg tasks. A particular computing task may be performed upon one processor while other processors perform unrelated computmg tasks Alternatively, components of a particular computmg task may be distributed among multiple processors to decrease the time required to perform the computing task as a whole. Generally speakmg, a processor is a device that executes programmed instructions to produce desired output signals, often in response to user-provided input data.
A popular architecture in commercial multiprocessor computer systems is the symmetnc multiprocessor (SMP) architecture. Typically, an SMP computer system compnses multiple processors each connected through a cache hierarchy to a shared bus. Additionally connected to the shared bus is a memory, which is shared among the processors m the system. Access to any particular memory location within the memory occurs m a similar amount of time as access to any other particular memory location. Since each location in the memory may be accessed m a uniform manner, this structure is often referred to as a uniform memory architecture (UMA).
Another architecture for multiprocessor computer systems is a distnbuted shared memory architecture. A distnbuted shared memory architecture includes multiple nodes that each include one or more processors and some local memory. The multiple nodes are coupled together by a network The memory included within the multiple nodes, when considered as a collective whole, forms the shared memory for the computer system.
Distnbuted shared memory systems are more scaleable than systems with a shared bus architecture. Since many of the processor accesses are completed within a node, nodes typically impose much lower bandwidth requirements upon the network than the same number of processors would impose on a shared bus The nodes may operate at high clock frequency and bandwidth, accessing the network only as needed Additional nodes may be added to the network without affecting the local bandwidth of the nodes. Instead, only the network bandwidth is affected.
Because of their high performance, multiprocessor computer systems are used for many different types of mission-cntical applications in the commercial marketplace. For these systems, downtime can have a dramatic and adverse impact on revenue. Thus system designs must meet the uptime demands of such mission cntical applications by providmg computmg platforms that are reliable, available for use when needed, and easy to diagnose and service.
One way to meet the uptime demands of these kinds of systems is to design in fault tolerance, redundancy, and reliability from the inception of the machine design. Reliability features incorporated in most multiprocessor computer systems include environmental monitoring, error correction code (ECC) data protection, and modular subsystem design More advanced fault tolerant multiprocessor systems also have several additional features, such as full hardware redundancy, fault tolerant power and coolmg subsystems, automatic recovery after power outage, and advanced system momtormg tools
For mission cntical applications such as transaction processmg, decision support systems, communications services, data warehousing, and file serving, no hardware failure in the system should halt processmg and bring the whole system down. Ideally, any failure should be transparent to users of the computer system and quickly isolated by the system. The system administrator must be informed of the failure so remedial action can be taken to bring the computer system back up to 100% operational status. Preferably, the remedial action can be made without bringing the system down.
In many modern multiprocessor systems, fault tolerance is provided by identifying and shutting down faulty processors and assigning their tasks to other functional processors. However, faults are not limited to processors and may occur m other portions of the system such as, e.g., interconnection traces and connector pins While these are easily tested when the system powers up, testing for faults while the system is running presents a much greater challenge. This may be a particularly crucial issue m systems that are "hot-swappable", i.e. systems that allow boards to be removed and replaced during normal operation so as to permit the system to be always available to users, even while the system is bemg repaired.
Further, some multiprocessor systems include a system controller, which is a dedicated processor or subsystem for configuring and allocating resources (processors and memory) among vaπous tasks. Fault tolerance for these systems may be provided m the form of a "back-up" system controller. It is desirable for the primary and redundant system controllers to each have the ability to disable the other if the other is determined to be faulty Further, it is desirable to be able to test either of the two subsystems during normal system operation without disrupting the normal system operation. This would be particularly useful for systems that allow the system controllers to be hot-swapped.
SUMMARY OF THE INVENTION Accordingly, there is disclosed herem a multiprocessor system that employs an apparatus and method for cagmg a redundant component to allow testmg of the redundant component without mterfermg with normal system operation. In one embodiment the multiprocessor system mcludes at least two system controllers and a set of processmg nodes interconnected by a network. The system controllers allocate and configure system resources, and the processmg nodes each include a node mterface that couple the nodes to the system controllers. The node interfaces can be individually and separately configured m a caged mode and an uncaged mode. In the uncaged mode, the node mterface communicates mformation from either of the system controllers to other components m the processmg node. In the caged mode, the node mterface censors mformation from at least one of the system controllers. When all node interfaces censor information from a common system controller, this system controller is effectively "caged" and communications from this system controller are thereby prevented from reachmg other node components. This allows the caged system controller along with all its associated interconnections to be tested without mterfermg with normal operation of the system. Normal system configuration tasks are handled by the uncaged system controller. The uncaged system controller can instruct the node mterfaces to uncage the caged system controller if the tests are successfully completed
BRIEF DESCRIPTION OF THE DRAWINGS A better understanding of the present invention can be obtamed when the following detailed description of the preferred embodiment is considered in conjunction with the following drawings, m which Fig. 1 is a functional block diagram of a multiprocessor system: and
Fig 2 is a functional block diagram of a processor node
While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example m the drawings and will herem be descπbed m detail It should be understood, however, that the drawings and detailed descnption thereto are not mtended to limit the invention to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spint and scope of the present invention as defined by the appended claims
DETAILED DESCRIPTION OF THE INVENTION Turning now to the figures, Fig. 1 shows a block diagram of a multiprocessor system. The system mcludes a center plane 102 that interconnects N nodes (designated Node 0 through Node N-l) with a network bus (not shown). The network bus is preferably a crossbar network The nodes preferably each mclude a node mterface board 104 which accepts up to two boards one of which is designated as a "Slot 0 board" 106 while the other is designated as a "Slot 1 board" 108. Slot 0 boards are preferably multiprocessor boards that each mclude four processors, a memory module, and a system mterface interconnected by a bus, and vaπous support chips. Slot 1 boards are preferably I/O boards that mterface to vaπous penpherals such as seπal and parallel ports, disk dπves, modems, printers, etc. In addition to the descπbed types of Slot 0 and Slot 1 boards, other board types may be used, and the mix of the various board types among the vanous nodes is preferably alterable.
The system also mcludes at least two system controllers 110 which are preferably coupled to the center plane 102 by corresponding system controller support boards 112. The center plane 102 preferably provides busses from the support boards 112 to the nodes for mamtenance, momtormg, and configuration of the nodes. The center plane 102 may also provide an arbitration bus 114 that allows the system controllers 110 to arbitrate for communication pπvileges to the nodes.
For a mission-cntical system, it is necessary that the vaπous components be hot-swappable so that defective components can be removed and replaced without brmgmg the system down. Accordmgly, each of the node mterface boards 104 and support boards 112, along with their dependent boards, can be removed and replaced while the system is operating. Smce insertion is an event that has a relatively high failure probability, it is desirable to test the newly inserted components along with their physical mterface to the system pπor to trusting them with substantive tasks. The ensuing descnption focuses on testmg of the system controllers 110 and support boards 112, but it is recognized that the nodes may be similarly tested Fig. 2 shows selected components common to each of the nodes. The node mterface board 104 mcludes a system data mterface chip 202, and each board 106, 108 mcludes a system data controller chip 204 that operates to configure and monitor vanous components on the board m accordance with information received from the system controller via the system data mterface chip 202 The system data mterface chip also operates to configure and monitor vanous components m the node mterface 104 m accordance with communications received from the system controller. Both chips 202 and 204 are preferably able to parse address mformation and route communications from the system controller to the components indicated by the address mformation. The chips 202, 204 may additionally convert the communications mto whatever form or bus protocol may be needed for the destmation component to understand the message
Referring concurrently to both Figs. 1 and 2, the system data interface chip 202 has a dedicated port for each of the system controllers 110, so that all communications with a given system controller are conducted via the associated port The system data mterface (SDI) chip 202 also mcludes some eπor detection and notification
circuitry If the SDI chip 202 detects that a communication from a given system controller is corrupted, the SDI chip 202 can communicate an error notification to that system controller However, if the SDI chip 202 is unable to determme the source of the eπor (e g when receivmg conflictmg communications from different system controllers) the SDI chip 202 may assert a system interrupt signal to alert the system controllers to the error event SDI chip 202 mcludes some status, configuration and test registers The status registers may be read by the system controllers to determine eπor conditions, for example One of the configuration registers mcludes "cage" mode bits that can be asseπed and de-asserted only by an "uncaged" system controller An uncaged system controller may put one or all of its mterfaces mto a cage mode, but an uncaged system controller will be required to put them back mto an uncaged mode It is noted that m situations where both node mterfaces are caged, or a caged system controller can not respond to a command to exit the cage, either system controller (whether caged or not) can initiate a bus reset that will force the node mterface back to an uncaged mode
Either of the system controllers can be caged by assertion of an associated cage mode bit The assertion of cage mode bits may be accomplished by one of the system controllers wπting an individual cagmg message to each of the nodes The SDI chips 202 m each of the nodes interpret the cagmg message and assert the cage mode bit for the designated system controller The system controller designated m the cagmg message to a node mterface is hereafter refeπed to as a caged system controller for that node mterface Conversely, a system controller for which the cage mode bits m a node mterface are not asserted is hereafter refeπed to as an uncaged system controller for that node mterface
Either of the system controllers can have one or more of its mterfaces caged by wπtmg a cage enable message to the pertinent node mterfaces If all node mterfaces have the same system controller mterface caged, then the system controller is said to be completely caged If not all not all node mterfaces have the same system controller mterface caged, then the system controller is incompletely caged, and it is permitted to communicate with mterfaces for which it is uncaged
Assertion of a cage mode bit causes the SDI chip 202 to censor any communications received from the caged system controller The SDI chip 202 may communicate responses to the caged system controller such as, e g eπor notification for corrupted communications The SDI chip 202 may also operate on communications from the caged system controller, e g storing values in the test registers However, the SDI chip 202 does not transmit any messages to other downstream components m response to communications received from the caged system controller This mcludes configuration messages for the boards 106, 108, as well as messages for other components on node mterface board 104 The SDI chip also suppresses interrupts tnggered by communications from the caged system controller, such as a protocol eπor interrupt that would normally be caused by a message from the caged system controller that conflicts with a message received from the other system controller
When the multiprocessor computer system is first powered up, the primary system controller runs through a Power-On-Self-Test (POST) which tests all of the components on the system controller and then tests all of the interconnections to other boards/components m the multiprocessor system Smce no user applications are active, seπous or fatal eπors will not cause service interruptions However, if the multiprocessor system is executmg user applications and a secondary system controller needs to be tested, then the cagmg mode may be employed to test the secondary system controller while the primary system controller contmues providmg services to the hardware and mastering all mamtenance buses required for testmg The cagmg mode will prevent the system controller under test from madvertently destrovmg state mformation m the active hardware or causmg an eπor which would not be
isolated and would be reported as an system eπor from the component under test. Such actions would probably bring down the multiprocessor system
Referring to Fig 1, a newly mserted system controller support board 112 with attached system controller 110 is caged by placmg all of its node mterfaces mto cagmg mode. This can be done by the newly mserted system controller or by the resident system controller A test process executmg on the caged system controller is then able to veπfy the functionality of not only the on-board components, but also the components on the support board 112, the center plane 102, and portions of the SDI chip 202 It is noted that the interconnections between the system controller 110, the support board 112, the center plane 102, and the node mterface 104 are also venfied by the test process. The uncaged system controller can check for successful completion of the test process, e.g. by readmg status registers m the caged system controller and/or status registers in the SDI chips 202, and broadcast an uncaging message to the SDI chips 202 if the test process is determined to have completed successfully
In addition to testmg itself, the caged system controller is able to test off-board interconnects without concern of mterfermg with running software applications and the primary system controller Without this ability, the caged system controller could not detect faulty mterconnections with the rest of the system. If untested mterconnections to the mserted system controller were faulty, this would not be known until after the primary system controller had failed. At that pomt the faults would appear and the system would probably crash. The detection of interconnect faults before failure of the primary system controller allows time for notification and for remedial action.
It is noted that discussion has centered on testmg of a redundant system controller by placmg all node mterfaces mto the cagmg mode. However, the descπbed embodiment also allows node mterfaces to be separately and individually placed mto the cagmg mode This allows testmg of individual bus connections while the system controller is able to maintain its duties elsewhere.
One embodiment of the invention has been generally descnbed above The following discussion descnbes vaπous details of one particular prefeπed implementation for explanatory purposes. However, the invention is not so limited.
The invention may be employed m a next generation, UltraSPARC III based, high-end enterpπse server system The system controllers may be smgle processor based subsystems which provide many global resources to all of the multiprocessor hardware. The system controllers may employ a variety of buses to obtam complete access to all hardware. Preferably, more than one system controller is present at any given time one but only one is active The second system controller preferably waits m a stand-by mode m case the primary system controller expeπences a hardware failure.
The system controller mterconnections to other hardware occur through vaπous buses such as I2C (Inter- Integrated Circuit), JTAG (Jomt Test Activity Group), and Console Bus. In a normal (uncaged) operating mode, the node mterfaces multiplex together both system controller's Console Buses and provide no hardware isolation Thus, all boards and components m the system see all transactions emanatmg from either system controller In cagmg mode, the node mterfaces isolate a system controller and its Console Bus interconnections to the vanous hardware boards to prevent faults and protocol eπors from propagatmg. This allows the system to properly test a system controller board while the system is running user application programs without causmg a complete system crash.
The center plane may be a 16x16 crossbar interconnection network such as Sun Microsystems' Inc. Gigaplane-XB. This center plane contams two symmetncal sides which can each mount up to eight system boards, a support board and a system controller board. The system boards reside on node mterface boards that connect to
the center plane through I2C bus and Console Bus. The I2C bus is a seπal data bus developed by Philips Corporation consisting of a two lme mterface. One lme consists of a data pin for input and output functions and the other lme is a clock for reference and control
Console Bus is a bus developed by Sun Microsystems Inc. and is used by the system controller as a pathway for status and control of all system functions. The System Data Interface (SDI) chip contams a console bus mterface used by the system controller as a pathway for stams momtormg and configuration control of all system functions The console bus is the primary system control/diagnostics bus and is required to operate coπectly at all times while the system is operational. Dual console bus mterfaces, one to each system controller, are provided for redundancy Because of its cntical importance, the SDI also contams a console bus cage mechanism to facilitate diagnostic testmg of one of the two console bus mterfaces while the other console bus mterface is actively bemg used by the system for momtormg and configuration functions Additionally, both mterfaces of an SDI chip may be caged and tested mdependently if the situation requires (e.g. when a new node is mserted mto a working system) The console bus cage operates to ensure that any event (coπect or eπoneous) that occurs while accessmg a caged console bus has no impact on the normal functioning of the rest of the system, and specifically not on the other console bus operations. If a system controller after bemg caged and tested is not functioning coπectly, the uncaged system controller can access SDI status registers that contam diagnostic identification information to determme the nature of the eπors.
During normal operation, the uncaged Console Bus mterface m the SDI chip handles any address translations required from the node mterface board to either of the resident Slot 0 and Slot 1 boards. In this mode, a smgle state machine may be shared between the SDI console bus ports. In the caged mode, a second state machme may be used to handle transactions with the caged system controller. The transition from uncaged mode to caged mode can occur at any tune. However, to avoid protocol eπors, the transition from the caged mode to uncaged mode can occur only when the uncaged mode state machme is quiescent. A cage control register mcludes bits for indicating the activity of the state machines and the cage mode bits for cagmg the system controllers
All accesses from a caged system controller are examined to determme if they are within the allowed range of addresses for status and test registers Accesses outside this range are responded to with an lllegal-access-eπor acknowledgement, and the access is suppressed. Eπor notifications may be posted m an eπor (status) register, but no interrupts are caused by caged transactions. An example of a Slot 0 board which may be employed m the system is multiprocessor system board that holds four Sun Microsystems UltraSPARC III microprocessors with supporting level two cache An example of a Slot 1 board which may be employed m the system is an I/O board with multiple PCI mterfaces with slots for networking and I/O adapters. PCI bus is a standard bus used m computer systems to communicate data, instructions and control mformation between logic circuits The system controller support boards connect the system controllers to the center plane through the
Console Bus and the I2C bus. The system controller support boards are repeaters, that is, they amplify and multiply the output signals from the system controller board and send them to the center plane for output to the node mterface boards The system controller board contams some system-level logic that mcludes a system clock generator, temperature and airflow momtormg circuitry, and a PCI mterface to a computer system which handles diagnostics, boot, shutdown and environmental momtormg The multiprocessor computer system requires only one
system control board for proper operation However, for higher levels of system availability, a second optional system control board may be installed
The computer system mcluded m the system controller board mcludes an UltraSparc Hi microprocessor and vanous programmable read only memoπes (PROM) contammg software for configuration and testing of the hardware m the multiprocessor computer system The system level logic converts PCI signals mto I2C and Console Bus and, after amplification and multiplication m the support board, sends these signals to the center plane The system-level logic also controls JTAG scan chams which connect through the center plane and all hardware boards in the multiprocessor computer system JTAG test access ports are present throughout the center plane and vanous boards of the multiprocessor computer system and allow for greater visibility and veπfication of system boards when the system controller performs the POST
During operation of the multiprocessor computer system, a replacement microprocessor board or I/O board after it has been mserted must be attached electncally to the remamder of the hardware The component must be isolated from the other hardware present m the computer system and tested pnor to and during attachment Finally, the hardware component must be incorporated logically mto the running multiprocessor computer system to run the operatmg system and execute application programs for users
In one prefeπed embodiment the replacement microprocessor or I O board becomes a part of a dynamic system domam after it has been mserted mto the center plane of the multiprocessor computer system Dynamic system domams are software partitions that permit the multiprocessor computer system to be dynamically subdivided mto multiple computers A dynamic system domam may consist of one or more system boards Each domam is a separate shared-memory SMP system that runs its own local copy of a multiprocessor operatmg system such as Sun Microsystems Solans and has its own disk storage and network connections Because individual system domams are logically isolated from other system domams, hardware and software eπors are confined to the domam m which they occur and do not affect the rest of the system After a system administrator requests a particular domam configuration, the system controller configures the vanous microprocessor and I/O boards mto dynamic system domams m the multiprocessor computer system
Modifications to the hardware makeup of a domam may be required while the multiprocessor computer system is m operation In order to facilitate run rime changes in dynamic system domam configuration, system administrators should be able to dynamically switch system boards between domams or remove them from active domams for testmg, upgrades, or servicing Ideally after testmg or service hardware boards should be easily remtroduced mto one of the active domams without mterruptmg system operation Each system domam is administered through the system controller which services all the domams The system controllers may mterface to a SPARC workstation or equivalent computer system board that runs a standard operatmg system such as Microsoft Wmdows NT or Microsoft Wmdows 98, Sun Microsystems Solans, IBM AIX, Hewlett Packard UX or some similar equivalent and a suite of diagnostic and management programs The external computer system may be connected via a network mterface card such as Ethernet to the system controller located m the multiprocessor computer system The microprocessor m the system controller board interprets the network interface card (e g TCP/IP Ethernet) traffic and converts it to encoded control mformation
Numerous vaπations and modifications will become apparent to those skilled m the art once the above disclosure is fully appreciated It is mtended that the following claims be interpreted to embrace all such vanations and modifications