|Publication number||US5179707 A|
|Application number||US 07/533,192|
|Publication date||Jan 12, 1993|
|Filing date||Jun 1, 1990|
|Priority date||Jun 1, 1990|
|Also published as||DE69129477D1, DE69129477T2, EP0459714A2, EP0459714A3, EP0459714B1|
|Publication number||07533192, 533192, US 5179707 A, US 5179707A, US-A-5179707, US5179707 A, US5179707A|
|Inventors||Richard S. Piepho|
|Original Assignee||At&T Bell Laboratories|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (4), Non-Patent Citations (4), Referenced by (37), Classifications (13), Legal Events (6)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This invention relates to apparatus for processing interrupts in a shared memory multiprocessing system.
A common way of achieving high throughput in processor systems, and more specifically for achieving the advantages of steady modular growth of capacity for such systems, is to use a multiprocessor system having a plurality of similar or identical processors, each of which can access a common memory. The processors of such systems typically perform both background tasks, i.e., tasks typically requiring a longer period of time for their completion, and interrupt tasks, i.e., tasks which are performed in response to specific interrupt signals and which must be performed almost immediately after an interrupt signal is received. The interrupt signal typically is generated by an input/output device such as a disk or tape unit and requires some immediate processing. In such a system where there may be a large number of separate sources of these interrupt signals, it is necessary to allocate the processing of each new interrupt to one of the processors of the multiprocessor system. In one prior art arrangement, all interrupts go to one processor. This arrangement is limited in the size of the system that can be supported because all of the interrupt processing is necessarily funneled to a common processor which must handle not only the average interrupt processing load, but also must handle clustered interrupt processing.
Some of these difficulties are overcome by a second arrangement wherein all interrupts are initially channeled to a single processor but wherein this processor in turn interrupts one of the other processors to handle each specific interrupt. The disadvantage of this approach is that the response time is necessarily slowed down because the initial processor's time is added to that of the processor that actually handles the interrupt. Further, this arrangement is wasteful of resources because the initial processor's resources which are dedicated in part or in whole to the initial processing of these interrupts cannot be used for other purposes. Further, if interrupts are clustered, these clustered interrupts are backed up in the initial processor and so the delay in processing the last of the clustered interrupts may be unacceptable. In an alternate arrangement, interrupts are placed on either a dedicated interrupt bus or a multiprocessor system memory bus. The interrupt source wait for an idle interval on either of these buses and upon the detection of this idle interval place a signal identifying the interrupt source. This interrupt is then passed by each of the processors of the multiprocessor system and through some arbitration scheme, one of the processors picks up the interrupt. The specific processor that carries out the interrupt processing is selected either by having all the processors pick up the interrupt signal and through some arbitration arrangement automatically selecting among themselves the particular processor which will handle the interrupt or by having the interrupt signal sequentially pass each processor wherein each processor either picks up the interrupt signal and prevents the signal from being passed to other processors or does not pick up the interrupt signal and permits the signal to pass to the other processors, one of whom will pick it up.
In many of these multiprocessor systems, the interrupts are clustered so that each interrupt signal in fact represents a plurality of sources. For example, one interrupt signal may represent interrupts generated by three or four tape units. Therefore, each interrupt signal in fact represents a plurality of basic interrupt sources. A problem, even with the last of these arrangements, is that it is very difficult to arrange that the work of processing the interrupts is evenly distributed. For example, if an interrupt bus is used that is connected to each of the processors in sequence, and each processor makes the decision on whether or not to accept an interrupt based on the level of the interrupt, the first of the processors may be either overloaded or underloaded because of changes in the rate of interrupts at different levels and because of the inability to predict adequately the portion of the interrupts that will be, in fact, associated with each level. A problem of the prior art therefore is that there is no satisfactory arrangement for distributing interrupts among a plurality of processors in a shared memory multiprocessor system such that each processor performs an approximately "fair" share of the interrupt processing load, that interrupts are processed expeditiously and efficiently.
A large body of programs exists which have been written for personal computers (PCs) and which have been written to interface with standard software and hardware interrupt drivers that work with such PCs. The industry standard for PC interrupt hardware (controlled by software and interrupt drivers) is specified in the Extended Industry Standard Architecture (EISA) Specification, Version 3.1 (1989) published by BCPR Services, Inc. Such drivers are used, for example, for controlling the video displays of PC terminals; one example of such a program is a typical PC console driver. The PC interrupt signals of this class must be routed to a particular processor and to a particular set of register locations within that processor. The types of interrupt drivers used in multiprocessor systems are not compatible with PC interrupt drivers. A problem of the prior art is that no arrangement exists for both fairly distributing interrupts among processors of a multiprocessor system having shared memory and distributing a subset of these interrupts to particular preselected processors (acting as PC processors) and to transmit this subset of interrupts in arrangements compatible with PC hardware and software.
The above problem is solved and an advance is made over the prior art in accordance with the principles of my invention wherein, in one exemplary embodiment, interrupts are distributed among a plurality of processors of a shared memory multiprocessor system; within that plurality, the available processor with the highest rank, as modified by a prefix representing the recent interrupt processing history of a processor, is selected by an arbitration arrangement to process the interrupt.
In one embodiment of the invention, each multiprocessor interrupt signal is distributed to all the processors and an arbitration scheme is used to select the processor that will actually handle the interrupt; each processor has a processor rank identification and the available processor with the highest rank identification is selected for processing the interrupt. In accordance with the invention, the rank identification of the processor is modified by an additional indication which effectively prefixes the processor rank identification, the indication being representative of the time that has elapsed since the last interrupt was processed and/or the processing load of that interrupt. In one specific embodiment, the indication is simply one bit which is reset for a prespecified time after any interrupt has been processed.
In one specific embodiment, the arbitration is based on transmitting signals among the processors in such a way that signals from lower ranked processors are overridden by signals from higher ranked processors. The designation of a processor is built into the backplane wiring such that the position of the processor at the position in the backplane where a processor is plugged in defines the rank of the processor. The circuitry used for accepting interrupt signals from the distribution bus comprises interrupt distribution logic which comprises logic for recognizing that an active signal of a type which the processor is required to handle if the processor is the highest ranking processor for that kind of signal and for performing the arbitration and for accepting additional information on the interrupt distribution medium defining the data of the interrupt. An interrupt register is attached to the interrupt distribution logic to accept the signal representing the rank of the interrupt. This rank is logically combined with the contents of an interrupt mask register used within the processor to select the levels of interrupt that the processor will process. If the interrupt signal is of a level which passes through the mask then this processor can bid for serving the interrupt and the arbitration logic determines which of the bidding processors will actually process the interrupt. In the arbitration arrangement, a bidding signal from a higher ranking processor suppresses the bidding signal of any lower ranking processor.
In this embodiment, some of the processors of the shared memory multiprocessor system are arranged to receive PC compatible input including interrupt signals and to transmit PC compatible output and also to receive and transmit interrupt signals and other I/O signals of the multiprocessing system. In accordance with one specific embodiment of the invention, each of the processors of the system receives multiprocessor I/O interrupt signals. The received multiprocessor I/O interrupt signals are distributed among all the processors by an interrupt distribution medium and PC compatible interrupt signals are distributed among some of the processors. Each processor comprises internal arbitration logic to select which processor will process any given interrupt. In this embodiment the arbitration scheme is based on each processor having a rank different from the rank of all other processors wherein the rank is prefaced by an indication of preference for accepting interrupts based on the previous interrupt processing history. The processor with the highest rank, as modified by this indication, then processes the interrupt.
In accordance with one aspect of the invention, the PC compatible mode can be used to drive and receive input from a network interface to a network connectable to a plurality of PC's. Advantageously, this arrangement permits PC compatible uniprocessor interface software to be used for communicating between uniprocessors at the other terminals of the network and a multiprocessor system.
Advantageously, this invention permits existing PC compatible software that meets the protocols of connected processor or communication system to be used for interfacing to a multiprocessor system wherein the multiprocessor system uses a different protocol, for improved performance, to communicate within the multiprocessor system and with other devices.
FIG. 1 is a block diagram of a multiprocessing system for practicing applicant's invention; and
FIG. 2 is a block diagram of the interrupt handling mechanism within one of the processors of the multiprocessing system.
FIG. 1 is a block diagram of a multiprocessor system. It includes compatible processor element 1 (block 100), . . . , compatible processing element-2 (block 101). These are processing elements, shown in detail in FIG. 2, for interfacing with input/output (I/O) equipment in a manner compatible with standard personal computer hardware and software standards. Also shown are input/output channel 1 (block 115), . . . , input/output channel 2 (block 116) for taking signals from the multiprocessor compatible input/output equipment and communicating with the main memory bus 123 and interrupt signal distribution medium (block 121). There are further multiprocessing elements MPE-1 (block 110), . . . , MPE-2 (block 111). Note that while CPE-1 in connected to both the interrupt signal bus 125 and the interrupt distribution medium 121, MPE-1 is only connected to the interrupt distribution medium 121. This is because whereas CPE-1 is capable of communicating with units such as the network interface 133 which operates only in the PC standard I/O mode, and with I/O devices such as disk 131 which is arranged to operate in the multiprocessor mode by signaling its interrupt signals over the interrupt distribution medium 121 MPE-1 only receives interrupt signals from IDM 121.
Each of the I/O channels 115, . . . , 116 accepts one or more of the I/O control signals from I/O signal bus 125 and uses them to generate a signal that is placed on IDM 121. In addition, I/O channels 115, . . . , 116 are used to communicate data between a peripheral device and the main memory bus 123. The main memory bus 123 is connected to both the CPE elements 100, . . . , 101 and MP elements 110, . . . , 111, all of which communicate with shared main memory 135 via the main memory bus 123. Multiprocessor I/O elements communicate with memory as follows: if CPE-1 wishes to access disk 131, for example, CPE-1 writes a request in main memory 135, including the address of memory where the disk output is to be written. CPE-1 then writes a request indication into a known location of a controller of disk 131 via data bus 126. The disk controller then reads main memory 135, via data bus 126, I/O channel-1 (115), and memory bus 123 to obtain addressing and data transfer length information. Disk 131 then accesses the requested data and delivers that data to main memory via data bus 126, I/O channel-1 (115) and main When the disk action is completed, the disk sends an interrupt signal over interrupt signal bus 125 to I/O channel 1, which then sends an interrupt signal over IDM 121 to notify the multiprocessor system that the disk action has been completed. Interrupt signal bus 125 comprises a bundle of wires carrying interrupt signals from sources to those processors and/or channels which are attached to particular ones of the interrupt signals. If PC compatible network interface 133 wishes to transmit one or more characters, it interrupts an attached compatible processing element, in this case, CPE-1 (block 100) over interrupt signal bus 125. CPE-1 then accepts the data from network interface 133 and controls the storage of that data in main memory using connections from CPE-1 to the main memory bus 123 for storage in main memory 135. Data from network interface 133 is transmitted to main memory 135 via data bus 126, CPE-1, and main memory bus 135. CPE-1 sends data to network interface 133 by first writing a request into a known location of interface 133 via data bus 126. Subsequently, data is sent from CPE-1 to interface 133 over data bus 126.
In one alternative mode, CPE-1 has a direct memory access unit for directly controlling the transmission of data from the network interface 133 via the data bus 126 to memory bus 123. In another alternative mode, the network interface 133 has a direct memory access unit for controlling transfer of data between the memory and the interface, and uses CPE-1 as an access means for accessing the memory bus 123.
FIG. 2 shows details of CPE-1 (block 100). Note that CPE-1 receives interrupt source signals on the top from interrupt signal bus 125, carrying a signal from PC compatible network interface 133 and on the bottom from multiprocessor system interrupt distribution medium 121. Consider first the signals received from the interrupt distribution medium. These signals go into interrupt distribution logic 201 and specifically are terminated within block 201 in an interrupt identifier register 203. The output of this register is connected to an arbitration (ARB) block 205 whose other inputs include an interrupt level mask register 209, a rank prefix 206, processor rank identifier 207 and interrupt distribution medium 121 which also carries signals from the output of ARB blocks 205 in other processing elements. The rank prefix 206 represents a modifier to the processor rank identifier 207, the rank being determined within the identifier by the position of the processor plugged into a backplane for interconnecting the processor elements. The rank prefix in this embodiment is set to 0 after processing of an interrupt has begun, and is set to 1 after a lapse of 100 microseconds following processing of the interrupt. If the CPE-1 is an available candidate for processing an interrupt received from medium 121, ARB 205 transmit a serial signal representing the modified rank of CPE-1. While transmitting this rank, CPE-1 also monitors medium 121. If CPE-1 transmits a "0" and monitors a "1", this is a signal that a processor element of higher modified rank is an available candidate, and that CPE-1 will therefore not process this interrupt. Otherwise, CPE-1 will process the interrupt; in that case, the output of ARB 205 is active.
If the output of ARB 205 is active, then a coded version of the level of the interrupt is gated by gate G1 (block 221) and stored in interrupt level register (XIR) 223. XIR 223 stores the interrupt level as a 1 out of 16 signal. This makes it easy to combine the output of XIR 223 with that of XIM 209 in gate G3 (block 227) to ensure that the level of the interrupt is still within the range that the processor can handle. XIM is then altered under software control to filter out subsequent interrupt signals at the level of the present interrupt signal at the same or lower levels as the present interrupt signal. The output of G3 is a single bit which enters a register in peripheral interrupt control-2 (PIC-2) (block 231). The signal enters PIC-2 at a position which identifies interrupt signals received from the interrupt distribution medium 121. Interrupt source signals which are directly received by CPE-1 are gated into one of the other positions of PIC-1 (block 233) or PIC-2 (block 231). PIC-1 also receives inputs from internal interrupt sources within CPE-1. If any of the interrupt signal indicators of PIC-2 are high, then the output of PIC-2 is placed into a prearranged position of PIC-1. Processor 240 then examines the state of PIC-1 and PIC-2 if any of the interrupt indicators of PIC-1 are high. If appropriate, as indicated by an active signal in the appropriate bit position of PIC-2, the processor than examines the contents of XIR 223 and decides which interrupt to process based on the contents of PIC-1, PIC-2 and XIR 223. Processor 240 then communicates as appropriate with main memory bus 123.
MPE-1 (block 110) is similar to CPE-1 except that the connections to the interrupt source signals shown at the top of block 100 are missing; consequently, MPE-1 only handles interrupts which have been gated from G3 (block 227).
Using this arrangement, communications with the network interface 133 can be carried out entirely in a PC compatible mode so that if software had been previously been arranged to process such communications using PC compatible mode networking software, there would be no need to write such software. On the other hand the multiprocessor system is capable of communicating with disk 131 in a conventional multiprocessor mode using software for controlling such communications.
In this particular embodiment, 11 outputs from interrupt signal source bus 125 are terminated in CPE-1. Each of the PIC units 231 and 233 can accept 7 input signals, therefore 6 of the interrupt source signals can be terminated in PIC-1 (block 233) and 5 can be terminated in PIC-2 (block 231). This allows the output of PIC-2 to be used for setting the seventh input of PIC-1 and allows the output of G3 to be used for setting the sixth or seventh input to PIC-2.
In other embodiments of the invention, the rank prefix is a counter whose initial value is set by software according to the processing load of the interrupt, and which is automatically incremented at a predetermined rate, such as once every millisecond. Advantageously, this arrangement permits a processor that has processed a complex interrupt to wait, on the average, a longer time before processing the next interrupt.
It is to be understood that the above description is only of one preferred embodiment of the invention. Numerous other arrangements may be devised by one skilled in the art without departing from the scope of the invention. The invention is thus limited only as defined in the accompanying claims.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4080649 *||Dec 16, 1976||Mar 21, 1978||Honeywell Information Systems Inc.||Balancing the utilization of I/O system processors|
|US4816990 *||Nov 5, 1986||Mar 28, 1989||Stratus Computer, Inc.||Method and apparatus for fault-tolerant computer system having expandable processor section|
|US5067071 *||Feb 27, 1985||Nov 19, 1991||Encore Computer Corporation||Multiprocessor computer system employing a plurality of tightly coupled processors with interrupt vector bus|
|EP0348076A1 *||Jun 8, 1989||Dec 27, 1989||International Computers Limited||Data processing device|
|1||A. Cantoni, "A Technique for Interrupt Distribution in a Multiprocessor System", Software & Microsystems, vol. 1, No. 6, Oct. 1982, pp. 153-160.|
|2||*||A. Cantoni, A Technique for Interrupt Distribution in a Multiprocessor System , Software & Microsystems, vol. 1, No. 6, Oct. 1982, pp. 153 160.|
|3||B. Beck et al., "VLSI Assist for a Multiprocessor", Proceedings Second International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS II), Palo Alto, Calif., Oct. 5-8, 1987, pp. 10-20.|
|4||*||B. Beck et al., VLSI Assist for a Multiprocessor , Proceedings Second International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS II), Palo Alto, Calif., Oct. 5 8, 1987, pp. 10 20.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US5283904 *||Jan 22, 1993||Feb 1, 1994||Intel Corporation||Multi-processor programmable interrupt controller system|
|US5379434 *||Jun 10, 1994||Jan 3, 1995||International Business Machines Corporation||Apparatus and method for managing interrupts in a multiprocessor system|
|US5381541 *||May 26, 1993||Jan 10, 1995||International Business Machines Corp.||Computer system having planar board with single interrupt controller and processor card with plural processors and interrupt director|
|US5410710 *||Dec 30, 1993||Apr 25, 1995||Intel Corporation||Multiprocessor programmable interrupt controller system adapted to functional redundancy checking processor systems|
|US5423049 *||Dec 14, 1993||Jun 6, 1995||Nec Corporation||Multiprocessor circuit|
|US5495615 *||Dec 30, 1993||Feb 27, 1996||Intel Corp||Multiprocessor interrupt controller with remote reading of interrupt control registers|
|US5555420 *||Dec 30, 1993||Sep 10, 1996||Intel Corporation||Multiprocessor programmable interrupt controller system with separate interrupt bus and bus retry management|
|US5613128 *||May 6, 1996||Mar 18, 1997||Intel Corporation||Programmable multi-processor interrupt controller system with a processor integrated local interrupt controller|
|US5619705 *||Jun 12, 1996||Apr 8, 1997||Intel Corporation||System and method for cascading multiple programmable interrupt controllers utilizing separate bus for broadcasting interrupt request data packet in a multi-processor system|
|US5634130 *||Oct 16, 1995||May 27, 1997||International Business Machines Corporation||Method and apparatus for spurious interrupt detection in a data processing system|
|US5696976 *||Sep 17, 1996||Dec 9, 1997||Intel Corporation||Protocol for interrupt bus arbitration in a multi-processor system|
|US5701495 *||Dec 18, 1995||Dec 23, 1997||International Business Machines Corporation||Scalable system interrupt structure for a multi-processing system|
|US5701496 *||Sep 17, 1996||Dec 23, 1997||Intel Corporation||Multi-processor computer system with interrupt controllers providing remote reading|
|US5758169 *||Jun 3, 1997||May 26, 1998||Intel Corporation||Protocol for interrupt bus arbitration in a multi-processor system|
|US5802350 *||Jan 18, 1996||Sep 1, 1998||International Business Machines Corporation||System and method for selecting an interrupt system based upon the operating system of a multiprocessor system|
|US5845131 *||May 31, 1996||Dec 1, 1998||Daewoo Telecom Ltd.||Multiprocessor system employing an improved self-coded distributed interrupt arbitration technique|
|US5848277 *||Feb 12, 1996||Dec 8, 1998||Ford Motor Company||Method for providing both level-sensitive and edge-sensitive interrupt signals on a serial interface between a peripheral and host|
|US5905897 *||Mar 20, 1997||May 18, 1999||Industrial Technology Research Institute||Method and apparatus for selecting a nonblocked interrupt request|
|US6003109 *||Aug 15, 1997||Dec 14, 1999||Lsi Logic Corporation||Method and apparatus for processing interrupts in a data processing system|
|US6189065 *||Sep 28, 1998||Feb 13, 2001||International Business Machines Corporation||Method and apparatus for interrupt load balancing for powerPC processors|
|US6606676 *||Nov 8, 1999||Aug 12, 2003||International Business Machines Corporation||Method and apparatus to distribute interrupts to multiple interrupt handlers in a distributed symmetric multiprocessor system|
|US6701429||Nov 22, 1999||Mar 2, 2004||Telefonaktiebolaget Lm Ericsson(Publ)||System and method of start-up in efficient way for multi-processor systems based on returned identification information read from pre-determined memory location|
|US6813665 *||Sep 21, 2001||Nov 2, 2004||Intel Corporation||Interrupt method, system and medium|
|US6859851 *||Dec 20, 1999||Feb 22, 2005||Intel Corporation||Buffer pre-loading for memory service interruptions|
|US7328294 *||Dec 3, 2001||Feb 5, 2008||Sun Microsystems, Inc.||Methods and apparatus for distributing interrupts|
|US7500040 *||Mar 12, 2007||Mar 3, 2009||Dell Products L.P.||Method for synchronizing processors following a memory hot plug event|
|US7529875 *||Aug 20, 2003||May 5, 2009||International Business Machines Corporation||Assigning interrupts for input/output (I/O) devices among nodes of a non-uniform memory access (NUMA) system|
|US7707344 *||Jan 29, 2008||Apr 27, 2010||International Business Machines Corporation||Interrupt mitigation on multiple network adapters|
|US7996595 *||Apr 14, 2009||Aug 9, 2011||Lstar Technologies Llc||Interrupt arbitration for multiprocessors|
|US8024504||Jun 26, 2008||Sep 20, 2011||Microsoft Corporation||Processor interrupt determination|
|US8234431||Oct 13, 2009||Jul 31, 2012||Empire Technology Development Llc||Interrupt masking for multi-core processors|
|US8260996||Apr 24, 2009||Sep 4, 2012||Empire Technology Development Llc||Interrupt optimization for multiprocessors|
|US8321614||Apr 24, 2009||Nov 27, 2012||Empire Technology Development Llc||Dynamic scheduling interrupt controller for multiprocessors|
|US20040111549 *||Dec 10, 2002||Jun 10, 2004||Intel Corporation||Method, system, and program for improved interrupt processing|
|US20050060460 *||Aug 20, 2003||Mar 17, 2005||International Business Machines Corporation||Assigning interrupts for input/output (I/O) devices among nodes of a non-uniform memory access (NUMA) system|
|DE4413459C2 *||Apr 18, 1994||Apr 6, 2000||Intel Corp||Programmierbares Interrupt-Controller-System|
|WO1995018416A1 *||Dec 21, 1994||Jul 6, 1995||Intel Corp||A multiprocessor programmable interrupt controller system with processor-integrated interrupt controllers|
|U.S. Classification||710/260, 710/242|
|International Classification||G06F9/48, G06F9/50, G06F13/26, G06F15/16, G06F15/177|
|Cooperative Classification||G06F13/26, G06F9/4812, G06F9/505|
|European Classification||G06F9/48C2, G06F13/26, G06F9/50A6L|
|Jun 1, 1990||AS||Assignment|
Owner name: AMERICAN TELEPHONE AND TELEGRAPH COMPANY, NEW YORK
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST.;ASSIGNOR:PIEPHO, RICHARD S.;REEL/FRAME:005345/0895
Effective date: 19900601
|May 31, 1996||FPAY||Fee payment|
Year of fee payment: 4
|Oct 28, 1996||AS||Assignment|
Owner name: NCR CORPORATION, OHIO
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AT&T CORPORATION;REEL/FRAME:008194/0528
Effective date: 19960329
|Apr 6, 2000||FPAY||Fee payment|
Year of fee payment: 8
|May 7, 2004||FPAY||Fee payment|
Year of fee payment: 12
|Dec 21, 2007||AS||Assignment|
Owner name: TERADATA US, INC., OHIO
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NCR CORPORATION;REEL/FRAME:020540/0786
Effective date: 20070924