US 20060143341 A1
A system with a first random access memory (RAM), a second RAM, a first processor coupled to the first RAM and a second processor coupled to the second RAM. The first RAM is configured to store input/output (I/O) completions from at least two engines. The second RAM is also configured to store I/O completions from at least two engines. When all engines are active, the system writes I/O completions from the engines to the first and second RAMs. The first processor processes I/O completions stored in the first RAM. The second processor processes I/O completions stored in the second RAM.
1. A storage network interface comprising:
a first protocol engine coupled to transmit and receive data from a first port on a storage network;
a second protocol engine coupled to transmit and receive data from a second port; and
first and second processors having coordination connections therebetween and coupled to the first and second protocol engines for controlling a transfer of data through the first and second ports.
2. The storage network interface as recited in
3. The storage network interface as recited in
4. The storage network interface as recited in
5. The storage network interface as recited in
6. The storage network interface as recited in
7. The storage network interface as recited in
8. A host bus adapter (HBA) comprising the storage network interface of
9. A server comprising the host bus adapter (HBA) of
10. A storage area network (SAN) comprising the server of
11. A method for providing a storage network interface, comprising:
transmitting and receiving data from a first port on a storage network using a first protocol engine;
transmitting and receiving data from a second port using a second protocol engine; and
coordinating operations between first and second processors coupled to the first and second protocol engines for controlling a transfer of data through the first and second ports.
12. The method as recited in
13. The method as recited in
14. The method as recited in
15. The method as recited in
16. The method as recited in
This application is a continuation of U.S. application Ser. No. 10/995,456 filed on Nov. 22, 2004, which in turn is a divisional of U.S. application Ser. No. 10/316,604 filed on Dec. 10, 2002 (now U.S. Pat. No. 6,829,660 B2), which in turn claims priority to U.S. Provisional Application No. 60/340,386 filed on Dec. 12, 2001, entitled “Supercharge Message Exchanger,” all of which are incorporated herein by reference.
A data processing system may receive data from a plurality of sources.
The application relates to a system for receiving and processing data from a plurality of sources. A message exchange system is provided to allow two or more processors to communicate more efficiently, which may improve input/output (I/O) processing speed.
In an embodiment, the system allows I/O completions to be posted to two memories associated with two processors. Either processor can access an I/O completion notification and prepare the I/O for a subsequent operation.
In an embodiment, the system may use zero-wait-state random access memories (RAMs) to store I/O completion notices from the DMA and messages between processors. The zero-wait-state RAMs reduce RAM access latency. Alternatively, the system may use higher latency RAMs with wait-states to achieve more information capacity at reduced performance.
When a plurality of different sources try to write to a RAM at the same time via a plurality of data paths, the system may overcome the possibility of data corruption by storing data in registers. The system also allows fair and quick access to a plurality of logical partitions in a RAM. Registers also overcome the possibility of data corruption when the RAM is full and a source tries to continue writing to the RAM.
The system may be configured to handle I/O completions that originate from different clock domains. For example, two I/O completions may come from a peripheral component interface (PCI) side, which may be running at 33 MHz, 66 MHz or 133 MHz. Two I/O completions may come from a Fibre Channel Link side, which may be synchronized to 106 MHz or 200 MHz clocks. And an ARM processor may post an I/O completion or inter-processor message with a 200 MHz clock.
The system may keep track of an empty or full status of a RAM and how many locations are filled in the RAM.
In addition, the system may also use door bell registers to notify a processor about important events. The door bell registers allow a first processor to set a particular door bell bit and allow a second processor to read and clear the particular door bell bit. The first processor may also be able to generate an interrupt when a particular door bell bit is set by the second processor.
The system may be implemented in an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) device.
An aspect of the application relates to a system comprising a first channel and a second channel. The first channel comprises a first direct memory access (DMA) engine, a second DMA engine, a first random access memory (RAM) and a first processor. The second channel comprises a third DMA engine, a fourth DMA engine, a second RAM and a second processor.
Another aspect relates to a channel comprising a random access memory, first and second engines and a processor. The random access memory is configured to store input/output completions. The first and second engines are configured to send input/output completions to the random access memory. The processor is configured to process the input/output completions stored in the random access memory.
Another aspect relates to a method comprising: configuring a random access memory (RAM) of a first channel to store input/output (I/O) completions from an engine in a second channel; and processing I/O completions in the RAM of the first channel.
Another aspect relates to a method comprising: detecting whether an engine of a first channel is inactive; if the engine of the first channel is inactive, storing input/output (I/O) completions from an engine in a second channel to a random access memory of the first channel; and processing the I/O completions in the random access memory of the first channel.
Another aspect relates to a method comprising: receiving input/output (I/O) completions from a plurality of data paths; storing the I/O completions in a plurality of registers, each register being configured to store an I/O completion from a predetermined data path; and transferring the I/O completions from the registers to a plurality of logical blocks of a random access memory, each logical block being configured to store I/O completions from a predetermined data path.
The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features and advantages will be apparent from the description and drawings, and from the claims.
Like reference symbols in the various drawings indicate like elements.
The PCI Rx/Tx DMA engines 102A, 102B may transfer data between a host memory 140 (via a PCI bus) and an external, intermediate memory 120 (same as external memory 605 in
100. The Link Rx/Tx DMA engines 104A, 104B may transfer data between the intermediate memory 120 and a Fibre Channel Link/interface 150. The Fibre Channel Link/interface 150 may also be referred to as an N-port and an L-port or an NL-Port.
The system 100 includes a Channel A portion (side) 101A and a Channel B portion (side) 101B. Each “channel” 101 includes a PCI Rx/Tx DMA engine 102, a Link Rx/Tx DMA engine 104, completion RAM interface logic 106, a completion RAM 108 and a processor 114.
PCI and Fibre Channel DMA are used here as examples. In other implementations, the system 100 may be used with other types of buses, networks or protocols.
In an embodiment, the processors 114A, 114B are ARM946 core processors developed by ARM Ltd., but other types of processors may be used. The processors 114A, 114B are not limited to ARM946.
A data transfer may be a 2-step process. For a “transmit path,” one of the PCI DMA engines 102A or 102B transfers data from the PCI-side host memory 140 to the intermediate memory 120 via a PCI bus. One of the Link DMA engines 104A or 104B then transfers the data from the intermediate memory 120 to the Fibre Channel Link/interface 150.
For a “receive path,” one of the Link DMA engines 104A, 104B transfers data from the Fibre Channel Link interface 150 to the intermediate memory 120. One of the PCI DMA engines 102A or 102B then transfers the data from the intermediate memory 120 to the PCI side host memory 140.
When data is successfully stored or retrieved from the intermediate memory 120 by one of the engines 102A, 102B, 104A or 104B, the engine 102A, 102B, 104A or 104B outputs a corresponding “I/O completion” notification (notice) to a corresponding completion RAM interface logic 106A or 106B.
An “I/O completion” is when an input (I) or output (O) operation successfully completes or completes in error. Examples of I/Os include: (1) transferring data from the host memory 140 to the intermediate memory 120 (transmit path), (2) transferring data from the intermediate memory 120 to the host memory 140 (receive path), (3) transferring data from intermediate memory 120 to the Link 150 (transmit path), and (4) transferring data from the Link 150 to the intermediate memory 120 (receive path). One of the processors 114A or 114B needs to know when each operation completes successfully. The processor 114 also needs to know if the operation completed but had an error, so the error recovery methods can be invoked.
In a first configuration, hardware may be configured to provide mode control bits to firmware, where the mode control bits set the system 100 in a single channel mode, a dual channel mode or a supercharge mode.
In a second configuration, hardware and/or firmware may switch the system 100 between single channel mode, dual channel mode and/or a supercharge mode. For example, hardware or firmware may dynamically determine whether one set of DMA engines 102 and 104 (e.g., 102A and 104A) is idle and switch from single channel mode to supercharge mode.
Single Channel Mode
In a non-supercharge, “single channel” mode, the “transmit” and “receive” paths are both operating simultaneously for one “channel” 101 to process the I/O completion notifications. Thus, only one processor 114 is controlling the entire processing of I/O completions. In single channel mode, only one channel 101 is operational, and the “other channel” 101 is non-operational or idle. A “non-operational” channel is defined as a channel 101 with no I/O completions to be processed.
Dual Channel Mode
In a “dual channel” mode, both channels 101A, 101B are performing both transmit and receive operations simultaneously and independently. Both channels A 101A and B 101B are also processing I/O completions simultaneously. Essentially, “dual channel” mode is when single channel operation is occurring on both channels.
In non-supercharge “single channel” and “dual channel” modes, I/O completions are posted to only one completion RAM 108, which is connected to a processor's DTCM port.
When only one channel 101A or 101B is operating in single channel mode, the other channel is in a non-operational state. The processing resources of a non-operational channel are unused.
According to mode control bits (or hardware or firmware detecting a single channel mode), the processors 114A, 114B may activate a “supercharge” mode to increase I/O processing speed and boost performance. In supercharge mode, the active channel 101 uses hardware components from the non-operational channel 101. Specifically, each Rx/Tx DMA engine 102A and 104A (or 102B and 104B if channel B is active) writes/posts PCI and link I/O completion notifications to both completion RAMs 108A and 108B. This feature allows either processor 114A or 114B to access an I/O completion notification, prepare the I/O for a subsequent processing operation (i.e., the next DMA stage), if any, and notify the other processor 114 to prepare the I/O for the subsequent processing operation.
This feature prevents I/O processing bottlenecks. A processor 114 that originated the I/O (i.e., initially set up the DMA operation) does not need to be involved during the post processing of the I/O completion notification when the DMA operation is complete. Instead, the DMA completion is directly passed into the completion RAM 108 for the other processor 114 to begin work without involving firmware.
In “supercharge” mode, both processors 114A, 114B may control the single active channel 101, i.e., both processors 114A, 114B can operate on a single active channel's I/O completions. There is no idle channel in supercharge mode. However, there may be some hardware that is not being used, depending on the configuration selected, as described below.
The supercharge mode is flexible in how the processors 114A, 114B are used to divide the processing work. Writing I/O completion notifications from each Rx/Tx DMA engine 102, 104 to both completion RAMs 108A and 108B allows the firmware in the system 100 to experiment with different configurations to determine an optimal solution.
In a supercharge mode configuration, hardware or firmware may dedicate one processor 114A to process I/O completions from PCI Tx and Rx DMA operations, while the other processor 114B may be dedicated to process I/O completions from Fibre Channel Link Tx and Rx DMA operations (see
In another supercharge mode configuration, hardware or firmware may be configured to allow one processor 114A to control the entire “transmit path” from the host memory 140 to the Fibre Channel Link/interface 150 (i.e., PCI Tx DMA and Link Tx DMA), and the other processor 114B to control the entire “receive path” from the Fibre Channel Link/interface 150 to the host memory 140 (i.e., Link Rx DMA and PCI Rx DMA).
In another supercharge mode configuration, hardware or firmware dedicates one processor 114 to handle the PCI TX and NL-Port RX, and the other processor 114 to handle the PCI RX and NL-Port TX.
The hardware or firmware may also permit varying degrees of these configurations.
In the supercharge configuration where one processor 114A controls PCI Tx and Rx DMA operations, when the processor 114A is finished with a PCI Tx DMA operation, the processor 114A may send a message to the other processor 114B (which controls Link Tx and Rx DMA operations) via a line 130B to notify the processor 114B that the data is ready for transmission to the Fibre Channel link/interface 150.
The system 100 may use low latency, zero-wait-state RAMs as the RAMs 108A, 108B to store I/O completion notifications so either processor 114A or 114B can quickly prepare the I/O for the next DMA stage. Zero-wait-state RAMs 108A, 108B reduce RAM access time by the processors 114A, 114B. Low latency, zero-wait-state RAMs and other specialized hardware may provide a high performance solution for improving the number of I/O completions per second (IOPS).
In an alternative system, software may manually move the I/O completion messages from one processor to another processor using higher latency memory (memory with wait states). This type of alternative system may result in lower performance.
Communication Between the Processors
Communication between the two processors 114A, 114B may occur in at least two ways. First, lines 130A, 130B may provide a fast and efficient communication exchange mechanism between the two processors 114A and 114B. The lines 130A, 130B improve data processing speed. The lines 130A, 130B may allow each processor 114 to write a single word or a burst of eight words to the other processor's zero-wait state completion RAM 108.
For example, in a supercharge mode, DMA I/O completions from the PCI TX/RX engine 102A and the Link TX/RX engine 104A may be posted to both processors' completion RAMs 108A, 108B. When one processor 114A processes an I/O completion, the processor 114A may send a message to the other processor 114B via a line 130B. Communication messages between the processors 114A, 114B may convey information such as “Processor A has completed DMAing data for a particular I/O, and the I/O is ready for the next processing step.” When the system 100 is running in “supercharge mode,” an efficient message exchange between the two ARM processors 114A, 114B via lines 130A, 130B may dramatically improve I/O processing speed.
Second, each processor 114 can notify the other processor by using doorbell registers and control logic 602A, 602B, as described below with
Control Logic and RAM
The DTCM RAM 214 in
The completion control logic 200 in
Although register symbols 250-257 are shown as single-bit registers, the register symbols 250-257 may represent registers configured to store any desired number of bits. The fifth register 257 may be coupled to the decoder 226 as shown or within the decoder 226.
In an embodiment, the DTCM RAM 214 has 2 read ports and 2 write ports. The DTCM RAM 214 may be a synchronous RAM. In a configuration, the RAM 214 is a 1024×32 RAM, but RAMs with various other word lengths and any number of words may be used.
In an embodiment, the DTCM RAM 214 is divided into a plurality of logical blocks (also called “queues” or “partitions”), such as an FRXQueue completions block 216, an FTXQueue completions block 218, a DXBRx completions block 220, a DXBTx completions block 222. These blocks 216, 218, 220 and 222 may correspond to the RAM 108 in
Each block is operable to store I/O completions from its respective source. For example, the FRXQueue completion block 216 may store Rx I/O completions from Fibre Channel Link Receive DMA operations. The Rx I/O completion relates to data transferred from the Fibre Channel Link/interface 150 to the intermediate memory 120. As another example, the DXBRx completion block 220 may store I/O completions from PCI Receive DMA operations.
The control logic 200 may be configured to allow each I/O completion to be written to only one RAM block 216, 218, 220, 222 or 110 if the block has sufficient space. The control logic 200 may generate a processor interrupt to a processor 114 whenever an I/O completion is written to the DTCM RAM 214.
If a RAM block is full and new I/O completions arrive, the new I/O completions should not over-write the data in the DTCM RAM 214. Instead, the control logic 200 allows the data source to write the new arriving I/O completions first to a register, such as registers 250-257. Then the control logic 200 writes the data into the DTCM RAM 214.
If a register is already filled with a new I/O completion, the register may not be ready to store new data. An error flag may be asserted by the control logic 200 or firmware, and a source may be instructed to resend the I/O completion again later.
After a system reset, the completion RAM control logic 200 may generate five READY signals to data sources coupled to the five data paths 240-248. Upon detecting a set READY signal, each data source may generate a write enable signal if the data source has data (I/O completions) to write to the DTCM RAM 214. Five data paths 240-248 may start to send their first data to five different “datain_reg” registers 250-257 once the write enable signal is asserted. When the registers 250-257 are filled, the READY signal may be de-asserted to prevent the data sources from writing more data to the “datain_reg” registers 250-257.
Write enable signals and data signals (I/O completions) from different sources are sometimes transmitted at different speeds (clock domains) compared to the operating speed of the completion RAM control logic 200. For example, data signals (I/O completions) from the DXB receive path 244 (PCI Receive) may be received at 33 MHz, 66 MHz or 133 MHz. Data from the Link FRxQ path 240 (Fibre Channel Link Receive) may be synchronized to a 200 MHz clock. Data from the Link FTXQ 242 (Fibre Channel Link transmit) may be synchronized to a 106 MHz clock. An ARM processor may post I/O completions or an inter-processor message with a 200 MHz clock. If these signals are not synchronized, the system 100 may miss important completion information. The system 100 may be configured to synchronize or re-clock these different signals to a common clock domain before the signals are processed.
When the registers 250-257 are filled, the control logic 200 determines whether to move the data in the “datain_reg” registers 250-257 to specific RAM blocks 216-222, 110. If a specific RAM block 216-222, 110 is not full, the control logic 200 moves the data from a “datain_reg” register to a specific RAM block 216-222, 110. The control logic 200 may also assert a READY signal to a source to indicate that the associated “datain_reg” register is ready to store the next I/O completion message.
If a specific RAM block 216-222, 110 is full, the control logic 200 will not move the data in the “datain_reg” register to the specific RAM block 216-222, 110. The READY signal to the source will remain active low.
If a source tries to continue writing data to the source's associated “datain_reg” register, the previously-loaded data in the register may be overwritten. The control logic 200 may set a flag or send an interrupt signal to the source. The source will know when and which data has been corrupted. The source may re-send the data later. In this case, no data in the DTCM RAM 214 is overwritten.
Five data paths 240-248 may try to access five different blocks 216-222, 110 of the DTCM RAM 214. Five different registers 250-257 are configured to store data from five different sources. When a PCI DMA 102, a LINK DMA 104 and an ARM processor 114 try to write I/O completions via a plurality of data paths 240-248 to the DTCM RAM 214 at the same time, the control logic 200 may overcome the possibility of data corruption.
When all five registers 250-257 are loaded, the arbitrator 212 decides which register data is written to the DTCM RAM 214 first. The arbitrator 212 may use a quick and fair arbitration method, such as a round robin method, to move data to different RAM blocks 216, 218, 220, 222, 110. The arbitration method ensures that data from different sources have fair and quick access to the RAM blocks 216-222, 110.
The control logic 200 may also overcome the possibility of data corruption by ensuring that the RAM write cycle for a particular data source is complete before a second write is attempted by the same data source. An interrupt may be generated by the control logic 100 or firmware to notify the data source and the firmware if an I/O completion notification was overwritten. The data source may resend the completion notification.
In an alternative method, when the RAM blocks 216-222, 110 and/or registers 250-257 are filled, the READY signal (“rdy” in
To determine whether the first four RAM blocks 216, 218, 220 and 222 in
The put_ctl controller 228 passes the put pointer value to the CR Reg registers 232, where a comparison is made between the put pointer 304 and a get pointer 302 that corresponds to the particular RAM block 300. When the put pointer 304 is greater than the get pointer 302, an interrupt is generated to firmware 306 indicating that there is at least one completion message in the completion RAM block 300.
When firmware 306 retrieves the I/O completion message, the firmware 306 increments the corresponding get pointer 302. When firmware 306 retrieves all of the outstanding messages and increments the appropriate get pointer 302, the get pointer 302 will equal the put pointer 304, and the interrupt condition will be removed.
The get pointer 302 and the put pointer 304 of the RAM block 300 may be easily tracked by the firmware 306. In
The firmware 306 manages four get pointers, such as the get pointer 302 in
As soon as data (I/O completion) is written to the RAM block 300, an interrupt may be generated by the RAM control logic 308. The firmware 306 may start to read the data out from the RAM block 300 through the DTCM port 112 to the processor 114.
The control logic 308 or firmware 306 may use the values of the put and get pointers 304, 302 to keep track of the status of each RAM block 216-222 and predict how many Completion RAM entries are filled. The firmware 306 or control logic 308 may also use the put and get pointers 302, 304 to predict when the RAM block 300 is full or empty. For example, if the put pointer 304 is equal to the get pointer 302 minus one (Put=Get−1), the block or queue 300 is full. This condition may mean there is one available location in the RAM 214 that cannot be used, and the data in the last entry is stored in a “datain_reg” register 250, 252, 254, 256, 257 (
If the put and get pointers 304, 302 are equal, i.e., put=get, for the block 300, the block or queue 300 is empty. After a chip reset, the put and get pointers 304, 302 may be equal to a default start value.
Storing Messages From Another Processor
The fifth RAM partition/block 110 in
The decoder 226 will provide the initial write address to write an I/O completion message in the RAM block 110. Once an I/O completion message is written to the block 110, the decoder 226 will increment the write address for the next write operation. In the next message write cycle, the processor interface 248 does not need to provide the address. The hardware 200 therefore includes information indicating where the new message should be stored.
If the Channel B processor 114B signals a write of a message to a completely new address, the processor 114B should provide both the message and an address. The Channel B “DTCM” port 112B in
Since a DTCM can be physically placed very close to the processor, the system 100 can handle zero wait state access.
In a block 404, the first processor 114A finishes an I/O completion and sends a message to the second processor 114B to notify the second processor 114B to prepare the I/O for the next processing stage.
In a block 502, each data source detects the set READY signal and generates a write enable signal if the data source has data (I/O completions) to write to the DTCM RAM 214.
In a block 504, once the write enable signal is asserted, a plurality of data paths start to send data to a plurality of data-in registers (e.g., 250-256).
In a block 506, when the registers are filled, the READY signal is de-asserted to prevent the data sources from writing more data to the data-in registers.
In a block 508, when the registers are filled, the control logic 200 determines whether to move the data from the data-in registers to the DTCM RAM 214, which depends on whether each RAM block 216-222 is full.
In a block 510, if a RAM block is not full, the control logic 200 moves the data to the RAM block. The control logic 200 may also assert a READY signal to a source to indicate that the associated data-in register is ready to store the next data segment.
In a block 511, when all or some of the five registers 250-256 are loaded, the arbitrator 212 decides which register data is written to the DTCM RAM 214 first. The arbitrator 212 may use a round robin method for fair and quick access to the RAM 214.
In a block 512, if a RAM block 216-222 is full, the control logic 200 does not move the data in the data-in register to the RAM block 216-222. The READY signal to the source remains active low.
In a block 514, a source may try to continue writing data to the source's associated data-in register. The control logic 200 sets a flag or sends an interrupt signal to source if the register already stores data.
Door Bell Registers and Control Logic
For example, processor A 114A can “ring” processor B 114B by writing a bit to the Channel A Door Bell Other Channel register 708A. The Channel A Door Bell Other Channel register 708A is mapped into Processor B address decode space as the Channel B Door Bell Register 704B, as shown by the arrow between the two registers 708A, 704B. Thus, setting a bit in the Channel A Door Bell Other Channel register 708A causes a bit to be set in the Channel B Door Bell Register 704B.
Processor B 114B can allow an interrupt to be generated for a particular door bell bit in the door bell register 704B by setting an appropriate enable bit in the Channel B Door Bell Enable Register 706B. Processor B 114B reads and clears the bit set in the Door Bell register 704B.
Processor A 114A has a read-only capability for Processor B's Door Bell Enable Register 706B, as shown by an arrow from the register 706B to the Channel A Door Bell Enable Other Channel register 710A.
There is also a corresponding set of registers 704A, 706A, 708B, 710B that allow Processor B 114B to ring Processor A 114A. Thus, processor A 114A may also be able to generate an interrupt when a particular door bell bit is set by processor B 114B.
A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the application. Accordingly, other embodiments are within the scope of the following claims.