|Publication number||US7451248 B2|
|Application number||US 11/054,183|
|Publication date||Nov 11, 2008|
|Filing date||Feb 9, 2005|
|Priority date||Feb 9, 2005|
|Also published as||US7788423, US20060190636, US20080294807|
|Publication number||054183, 11054183, US 7451248 B2, US 7451248B2, US-B2-7451248, US7451248 B2, US7451248B2|
|Inventors||George W. Daly, Jr., James S. Fields, Jr.|
|Original Assignee||International Business Machines Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (5), Classifications (15), Legal Events (7)|
|External Links: USPTO, USPTO Assignment, Espacenet|
1. Technical Field
The present invention relates to direct memory access (DMA) in general, and, in particular, to a method and apparatus for performing DMA write operations. Still more particularly, the present invention relates to a method and apparatus for invalidating cache lines during a DMA Write operation by a Peripheral Component Interconnect device.
2. Description of Related Art
Peripheral Component Interconnect (PCI) bus technology uses memory mapping techniques for performing input/output (I/O) operations and DMA operations. In a data processing system that is capable of handling PCI devices, a range of addresses called PCI address space is allocated within a system memory for all the PCI devices associated with the data processing system. Within the PCI address space, there is a region reserved by the operating system for programmable I/O operations that are performed by a processor to read or change the contents of PCI device registers within the PCI devices. In addition, a separate region is allocated within the PCI address space by the operating system for DMA accesses to the system memory by the PCI devices. The allocated addresses are dynamically mapped to a section of the system memory. Each of the PCI device can use the mapped addresses to perform DMA Read or Write operations by directly reading and writing in the PCI address space with the mapped addresses.
DMA Write operations from each of the PCI devices must be performed in a specific order as observed by any potential data consumer within the data processing system. Because the location of a DMA Write operation may have data that are shared by a cache memory of a processor, in order to maintain correct ordering, a DMA Write operation must invalidate any copies of the data from all cache memories within the data processing system before completing the DMA Write operation. In fact, the completion must occur before any subsequent DMA Write operations from the same PCI device can become visible to any data consumer; otherwise, ordering rules will be violated.
In order to sustain full DMA Write throughput, the data processing system must balance between the amount of time to resolve cache coherence and the amount of data transferred per DMA Write request. Typically, as a data processing system becomes larger, the time required to resolve cache coherence also increases, which effectively limits the bandwidth that a PCI device is able to achieve in the data processing system. One method of improving the bandwidth is to design the data processing system with a longer cache line such that more data can be invalidated per cache line invalidation request. However, there are also drawbacks associated with a relatively long cache line length. For example, an entire cache line's worth of data needs to be transferred even when only a small portion of the cache line contains the required data, which effectively leads to a reduction in bus bandwidth. A longer cache line length also increases the likelihood of false sharing of data within the same cache line by multiple processors.
The present disclosure provides an improved method and apparatus for invalidating cache lines during a DMA Write operation by a PCI device.
In accordance with a preferred embodiment of the present invention, a multi-cache line DMA request initially issued by a peripheral device. The multi-cache line DMA request is snooped by a cache memory. A determination is made as to whether or not the cache memory includes a copy of data stored in the system memory locations to which the multi-cache line DMA request are directed. In response to a determination that the cache memory includes a copy of data stored in the system memory locations to which the multi-cache line DMA request are directed, multiple cache lines within the cache memory are consecutively invalidated.
All features and advantages of the present invention will become apparent in the following detailed written description.
The invention itself, as well as a preferred mode of use, further objects, and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein:
Referring now to the drawings and in particular to
Multi-processor system 10 also includes Peripheral Component Interconnect (PCI) adapters 18 a-18 c coupled to interconnect 14 via a PCI host bridge 17 and an input/output (I/O) controller 19. I/O controller 19 is provided with a direct memory access (DMA) controller capability. PCI host bridge 17 provides an address translation function between a PCI address space and a system memory address space, which is commonly referred to as a translation control entry (TCE) translation. PCI host bridge 17 and PCI adapters 18 a-18 c can support a variety of I/O devices that are conformed to the PCI standard. Details of the PCI standard can be found in PCI Local Bus Specification, Revision 2.2, dated Dec. 18, 1998.
Multiprocessor system 10 supports DMA transfers from a PCI device to a system memory. A PCI address space is allocated within system memory 16 for all PCI devices connected to PCI adapters 18 a-18 c. A first region within the PCI address space is reserved by the operating system for programmable I/O operations that are performed by processors 11 a-11 n to read or change the contents of PCI device registers within PCI devices connected to PCI adapters 18 a-18 c. In addition, a second region within the PCI address space is allocated by the operating system for DMA accesses to system memory 16 by PCI devices connected to PCI adapters 18 a-18 c. Each of PCI devices connected to PCI adapters 18 a-18 c can use the mapped addresses to perform DMA Read or DMA Write operations by directly reading and writing in the PCI address space with the mapped addresses.
As such, a PCI device connected one of PCI adapters 18 a-18 c is able to transfer data via a DMA Write operation to system memory 16 through I/O controller 19. If the DMA Write operation is going to be performed on a location within system memory 16 having data that are shared by one or more of cache memories 12 a-12 n, the DMA Write operation must invalidate any copies of the data from those cache memories before completing the DMA Write operation in order to maintain data coherence.
With reference now to
Next, I/O controller 19 sends the multi-cache DMA Write request to interconnect 14. The multi-cache line DMA Write request is then snooped by cache memories 12 a-12 n, as depicted in
If, for example, L2 cache memory 12 b has a copy of the data stored in the consecutive locations of system memory 16 to which the multi-cache line DMA Write request is directed, L2 cache memory 12 b will invalidate the two cache lines that contains the copy of the data stored in the two consecutive locations of system memory 16.
If none of L2 cache memories 12 a-12 n has a copy of the data stored in the consecutive locations of system memory 16 to which the multi-cache line DMA Write request is directed, each of the cache slices of L2 cache memories 12 a-12 n generates a separate coherence response. For each of the cache slices of L2 cache memories 12 a-12 n, the two coherence responses are then combined by a respective one of system bus interfaces 13 a-13 n to form a combined clean response, and all the combined clean responses are subsequently sent to interconnect 14, as shown in
After snooping all the combined clean responses from L2 cache memories 12 a-12 n on interconnect 14, I/O controller 19 directs memory controller 15 to perform the DMA Write operation in system memory 16, as depicted in
During the DMA Write operation, memory controller 15 allows two cache lines's worth of data to be written to system memory 16 by using a Page mode access because Page mode access saves power as compared to sequential non-Page mode accesses.
As has been described, the present invention provide an improved method and apparatus for invalidating cache lines during a DMA Write operation by a PCI device. Although a multi-line data prefetch request for two cache lines is used to illustrate the present invention, it is understood by those skilled in the art that the multi-line data prefetch request of the present invention can be extended to more than two cache lines, such as four or more cache lines.
It is also important to note that although the present invention has been described in the context of a fully functional computer system, those skilled in the art will appreciate that the mechanisms of the present invention are capable of being distributed as a program product in a variety of forms, and that the present invention applies equally regardless of the particular type of signal bearing media utilized to actually carry out the distribution. Examples of signal bearing media include, without limitation, recordable type media such as floppy disks or CD ROMs and transmission type media such as analog or digital communications links.
While the invention has been particularly shown and described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US5802576 *||Jul 1, 1996||Sep 1, 1998||Sun Microsystems, Inc.||Speculative cache snoop during DMA line update|
|US6275909 *||Aug 4, 1999||Aug 14, 2001||International Business Machines Corporation||Multiprocessor system bus with system controller explicitly updating snooper cache state information|
|US6535958 *||Jun 26, 2000||Mar 18, 2003||Texas Instruments Incorporated||Multilevel cache system coherence with memory selectively configured as cache or direct access memory and direct memory access|
|US6748463 *||Mar 13, 1996||Jun 8, 2004||Hitachi, Ltd.||Information processor with snoop suppressing function, memory controller, and direct memory access processing method|
|US20040158681 *||Feb 12, 2003||Aug 12, 2004||Ip-First Llc||Write back and invalidate mechanism for multiple cache lines|
|U.S. Classification||710/22, 710/7, 711/141, 711/117, 711/E12.035, 711/129, 711/127, 711/3, 710/26, 710/20|
|Cooperative Classification||G06F13/28, G06F12/0835|
|European Classification||G06F13/28, G06F12/08B4P4P|
|Mar 10, 2005||AS||Assignment|
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:DALY, GEORGE W. JR.;FIELDS, JAMES S. JR.;REEL/FRAME:015869/0903
Effective date: 20050207
|Jun 25, 2012||REMI||Maintenance fee reminder mailed|
|Sep 28, 2012||SULP||Surcharge for late payment|
|Sep 28, 2012||FPAY||Fee payment|
Year of fee payment: 4
|Jun 24, 2016||REMI||Maintenance fee reminder mailed|
|Nov 11, 2016||LAPS||Lapse for failure to pay maintenance fees|
|Jan 3, 2017||FP||Expired due to failure to pay maintenance fee|
Effective date: 20161111