WO1998000776A1 - Cache memory controller in a raid interface - Google Patents
Cache memory controller in a raid interface Download PDFInfo
- Publication number
- WO1998000776A1 WO1998000776A1 PCT/GB1997/001474 GB9701474W WO9800776A1 WO 1998000776 A1 WO1998000776 A1 WO 1998000776A1 GB 9701474 W GB9701474 W GB 9701474W WO 9800776 A1 WO9800776 A1 WO 9800776A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- bus
- controller
- memory
- cache memory
- raid
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0866—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches for peripheral storage systems, e.g. disk cache
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0877—Cache access modes
- G06F12/0879—Burst mode
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/38—Information transfer, e.g. on bus
- G06F13/40—Bus structure
- G06F13/4004—Coupling between buses
- G06F13/4027—Coupling between buses using bus bridges
- G06F13/405—Coupling between buses using bus bridges where the bridge performs a synchronising function
- G06F13/4059—Coupling between buses using bus bridges where the bridge performs a synchronising function where the synchronisation uses buffers, e.g. for speed matching between buses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0658—Controller construction arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/31—Providing disk cache in a specific location of a storage system
- G06F2212/312—In storage controller
Definitions
- the present invention relates to a cache memory controller and in particular, but not exclusively, to a RAID storage subsystem having such a controller.
- Modern mass storage subsystems are continuing to provide increasing storage capacities to fulfill user demands from host computer system applications. Due to this critical reliance on large capacity mass storage, demands for enhanced reliability are also high. Various storage device configurations and geometries are commonly applied to meet the demands for higher storage capacity while maintaining or enhancing reliability of the mass storage subsystems.
- RAID level 1 There are five “levels” of standard geometries defined in the Patterson publication.
- the simplest array, a RAID level 1 system comprises one or more disks for storing data and an equal number of additional mirror disks for storing copies of the information written to the data disks.
- the remaining RAID levels identified as RAID level 2,3,4 and 5 systems, segment the data into portions for storage across several data disks.
- One of more additional disks are utilized to store error check or parity information.
- a single unit of storage is spread across the several disk drives and is commonly referred to as a "stripe."
- the stripe consists of the related data written in each of the disk drive containing data plus the parity (error recovery) information written to the parity disk drive.
- RAID storage subsystems typically utilize a control module that shields the user or host system from the details of managing the redundant array.
- the controller makes the subsystem appear to the host computer as a single, highly reliable, high capacity disk drive.
- the RAID controller may distribute the host computer system supplied data across a plurality of the small independent drives with redundancy and error checking information so as to improve subsystem performance and reliability.
- RAID subsystems provide large cache memory structures to further improve the performance of the RAID subsystem.
- the cache memory is associated with the control module such that the storage blocks on the disk array are mapped to blocks in the cache. This mapping is also transparent to the host system. The host system simply requests blocks of data to be read or written and the RAID controller manipulates the disk array and cache memory as required.
- I/O requests generated by a host computer requires significant data bandwidth transferring data to and from the cache memory.
- data read from the disk array of the RAID storage subsystem into the cache memory may be transferred to a requesting host computer system to satisfy a read I/O request.
- Other data may be received from a host computer system and transferred to the cache memory to satisfy a write I/O request.
- background operations performed by the RAID controller of the RAID storage subsystem may post data in the cache memory to the drive array or read data from the disk array to the cache memory. All these exemplary data transfer operations between a host computer and the cache memory and between the disk array and the cache memory require a portion of the total available bandwidth on the bus to the cache memory.
- the present invention seeks to provide for a memory controller having advantages over known memory controllers.
- a cache memory controller comprising main memory controller means, coupled to a central cache memory through a high speed memory bus, for controlling access to said central cache memory, bus bridge means, coupled to said main memory controller means through an intermediate shared memory bus and coupled to a controller specific bus, for accessing said central cache memory by converting signals exchanged on said controller specific bus into signals appropriate for exchange on said intermediate shared memory bus and by converting signals exchanged on said intermediate shared memory bus into signals appropriate for exchange on said high speed memory bus.
- a cache memory control architecture within a RAID storage subsystem which simplifies the migration and porting of existing ("legacy") control methods and structures to newer high performance cache memory designs.
- a centralized high speed cache memory is controlled by a main memory controller circuit.
- One or more bus bridge circuits adapt the signals from the bus architecture used by the legacy systems to the high speed cache memory.
- the bus bridge circuits each adapt, for example, a PCI bus used for a particular cache access purpose to the signal standards of an intermediate shared memory bus.
- the main memory controller circuit adapts the signals applied to the intermediate shared memory bus to the high speed cache memory bus.
- the hierarchical bus architecture permits older "legacy" control methods and structures to be easily adapted to newer cache memory architectures.
- the centralized high speed cache memory and associated legacy system busses serve to distribute the load of cache memory access over simultaneously operable busses.
- the cache memory architecture of the present invention therefore permits rapid porting and re-usability of older "legacy" control methods and structures while permitting the overall cache memory performance to be scaled up to higher bandwidth demands of modern RAID subsystems.
- the present invention solves the above and other problems, and thereby advances the useful arts, by disclosing a RAID controller architecture which provides a high performance cache memory design while simplifying the task of porting (supporting) older "legacy" RAID control software methods.
- the present invention provides for a hierarchy of lower level interface busses to attach "legacy" RAID control software systems to a centralized high performance cache memory through an intermediate shared memory bus structure.
- the hierarchical levels of busses are used to distribute the load of the various data exchange and manipulation tasks involving the cache memory.
- the intermediate shared memory bus is designed to meet the higher performance requirements of modern cached RAID controllers and is shared by the lower level hierarchical busses in order to provide access to the high speed cache memory subsystem.
- the present invention includes a main memory controller circuit which manages a wide (preferably 64-bit wide), high speed, cache memory and is adaptable to a wide variety of memory speeds, memory sizes, and memory error detection/correction methods.
- This main memory controller circuit includes a high performance RAID parity generation and detection portion to enable high speed parity computation functions in parallel with other access to the data stored in the high speed cache memory.
- Bus bridge circuits of the present invention are used to adapt older "legacy" software control methods and structures to the new high speed cache architecture.
- the bus bridge adapts signals applied to a bus used by the legacy subsystems to the signal standards of an intermediate shared memory bus.
- a specific bus bridge circuit is used for each specific type of legacy system cache memory architecture.
- the intermediate shared memory bus connects one or more bus bridge circuits to the main memory controller circuit.
- the legacy system control methods and structures may therefore be easily ported and re-used in the higher performance cache environment of the present invention.
- the main memory controller circuit is configurable to manage various sizes, geometries, and speeds of memory devices which comprise the high performance cache memory.
- the high speed cache memory managed by the main memory controller chip is 72-bits wide (64 data bits plus 8 bits for error correcting codes).
- the main memory controller circuit is also configurable to utilize a variety of memory error detection and correction techniques to permit scaling to newer memory error correction techniques.
- the main memory controller circuit permits access to the high performance cache memory through an intermediate shared memory bus for shared connectivity to cache memory bus designs of legacy systems (such as 32 and 64 bit PCI busses).
- the intermediate shared memory bus provides for high performance shared access to the main memory controller circuit by the bus bridge circuits.
- Legacy systems are connected to the intermediate shared memory bus by bus bridge circuits of the present invention which adapt the legacy system's PCI bus (for example) to the intermediate shared memory bus of the main memory controller circuit.
- the intermediate shared memory bus may utilize a plurality of gigabit-per- second parallel to serial transceiver devices to provide a high speed bus connection with a minimum pin count. For example, eight such transceivers may be utilized to provide a
- the main memory controller circuit and the bus bridge circuits also connect to a
- the central processor of the RAID controller connects to the 32-bit processor bus to control the operations of the main memory controller circuit, the bus bridge circuits, and all circuits connected to the legacy bus through the bus bridge circuits.
- the processor can therefore manage the operation of the bus bridge circuits and the main memory controller circuit without interfering with the operation of the intermediate shared memory bus and associated access to the high performance cache memory.
- This feature of the present invention enables the main processor to control operations on one bus (the legacy bus and the processor bus) while DMA transfers to the high performance cache memory take place simultaneously on another independent bus (the intermediate shared memory bus).
- the circuits of the present invention thereby implement a hierarchical cache memory management structure in which RAID control subsystems, representing a potentially heterogeneous mix of current and legacy control architectures, may share access to a common high performance cache memory subsystem.
- Each RAID control subsystem connects to the a bus bridge circuit of the present invention which adapts that subsystem's particular cache memory bus architecture (e.g., 32 or 64 bit PCI) to a high performance intermediate shared memory bus architecture of the present invention.
- the intermediate shared memory bus connects each of the bus bridge circuits to a main memory controller circuit which, in turn, adapts the intermediate shared memory bus to the high performance cache memory.
- the structure and operation of the high performance cache memory of the present invention is therefore independent of any particular RAID controller architecture.
- the bus bridge circuits of the present invention provide any translation and adaptation of signals required to mate the cache memory bus of a legacy RAID controller to the high performance cache memory and the main memory controller circuit. This independence of the high performance cache memory from any particular legacy RAID control subsystem architecture permits the high performance cache memory to be updated and improved without impacting the support for older legacy RAID control subsystems.
- a 32- bit processor bus connects a central processing unit to each of the bus bridge circuits and to the main memory controller circuit to control their operation.
- This RAID controller architecture is easily scaled up to higher performance cache memory designs by adapting the main memory controller circuit to the required performance enhancing designs. Older legacy subsystems are maintained through such enhancements by adapting the legacy subsystem's cache memory bus to the main memory controller circuit via a bus bridge circuit.
- the present invention can therefore provide a RAID cache memory architecture with improved data bandwidth capacity.
- the present invention can advantageously provide a RAID cache memory architecture having a centralized high performance cache memory and intermediate high performance bus for the sharing of the centralized cache memory.
- the present invention can provide a RAID cache memory architecture having a main memory controller for the control of a centralized high performance cache memory system, a plurality of bus bridges for adapting other bus architectures to the main memory controller, and an intermediate bus connecting the plurality of bus bridges and the main memory controller.
- the present invention can provide for RAID cache memory architecture which reduces the complexity of re-using legacy control structures from previous RAID controllers.
- the present invention also allows for a RAID cache memory architecture including a centralized high performance cache memory and at least one bus bridge for adapting a slower and/or smaller bus to the centralized high performance cache memory thereby enabling re-use of legacy control structures from previous RAID controllers.
- the present invention can provide for a RAID cache memory architecture including a centralized high performance cache memory, a plurality of bus bridges for adapting slower and/or smaller busses to the centralized cache memory, and an intermediate bus connecting the plurality of bus bridges and the main memory controller thereby enabling re-use of legacy control structures from previous RAID controllers.
- the invention can also provide for a hierarchial memory interface which provides high bandwidth for RAID data transfer while permitting easy scalability to support slower "legacy" RAID controller software.
- Fig. 1 is a block diagram of a typical RAID storage subsystem as known in the art in which each of a plurality of RAID controllers has a unique cache memory architecture
- Fig. 2 is a block diagram of a RAID storage subsystem and embodying centralized high performance cache memory control structures of the present invention
- Fig. 3 is a block diagram providing additional details of the structure of the main memory controller circuit of Fig. 2;
- Fig. 4 is a block diagram providing additional details of the structure of the bus bridge circuit of Fig. 2;
- Fig. 5 is a block diagram of one embodiment of the intermediate shared memory bus of the present invention as a 64-bit-wide parallel bus;
- Fig. 6 is a block diagram of another embodiment of the intermediate shared memory bus of the present invention as a 128-bit-wide parallel to byte serial bus.
- Fig. 1 is a block diagram depicting a typical RAID storage subsystem 100 as is known in the art.
- RAID storage subsystem 100 is connected by its RAID controller 101 to one or more host systems 108 via bus 120.
- bus 120 may commonly be a SCSI bus connection, a Fibre Channel connection, a network connection such as Ethernet or Token Ring, or any of several standard interconnections between host computer systems and storage peripheral devices.
- a RAID controller 101 processes host computer system generated I/O requests to store and retrieve information from the RAID storage subsystem.
- RAID controller 101 manages the disk drives of the RAID subsystem to manage the redundancy features of the RAID geometries.
- RAID controller 101 is connected to RAID array 104 via bus 150.
- RAID array 104 is comprised of a plurality of storage medium devices such as disk drives 106.
- bus 150 may be any of several standard busses utilized in the storage industry to communicate with storage medium devices such as disk drives 106. For example, SCSI, EIDE, IPI, Fibre Channel (FCAL), and other such busses and protocols are commonly used in the storage medium industry to communicate with disk drives.
- RAID controller 101 utilizes subsets of the disk drives 106 of the RAID array 104 to configure and manage one or more RAID devices on behalf of attached host computer systems.
- RAID controller 101 includes CPU 122 in which the RAID control methods are operable.
- Standard integrated circuit chip sets well known in the art, are typically utilized to connect CPU 122 to a second level cache (L2 cache 124) and to its main memory (RAM).
- Cache DRAM controller (CDC 126) and data path unit (DPU 130) exemplify such standard chip sets known (together with system I/O (SIO 140)) as the "Saturn II" chip set manufactured as part number 82420 by Intel Corporation.
- CDC 126 and DPU 130 also serve to connect CPU 122 to a standard I/O bus (such as PCI bus 102) for connection with the peripheral I/O devices used by CPU 122.
- SIO 140 connects standard lower speed devices used in initialization of the RAID controller 101 or used in the maintenance or development of the RAID controller 101.
- NVRAM 142, serial port 144, and debug/maintenance port 146 may be used ,for example, to permanently store the operational programmed instructions for CPU 122 (typically copied to RAM 128 for faster fetch and execution), and to communicate maintenance and debug information to service or design engineers.
- Host interface 132 Host interface 132, RAID parity assist (RPA) 134, and device interfaces 138 all attach to PCI bus 102 for the exchange of data there between via the common PCI bus 102.
- Host interface 132 provides a connection between RAID controller 101 and the host connection bus 120 to exchange data between PCI bus 102 and an attached host computer 108.
- connection bus 120 may be a SCSI bus, Fibre Channel, network, or other common connection standards between host computers and peripheral storage devices.
- RPA 134 provides connectivity between PCI bus 102 and cache buffer 136 (via bus 152) and provides parity computation assistance for the storage and retrieval of data in cache buffer 136. Host write requests are typically completed by writing the requested data into the cache buffer 136 for later posting to the RAID array 104.
- Device interfaces 138 connect the disk drives 106 of RAID array 104 to PCI bus 102 for the storage and retrieval of information therefrom.
- the RAID parity assist 134 operates on data stored in a local memory (not shown) associated therewith. It calculates parity a burst at a time utilizing an internal 128 byte
- the RAID parity assist 134 operates at the full speed of the local memory bus to provide the fastest possible performance for the memory bandwidth.
- the parity assist engine contains 4 separate sections that allow additional tasks to be queued up while one task executes. Each task maintains its own control and status register so that task scheduling does not interfere with the currently executing task.
- the engine can be configured as a single engine which works on wide arrays up to 22+1 drives wide, or as a four engine machine which operates with arrays as wide as 4+1 drives.
- An intermediate mode provides a two engine machine with up to 10+1 drives.
- the parity engine includes exclusive-OR logic providing RAID 5 and RAID 3 parity generation/checking as well as move and zero check modes.
- RPA 134 Further details regarding the structure and operation of RPA 134 are contained in the aforementioned US and European patent applications.
- CPU 122 The principle operational purpose of CPU 122 is to exchange data between an attached host computer system 108 and the RAID array 104 through cache buffer 136 in satisfaction of host computer generated I/O requests. As can bee seen from the above discussed structure, all such exchanges of data pass through PCI bus 102. Specifically, host generated I/O requests typically cause the exchange of data between cache buffer 136 and host computer 108 through PCI bus 102 (and through host interface 132 and RPA 134). Likewise, later posting of data by cached RAID controller 101 typically causes the exchange of data between cache buffer 136 and RAID array 104 through PCI bus 102 (and through RPA 134 and device interfaces 138.
- PCI bus 102 may be replaced or enhanced by higher performance bus architectures.
- fundamental architectural changes in the bus structure often require significant, costly changes in the control methods (the control software) operable within RAID controller 101.
- Fig. 2 is a block diagram describing the structure of a RAID storage subsystem 200 in which the present invention is applied to improve the cache memory bandwidth capacity while minimizing the need for costly changes in existing "legacy" control systems of the
- RAID controller within RAID subsystem 200 is similar in many respects, other than the cache memory architecture, to RAID controller 101 of RAID subsystem 100 of FIG. 1.
- CPU 122, L2 cache 124, CDC 126, RAM 128, and DPU 130 of FIG. 2 are connected through processor bus 156 as described above with respect to FIG.
- NVRAM 142, serial port 144, and debug/maintenance port I/O (SIO) 140 and bus 154.
- CDC 126 are exemplary of such chip sets presently commercially available from Intel and other integrated circuit manufacturers.
- High speed cache buffer 214 comprises a memory array which is both fast and wide to provide adequate bandwidth capacity for aggregated cache buffer accesses in the exchange of information.
- Main memory controller 212 manages the memory array of high speed cache buffer 214 exchanging data therewith via bus 254.
- Intermediate shared memory bus 250 connects all other components involved in the exchange of data with high speed cache buffer 214 via main memory controller and RPA 212 and bus 254. All exchange of data between high speed cache buffer 214 and other components of RAID subsystem 200 are managed by the centralized main memory controller 212 and provided thereto via intermediate shared memory bus 250.
- Main memory controller 212 is therefore isolated from the specific protocols and bus structure of each component within RAID subsystem 200 wishing to exchange data with high speed cache buffer 214 and may therefore be adapted in future evolutionary designs to further enhance bandwidth capacity of the high speed cache buffer 214.
- bus bridge 206, 208, and 210 adapts the signals applied to their respective, unique, connected bus architecture to the intermediate shared memory bus 250.
- bus bridge 206 adapts signals on bus 256 (e.g., a 32-bit or 64-bit PCI bus) to appropriate signals on intermediate shared memory bus 250 (and vice versa) for purposes of exchanging data between an attached host computer 108, through bus 120 and high speed host interface 204, and the high speed cache buffer 214 under the control of main memory controller 212.
- bus bridges 208 and 210 adapt signals on busses 258 and 260, respectively, for application to intermediate shared memory bus 250 (and vice versa). Bus bridges 208 and 210 thereby provide for the exchange of information between RAID array 104 (through device interfaces 138.1, 138.2) and high speed cache buffer 214 under the control of main memory controller 212.
- busses 256, 258, and 260 may be implemented for compatibility with older "legacy" control systems operable within RAID controller 201.
- Legacy systems that depended upon, for example, a 32-bit PCI bus for access to device interfaces 138.1 and 138.2 may be utilized within the RAID controller 201 with the cache architecture of the present invention. Use of such legacy systems reduces the costs and complexity of redesigning new systems compatible with a new cache memory architecture.
- a second feature of the present invention is the enhanced aggregate bandwidth for data exchange realized by the plurality of busses hierarchically arranged as described above.
- a high speed cache memory bus architecture is supported by high speed cache buffer 214, bus 254, and main memory controller 212.
- This high speed cache architecture may be enhanced to utilize the highest performance memory designs available in a particular memory technology.
- the high speed cache may, for example, make use of DRAM, SRAM, SDRAM, EDRAM, EDO RAM, Burst EDO RAM, or RAMBUS architectures without impacting the portability of legacy control systems in the RAID controller 201.
- the intermediate shared memory bus 250 provides a high bandwidth shared access path for each of a plurality of legacy system supported busses to exchange data with the high speed cache.
- a plurality of legacy system supported busses may be intermixed and used, each connected to the intermediate shared memory bus 250 through a bus bridge, to distribute the data exchange bandwidth requirements over a plurality of busses.
- Each individual bus is therefore less likely to be saturated to its capacity for data exchange.
- data exchange between an attached host computer 108 and the high speed cache buffer 214 may take place concurrently with data exchange between the high speed cache buffer 214 and the disk drives 106 of the RAID array 104.
- Each of these data exchange operations consumes bandwidth on its own legacy system supported bus (e.g., bus 256 and 258 or 260, respectively).
- bus 256 and 258 or 260 respectively.
- the concurrent data exchanges will require use of the intermediate shared memory bus and the main memory controller 212 and bus 254, these components are designed to maximize bandwidth capacity.
- the cache memory architecture of the present invention is therefore less likely to saturate available data exchange bandwidth.
- Main memory controller 212 also may provide RAID parity assist logic (RPA) to provide centralized high speed assistance for the generation and checking of RAID redundancy information. It is common in many RAID storage subsystem to utilize exclusive or (XOR) parity generation and checking for the redundancy information.
- redundancy information includes well known exclusive or parity techniques as well as other encoding methods used to generate and check information used to regenerate erroneous or missing stored data.
- This feature of the present invention serves to isolate the RAID parity transfers from the individual legacy bus structures as well as the shared memory bus 250. Parity computations are performed internally within RPA circuits of main memory controller 212. This further enhances the distribution of the memory access load from a plurality of legacy systems to a centralized high speed bus structure. The individual legacy busses are therefore less likely to be saturated by such parity accesses.
- the main memory controller 212 may be directed to allow the device attached to a bus bridge to manage all redundancy information generation and checking.
- Processor bus 252 connects CPU 122 to the hierarchical cache memory architecture of the present invention to permit methods operable within CPU 122 to control the operations of the bus bridges 206, 208, and 210, and of the main memory controller 212.
- This aspect of the present invention enables methods operable within processor 122 to access an individual one of busses (e.g., 256, 258, and 260) without impacting continued high speed data access and processing by the other busses.
- the higher bandwidth capacity of the cache memory architecture of the present invention permits the RAID controller 201 to be scaled up for higher performance while minimizing the needs for redesign of older legacy control systems within the RAID controller 201.
- host interface 204 may be easily scaled up to Fibre Channel host system connections.
- the higher data bandwidth available in the cache memory architecture of the present invention provides sufficient bandwidth therefor while permitting easier re-use of legacy control systems in the controller.
- device interfaces 138.1 and 138.2 may be scaled up to higher performance disk drive control architectures (e.g., fast SCSI, wide SCSI, SCSI 3, Fibre Channel (FCAL), etc.).
- the data bandwidth available in the cache architecture of the present invention enables the use of such high performance connections while minimizing the re-design necessary for legacy control structures.
- Fig. 3 is a block diagram showing additional details of the main memory controller 212 of Fig. 2.
- Memory control 308 within main memory controller 212 receives address, data, and control information on intermediate shared memory bus 250. The address, data, and control information together define a requested operation within the attached high speed cache buffer (214 of FIG. 2).
- Memory interface 310 within memory control 308 manages the signals received from and applied to intermediate shared memory bus 250.
- RAM control 312 within memory control 308 generates appropriate RAM circuit controls and applies them to bus 254 to perform the requested operation in the attached high speed cache buffer (214 of FIG. 2).
- Memory control 308 may be configured to appropriately control a wide variety of RAM geometries and technologies including DRAM, SRAM, SDRAM, EDO RAM, etc.
- memory control 308 implements a 64-bit wide memory bus on intermediate shared memory bus 250 and controls a 72-bit wide memory bus on bus 254 (64 data bits plus 8 ECC bits).
- both intermediate shared memory bus 250 and bus 254 are clocked at a frequency in excess of 60 megahertz (preferably 66 megahertz).
- intermediate shared memory bus 250 and bus 254 can therefore sustain a bandwidth in excess of 500 megabytes per second.
- RAID parity assist engine 300 within main memory controller 212 performs all required RAID parity generation and checking as data is stored in and retrieved from the high speed cache buffer (214 of FIG. 2).
- the parity calculations are performed within the RAID parity assist engine 300 without imposing additional load on the various busses of the RAID controller architecture.
- the parity computations are performed with a minimum of access to the cache memory 214 by main memory controller 212.
- the computed parity data may be retrieved from the RAID parity assist engine 300 for direct storage in the high speed cache buffer 214 or the disk array 104, or may be retrieved by the CPU 122 via processor bus 252 for further computation and processing.
- Processor bus interface 316 within main memory controller 212 comprises the connection logic and circuits to connect the main memory controller 212 to the processor bus 252.
- CPU 122 of FIG. 2 configures and controls the operation of main memory controller 212 via processor bus 252.
- CPU 122 may access high speed cache memory 214 of FIG. 2 through processor bus 252 and processor bus interface 316 of main memory controller 212.
- Memory arbitration logic 314 arbitrates access to bus 254 requested through intermediate shared memory bus 250 and through processor bus 252. Such memory arbitration and control functions, and commercially available circuits to perform these functions, are well known to those of ordinary skill in the art.
- Fig. 4 is a block diagram providing additional detail of the structure of bus bridges 206, 208, and 210 of Fig. 3.
- Bus bridges 206, 208, and 210 exchange signals over their respective attached busses 256, 258, and 260, respectively.
- busses 256, 258, and 260 may be, for example, 32 and 64-bit PCI busses to maintain structural compatibility with older "legacy" control systems within RAID controller 201 of Fig. 2.
- the signals exchanged over busses 256, 258, and 260 are adapted for exchange over intermediate shared memory bus 250.
- Each bus bridge, 206, 208, or 210 therefore adapts the legacy control system's preferred bus, 256, 258, or 260, respectively, to the intermediate shared memory bus 250 for access to the high speed cache buffer 214 of Fig. 2.
- a PCI bus interface 400 manages the exchange of signals on the legacy control system's preferred bus 256, 258, or 260, respectively.
- PCI bus converter 404 converts the respective PCI bus signals to the signal levels and timing required for application to the intermediate shared memory bus 250.
- Memory interface 406 applies the converted PCI bus signals to the intermediate shared memory bus 250 as appropriate for control of the shared bus.
- Memory interface 406 performs all required bus arbitration and negotiation to share the intermediate shared memory bus among the plurality of bus bridges 206, 208, and 210.
- signals received on the intermediate shared memory bus 250 by memory interface 406 are converted to equivalent PCI bus signals by a PCI bus converter 404 within the bus bridge to which the signals are directed.
- PCI bus interface 400 within the bus bridge then applies the converted signals to the connected bus 256, 258, or 260 as appropriate.
- selection signals in intermediate shared memory bus 250 are used to arbitrate the use of the bus to exchange converted signals between the intermediate shared memory bus 250 and a selected one of the plurality of bus bridges 206, 208, and 210.
- Processor bus 252 is connected to bus bridges 206, 208, and 210 to permit a CPU (122 of FIG. 2) to control and configure the operation of the bus bridges.
- CPU 1 12 exchanges information and configures peripheral devices directly over the connected PCI busses 256, 258, or 260 vi the bus bridges 206, 208, or 210, respectively, without interfering with the exchange of data via intermediate shared memory bus 250.
- PCI bus converter 404 converts signals between levels and timing required for processor bus 252 as received and generated by processor bus interface 402, and signal levels and timing required for the connected PCI bus 256, 258, or 260 as received and generated by PCI bus interface 400 of the bus bridges.
- Bus bridges 206. 208, and 210 convert all signals for exchange between the connected PCI bus 256, 258, or 260 and the intermediate shared memory bus 250 or between the connected PCI bus 256, 258, or 260 and the processor bus 252. This conversion includes all signal level and timing conversions as well as bus width conversions.
- PCI busses 256, 258, and 260 may be either 32-bit or 64-bit wide PCI busses and may be operated at either 33 or 66 megahertz while the processor bus is a bus appropriate to the selected CPU (for example a 32-bit wide, 33 megahertz PCI bus) and the intermediate shared memory bus is preferably 64-bits wide operating at 66 megahertz.
- the shared memory bus 250 is preferably a parallel signal bus providing 64 bit wide data path
- the shared memory bus 250 is implemented as a 128 bit wide data path.
- the 128 bit wide bus is implemented through use of a plurality of high speed parallel to serial transceivers. For example, eight, sixteen bit wide parallel to serial high speed transceivers may be used in both the bus bridge circuits 206, 208, and 210 and the corresponding memory interface circuits within main memory controller 212 to implement the shared memory bus 250.
- Each bus interface, PCI bus interface 400, processor bus interface 402, and memory interface 406, may include FIFO devices as required for speed and bus width matching purposes.
- the FIFOs permit each bus interface component and the associated bus to operate at peak efficiency for extended bursts though exchanging data with a different bus architecture.
- the processor bus interface 402 of the bus bridge circuit essentially implements a bus bridge to translate the signals exchanged between processor bus 252 and the external legacy system bus 256, 258, or 260.
- the memory interface 406 portion uses FIFO technologies to buffer the exchange of bursts of data between the legacy system bus 256, 258, or 260 and the shared memory bus 250. This buffering enables speed matching between the legacy system bus 256, 258, or 260 and the higher speed shared memory bus 250.
- FIG. 5 is a block diagram depicting such a structure for intermediate shared memory bus 250 of Fig. 2.
- Intermediate memory bus 250 of Fig. 5 is connected to the memory interface (FIFO) 406 of bus bridge circuits 206, 208, or 210.
- FIFO memory interface
- Fig. 6 is a block diagram of an alternative embodiment of the intermediate shared memory bus 250 of the present invention in which eight, high speed, 16-bit-wide parallel to bit serial transceivers 500 are integrated into memory interface 406.
- This structure provides a 128-bit-wide internal bus structure but an 8-bit-wide (byte) serial interface as applied to shared intermediate memory bus 250.
- This alternate embodiment of the present invention serves to reduce the external pin count required for interconnect of bus bridge circuits 206, 208, and 210 and main memory controller circuit 212 via intermediate shared memory bus 250.
- the transceivers 500 are capable of data transfer speeds of at least 1 gigabit per second each to thereby provide bandwidth of 1 gigabyte per second over intermediate shared memory bus 250.
Abstract
Description
Claims
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP97924140A EP0907917B1 (en) | 1996-06-28 | 1997-05-30 | Cache memory controller in a raid interface |
DE69702523T DE69702523T2 (en) | 1996-06-28 | 1997-05-30 | CACHE STORAGE CONTROL UNIT IN A RAID INTERFACE |
AU29707/97A AU2970797A (en) | 1996-06-28 | 1997-05-30 | Cache memory controller in a raid interface |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US08/673,654 US5937174A (en) | 1996-06-28 | 1996-06-28 | Scalable hierarchial memory structure for high data bandwidth raid applications |
US08/673,654 | 1996-06-28 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO1998000776A1 true WO1998000776A1 (en) | 1998-01-08 |
Family
ID=24703557
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/GB1997/001474 WO1998000776A1 (en) | 1996-06-28 | 1997-05-30 | Cache memory controller in a raid interface |
Country Status (5)
Country | Link |
---|---|
US (1) | US5937174A (en) |
EP (1) | EP0907917B1 (en) |
AU (1) | AU2970797A (en) |
DE (1) | DE69702523T2 (en) |
WO (1) | WO1998000776A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7483757B2 (en) | 2005-07-22 | 2009-01-27 | Honeywell International, Inc. | Control system migration |
Families Citing this family (27)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6161165A (en) * | 1996-11-14 | 2000-12-12 | Emc Corporation | High performance data path with XOR on the fly |
US6118776A (en) | 1997-02-18 | 2000-09-12 | Vixel Corporation | Methods and apparatus for fiber channel interconnection of private loop devices |
US6324608B1 (en) * | 1997-05-13 | 2001-11-27 | Micron Electronics | Method for hot swapping of network components |
US6665703B1 (en) * | 1998-01-02 | 2003-12-16 | Emc Corporation | Storage system with internal LAN |
US6112311A (en) * | 1998-02-20 | 2000-08-29 | International Business Machines Corporation | Bridge failover system |
US6098140A (en) * | 1998-06-11 | 2000-08-01 | Adaptec, Inc. | Modular bus bridge system compatible with multiple bus pin configurations |
US6434654B1 (en) * | 1999-03-26 | 2002-08-13 | Koninklijke Philips Electronics N.V. | System bus with a variable width selectivity configurable at initialization |
US6591337B1 (en) * | 1999-04-05 | 2003-07-08 | Lsi Logic Corporation | Method and apparatus for caching objects in a disparate management environment |
WO2001031456A1 (en) * | 1999-10-28 | 2001-05-03 | Connectcom Solutions, Inc. | Caching techniques for improving system performance in raid applications |
US6615890B1 (en) * | 2000-06-09 | 2003-09-09 | Venture Tape Corp. | Tape applicator for glazing applications |
US6530004B1 (en) * | 2000-06-20 | 2003-03-04 | International Business Machines Corporation | Efficient fault-tolerant preservation of data integrity during dynamic RAID data migration |
WO2002001375A1 (en) * | 2000-06-27 | 2002-01-03 | Koninklijke Philips Electronics N.V. | Integrated circuit with flash |
JP2003015826A (en) * | 2001-07-04 | 2003-01-17 | Hitachi Ltd | Shared memory copy function in disk array controller |
US6789169B2 (en) * | 2001-10-04 | 2004-09-07 | Micron Technology, Inc. | Embedded DRAM cache memory and method having reduced latency |
US6985995B2 (en) | 2002-03-29 | 2006-01-10 | Panasas, Inc. | Data file migration from a mirrored RAID to a non-mirrored XOR-based RAID without rewriting the data |
JP2003345528A (en) * | 2002-05-22 | 2003-12-05 | Hitachi Ltd | Storage system |
US20030236943A1 (en) * | 2002-06-24 | 2003-12-25 | Delaney William P. | Method and systems for flyby raid parity generation |
US7103826B2 (en) * | 2003-07-31 | 2006-09-05 | Hewlett-Packard Development Company, L.P. | Memory system and controller for same |
JP4468666B2 (en) * | 2003-08-22 | 2010-05-26 | 富士通株式会社 | Apparatus and storage control apparatus having dual writing function |
US7099994B2 (en) * | 2003-09-29 | 2006-08-29 | Hewlett-Packard Development Company, L.P. | RAID memory system |
JP4307202B2 (en) * | 2003-09-29 | 2009-08-05 | 株式会社日立製作所 | Storage system and storage control device |
DE602004029480D1 (en) * | 2004-11-24 | 2010-11-18 | Fujitsu Ltd | Device with double writing function and memory control device |
US7230851B2 (en) * | 2004-12-23 | 2007-06-12 | Sandisk Corporation | Reducing floating gate to floating gate coupling effect |
DE102006027355A1 (en) * | 2006-06-13 | 2007-12-20 | Mahle International Gmbh | Piston for an internal combustion engine and method for its production |
US8756371B2 (en) | 2011-10-12 | 2014-06-17 | Lsi Corporation | Methods and apparatus for improved raid parity computation in a storage controller |
US8972645B2 (en) * | 2012-09-19 | 2015-03-03 | Hewlett-Packard Development Company, L.P. | Request sent to storage device based on moving average |
KR102160414B1 (en) * | 2016-01-22 | 2020-09-28 | 주식회사 소니 인터랙티브 엔터테인먼트 | Legacy bus behavior simulation for backwards compatibility |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1993014455A1 (en) * | 1992-01-06 | 1993-07-22 | Array Data Limited | Computer memory array control |
US5353415A (en) * | 1992-10-02 | 1994-10-04 | Compaq Computer Corporation | Method and apparatus for concurrency of bus operations |
EP0631241A1 (en) * | 1993-05-28 | 1994-12-28 | International Business Machines Corporation | Initializing multiple bus networks |
US5379384A (en) * | 1992-06-05 | 1995-01-03 | Intel Corporation | Configuration data loopback in a bus bridge circuit |
EP0756235A1 (en) * | 1995-07-24 | 1997-01-29 | Symbios Logic Inc. | Method and apparatus for enhancing throughput of disk array data transfers in a controller |
Family Cites Families (28)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPS6226561A (en) * | 1985-07-26 | 1987-02-04 | Toshiba Corp | Personal computer |
US4965723A (en) * | 1987-10-23 | 1990-10-23 | Digital Equipment Corporation | Bus data path control scheme |
JPH0786853B2 (en) * | 1988-02-29 | 1995-09-20 | 株式会社ピーエフユー | Bus transfer control method |
DE69124946T2 (en) * | 1990-11-30 | 1997-09-18 | Ibm | Bi-directional FIFO buffer for interface formation between two buses |
US5345565A (en) * | 1991-03-13 | 1994-09-06 | Ncr Corporation | Multiple configuration data path architecture for a disk array controller |
US5301281A (en) * | 1991-06-26 | 1994-04-05 | Ast Research, Inc. | Method and apparatus for expanding a backplane interconnecting bus in a multiprocessor computer system without additional byte select signals |
US5257391A (en) * | 1991-08-16 | 1993-10-26 | Ncr Corporation | Disk controller having host interface and bus switches for selecting buffer and drive busses respectively based on configuration control signals |
JP2836321B2 (en) * | 1991-11-05 | 1998-12-14 | 三菱電機株式会社 | Data processing device |
US5333277A (en) * | 1992-01-10 | 1994-07-26 | Exportech Trading Company | Data buss interface and expansion system |
US5263139A (en) * | 1992-05-19 | 1993-11-16 | Sun Microsystems, Inc. | Multiple bus architecture for flexible communication among processor modules and memory subsystems and specialized subsystems |
US5392407A (en) * | 1992-12-24 | 1995-02-21 | Ncr Corporation | Multi-port processor with peripheral component interconnect port and rambus port |
US5522050A (en) * | 1993-05-28 | 1996-05-28 | International Business Machines Corporation | Bus-to-bus bridge for a multiple bus information handling system that optimizes data transfers between a system bus and a peripheral bus |
US5396602A (en) * | 1993-05-28 | 1995-03-07 | International Business Machines Corp. | Arbitration logic for multiple bus computer system |
US5664122A (en) * | 1993-11-12 | 1997-09-02 | Intel Corporation | Method and apparatus for sequencing buffers for fast transfer of data between buses |
US5533204A (en) * | 1994-04-18 | 1996-07-02 | Compaq Computer Corporation | Split transaction protocol for the peripheral component interconnect bus |
US5560022A (en) * | 1994-07-19 | 1996-09-24 | Intel Corporation | Power management coordinator system and interface |
US5548730A (en) * | 1994-09-20 | 1996-08-20 | Intel Corporation | Intelligent bus bridge for input/output subsystems in a computer system |
US5557758A (en) * | 1994-11-30 | 1996-09-17 | International Business Machines Corporation | Bridge between two buses of a computer system that determines the location of memory or accesses from bus masters on one of the buses |
US5634033A (en) * | 1994-12-16 | 1997-05-27 | At&T Global Information Solutions Company | Disk array storage system architecture for parity operations simultaneous with other data operations |
US5594882A (en) * | 1995-01-04 | 1997-01-14 | Intel Corporation | PCI split transactions utilizing dual address cycle |
US5574937A (en) * | 1995-01-30 | 1996-11-12 | Intel Corporation | Method and apparatus for improving instruction tracing operations in a computer system |
US5586268A (en) * | 1995-03-03 | 1996-12-17 | Advanced System Products, Inc. | Multiple peripheral adapter device driver architecture |
US5559968A (en) * | 1995-03-03 | 1996-09-24 | Compaq Computer Corporation | Non-conforming PCI bus master timing compensation circuit |
US5608876A (en) * | 1995-05-22 | 1997-03-04 | International Business Machines Corporation | Add-in board with enable-disable expansion ROM for PCI bus computers |
US5590377A (en) * | 1995-06-07 | 1996-12-31 | Ast Research, Inc. | Automatic control of distributed DMAs in a PCI bus system supporting dual ISA buses |
US5632021A (en) * | 1995-10-25 | 1997-05-20 | Cisco Systems Inc. | Computer system with cascaded peripheral component interconnect (PCI) buses |
US5644470A (en) * | 1995-11-02 | 1997-07-01 | International Business Machines Corporation | Autodocking hardware for installing and/or removing adapter cards without opening the computer system cover |
US5631912A (en) * | 1995-12-19 | 1997-05-20 | Samsung Electronics Co., Ltd. | High impedance test mode for JTAG |
-
1996
- 1996-06-28 US US08/673,654 patent/US5937174A/en not_active Expired - Lifetime
-
1997
- 1997-05-30 WO PCT/GB1997/001474 patent/WO1998000776A1/en active IP Right Grant
- 1997-05-30 AU AU29707/97A patent/AU2970797A/en not_active Abandoned
- 1997-05-30 DE DE69702523T patent/DE69702523T2/en not_active Expired - Lifetime
- 1997-05-30 EP EP97924140A patent/EP0907917B1/en not_active Expired - Lifetime
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1993014455A1 (en) * | 1992-01-06 | 1993-07-22 | Array Data Limited | Computer memory array control |
US5379384A (en) * | 1992-06-05 | 1995-01-03 | Intel Corporation | Configuration data loopback in a bus bridge circuit |
US5353415A (en) * | 1992-10-02 | 1994-10-04 | Compaq Computer Corporation | Method and apparatus for concurrency of bus operations |
EP0631241A1 (en) * | 1993-05-28 | 1994-12-28 | International Business Machines Corporation | Initializing multiple bus networks |
EP0756235A1 (en) * | 1995-07-24 | 1997-01-29 | Symbios Logic Inc. | Method and apparatus for enhancing throughput of disk array data transfers in a controller |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7483757B2 (en) | 2005-07-22 | 2009-01-27 | Honeywell International, Inc. | Control system migration |
Also Published As
Publication number | Publication date |
---|---|
EP0907917B1 (en) | 2000-07-12 |
DE69702523T2 (en) | 2001-04-19 |
AU2970797A (en) | 1998-01-21 |
DE69702523D1 (en) | 2000-08-17 |
US5937174A (en) | 1999-08-10 |
EP0907917A1 (en) | 1999-04-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP0907917B1 (en) | Cache memory controller in a raid interface | |
US6370611B1 (en) | Raid XOR operations to synchronous DRAM using a read buffer and pipelining of synchronous DRAM burst read data | |
US5881254A (en) | Inter-bus bridge circuit with integrated memory port | |
US7062591B2 (en) | Controller data sharing using a modular DMA architecture | |
US7437493B2 (en) | Modular architecture for a network storage controller | |
JP3211976B2 (en) | Device and method for generating interrupt signal for disk array system | |
US7240139B2 (en) | Disk array control device with two different internal connection systems | |
EP0717357B1 (en) | Disk array apparatus | |
US6839788B2 (en) | Bus zoning in a channel independent storage controller architecture | |
US7093043B2 (en) | Data array having redundancy messaging between array controllers over the host bus | |
US5884055A (en) | Method and apparatus including a shared resource and multiple processors running a common control program accessing the shared resource | |
US8478835B2 (en) | Method and system for using shared memory with optimized data flow to improve input/output throughout and latency | |
US5522065A (en) | Method for performing write operations in a parity fault tolerant disk array | |
JPH06180623A (en) | Multiplex-configuration data- path structure for disk array controller | |
US6529989B1 (en) | Intelligent expansion ROM sharing bus subsystem | |
US20160259568A1 (en) | Method and apparatus for storing data | |
EP1092188A1 (en) | Split directory-based cache coherency technique for a multi-processor computer system | |
WO2005066761A2 (en) | Method, system, and program for managing parity raid data reconstruction | |
WO1996018141A1 (en) | Computer system | |
US6370616B1 (en) | Memory interface controller for datum raid operations with a datum multiplier | |
WO1993013475A1 (en) | Method for performing disk array operations using a nonuniform stripe size mapping scheme | |
US7136972B2 (en) | Apparatus, system, and method for distributed management in a storage system | |
US5375217A (en) | Method and apparatus for synchronizing disk drive requests within a disk array | |
KR19990047968A (en) | Structure of Array Controller with Dual I / O Bus for Raid System | |
KR100447267B1 (en) | A distributed controlling apparatus of a RAID system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AK | Designated states |
Kind code of ref document: A1 Designated state(s): AL AM AT AU AZ BA BB BG BR BY CA CH CN CU CZ DE DK EE ES FI GB GE GH HU IL IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MD MG MK MN MW MX NO NZ PL PT RO RU SD SE SG SI SK TJ TM TR TT UA UG UZ VN YU AM AZ BY KG KZ MD RU TJ TM |
|
AL | Designated countries for regional patents |
Kind code of ref document: A1 Designated state(s): GH KE LS MW SD SZ UG AT BE CH DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ |
|
DFPE | Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101) | ||
DFPE | Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101) |
Free format text: MG |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
WWE | Wipo information: entry into national phase |
Ref document number: 1997924140 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: JP Ref document number: 98503896 Format of ref document f/p: F |
|
WWP | Wipo information: published in national office |
Ref document number: 1997924140 Country of ref document: EP |
|
REG | Reference to national code |
Ref country code: DE Ref legal event code: 8642 |
|
NENP | Non-entry into the national phase |
Ref country code: CA |
|
WWG | Wipo information: grant in national office |
Ref document number: 1997924140 Country of ref document: EP |