« PreviousContinue »
Storage access systems are designed for extracting needed information (e.g. user data files, programs, or data base records) from large volumes of stored data in response to user or application needs. Such storage access systems have a wide variety of applications including computer workstation networks, and large, high performance multiprocessor general purpose computing systems. Such storage access systems are also useful for providing high throughput inquiry-response database-systems for service control points in public communications networks, large video-text systems, directory assistance systems, and airline reservation systems.
Typically, a general purpose computer system for inquiry-response data base or other applications comprises a central processor. At the front end, the central processor interfaces with the users who generate inquiries or otherwise interact with applications being executed in the central processor. At the back end the 35 central processor interfaces with storage devices which store the user files, programs and/or a data base containing records to be accessed in response to user requests.
Illustratively, the storage devices are magnetic disks. 40 If a data base is stored, either on disk or in the local memory of the central processor an index may be provided so that the central processor knows where in the data base to look for particular records required to respond to particular inquiries. Similarly, directories or 45 other tables may be provided to indicate where relevant data files or programs are stored.
The design of high performance data base or application-processing systems—i.e. those with high transaction volumes accessing very large storage volumes with 50 stringent response time criteria—is limited by two major factors. These are processor real time and the time required for physical access to storage.
Processor real time becomes a resource limit when the central processor or other processing unit in the 55 system cannot execute instructions rapidly enough to respond to user requests in a reasonable period of time. Processor real time is required to execute instructions for front end operations (communications with users)
inquiry-response transactions that can be handled per second using the central processor dominated application processing system architecture described above. For this type of system, transaction handling capacity may be increased by using higher performance processing elements and/or by adding processing elements in series (e.g. a front end processor) to off load the main processor and/or by adding main processors in parallel to share the processing load. Since modular, multiprocessor configurations permit the assembly of very large amounts of processing capacity to execute applications in parallel, physical access to storage and control over modifications to storage by the several parallel processing elements are often important elements in creating very high capacity inquiry-response data storage access systems or application processing systems.
With higher performance processors or with multiprocessor configurations, physical access to storage can become a system bottleneck. Illustratively, physical access to records stored on magnetic disk systems is limited by the time required for the disk head to "seek" to the desired cylinder, time for the platter to rotate to the desired position, and time to transfer the data over the channel connecting the storage device to the associated processor.
Several strategies have been devised to deal with the storage access bottleneck. One such strategy is for each processing element to maintain in its local memory all (or the most frequently accessed part) of the data base. Alternatively, each processor may maintain a different part of the data base in its local memory. This approach handles larger data bases than the previously-mentioned approach but requires the preprocessing of each incoming transaction to direct it to the responsible processing element. This may be difficult to achieve, if for example the query contents do not imply the correct partition or if the data base is large. In yet another approach, replicated disk storage devices with optimizing controllers may be used. In this approach, processing elements send inquiries to an intelligent mass storage controller which assigns the inquiries to a set of mirrored disk drives containing duplicate copies of the data base. The storage controller attempts to minimize the overall average access latency by allocating queries among the various drives to minimize seek time and waiting time for disk rotation across the set of outstanding queries. The speed and volume capacity of this approach for retrieving records are limited by the capability of the intelligent storage controller and by the number and performance of the magnetic disk drives which are attached to the controller.
In short, the foregoing approaches to increasing the capabilities of inquiry and response record handling systems invoke the use of parallel processing and/or parallel storage architectures. These prior art architectures for high performance inquiry and response data base systems have limitations. First, by adding parallelism in processing and storage to increase overall system throughput, a need may be created for a mechanism to
and back-end operators (communications with storage £0 coordinate access to and modification of multiple copies
devices). Processor real time is also required to interpret each incoming inquiry to specify the required records, files or programs, and once the required records, files or programs have been obtained, to process the records, files or programs to determine the appropriate 65 response.
The processor real time requirements for each inquiry-response transaction severely limits the number of
of records or other data by the various transactions running on the various processing elements. This need for concurrency control may not be severe for applications where occasional inconsistent views of the data base are tolerable, or where changes to the data base can be queued and executed in batch form to prevent inconsistencies from developing. However, for more general applications, the system overhead incurred in
the software mechanisms, which guarantee that each transaction receives a "true" view of the data base and which resolve conflicting attempts to read or write database records, limits the ultimate capacity of the system architectures. 5
Second, some applications like 800 service in the public telephone network employ data bases with defined structures. A single key (e.g. the dialed 800 number) could be used with a memory resident index to identify the location of the relevant record in storage, 10 avoiding the need to search the entire data base to find the relevant record. For very large data bases, or for data bases which permit access through a number of keys (e.g. by phone number, by name, by address) the memory resident indices themselves may become a 15 significant size component of the overall data base, thus requiring multiple accesses to physical storage to finally retrieve a desired record. Additionally, if "wild card" inquiries occur, i.e. inquiries which require retrieval of records on keys for which no indices were created a 20 priori, then every record in the data base must be retrieved and the contents examined to determine which records satisfy the request. This search process may be enormously demanding on processor real time and may also saturate the storage input output bandwidth if the 25 records reside on disk.
In another prior art approach to record access systems, content addressable storage systems have been implemented to improve efficiency in accessing records from storage. In response to an inquiry, these systems 30 typically read an entire disk cylinder at a time and employ "high performance processors" which examine the output data stream on the fly, identifying and moving to buffers those records whose contents match a particular record request. Some indexing is often required in these 35 systems to identify when possible the set of cylinders likely to contain the requested records. A system of this type is available from the British computer manufacturer ICL; see e.g. "Mailer V.A.J. The Content Addressable File Store-CAFS", ICL Technical Journal, 40 Vol. 1, No. 3, November 1979. This type of system permits much shorter response time to arbitrary inquiries than do systems in which a single central processor itself must sort through and retrieve a large number of candidate records. Still, based on one directory assist- 45 ance benchmark for the ICL system (17 queries/sec), for limited predictable queries, this technology is still inadequate to compete effectively with highly parallel central processor type systems which utilize indexing to enable processing elements to find particular requested 50 records. In short, the above-described inquiry and response system architectures all have limitations on the volume of transactions which can be handled per unit of time.
Accordingly, it is an object of the present invention 55 to provide a new architecture for an inquiry and response data storage access system, which can handle a larger volume of transactions than prior inquiry and response data storage access systems.
SUMMARY OF THE INVENTION 60
The present invention is an architecture for a storage access system which permits implementation of high capacity query intensive data bases required for mass market public applications such as directory assistance. 65 Since the architecture and processes for managing records, which are defined as associated items of data, also permit the management of files (associated sets of
records), the architecture is applicable to high performance multiprocessor computer configurations including networks of diskless workstations.
The physical components of the architecture include one or more record storage "pumps", a high bandwidth distribution medium (e.g. optical fiber) and a plurality of record retrieval elements which serve as interfaces between traditional general purpose computer processing elements and the information present on the distribution medium. The computer processing elements, illustratively enable the user to interact with the architecture.
A data base comprising the records to be accessed is stored in the pump and is transmitted over the distribution medium in a time cyclic fashion, i.e. all records in the data base will be transmitted between time U> and time t<,+T at which time the cycle will repeat. In some embodiments of the invention, some number of these records may be transmitted multiple times within an interval T. The general purpose user computer processing elements communicate record retrieval requests (resulting from user inquiries) to associated record retrieval elements. The record retrieval elements view all records transmitted on the distribution medium in any time interval T and identify and store those records whose contents contain the specific attributes defined by the specific record retrieval requests. Any request for a record or record set from a particular user processor to a particular record retrieval element can be satisfied (or known to have failed) within about T seconds of receipt of the request, independent of the total number of requests pending. To select files, instead of individual records, each record may include an attribute known as a file identifier; files are retrieved from the storage access system by specifying the file identifier for the desired file. The record retrieval elements will identify and store all records with the specified file identifier, thereby retrieving the desired file, since the processes basic to the operation of the data storage access system apply to sets of records (i.e., files) as well as to individual records, description of the invention will primarily deal with records.
Preferably, the data storage access system architecture will also include an upstream channel from the record retrieval elements to the storage pump and a mechanism at the pump to introduce changed or new records into pump storage. A record management system may also be provided to resolve attempts by different record retrieval elements to "write" changes to the same records. The ability to write into pump storage, however, should not impair the ability to cyclically broadcast the data base contents at high rates.
The broadcast oriented record retrieval architecture has a number of important advantages. These are:
A. Multiprocessor Access to Records
The architecture permits parallel, non-interfering access to the data base by a plurality of user processing elements by broadcasting the entire data base in a cyclical fashion to record retrieval elements associated with the plurality of processors. Thus, each user processor has independent access to the data as though it had a dedicated disk or cache copy of the data base. Yet, only a single physical copy of the data base exists—at the storage pump—greatly reducing the overhead of assuring that each processor has the same view of the data base given to other processors. For record retrieval, the architecture permits an unlimited number of user processors to be attached to the system.