BACKGROUND OF THE INVENTION
1. Field of Invention
The present invention relates generally to multi-dimensional relational databases and, more specifically to mechanisms for aggregating data elements in a multi-dimensional relational database system and for processing queries on such aggregated data elements, and also to informational database systems that utilize multi-dimensional relational databases and such aggregation/query mechanisms.
2. Brief Description Of The State Of The Art
Information technology (IT) enables an enterprise to manage and optimize its internal business practices through the analysis and sharing of data internally within the enterprise. In addition, IT enables an enterprise to manage and optimize its external business practices through the sharing of data with external parties such as suppliers, customers and investors, and through on-line transactions between the enterprise and external parties. Informational database systems (systems that store data, support query processing on the stored data, and possibly support analysis of the stored data) play a central role in many different parts of today's IT systems.
FIG. 1 illustrates exemplary domains where informational database systems are used. As shown, an operational environment 10 generates data which is stored in a data store 22 in the informational database system 20. These domains include data analysis systems (spread-sheet modeling programs, snap-shots, extraction, denormalization), data warehousing, data marts, OLAP systems, data mining systems, electronic commerce-enabled web servers, and business-to-business exchanges. Modem informational database systems typically use a relational database management system (RDBMS) as a repository for storing the data and querying the data.
FIG. 2 illustrates a data warehouse-OLAP domain that utilizes the prior art approaches described above. The data warehouse is an enterprise-wide data store. It is becoming an integral part of many information delivery systems because it provides a single, central location where a reconciled version of data extracted from a wide variety of operational systems is stored. Details on methods of data integration and constructing data warehouses can be found in the white paper entitled “Data Integration: The Warehouse Foundation” by Louis Rollleigh and Joe Thomas, published at http://www.acxiom.com/whitepapers/wP-11.asp. Building a Data Warehouse has its own special challenges (e.g. using common data model, common business dictionary, etc.) and is a complex endeavor. However, just having a Data Warehouse does not provide organizations with the often-heralded business benefits of data warehousing. To complete the supply chain from transactional systems to decision maker, organizations need to deliver systems that allow knowledge workers to make strategic and tactical decisions based on the information stored in these warehouses. These decision support systems are referred to as On-Line Analytical Processing (OLAP) systems. Such OLAP systems are commonly classified as Relation OLAP systems or Multi-Dimensional OLAP systems.
The Relational OLAP (ROLAP) system accesses data stored in a Data Warehouse to provide OLAP analyses. The premise of ROLAP is that OLAP capabilities are best provided directly against the relational database, i.e. the Data Warehouse. The ROLAP architecture was invented to enable direct access of data from Data Warehouses, and therefore support optimization techniques to meet batch window requirements and provide fast response times. Typically, these optimization techniques include application-level table partitioning, pre-aggregate inferencing, denormalization support, and the joining of multiple fact tables.
A typical ROLAP system has a three-tier or layer client/server architecture. The “database layer” utilizes relational databases for data storage, access, and retrieval processes. The “application logic layer” is the ROLAP engine which executes the multidimensional reports from multiple users. The ROLAP engine integrates with a variety of “presentation layers,” through which users perform OLAP analyses. After the data model for the data warehouse is defined, data from on-line transaction-processing (OLTP) systems is loaded into the relational database management system (RDBMS). If required by the data model, database routines are run to pre-aggregate the data within the RDBMS. Indices are then created to optimize query access times. End users submit multidimensional analyses to the ROLAP engine, which then dynamically transforms the requests into SQL execution plans. The SQL execution plans are submitted to the relational database for processing, the relational query results are cross-tabulated, and a multidimensional result data set is returned to the end user. ROLAP is a fully dynamic architecture capable of utilizing pre-calculated results when they are available, or dynamically generating results from the raw information when necessary.
The Multidimensional OLAP (MOLAP) systems utilize a MDD or “cube” to provide OLAP analyses. The main premise of this architecture is that data must be stored multidimensionally to be accessed and viewed multidimensionally. Such non-relational MDD data structures typically can be queried by users to enable the users to “slice and dice” the aggregated data. As shown in FIG. 2, such MOLAP systems have an Aggregation module which is responsible for all data storage, access, and retrieval processes, including data aggregation (i.e. pre-aggregation) in the MDDB, and an analytical processing and GUI module responsible for interfacing with a user to provide analytical analysis, query input, and reporting of query results to the user.
A more detailed description of the data warehouse and OLAP environment may be found in copending U.S. patent application Ser. No. 09/514,611 to R. Bakalash, G. Shaked, and J. Caspi, commonly assigned to HyperRoll Israel, Limited, incorporated by reference above in its entirety.
In a RDBMS, users view data stored in tables. By contrast, users of a non-relation database system can view other data structures, either instead of or in addition to the tables of the RDBMS system. FIG. 3A illustrates an exemplary table in an RDBMS; and FIGS. 3B and 3C illustrate operators (queries) on the table of FIG. 3A, and the result of such queries, respectively. The operators illustrated in FIGS. 3B and 3C are expressed as Structured Query Language (SQL) statements as is conventional in the art.
The choice of using a RDBMS as the data repository in information database systems naturally stems from the realities of SQL standardization, the wealth of RDBMS-related tools, and readily available expertise in RDBMS systems. However, the querying component of RDBMS technology suffers from performance and optimization problems stemming from the very nature of the relational data model. More specifically, during query processing, the relational data model requires a mechanism that locates the raw data elements that match the query. Moreover, to support queries that involve aggregation operations, such aggregation operations must be performed over the raw data elements that match the query. For large multidimensional databases, a naive implementation of these operations involves computational intensive table scans that leads to unacceptable query response times.
In order to better understand how the prior art has approached this problem, it will be helpful to briefly describe the relational database model. According to the relational database model, a relational database is represented by a logical schema and tables that implement the schema. The logical schema is represented by a set of templates that define one or more dimensions (entities) and attributes associated with a given dimension. The attributes associated with a given dimension includes one or more attributes that distinguish it from every other dimension in the database (a dimension identifier). Relationships amongst dimensions are formed by joining attributes. The data structure that represents the set of templates and relations of the logical schema is typically referred to as a catalog or dictionary. Note that the logical schema represents the relational organization of the database, but does not hold any fact data per se. This fact data is stored in tables that implement the logical schema.
Star schemas are frequently used to represent the logical structure of a relational database. The basic premise of star schemas is that information can be classified into two groups: facts and dimensions. Facts are the core data elements being analyzed. For example, units of individual item sold are facts, while dimensions are attributes about the facts. For example, dimensions are the product types purchased and the data purchase. Business questions against this schema are asked looking up specific facts (UNITS) through a set of dimensions (MARKETS, PRODUCTS, PERIOD). The central fact table is typically much larger than any of its dimension tables.
An exemplary star schema is illustrated in FIG. 4A for suppliers (the “Supplier” dimension) and parts (the “Parts” dimension) over time periods (the “Time-Period” dimension). It includes a central fact table “Supplied-Parts” that relates to multiple dimensions—the “Supplier”, “Parts” and “Time-Period” dimensions. FIG. 4B illustrates the tables used to implement the star schema of FIG. 4A. More specifically, these tables include a central fact table and a dimension table for each dimension in the logical schema of FIG. 4A. A given dimension table stores rows (instances) of the dimension defined in the logical schema. For the sake of description, FIG. 4B illustrates the dimension table for the “Time-Period” dimension only. Similar dimension tables for the “Supplier” and “Part” dimensions (not shown) are also included in such an implementation. Each row within the central fact table includes a multi-part key associated with a set of facts (in this example, a number representing a quantity). The multipart key of a given row (values stored in the S#, P#, TP# fields as shown) points to rows (instances) stored in the dimension tables described above. A more detailed description of star schemas and the tables used to implement star schemas may be found in C. J. Date, “An Introduction to Database Systems,” Seventh Edition, Addison-Wesley, 2000, pp. 711-715, herein incorporated by reference in its entirety.
When processing a query, the tables that implement the schema are accessed to retrieve the facts that match the query. For example, in a star schema implementation as described above, the facts are retrieved from the central fact table and/or the dimension tables. Locating the facts that match a given query involves one or more join operations. Moreover, to support queries that involve aggregation operations, such aggregation operations must be performed over the facts that match the query. For large multi-dimensional databases, a naive implementation of these operations involves computational intensive table scans that typically leads to unacceptable query response times. Moreover, since the fact tables are pre-summarized and aggregated along business dimensions, these tables tend to be very large. This point becomes an important consideration of the performance issues associated with star schemas. A more detailed discussion of the performance issues (and proposed approaches that address such issues) related to joining and aggregation of star schema is now set forth.
The first performance issue arises from computationally intensive table scans that are performed by a naive implementation of data joining. Indexing schemes may be used to bypass these scans when performing joining operations. Such schemes include B-tree indexing, inverted list indexing and aggregate indexing. A more detailed description of such indexing schemes can be found in “The Art of Indexing”, Dynamic Information Systems Corporation, October 1999, available at http://www.disc.com/artindex.pdf. All of these indexing schemes replaces table scan operations (involved in locating the data elements that match a query) with one ore more index lookup operation. Inverted list indexing associates an index with a group of data elements, and stores (at a location identified by the index) a group of pointers to the associated data elements. During query processing, in the event that the query matches the index, the pointers stored in the index are used to retrieve the corresponding data elements pointed therefrom. Aggregation indexing integrates an aggregation index with an inverted list index to provide pointers to raw data elements that require aggregation, thereby providing for dynamic summarization of the raw data elements that match the user-submitted query.
These indexing schemes are intended to improve join operations by replacing table scan operations with one or more index lookup operation in order to locate the data elements that match a query. However, these indexing schemes suffer from various performance issues as follows:
Since the tables in the star schema design typically contain the entire hierarchy of attributes (e.g. in a PERIOD dimension, this hierarchy could be day>week>month>quarter>year), a multipart key of day, week, month, quarter, year has to be created; thus, multiple meta-data definitions are required (one of each key component) to define a single relationship; this adds to the design complexity, and sluggishness in performance.
Addition or deletion of levels in the hierarchy will require physical modification of the fact table, which is time consuming process that limits flexibility.
Carrying all the segments of the compound dimensional key in the fact table increases the size of the index, thus impacting both performance and scalability.
Another performance issue arises from dimension tables that contain multiple hierarchies. In such cases, the dimensional table often includes a level of hierarchy indicator for every record. Every retrieval from fact table that stores details and aggregates must use the indicator to obtain the correct result, which impacts performance. The best alternative to using the level indicator is the snowflake schema. In this schema aggregate tables are created separately from the detail tables. In addition to the main fact tables, snowflake schema contains separate fact tables for each level of aggregation. Notably, the snowflake schema is even more complicated than a star schema, and often requires multiple SQL statements to get the results that are required.
Another performance issue arises from the pairwise join problem. Traditional RDBMS engines are not design for the rich set of complex queries that are issued against a star schema. The need to retrieve related information from several tables in a single query—“join processing”—is severely limited. Many RDBMSs can join only two tables at a time. If a complex join involves more than two tables, the RDBMS needs to break the query into a series of pairwise joins. Selecting the order of these joins has a dramatic performance impact. There are optimizers that spend a lot of CPU cycles to find the best order in which to execute those joins. Unfortunately, because the number of combinations to be evaluated grows exponentially with the number of tables being joined, the problem of selecting the best order of pairwise joins rarely can be solved in a reasonable amount of time.
Moreover, because the number of combinations is often too large, optimizers limit the selection on the basis of a criterion of directly related tables. In a star schema, the fact table is the only table directly related to most other tables, meaning that the fact table is a natural candidate for the first pairwise join. Unfortunately, the fact table is the very largest table in the query, so this strategy leads to selecting a pairwise join order that generates a very large intermediate result set, severely affecting query performance.
This is an optimization strategy, typically referred to as Cartesian Joins, that lessens the performance impact of the pairwise join problem by allowing joining of unrelated tables. The join to the fact table, which is the largest one, is deferred until the very end, thus reducing the size of intermediate result sets. In a join of two unrelated tables every combination of the two tables' rows is produced, a Cartesian product. Such a Cartesian product improves query performance. However, this strategy is viable only if the Cartesian product of dimension rows selected is much smaller than the number of rows in the fact table. The multiplicative nature of the Cartesian join makes the optimization helpful only for relatively small databases.
In addition, systems that exploit hardware and software parallelism have been developed that lessens the performance issues set forth above. Parallelism can help reduce the execution time of a single query (speed-up), or handle additional work without degrading execution time (scale-up).). For example, Red Brick™ has developed STARjoin™ technology that provides high speed, parallelizable multi-table joins in a single pass, thus allowing more than two tables can be joined in a single operation. The core technology is an innovative approach to indexing that accelerates multiple joins. Unfortunately, parallelism can only reduce, not eliminate, the performance degradation issues related to the star schema.
One of the most fundamental principles of the multidimensional database is the idea of aggregation. The most common aggregation is called a roll-up aggregation. This type is relatively easy to compute: e.g. taking daily sales totals and rolling them up into a monthly sales table. The more difficult are analytical calculations, the aggregation of Boolean and comparative operators. However these are also considered as a subset of aggregation.
In a star schema, the results of aggregation are summary tables. Typically, summary tables are generated by database administrators who attempt to anticipate the data aggregations that the users will request, and then pre-build such tables. In such systems, when processing a user-generated query that involves aggregation operations, the pre-built aggregated data that matches the query is retrieved from the summary tables (if such data exists). FIGS. 5A and 5B illustrate a multi-dimensional relational database using a star schema and summary tables. In this example, the summary tables are generated over the “time” dimension storing aggregated data for “month”, “quarter” and “year” time periods as shown in FIG. 5B. Summary tables are in essence additional fact tables, of higher levels. They are attached to the basic fact table creating a snowflake extension of the star schema There are hierarchies among summary tables because users at different levels of management require different levels of summarization. Choosing the level of aggregation is accomplished via the “drill-down” feature.
Summary tables containing pre-aggregated results typically provide for improved query response time with respect to on-the-fly aggregation. However, summary tables suffer from some disadvantages:
summary tables require that database administrators anticipate the data aggregation operations that users will require; this is a difficult task in large multi-dimensional databases (for example, in data warehouses and data mining systems), where users always need to query in new ways looking for new information and patterns.
summary tables do not provide a mechanism that allows efficient drill down to view the raw data that makes up the summary table—typically a table scan of one or more large tables is required.
querying is delayed until pre-aggregation is completed.
there is a heavy time overhead because the vast majority of the generated information remains unvisited.
there is a need to synchronize the summary tables before the use.
the degree of viable parallelism is limited because the subsequent levels of summary tables must be performed in pipeline, due to their hierarchies.
for very large databases, this option is not valid because of time and storage space.
Note that it is common to utilize both pre-aggregated results and on-the-fly aggregation in support aggregation. In these system, partial pre-aggregation of the facts results in a small set of summary tables. On-the-fly aggregation is used in the case the required aggregated data does not exist in the summary tables.
Note that in the event that the aggregated data does not exist in the summary tables, table join operations and aggregation operations are performed over the raw facts in order to generate such aggregated data. This is typically referred to as on-the-fly aggregation. In such instances, aggregation indexing is used to mitigate the performance of multiple data joins associated with dynamic aggregation of the raw data Thus, in large multi-dimensional databases, such dynamic aggregation may lead to unacceptable query response times.
In view of the problems associated with joining and aggregation within RDBMS, prior art ROLAP systems have suffered from essentially the same shortcomings and drawbacks of their underlying RDBMS.
While prior art MOLAP systems provide for improved access time to aggregated data within their underlying MDD structures, and have performance advantages when carrying out joining and aggregations operations, prior art MOLAP architectures have suffered from a number of shortcomings and drawbacks which Applicants have detailed in their copending U.S. application Ser. Nos. 09/368,241 and 09/514,611 incorporated herein by reference.
In summary, such shortcomings and drawbacks stem from the fact that there is unidirectional data flow from the RDBMS to the MOLAP system. More specifically, atomic (raw) data is moved, in a single transfer, to the MOLAP system for aggregation, analysis and querying. Importantly, the aggregation results are external to the RDBMS. Thus, users of the RDBMS cannot directly view these results. Such results are accessible only from the MOLAP system. Because the MDD query processing logic in prior art MOLAP systems is separate from that of the RDBMS, users must procure rights to access to the MOLAP system and be instructed (and be careful to conform to such instructions) to access the MDD (or the RDBMS) under certain conditions. Such requirements can present security issues, highly undesirable for system administration. Satisfying such requirements is a costly and logistically cumbersome process. As a result, the widespread applicability of MOLAP systems has been limited.
Thus, there is a great need in the art for an improved mechanism for joining and aggregating data elements within a relational database management system, and for integrating the improved relational database management system into informational database systems (including the data warehouse and OLAP domains), while avoiding the shortcomings and drawbacks of prior art systems and methodologies.
SUMMARY AND OBJECTS OF PRESENT INVENTION
Accordingly, it is an object of the present invention to provide an improved method of and system for joining and aggregating data elements integrated within a relational database management system (RDBMS) using a non-relational multi-dimensional data structure (MDD), achieving a significant increase in system performance (e.g. deceased access/search time), user flexibility and ease of use.
Another object of the present invention is to provide such an RDBMS wherein its integrated data aggregation module supports high-performance aggregation (i.e. data roll-up) processes to maximize query performance of large data volumes.
Another object of the present invention is to provide such an RDBMS system, wherein its integrated data aggregation (i.e. roll-up) module speeds up the aggregation process by orders of magnitude, enabling larger database analysis by lowering the aggregation times.
Another object of the present invention is to provide such a novel RDBMS system for use in OLAP operations.
Another object of the present invention is to provide a novel RDBMS system having an integrated aggregation module that carries out an novel rollup (i.e. down-up) and spread down (i.e. top-down) aggregation algorithms.
Another object of the present invention is to provide a novel RDBMS system having an integrated aggregation module that carries out full pre-aggregation and/or “on-the-fly” aggregation processes.
Another object of the present invention is to provide a novel RDBMS system having an integrated aggregation module which is capable of supporting a MDD having a multi-hierarchy dimensionality.
These and other object of the present invention will become apparent hereinafter and in the claims to Invention set forth herein.
BRIEF DESCRIPTION OF THE DRAWINGS
In order to more fully appreciate the objects of the present invention, the following Detailed Description of the Illustrative Embodiments should be read in conjunction with the accompanying Drawings, wherein:
FIG. 1A is a schematic representation of a prior art information database system, wherein the present invention may be embodied.
FIG. 2 is a schematic representation of the prior art data warehouse and OLAP system, wherein the present invention may be embodied.
FIGS. 3A-3C are schematic representations of exemplary tables employed in a prior art Relational Database Management System (RDBMS); FIGS. 3B and 3C illustrate operators (queries) on the table of FIG. 3A, and the result of such queries, respectively.
FIG. 4A is a schematic representation of an exemplary dimensional schema (star schema) of a relational database.
FIG. 4B is a schematic representation of tables used to implement the schema shown in FIG. 4A.
FIG. 5A is a schematic representation of an exemplary multidimensional schema (star schema).
FIG. 5B is a schematic representation of tables used to implement the schema of FIG. 5A, including summary tables storing results of aggregation operations performed on the facts of the central fact table along the time-period dimension, in accordance with conventional teachings.
FIG. 6A is a schematic representation of a generalized embodiment of an RDBMS of the present invention comprising a relational database having an integrated multidimensional (MDD) aggregation module supporting queries from a plurality of clients, wherein the aggregation engine performs aggregation functions (e.g. summation of numbers, as well as other mathematical operations, such as multiplication, subtraction, division etc.) and non-relational multi-dimensional data storage functions.
FIG. 6B is a schematic block diagram of the MDD aggregation module of the illustrative embodiment of the present invention shown in FIG. 6A.
FIGS. 6C(i) and 6C(ii), taken together, set forth a flow chart representation of the primary operations carried out within the RDBMS of the present invention when performing data aggregation and related support operations, including the servicing of user-submitted (e.g. natural language) queries made on such aggregated database of the present invention.
FIG. 6D is a flow chart representation of the primary operations carried out by the (DB) request serving mechanism within the MDD control module shown in FIG. 6B.
FIG. 6E is a schematic representation of the view mechanism of the RDBMS that enables users to query on the aggregated data generated and/or stored in the MDD Aggregation module according to the present invention.
FIG. 6F is a schematic representation of the trigger mechanism of the RDBMS that enables users to query on the aggregated data generated and/or stored in the MDD Aggregation module according to the present invention.
FIG. 7A shows a separate-platform implementation of the RDBMS system of the illustrative embodiment shown in FIG. 6A, wherein the query handling, fact table(s) and dictionary of the RDBMS resides on a separate hardware platform and/or OS system from that used to run the MDD Aggregation Module of the present invention.
FIG. 7B shows a common-platform implementation of the RDBMS system of the illustrative embodiment shown in Fig. A, wherein the query handling, fact table(s) and dictionary of the RDBMS shares the same hardware platform and operating system (OS) that used to run the MDD Aggregation Module of the present invention.
FIG. 8A is a data table setting forth information representative of performance benchmarks obtained by the shared-platform type implementation of the MDD Aggregation Module of the illustrative embodiment serving the conventional OLAP server (i.e. Oracle EXPRESS Server, wherein the common hardware/software platform is realized using a Pentium II 450 Mhz, 1 GB RAM, 18 GB Disk, running the Microsoft NT operating system (OS);
FIG. 9A is a schematic representation of the first stage in the method of segmented aggregation according to the principles of the present invention, showing initial aggregation along the 1st dimension.
FIG. 9B is a schematic representation of the next stage in the method of segmented aggregation according to the principles of the present invention, showing that any segment along dimension 1, such as the shown slice, can be separately aggregated along the remaining dimensions, 2 and 3, and that in general, for an N dimensional system, the second stage involves aggregation in N−1 dimensions. The principle of segmentation can be applied on the first stage as well, however, only a large enough data will justify such a sliced procedure in the first dimension. Actually, it is possible to consider each segment as an N−1 cube, enabling recursive computation.
FIG. 9C1 is a schematic representation of the Query Directed Roll-up (QDR) aggregation method/procedure of the present invention, showing data aggregation starting from existing basic data or previously aggregated data in the first dimension (D1), and such aggregated data being utilized as a basis for QDR aggregation along the second dimension (D2).
FIG. 9C2 is a schematic representation of the Query Directed Roll-up (QDR) aggregation method/procedure of the present invention, showing initial data aggregation starting from existing previously aggregated data in the second third (D3), and continuing along the third dimension (D3), and thereafter continuing aggregation along the second dimension (D2).
FIG. 10A is a schematic representation of the “slice-storage” method of storing sparse data in the disk storage devices of the MDDB of FIG. 6B in accordance with the principles of the present invention, based on an ascending-ordered index along aggregation direction, enabling fast retrieval of data.
FIG. 10B is a schematic representation of the data organization of data files and the directory file used in the storage of the MDDB of FIG. 6B, and the method of searching for a queried data point therein using a simple binary search technique due to the data files ascending order;
FIG. 11A is a schematic representation of three exemplary multi-hierarchical data structures for storage of data within the MDDB of FIG. 6B, having three levels of hierarchy, wherein the first level representative of base data is composed of items A, B, F, and G, the second level is composed of items C, E, H and I, and the third level is composed of a single item D, which is common to all three hierarchical structures.
FIG. 11B is a schematic representation of an optimized multi-hierarchical data structure merged from all three hierarchies of FIG. 11A, in accordance with the principles of the present invention.
FIG. 12 is a schematic representation showing the levels of operations performed by the stand-alone Aggregation Server of FIG. 6B, summarizing the different enabling components for carrying out the method of segmented aggregation in accordance with the principles of the present invention.
FIG. 13 is a schematic representation of the RDBMS of the present invention shown as a component of a central data warehouse, serving the data storage and aggregation needs of a ROLAP system (or other OLAP system).