|Publication number||US7263520 B2|
|Application number||US 10/789,426|
|Publication date||Aug 28, 2007|
|Filing date||Feb 27, 2004|
|Priority date||Feb 27, 2004|
|Also published as||US7650326, US20050192941, US20080016322|
|Publication number||10789426, 789426, US 7263520 B2, US 7263520B2, US-B2-7263520, US7263520 B2, US7263520B2|
|Inventors||Stefan Biedenstein, Jens-Peter Dittrich, Erich Marschall, Olaf Meincke, Klaus Nagel, Guenter Radestock, Andrew Ross, Stefan Unnebrink|
|Original Assignee||Sap Ag|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (10), Non-Patent Citations (10), Referenced by (6), Classifications (9), Legal Events (4)|
|External Links: USPTO, USPTO Assignment, Espacenet|
The following description relates to information management systems.
An information management system may include an information retrieval system and/or a database management system. An information management system can include a computer system and one or more databases, each of which is a collection of tables representing classes of physical or conceptual objects. Each object is represented by a record. The information contained in a record may be organized by multiple attributes. For example, if a database were used to keep track of employees in a corporation, each record might represent an employee and each record might include attributes such as a first name, last name, home address, and telephone number.
Relational database management systems are commonly employed to manage relational databases. A relational database is a database that includes one or more tables, where each table is organized by columns and rows. In such a database, each column is an attribute and each row is a record.
There are different ways to view the data in a database. One type of view is known as a multidimensional view. In a multidimensional view, each fact table has several dimensions such that each attribute of a table represents a dimension. Relational databases can be used to generate a multidimensional view of data. One use case for accessing data and performing operations on a database, when using a multidimensional view, is known as online analytical processing (OLAP). In accordance with OLAP, there are many database operations that manipulate data in response to a query, and many of these operations may need to aggregate data in order to generate a result. In relational databases, aggregating data is the process of summarizing a table by selecting columns and representing (e.g. by summing, averaging, or finding the maximum or minimum) the key figures for similar attribute values in each of the selected columns to generate a view with a “coarser granularity.” Aggregation may require a table scan. A table scan is the process of reading through a table record by record. Because aggregation may require access to many values in a database, aggregation can be a time and resource consumptive process. Also, because database operations may require a lot of memory, operations tend to have several accesses to a storage device of a computer system while performing the operation, rather than storing a table in a memory of the computer system.
In relational OLAP (ROLAP), the speed of aggregation queries may be increased by having frequently used aggregates precalculated and stored in the database. In multidimensional OLAP (MOLAP), the speed of aggregation queries may be increased by having all the aggregates precalculated and stored in special data structures. Precalculating aggregates tends to reduce response times for any queries that involve the precalculated aggregates; however, the ROLAP solution is advantageous only for those aggregates that are precalculated, and the MOLAP solution is computationally expensive.
The present disclosure includes systems and techniques relating to an information management system and methods of performing aggregation by table scans. The systems described here, and corresponding techniques for use, may include various combinations of the following features.
In one general aspect, the techniques feature a method of aggregating data in an information management system. The method includes receiving a query for a response to a search on a database, loading data from the database into a memory if the data necessary to generate the response to the query is absent from the memory, filtering the data based on the query to generate a list of results, buffering at least one key figure corresponding to a result in the list of results, buffering at least one dimension value corresponding to each key figure, aggregating the dimension values to generate an aggregate key, aggregating the key figures corresponding to the same aggregate key to generate one or more aggregate key figures, and displaying the response to the search on a display device. In that case, the response to the search includes at least one aggregate key figure.
Implementations may include one or more of the following features. The method may further include generating a hash key based on the aggregate key and storing in a hash table aggregate key figures corresponding to that hash key. Loading data from the database into a memory may include compressing data according to a compression algorithm. In that case, the compression algorithm may be dictionary-based compression. Loading data from the database may include loading data into memory. Loading data from the database into a memory may include organizing the data in the memory as columns of the database. Aggregating the dimension values may include concatenating the dimension values. Filtering the data based on the query may be performed blockwise.
In another aspect, an information management system includes a database and a computer system. In that case the computer system is programmed to load data from the database into a memory, where the data represents a table; filter the data based on a query, which includes generating a list of results, buffer at least one key figure corresponding to a result in the list of results; buffer at least one dimension value corresponding to each key figure; generate an aggregate key based on the dimension values; aggregate key figures with the same aggregate key to generate one or more aggregate key figures; and, display at least one aggregate key figure on a display device.
Implementations may include one or more of the following features. The computer system may be further programmed to generate a hash key based on the aggregate key and store in a hash table aggregate key figures corresponding to the hash key. The operation of loading data from the database into a memory may include compressing data according to a compression algorithm. In that case, the compression algorithm may be dictionary-based compression. The operation of loading data from the database may include loading data into memory. The operation of filtering the data based on the query may be performed blockwise. The operation of loading data from the database into a memory may include organizing the data in the memory as columns of the database. The operation of generating an aggregate key may include concatenating the dimension values.
Implementations of the systems and techniques described here may occur in hardware, firmware, software or a combination of these, and may include instructions for causing a machine to perform the operations described.
The system and method of performing table scans in memory, and related mechanisms and/or techniques described here may provide one or more of the following advantages. Aggregates may be generated each time a query is submitted by performing a table scan. Performing a table scan each time a query is submitted is advantageous because any aggregate can then be generated as required. Performance tends to be increased because data from the database is stored in the memory of at least one computer system when the table scan is performed. The amount of memory required to perform a table scan may be reduced by storing compressed data in the memory.
Details of one or more implementations are set forth in the accompanying drawings and the description below. Other features and advantages may be apparent from the description and drawings, and from the claims.
These and other aspects will now be described in detail with reference to the following drawings.
Like reference numerals and designations in the drawings indicate like elements.
The systems and techniques described here relate to an information management system and to a method of performing aggregation by table scans in memory.
At 120, the data in the memory is filtered based on criteria in the query. Filtering uses criteria in the query to generate a list of records that are relevant. For example, a Structured Query Language (SQL).query may include a “where” condition that is used to select the relevant rows from a table, and filtering may be performed based on the “where” condition, such that results are limited to those records that meet the condition. The filtering may be performed blockwise (i.e. in blocks of N rows).
At 130, a block of N row identifiers (IDs) is loaded into a temporary memory location for hashing and aggregation. The value of N may be 1, but depending on the technical specification of the caching mechanism, N may advantageously be set to a value greater than 1, such as 128 or more. If the key figures for a row are still at their initial value (i.e. if they have not been maintained), the row need not be loaded into the cache, since it will have no effect on the aggregate key figures that are to be calculated. For example, a database may exist that organizes customer information by the attributes of customer key or customer identification (ID), time (in months), and the number of products purchased. Some customers in the database might not have purchased any products. In that case, the N block may include a list of row identifiers for records with only those customers who have purchased products. The N block may include row identifiers. Key figures are buffered for each record identified in the N block. The key figure columns contain the data values that are to be aggregated. In the prior example, the key figures may be the number of products purchased. Dimension values may be buffered next to each key figure so that a key figure is identifiable by nearby dimension values. The dimension values are keys for attributes or characteristics that belong together, in a record. For example, if a dimension represents customers, a dimension value may represent a particular customer and the attributes or characteristics that belong together may be customer name and address. In a record, a dimension value appears with one or more key figures. Each row identifier in the N block refers to a dimension value that is buffered. If dictionary-based compression is used, the dimension values that are buffered may be the dictionary references corresponding to the dimension values. All of the buffering should be handled in the memory of a computer system. In alternative implementations the data need not be processed blockwise, i.e. in blocks of N rows.
Aggregate keys are generated at 140. An aggregate key is generated for each combination of dimension values in the buffer. The aggregate keys may be generated by concatenating the dimension values. If dictionary-based compression is used, the aggregate keys may be a concatenation of the dictionary references. For example, if dictionary references 0122 and 06 are used to represent dimension values CompanyA and June, respectively, the aggregate key may be 012206.
At 150, a hash key is generated for each aggregate key and each hash key is written into a hash table. Generating a hash key involves the use of a hash function. In the case that dictionary-based compression is used, the length of the hash key may be set to n bits, where n is greater than or equal to the number of bits needed to represent the product of the cardinalities (i.e. the total number of different values for a dimension value) of the respective dimensions—i.e. 2n≧|D1×D2×. . . DY|, where each D is the cardinality of a dimension value. In any case, a suitable procedure must be available for handling hash key collisions.
For each hash key, key figures are aggregated at 160 during a table scan. A table scan is performed over the buffer where the key figures and respective dimension values are stored. The table scan involves the aggregation of the key figures with dimension values that correspond to the same aggregate key. If the table is divided among several computer systems, aggregate key figures may be aggregated locally and a separate server may merge the locally aggregated key figures. The aggregate key figures are entered into a hash table using the respective hash keys generated at 150. The hash table may then be used to generate a suitable display, including the results of the aggregation, at a user interface. The processes of 120-160 of
Filtering the data 215 generates the filter results 220. The filter results 220 are a list of row IDs referring to rows that are determined to be relevant based on a query. Filtering may use criteria in the query. For example, “where” conditions of a query may be used such that results are limited to those records which meet the filter condition specified.
The N block 225 is a cached part of the memory that stores row identifiers. The row identifiers correspond to the records in the filter results 220. The N block 225 excludes records that have key figures that are null (i.e. still set at their initial value). Because the N block stores row identifiers that correspond to the filter results 220, the N block is used as a reference when storing the dimension values 230 and the key figures 235 in the buffer 240.
The dimension values 230 and the key figures 235 respectively correspond to row IDs in the N block 225. For each row identifier in the N block, the corresponding key figure and dimension values are stored in the buffer 240. In
An aggregate key, such as the aggregate key 245, is generated for each combination of dimension values that exists in the buffer 240. Thus, two key figures that have the same dimension values would generate the same aggregate key. The aggregate key 245 may be generated by concatenating the corresponding dimension values. If dictionary-based compression is used, references that represent dimension values may be concatenated.
Each aggregate key is used to generate a hash key, such as the hash key 250. The hash key 250 may be any length and may be generated according to any function for generating hash keys. For example, the hash keys may be at least as long as the aggregate key 245, or the hash key can be shorter than the aggregate key 245. In the case that the hash key 250 is smaller than the aggregate key 245, collisions in the hash table 255 need to be handled. In the case that dictionary-based compression is used, the length of the hash key may be set to n bits, where n is greater than or equal to the number of bits needed to represent the product of the cardinalities (i.e. the total number of different values for a dimension value) of the respective dimensions—i.e. 2n≧|D1×D2×. . . DY|, where each D is the cardinality of a dimension value.
The hash table 255 is configured to be indexed by the hash key 250. The hash table 255 stores aggregate key figures. The aggregate key figures are generated by scanning the buffer 240 and aggregating all key figures with the same corresponding aggregate keys. The aggregate key figures are then stored in the hash table 255. From the hash table 255, the aggregate key figures can be accessed so they may be displayed on a display device. In exceptional scenarios where the aggregate keys are short and the result sets are small, a hashing procedure may not be necessary and the results can be aggregated using a simply array. However, in scenarios where the proposed method can scale to nontrivial use cases, hashing is likely to be required.
Although the method of aggregating data is shown in
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US5884299 *||Feb 6, 1997||Mar 16, 1999||Ncr Corporation||Optimization of SQL queries involving aggregate expressions using a plurality of local and global aggregation operations|
|US6629102 *||Jul 28, 2000||Sep 30, 2003||International Business Machines Corporation||Efficiently updating a key table during outline restructure of a multi-dimensional database|
|US6775682 *||Feb 26, 2002||Aug 10, 2004||Oracle International Corporation||Evaluation of rollups with distinct aggregates by using sequence of sorts and partitioning by measures|
|US20020077997 *||Mar 25, 1999||Jun 20, 2002||Informix Software, Inc., A California Corporation||Server integrated systems and methods for processing precomputed views|
|US20020099691 *||Jun 24, 1998||Jul 25, 2002||Michael Dean Lore||Method and apparatus for aggregation of data in a database management system|
|US20020152219 *||Apr 16, 2001||Oct 17, 2002||Singh Monmohan L.||Data interexchange protocol|
|US20030101194 *||Nov 1, 2001||May 29, 2003||Michael Rys||System and method for loading hierarchical data into relational database systems|
|US20040199535 *||Apr 4, 2003||Oct 7, 2004||Nir Zuk||Attack database structure|
|US20060010147 *||Sep 11, 2003||Jan 12, 2006||Patrick Arras||Method and system for online analytical processing (olap)|
|WO2004036456A2 *||Sep 11, 2003||Apr 29, 2004||International Business Machines Corporation||Method and system for online analytical processing (olap)|
|1||Joachims, "Text Categorization with Support Vector Machines: Learning with Many Relevant Features", Machine Learning: ECML-98, Proceedings of the 10<SUP>th </SUP>European Conference on Machine Learning, Apr. 1998, Chemnitz, Germany, pp. 137-142.|
|2||Joachims, "Text Categorization with Support Vector Machines: Learning with Many Relevant Features", Machine Learning: ECML-98, Proceedings of the 10th European Conference on Machine Learning, Apr. 1998, Chemnitz, Germany, pp. 137-142.|
|3||Joachims, Learning to Classify Text Using Support Vector Machines. Dissertation, Cornell University, Department of Computer Science, Kluwer Academic Publishers, May 2002.|
|4||Salton, et al., "A Vector Space Model for Automatic Indexing", Journal of the American Society for Information Science, vol. 18, No. 11, Nov. 1975, pp. 613-620.|
|5||Vivisimo(R) Clustering Engine, Clustering Engine, http://www.vivisimo.com, no date & month.|
|6||Vivisimo(R) Clustering Engine, Content Integrator, http://www.vivisimo.com, no date & month.|
|7||VivisimoŽ Clustering Engine, Clustering Engine, http://www.vivisimo.com, no date & month.|
|8||VivisimoŽ Clustering Engine, Content Integrator, http://www.vivisimo.com, no date & month.|
|9||Witten, et al., Managing Gigabytes, Compressing and Indexing Documents and Images, Second Edition, Morgan Kaufmann Publishers, San Francisco, California, USA, 1999, no months.|
|10||Yang, et al., "A re-examination of text categorization methods", Proceedings of ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR, 1999, pp. 42-49, no months.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US8019831||Apr 17, 2008||Sep 13, 2011||Microsoft Corporation||Adaptive buffering of database server response data|
|US8195645||Jul 23, 2008||Jun 5, 2012||International Business Machines Corporation||Optimized bulk computations in data warehouse environments|
|US8612285||Sep 26, 2007||Dec 17, 2013||Sap Ag||Unified access of key figure values|
|US20090083131 *||Sep 26, 2007||Mar 26, 2009||Sap Ag||Unified Access of Key Figure Values|
|US20090265398 *||Apr 17, 2008||Oct 22, 2009||Microsoft Corporation||Adaptive Buffering of Database Server Response Data|
|US20100023477 *||Jul 23, 2008||Jan 28, 2010||International Business Machines Corporation||Optimized bulk computations in data warehouse environments|
|U.S. Classification||707/3, 707/101|
|International Classification||G06F17/30, G06F7/00|
|Cooperative Classification||Y10S707/99933, Y10S707/99942, Y10S707/99932, G06F17/30489|
|Jul 8, 2004||AS||Assignment|
Owner name: SAP AKTIENGESELLSCHAFT, GERMANY
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BIEDENSTEIN, STEFAN;DITTRICH, JENS-PETER;MARSCHALL, ERICH;AND OTHERS;REEL/FRAME:014829/0498;SIGNING DATES FROM 20040609 TO 20040624
|Feb 23, 2011||FPAY||Fee payment|
Year of fee payment: 4
|Aug 26, 2014||AS||Assignment|
Owner name: SAP SE, GERMANY
Free format text: CHANGE OF NAME;ASSIGNOR:SAP AG;REEL/FRAME:033625/0334
Effective date: 20140707
|Jan 28, 2015||FPAY||Fee payment|
Year of fee payment: 8