WO2001006413A1 - System for accessing database tables mapped into memory for high performance data retrieval - Google Patents
System for accessing database tables mapped into memory for high performance data retrieval Download PDFInfo
- Publication number
- WO2001006413A1 WO2001006413A1 PCT/US1999/016769 US9916769W WO0106413A1 WO 2001006413 A1 WO2001006413 A1 WO 2001006413A1 US 9916769 W US9916769 W US 9916769W WO 0106413 A1 WO0106413 A1 WO 0106413A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- database
- information
- file
- recited
- access
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24539—Query rewriting; Transformation using cached or materialised query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24547—Optimisations to support specific applications; Extensibility of optimisers
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99931—Database or file accessing
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99941—Database schema or data structure
- Y10S707/99943—Generating database or data structure, e.g. via user interface
Definitions
- a microfiche appendix having 3 microfiche and 216 frames is included herewith and includes a detailed design specification of the present invention.
- the present invention is directed to a system for enhancing application processing performance involving read-only access to database tables. More particularly, to a system to enhance execution of complex business transaction applications, such as those performed in telecommunications billing systems, that requires quick access to data stored in a database by placing a portion of the data into RAM memory using a master file and an incremental file.
- SUBSTITUTE SHEET (RULE 25) It is also an object of the present invention to provide a method of accessing a database by placing information extracted from the database into a first data structure in memory. Then the present invention updates the information in memory periodically or when requested when a change in the information occurs in the database by placing the updated information into a second data structure in memory which is linked to the first data structure.
- the above objects can be attained by a system and method that partially or fully extracts data from a database and places the data in RAM for access by application programs. Further, the foregoing objects can be attained by a system and method that updates data in RAM (random access memory), transparent to the application programs, whenever the data changes in the database.
- RAM random access memory
- Figure 2 is an overall hardware and software layer configuration of the present invention.
- Figure 3 is a data flow diagram of the present invention.
- Figure 4 is a flowchart of the algorithm to create full extractions in the present invention.
- Figure 5 is a diagram depicting an example of the full and incremental extraction of data in the present invention.
- Figure 6 is a flowchart of the algorithm used to create incremental extractions in the present invention.
- Figure 7 is a flowchart of the algorithm used to access incremental extractions in the present invention.
- Figure 8 is a flowchart of the algorithm used to advance to the next valid entry in the present invention.
- Figure 9 is a flowchart of the algorithm used to skip invalid entries in the present invention.
- Figure 10 is an example identifying incremental information using a database table and a trigger table in the present invention.
- Figure 11 further illustrates the example shown in figure 10.
- Figure 1 is an overall modular configuration of the present invention illustrating the major modular components thereof.
- Xref Framework 10 has three fundamental modules including: Xref Storage Manager 20 (XSM 20); ExtractManager 30; and the Xref Interface 40. Each of these modules will be discussed in detail below.
- the XSM 20 module is used to extract the information stored in database 60 and transfer it to Xref files 50.
- Xref files 50 comprise several standard flat files.
- a flat file is sequential file of a single record type with no linked structures or direct record access capability.
- Each Xref file 50 contains only entries (possibly converted) returned by a system query language ("SQL") select statement as configured for the database 60.
- SQL system query language
- Application 70 and application 80 are notified when full extracts or updates are available by XSM 20.
- Applications 70 and 80 utilize the high performance access to data provided by the Xref Framework 10 for the specific business process that they perform. An example of such a business process where performance is essential is rating and billing of large telecommunication customer accounts.
- XSM 20 is able to extract incremental updates from database 60 and place them into separate update files which are part of the Xref Files 50. These incremental updates contain information for rows in the database 60 that have been changed, added or deleted since the last
- SUBSTITUTE SHEET (RULE 25) full extraction of the database 60.
- the incremental extraction is needed because full extraction is a slow process and it would introduce a performance problem to force the applications 70 and 80 using the Xref Interface 40 to reload a new version of the full extracted Xref files 50 for each new extraction.
- the applications 70 and 80 are notified as soon as an incremental update is available by XSM 20 sending a message to the Xref interface 40.
- ExtractManager 30 is a module that sends messages to the XSM 20 and tells XSM 20 which database 60 table(s) to extract and whether to perform a full or incremental extraction.
- the ExtractManager 30 is run periodically to send a message to XSM 20 requesting XSM 20 to create incremental extracts. This ensures that applications 70 and 80 using extracted data, are provided with updates at periodic intervals.
- Xref Interface 40 is the part of the framework used by applications to manage and access data stored in
- the Xref Interface 40 manages the mapping of Xref files 50 to memory and handles access of incremental updates, transparent to applications 70 and 80. Memory (not shown) storing the Xref data extracted from database 60 is shared by applications 70 and 80 to improve performance and save resources.
- SUBST ⁇ UTE SHEET (RULE 25) as methods for stepping through some or all entries.
- the queries supported are: searching for a certain entries using a key; searching for the best matching entry using a key; searching using a key and an effective date. Also supported is stepping through some or all entries in the order provided in the Xref Files 50 starting at the entry returned by a query with respect to incremental updates and effective date.
- Figure 2 is an overall hardware and software layer configuration of the present invention. As shown in Figure 2, the present invention is implemented in a distributed client server system format comprising an application server 100 and a database server
- the application server 100 is implemented using application code 102 (C++ programming language) running on a HP-UNIX 110 platform.
- the application server 100 also uses an Oracle database 106 to store persistent objects.
- XSM 20, ExtractManager 30 and applications 70 and 80 using the Xref Interface 40 reside on the application server 100.
- the Xref Framework 10 uses the Oracle client libraries to access the database server 120.
- the Xref Framework 10 components communicate using an asynchronous message queuing system, see 130 figure 3, provided by the infrastructure 104. The de- coupling of the Xref components using queues allows distributed deployment and flexible configuration of the system.
- the database server 120 hosts the Oracle database 116 running on a HP JNIX 110 platform.
- the Xref Framework 10 uses Oracle database 114 triggers to identify data to be included in incremental updates.
- FIG. 3 is a data flow diagram of the present invention.
- the XSM 20 is started and sends a broadcast message to all applications
- Applications 70 and 80 announcing its availability via queue 130.
- Applications 70 and 80 send a registration message to XSM 20 via Xref Interface 40 and message queuing system 130 to indicate which database tables 62 they want to use.
- the ExtractManager 30 is started and sends a message using the message queuing system 130 to XSM 20 to generate a full extraction of one or multiple tables 62 in database 60.
- Database 60 contains table 62 and trigger table 64.
- XSM 20 When XSM 20 receives the message from the message queuing system 130 to perform a full extraction, XSM 20 executes a query to retrieve all data from table 62 and writes the information to the Xref master file 56. When the extraction process has finished, XSM 20 sends a message using the message queuing system 130, to all registered applications 70 and 80 interested in the extracted table, to announce that a full extraction is ready for use.
- Application 70 uses the Xref Interface 40 to load the data into shared memory 150 from master file 56 and also to access the data through the Xref Interface 40 from shared memory 150. If application 80 is interested in the same data as used by application 70 and both processes reside on the same application server 100, as shown in Figure 2, the Xref Interface 40 will share the memory 150 already used by Application
- SUBST ⁇ UTE SHEET (RULE 25) 70 The sharing of memory saves important resources and improves performance since the data only needs to be loaded to memory once. Still referring to Figure 3, application 90 modifies the data in the table 62 that was extracted earlier. This change in table 62 causes database triggers 114, shown in Figure 2, to be executed in database
- the database triggers 114 then copy the modified data to the trigger table 64 and mark it with a time stamp, sequence number and a type code for the modification.
- the ExtractManager 30 is executed periodically and sends a message using the message queuing system 130 instructing XSM 20 to create an incremental extraction.
- XSM 20 scans the corresponding trigger table 64 for rows that have not been included in the last full extraction. This is determined using the time stamp on each row.
- XSM 20 uses this data to create an incremental file 52 and an index file 54.
- XSM 20 then sends a notification message to all registered applications 70 and 80 using the message queuing system 130, announcing that a new incremental update is available.
- the Xref interface 40 is used by application 70 and 80 to load the incremental file 52 and the index file 54 into shared memory 150.
- Applications 70 and 80 are now provided with the updated information. This processing is transparent to the application logic of application 70 and 80. Once either application 70 or 80 terminate or are no longer interested in Xref data, application 70 or 80 send a message to XSM 20 to using the message queuing system 130. XSM 20 will not send further update messages to applications 70 or 80.
- the XSM 20 provides the capability of creating full database extractions and saving the extracted information to a Xref master file 56.
- a SQL seiect statement can be created for the extraction process. This provides the ability to do a partial or a full database extraction based on business needs.
- Creating a full extraction of a large table is processor intensive. Full table extractions need to be created in two situations. First, upon start up of the system, the Xref files 50 must be created. Second, when incremental update information reaches such a volume that the application accessing the data is noticeably slowed.
- the rows in the trigger table 64 can either be deleted or kept to provide the history of modifications applied to the table 60. in the case where the rows in the trigger table 64 are kept, the rows need to be marked so that they are not included in the next incremental extract. This marking can either be achieved by adding an additional column to the trigger table 64 in order to mark rows or to use a time stamp on each row to identify the updates that were performed after the last full extract took place. The description that is provided herein utilizes a time stamp for this purpose.
- Figure 4 is a flowchart of the algorithm used to create full extractions in the present invention. Once a request for a full database extraction is received, the extraction process starts by initiating XSM 20, shown in Figure 1 , in operation 1000 of Figure 4. In
- XSM 20 starts by retrieving 1010 the SQL select statement
- SUBSTITUTE SHEET (RULE 25) configured for the table to be extracted. This provides the flexibility to only extract the data required for the business process and helps to save resources.
- XSM 20 now extracts all the data in the database table using the created SQL select statement in operation 1030. If other applications are allowed to update the table while the extraction is in process, it is possible that the trigger table 64 will contain entries that are also contained in the full extraction. These additional entries are filtered out when generating an incremental extraction.
- the extracted data is converted into structures in memory 150 and data conversion is performed if necessary.
- the data structures are the data containers in memory 150 where the data is accessed. This provides a mechanism to have different data representations in applications using the database 60 and applications 70 and 80 using the present invention.
- Xref Interface 40 will only need to reload the updated information. As disk access is slow relative to RAM memory access, this is a major performance advantage.
- XSM 20 is able to extract incremental updates into separate files 52, 54.
- the incremental updates contain information for rows in the table 62 that have been changed, added or deleted since the last full extraction of the table 62.
- the Xref Interface 40 provides a completely transparent interface to the incremental information and an application 70 or 80 using a Xref table will not know if it is receiving an entry from the full extraction or from the incremental extraction. In this way incremental updates can be switched on or off for individual database tables 60 without having any impact on the applications 70 or 80 using those tables 60.
- the master file 56 and the incremental file 54 can be merged periodically. This keeps the size of the incremental file small, which in turn increases the access performance and avoids the need to perform performance intensive full extractions.
- This implementation uses trigger table 64 for this purpose.
- the trigger table 64 contains the complete row data, a field used to distinguish between insertions, updates and deletions and a sequence number that is needed to keep the database operations in a consistent order.
- the database trigger (not shown) provides the necessary logic for generating the sequence number.
- XSM 20 uses the primary key, see 641 in figure 10 and 561 in figure 5, to link entries in the trigger table 64 to entries in the master file 56 when performing an incremental extract.
- FIG. 10 depicts an example where table 62, shown in Figure 3, contains business data and trigger table 64, shown in figure
- SUBSTITUTE SHEET (RULE 25) incremental updates.
- table 62 is initially filled with three data rows 624, 625 and 626.
- the trigger table 64 is initially empty.
- the table 62 contains effective dated data. This means that for a particular primary key multiple entries can exist over time. For processing only the entry active on the processing date is considered.
- the table 62 comprises three items, First, the primary key column 621 used to identify logical objects. Second, the effective date column 622 used to specify the date when a particular entry is active. Third, the data column 623 that contains the business data contained in the row.
- the trigger table 64 consists of six columns.
- the first three columns 641 , 642 and 643 replicate the data of columns 621 , 622 and 623 in table 62.
- the update type 644 coiumn is used to distinguish between insertions, updates and deletions.
- the sequence number column 645 is needed to keep database operation in a consistent order.
- the time stamp column 646 is used to determine whether trigger table entries have been included in the last full extract when performing a full extract. Starting with this data scenario shown in figure 10 a full extract is created of the table 62.
- the master file 56 contains the data as depicted in figure 5, discussed in detail below.
- the master file 56 shown in figure 5, comprises the three fields 561 , 563, 565 containing the data of columns 621 , 622 and 623 in table 62.
- one or multiple applications 90 as shown in figure 3, modify table 62 containing the
- SUBSTITUTE SHEET (RULE 25) business data. Each modification fires database triggers that write the information about the modification to the trigger table 64.
- a row 627 with a primary key of 2 is inserted. This event is marked in the trigger table as row 649 with the sequence number 5002 in figure 11. 4.
- a new effective dated entry 628 is inserted for the row with the primary key 3. This event is marked in the trigger table as row 650 with the sequence number 5003 in figure 11. 5.
- a row 629 with a primary key of 4 gets inserted. This event gets marked in the trigger table as row 651 with the sequence number 5004 in figure 11.
- SUBSTITUTE SHEET (RULE 25) 6.
- the row 626 with a primary key of 5 is deleted. This event is marked in the trigger table as row 652 with the sequence number 5005 in figure 11.
- the incremental file 52 and the index file 54 contain the data as depicted in figure 5.
- the incremental file 52 layout contains the three fields 521 , 523, 525 containing the data of the columns 641 , 642 and 643 in trigger table 64.
- XSM 20 uses the trigger table 64 to identify the rows in the database 60 that have been altered since the last full extraction.
- the ExtractManager 30 is executed periodically and sends a message to XSM 20 requesting a new incremental extract using the message queuing system 130. This ensures that applications are provided with up-to-date data without continuously accessing the database.
- XSM 20 only reads the rows that have not been included in the last fuil extract, this is determined using the time stamp on each row.
- the entries are ordered by primary key and sequence number. By sorting according to primary key and sequence number, all entries with the same primary key are grouped together and all but the last one (with the largest sequence-number) are disregarded. Only the entry for a primary key with the largest sequence number is used, as discussed above, since all other entries represent outdated
- Incremental information is stored in a separate file called the incremental file 52 shown in figures 3 and 5.
- the reason for use of a separate file is it is not possible to extend the master file without forcing all applications 70 and 80 using it to first unload the old version and then load the new version which would have a negative performance impact on the applications.
- the incremental file 52 shown in Figure 5 only contains the inserted/updated rows (deleted rows are managed using the index file), and additional information needed for quick access of incremental data.
- the incremental file 52 is created by the XSM 20 shown in Figure 3.
- the first is the update type field 527 that contains the update type code which can be either 'U'(pdated), 'l'(nserted) or 'D'(eleted).
- the XSM 20 searches the master file 56 for the updated entry to determine how the row should be stored in the incremental file 52.
- the update type 527 is transformed as provided in the following table.
- the second field in the incremental file 52 is the index field 529 that is an index to predecessor row in the master file 56.
- the index is the row number in the master file 56 for the updated entry.
- the index field 529 is the row number for the last entry in the master file 56 that has a primary key 561 with a value less than the inserted row (i.e. the row 'preceding' the inserted one). It is possible for several entries in the incremental file 52 to have the same index 529 in the case that they are all successors to the same entry in the master file 56.
- the index file 54 provides information about deleted rows and a reference to quickly find inserted or updated entries in the incremental file 52 from the master file 56.
- Each row in the master file 56 has a corresponding row in the index file 54.
- the index file 54 has three fields: deleted 545; updated 541 , and successor 543.
- the deleted 545 field is a fiag indicating if the
- SUBSTITUTE SHEET (RULE 25) corresponding row in the master file 56 has been deleted.
- the updated 541 field is an index to the incremental file 52.
- a row in the master file 56 has been updated, the corresponding row in the index file 54 will have the update 541 field set to the index of the corresponding entry in the incremental file 52 for the updated version of the row.
- the successor field 543 is used in the case when an insert has been made. Upon an insertion, the successor field 543 will then contain the index of the first inserted successor in the incremental file 52 of the corresponding row in the master file 56.
- this can be either a later effective date version for the same primary key 561 or a row with a primary key value that is larger than the value for the primary key of the row in the master file 56.
- a value -1 is used in the successor 543 field of the index file 54 for non-valid entries (i.e., to indicate that the row has not been updated or does not have any successors).
- the first row in the master file 56 has been updated which is the first row in the incremental file 52.
- the first row in the master file 56 also has a successor which is the second row in the incremental file 52.
- the successor has a primary key 561 value that is greater than the value for the row in the master file 56.
- the second row in the master file 56 has two successors as shown in the incremental file 52.
- the first one is an effective-date-successor that has the same primary key 561 equal to the value 3 as the row in the master file 56 but a different effective date 563 of 1/3/98 as shown in the incremental file 52.
- the second successor in the incremental file 52 has a primary key 521
- SUBST ⁇ SHEET (RULE 25) equal to the value 4 which is greater than the row in the master file 56. These two successors share the same predecessor entry in the index file 54 and they therefore will have the same index value of 1. The third entry in the master file 56 has been deleted as indicated in the index file 54 column 545.
- Figure 6 is a flowchart of the algorithm used to create incremental extractions in the present invention.
- the algorithm shown in Figure 6 is used by XSM 20 to create an incremental extraction from database 60.
- This incremental extraction combined with the previously extracted master file 56, is used to provide applications 70 and 80 using the Xref Interface 40 with the most up- to-date information.
- the XSM 20, shown in figure 3, is started in order to execute an incremental table extraction in operation 1110.
- the XSM 20 starts by retrieving the SQL select statement configured for table 62, shown in Figure 3, to be extracted. This provides the flexibility to only extract the data required for the business process and help to reduce resources required.
- the XSM 20 extracts the data in the trigger table 64, that has not been included in the last full extract, using the configured SQL select statement from operation 1 120. The time stamp on each row is used to determine whether the row has been included in the last full extract.
- the extracted data is converted into structures in memory and data conversion is performed if necessary. This provides a mechanism to have different data representations in applications using the database and applications using the present
- SUBST ⁇ SHEET (RULE 25) invention Data conversion or custom order requirements can make it necessary to sort the data at this point and this determination that sorting is required is made in operation 1145. If sorting is required this is done in operation 1150. This also allows applications 70 and 80 to use different primary keys or ordering rules than those defined in the database.
- XSM 20 compares the sorted entries to entries in the master file 56, shown in Figure 3. In this operation it is determined whether trigger table 64 entries represent update, insert or delete operations. If multiple entries for one primary key exist in trigger tabie 64, only the entry with the highest sequence number is considered. Only the entry for a primary key with the largest sequence number is used since all other entries represent outdated entries that have been superceded by updated with higher sequence numbers.
- XSM 20 determines the index 529 for each entry in incremental file 52 and maintains a list for all index 529 entries.
- the index 529 information is now written to index file 54.
- the sorted and filtered data is now written to the incremental file 52.
- XSM 20 sends a message to all registered applications 70 and 80 announcing the availability of a new incremental update information.
- the incremental extraction process terminates.
- XSM 20 When XSM 20 is done extracting the incremental information in operation 1180, it sends a message to applications 70 and 80 that have registered as Xref users that a new incremental file 52 is now available. The applications 70 and 80 then load the new incremental
- Figure 7 is a flowchart of the algorithm used to access incremental extractions in the present invention.
- the algorithm shown in Figure 7 is executed by the Xref Interface 40, shown in Figure 3, and is activated when an application uses the Xref Interface 40 to look up data entries. It is transparent to applications 70 and 80 whether the data returned is stored in the master file 56 or incremental file 52.
- the algorithm accepts a full primary key or partial primary key as a search parameter.
- Xref Interface 40 is activated to find an entry in operation 1300.
- the Xref interface is provided with a search key to perform the operation.
- Xref Interface 40 starts by performing a binary search for the search key in the master file 56 in operation 1310.
- the binary search returns the closest matching entry with a key value smaller than the search key. If the binary search operation 1310 determines that several matching entries exist, the first one is returned.
- the search key is smaller than the primary key 561 of the first entry in the master file 56, NULL is returned. If no entry was found by the binary search, then in operation 1340, the first entry of the incremental file 52 is used as the current entry.
- SUBST ⁇ SHEET (RULE 25) updated or deleted.
- the Xref interface 40 reads the next record until the next valid entry in the either the incremental file 52 or master file 56 is reached.
- a valid entry refers to an entry that is up-to-date and has not been updated or deleted. This is also discussed in the description of the skip invalid entries algorithm provided below.
- operation 1360 if the primary key of the current entry has a value smaller than the search key, then processing proceeds to operation 1370.
- the next logical successor of the current entry is taken from the master file 56 or incremental file 52. This is also discussed in the algorithm to advance to next valid entry described below.
- Xref interface 40 compares the primary key 561 of the current entry to determine if it matches the search key. If the two keys do not match, then, in operation 1420, a NULL value is returned and processing terminates in operation 1430. If the keys match, then in operation 1380 Xref interface 40 checks whether the extracted table contains effective dated data. Effective dated data can have multiple entries for the same primary key over time. Only one entry is active at one particular point in time. If the data is effective dated the algorithm has to find the entry active on the processing date. In operation 1410, if the data is not effective dated, the current entry is returned as the search result and processing terminates in operation 1430. If the data is effective dated, Xref
- SUBST ⁇ SHEET (RULE 25) interface 40 advances until an entry which is active on the search date is found in operation 1390. If such an entry is found in operation 1400, Xref interface 40 returns the entry in operation 1410. If no entry is found then in operation 1400, then in operation 1420 a NULL value is returned.
- Figure 8 is a flowchart of the algorithm used to advance to the next valid entry in the present invention.
- the algorithm of Figure 8 takes a master file 56 or incremental file 52 entry as a parameter and returns the next valid entry.
- Valid entries refer to entries that are up- to-date. Entries that have been updated or deleted are not considered valid entries.
- a master file 56 entry that has been updated is superceded by an entry in the incremental file and is therefore not valid.
- the next valid entry can either be located in the master file 56 or the incremental file 52.
- Figure 8 further details operation 1370 shown in figure 6.
- Operation 1500 advances to the next valid entry in master file 56 or incremental file 52.
- a determination is made whether the entry is a master file 56 entry or a incremental file 52 entry. If the entry is a master file 56 entry, then, in operation 1520, the index file 52 entry is used to determine whether a successor exist for the entry. If successors exist, then processing returns the first successor from the incremental file 52 in operation 1540. If no successors exist, then the next entry is retrieved from the master file 56, in operation 1550. In operation 1570, invalid entries are skipped, as further discussed below. In operation 1530, if the entry is an incremental file 52 entry, a determination is made whether the next
- SUBST ⁇ SHEET (RULE 25) entry in the incremental file 52 has the same predecessor index as the current entry. If the index is not the same, then the next entry from master file 56 is retrieved. Operation 1570 skips invalid entries as further described below. In operation 1560, If the index is the same, the next entry is returned in the incremental file 52.
- Figure 9 is a flowchart of the algorithm used to skip invalid entries in the present invention and serves to further detail operation 1370 shown in figure 6.
- the flowchart of Figure 9 further details the operation of operation 1570 shown in Figure 8 and discussed above.
- the algorithm shown in figure 9 takes a master file 56 entry as a parameter and skips the invalid entries starting at the current entry.
- Invalid entry refers to an entry that is not up-to-date. Invalid entries have been updated or deleted.
- a master file 56 entry for which a update exists is not valid or up-to-date.
- the next valid entry can either be located in the master file 56 or the incremental file 52. If the current entry is a valid entry the algorithm does not skip any entries.
- Figure 9 further details skip processing shown in operation 1350 of figure 7 and operation 1570 of figure 8.
- the index file 54 entry is examined corresponding to the master file 56 entry to determine if the entry has been updated. If an update entry exists, then in operation 1670 the update entry is returned from the incremental file 52. If no update exist, then, in operation 1620, the index file 54 entry is examined to determine if the entry has been deleted. In operation 1660, if the entry has not been deleted, it is a valid entry and is returned by the algorithm. In operation 1630, if the entry was deleted, the index file 54 entry is examined to determine
- SUBST ⁇ SHEET (RULE 25) whether successors exist for the entry. This is indicated by the successor field 543 in the index file 54.
- the first successor is returned from the incremental file 52.
- processing terminates in operation 1680.
- no successors are found to exist in operation 1630, then, in operation 1640, the next entry is retrieved from the master file 56 and continues processing until a valid entry is found.
- NULL is returned.
- any suitable operating system may be used.
- UNIX may also be used as well as any comparable operating system. Almost any processor or computer may be used such as a Sun computer.
Abstract
Description
Claims
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CA002302303A CA2302303A1 (en) | 1999-07-15 | 1999-07-26 | System for accessing database tables mapped into memory for high performance |
EP99937435A EP1204938A1 (en) | 1999-07-15 | 1999-07-26 | System for accessing database tables mapped into memory for high performance data retrieval |
AU52274/99A AU5227499A (en) | 1999-07-15 | 1999-07-26 | System for accessing database tables mapped into memory for high performance data retrieval |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/353,395 | 1999-07-15 | ||
US09/353,395 US6463439B1 (en) | 1999-07-15 | 1999-07-15 | System for accessing database tables mapped into memory for high performance data retrieval |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2001006413A1 true WO2001006413A1 (en) | 2001-01-25 |
Family
ID=23388909
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US1999/016769 WO2001006413A1 (en) | 1999-07-15 | 1999-07-26 | System for accessing database tables mapped into memory for high performance data retrieval |
Country Status (5)
Country | Link |
---|---|
US (1) | US6463439B1 (en) |
EP (1) | EP1204938A1 (en) |
AU (1) | AU5227499A (en) |
CA (1) | CA2302303A1 (en) |
WO (1) | WO2001006413A1 (en) |
Families Citing this family (51)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7181478B1 (en) * | 2000-08-11 | 2007-02-20 | General Electric Company | Method and system for exporting flight data for long term storage |
US6988095B1 (en) * | 2001-05-07 | 2006-01-17 | Hearst Business Communications, Inc. | Systems and methods for performing parametric searches |
US6687702B2 (en) * | 2001-06-15 | 2004-02-03 | Sybass, Inc. | Methodology providing high-speed shared memory access between database middle tier and database server |
JP3695581B2 (en) * | 2001-08-08 | 2005-09-14 | ソニー株式会社 | Recording apparatus, recording method, recording medium, and electronic camera |
US8099393B2 (en) | 2002-03-22 | 2012-01-17 | Oracle International Corporation | Transaction in memory object store |
US7206793B2 (en) * | 2002-05-22 | 2007-04-17 | Sbc Properties L.P. | Generalized file matching method and system |
US20040010524A1 (en) * | 2002-07-12 | 2004-01-15 | Wallace Michael W. | Efficient method and system for delivering resources in broadcast environment |
US7251683B1 (en) * | 2002-10-25 | 2007-07-31 | Sandeep Shah | Information handling system including arrangements for initiating an application in response to usage of cross reference between information and for initiating usage of a workflow flow chart associated with and information work |
US7430569B2 (en) * | 2002-11-27 | 2008-09-30 | Sap Ag | Computerized replication of data objects |
US7464091B2 (en) * | 2002-11-27 | 2008-12-09 | Sap Ag | Method and software for processing data objects in business applications |
US7225302B2 (en) * | 2002-11-27 | 2007-05-29 | Sap Ag | Method and software application for avoiding data loss |
US7409412B2 (en) | 2002-11-27 | 2008-08-05 | Sap Ag | Data element and structure for data processing |
US7149752B2 (en) | 2002-12-03 | 2006-12-12 | Jp Morgan Chase Bank | Method for simplifying databinding in application programs |
US7085759B2 (en) | 2002-12-06 | 2006-08-01 | Jpmorgan Chase Bank | System and method for communicating data to a process |
US8032439B2 (en) | 2003-01-07 | 2011-10-04 | Jpmorgan Chase Bank, N.A. | System and method for process scheduling |
US7165072B2 (en) * | 2003-03-26 | 2007-01-16 | Taiwan Semiconductor Manufacturing Company, Ltd. | Method for merging information from effective dated base tables |
US7379998B2 (en) | 2003-03-31 | 2008-05-27 | Jp Morgan Chase Bank | System and method for multi-platform queue queries |
US7366722B2 (en) | 2003-05-15 | 2008-04-29 | Jp Morgan Chase Bank | System and method for specifying application services and distributing them across multiple processors using XML |
US7509641B2 (en) | 2003-05-16 | 2009-03-24 | Jp Morgan Chase Bank | Job processing framework |
WO2004114160A2 (en) * | 2003-06-13 | 2004-12-29 | Equifax, Inc. | Systems and processes for automated criteria and attribute generation, searching, auditing and reporting of data |
US11132183B2 (en) | 2003-08-27 | 2021-09-28 | Equifax Inc. | Software development platform for testing and modifying decision algorithms |
US20070179827A1 (en) | 2003-08-27 | 2007-08-02 | Sandeep Gupta | Application processing and decision systems and processes |
US7424467B2 (en) | 2004-01-26 | 2008-09-09 | International Business Machines Corporation | Architecture for an indexer with fixed width sort and variable width sort |
US7293005B2 (en) * | 2004-01-26 | 2007-11-06 | International Business Machines Corporation | Pipelined architecture for global analysis and index building |
US8296304B2 (en) * | 2004-01-26 | 2012-10-23 | International Business Machines Corporation | Method, system, and program for handling redirects in a search engine |
US7499913B2 (en) | 2004-01-26 | 2009-03-03 | International Business Machines Corporation | Method for handling anchor text |
JP2005214779A (en) * | 2004-01-29 | 2005-08-11 | Xanavi Informatics Corp | Navigation system and method for updating map data |
CN100461164C (en) * | 2004-03-29 | 2009-02-11 | 微软公司 | Systems and methods for versioning based triggers |
CA2563354C (en) | 2004-04-26 | 2010-08-17 | Jp Morgan Chase Bank | System and method for routing messages |
US20060095413A1 (en) * | 2004-05-07 | 2006-05-04 | Lombardi Software, Inc. | Method of exposing normalized data as denormalized tables |
US7392471B1 (en) | 2004-07-28 | 2008-06-24 | Jp Morgan Chase Bank | System and method for comparing extensible markup language (XML) documents |
US7461064B2 (en) * | 2004-09-24 | 2008-12-02 | International Buiness Machines Corporation | Method for searching documents for ranges of numeric values |
US7792839B2 (en) * | 2005-01-13 | 2010-09-07 | International Business Machines Corporation | Incremental indexing of a database table in a database |
JP4675174B2 (en) * | 2005-07-12 | 2011-04-20 | 株式会社日立製作所 | Database processing method, system and program |
US8417693B2 (en) | 2005-07-14 | 2013-04-09 | International Business Machines Corporation | Enforcing native access control to indexed documents |
US7499933B1 (en) | 2005-11-12 | 2009-03-03 | Jpmorgan Chase Bank, N.A. | System and method for managing enterprise application configuration |
US20070124345A1 (en) * | 2005-11-30 | 2007-05-31 | Heinz Kathy K | Library backup |
US7610172B2 (en) | 2006-06-16 | 2009-10-27 | Jpmorgan Chase Bank, N.A. | Method and system for monitoring non-occurring events |
US8313005B2 (en) * | 2006-08-03 | 2012-11-20 | Kraft Foods Global Brands, Llc | Plastic coffee container with pinch grip |
US8417731B2 (en) * | 2006-12-28 | 2013-04-09 | Sap Ag | Article utilizing a generic update module with recursive calls identify, reformat the update parameters into the identified database table structure |
EP2750098A3 (en) * | 2007-02-16 | 2014-08-06 | BodyMedia, Inc. | Systems and methods for understanding and applying the physiological and contextual life patterns of an individual or set of individuals |
US20080201290A1 (en) * | 2007-02-16 | 2008-08-21 | International Business Machines Corporation | Computer-implemented methods, systems, and computer program products for enhanced batch mode processing of a relational database |
WO2009021011A2 (en) * | 2007-08-07 | 2009-02-12 | Equifax, Inc. | Systems and methods for managing statistical expressions |
US8584140B2 (en) * | 2007-09-21 | 2013-11-12 | Presenceid, Inc. | Systems and methods for receiving and sending messages about changes to data attributes |
JP5012628B2 (en) * | 2008-04-04 | 2012-08-29 | 日本電気株式会社 | Memory database, memory database system, and memory database update method |
CN103136315A (en) * | 2011-11-22 | 2013-06-05 | 中兴通讯股份有限公司 | Storage processing method and storage processing device for call detail record (CDR) data |
CN105224527B (en) * | 2014-05-27 | 2018-07-13 | 北京宸瑞科技股份有限公司 | General ETL methods suitable for a variety of purpose table update modes |
US10417212B1 (en) * | 2015-01-15 | 2019-09-17 | Qad, Inc. | Database modification replay |
CN110019181B (en) * | 2017-08-10 | 2024-04-16 | 中兴通讯股份有限公司 | Warehouse-in method and device computer readable storage medium |
CN116719971B (en) * | 2023-05-08 | 2024-04-09 | 中银金融科技有限公司 | Pull chain table data loading method and device and electronic equipment |
US11899636B1 (en) * | 2023-07-13 | 2024-02-13 | Fmr Llc | Capturing and maintaining a timeline of data changes in a relational database system |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5603024A (en) * | 1993-10-08 | 1997-02-11 | International Business Machines Corporation | Lossless distribution of time series data in a relational data base network |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5664177A (en) * | 1988-04-13 | 1997-09-02 | Digital Equipment Corporation | Data processing system having a data structure with a single, simple primitive |
US5729730A (en) * | 1995-03-28 | 1998-03-17 | Dex Information Systems, Inc. | Method and apparatus for improved information storage and retrieval system |
US5740423A (en) * | 1995-12-28 | 1998-04-14 | Csg Systems, Inc. | System and method for accessing distributed data on a plurality of databases |
US6208993B1 (en) * | 1996-07-26 | 2001-03-27 | Ori Software Development Ltd. | Method for organizing directories |
US5797008A (en) * | 1996-08-09 | 1998-08-18 | Digital Equipment Corporation | Memory storing an integrated index of database records |
US5974407A (en) * | 1997-09-29 | 1999-10-26 | Sacks; Jerome E. | Method and apparatus for implementing a hierarchical database management system (HDBMS) using a relational database management system (RDBMS) as the implementing apparatus |
US6061678A (en) * | 1997-10-31 | 2000-05-09 | Oracle Corporation | Approach for managing access to large objects in database systems using large object indexes |
-
1999
- 1999-07-15 US US09/353,395 patent/US6463439B1/en not_active Expired - Lifetime
- 1999-07-26 AU AU52274/99A patent/AU5227499A/en not_active Abandoned
- 1999-07-26 WO PCT/US1999/016769 patent/WO2001006413A1/en not_active Application Discontinuation
- 1999-07-26 CA CA002302303A patent/CA2302303A1/en not_active Abandoned
- 1999-07-26 EP EP99937435A patent/EP1204938A1/en not_active Withdrawn
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5603024A (en) * | 1993-10-08 | 1997-02-11 | International Business Machines Corporation | Lossless distribution of time series data in a relational data base network |
Non-Patent Citations (3)
Title |
---|
ADIBA M E ET AL: "DATABASE SNAPSHOTS", IBM RESEARCH REPORT,US,SAN JOSE, CA, 7 March 1980 (1980-03-07), pages 1 - 16, XP000747603 * |
LINDSAY B ET AL: "A snapshot differential refresh algorithm", PROCEEDINGS OF ACM SIGMOD '86. INTERNATIONAL CONFERENCE ON MANAGEMENT OF DATA, WASHINGTON, DC, USA, 28-30 MAY 1986, vol. 15, no. 2, SIGMOD Record, June 1986, USA, pages 53 - 60, XP002134864, ISSN: 0163-5808 * |
SEVERANCE D G ET AL: "Differential files: their application to the maintenance of large databases", ACM TRANSACTIONS ON DATABASE SYSTEMS, SEPT. 1976, USA, vol. 1, no. 3, pages 256 - 267, XP002134863, ISSN: 0362-5915 * |
Also Published As
Publication number | Publication date |
---|---|
CA2302303A1 (en) | 2001-01-15 |
US6463439B1 (en) | 2002-10-08 |
AU5227499A (en) | 2001-02-05 |
EP1204938A1 (en) | 2002-05-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6463439B1 (en) | System for accessing database tables mapped into memory for high performance data retrieval | |
US6035303A (en) | Object management system for digital libraries | |
US5802524A (en) | Method and product for integrating an object-based search engine with a parametrically archived database | |
EP2264617B1 (en) | Method and apparatus for replicating a database | |
US6058389A (en) | Apparatus and method for message queuing in a database system | |
US6349310B1 (en) | Database management system and method for accessing rows in a partitioned table | |
JP3848085B2 (en) | High-speed storage resident processing method and processing system for transaction data | |
JP4866495B2 (en) | Method and apparatus for executing combined queries in a database system | |
US4914569A (en) | Method for concurrent record access, insertion, deletion and alteration using an index tree | |
US6339772B1 (en) | System and method for performing database operations on a continuous stream of tuples | |
US6240422B1 (en) | Object to relational database mapping infrastructure in a customer care and billing system | |
US6453313B1 (en) | Database management system and method for dequeuing rows published to a database table | |
US6438562B1 (en) | Parallel index maintenance | |
US7103589B1 (en) | Method and system for searching, accessing and updating databases | |
US6397227B1 (en) | Database management system and method for updating specified tuple fields upon transaction rollback | |
US20020095421A1 (en) | Methods of organizing data and processing queries in a database system, and database system and software product for implementing such methods | |
US7246124B2 (en) | Methods of encoding and combining integer lists in a computer system, and computer software product for implementing such methods | |
US20020095397A1 (en) | Method of processing queries in a database system, and database system and software product for implementing such method | |
US20060041606A1 (en) | Indexing system for a computer file store | |
JP5844333B2 (en) | Sustainable data storage technology | |
EP0663641A1 (en) | Summary catalogs | |
US7499920B2 (en) | Multi-column multi-data type internationalized sort extension method for web applications | |
US20100274795A1 (en) | Method and system for implementing a composite database | |
JPH08235040A (en) | Data file management system | |
JP2675958B2 (en) | Information retrieval computer system and method of operating storage device thereof |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
ENP | Entry into the national phase |
Ref document number: 2302303 Country of ref document: CA |
|
WWE | Wipo information: entry into national phase |
Ref document number: 1999937435 Country of ref document: EP |
|
AK | Designated states |
Kind code of ref document: A1 Designated state(s): AL AM AT AU AZ BA BB BG BR BY CA CH CN CU CZ DE DK EE ES FI GB GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MD MG MK MN MW MX NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT UA UG US UZ VN YU ZW |
|
AL | Designated countries for regional patents |
Kind code of ref document: A1 Designated state(s): GH GM KE LS MW SD SL SZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG |
|
DFPE | Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101) | ||
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
REG | Reference to national code |
Ref country code: DE Ref legal event code: 8642 |
|
WWP | Wipo information: published in national office |
Ref document number: 1999937435 Country of ref document: EP |
|
WWW | Wipo information: withdrawn in national office |
Ref document number: 1999937435 Country of ref document: EP |