« PreviousContinue »
Identify Resource Group having Closest Center of Activity
Relative to Computing Node (Minimum Network Distance)
Any Storage Location (e.g., Another Computing Node) Closer to
Center of Activity than Computing Node?
Computing Node Joins Resource Group, Receives Data Replicated from Prior Nodes Belonging to Resource Group
Divide Database (DA, DB) Among Divided Resource Groups with New Centers (CGA for DA, CGB for DB)
MOVEMENT OF DATA IN A DISTRIBUTED
DATABASE SYSTEM TO A STORAGE
LOCATION CLOSEST TO A CENTER OF
ACTIVITY FOR THE DATA
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention is related to optimizing storage of data in distributed databases. In particular, the present inven- 10 tion is related to moving data in distributed databases to storage locations that are closest to the client nodes that will perform the largest frequency of access for the related data.
2. Description of the Related Art
Distributed computing has evolved to technologies that 15 enable multiple devices to share in providing a given service. For example, grid computing has been proposed as a model for solving substantially large computational problems using large numbers of computers arranged as clusters embedded in a distributed infrastructure. Grid computing has been 20 described as differentiated from distributed computing through an increased focus on resource sharing, coordination, manageability, and high performance. The focus on resource sharing has been referred to as "the grid problem", which has been defined as the set of problems associated with resource 25 sharing among a set of individuals or groups.
One form of distributed computing involves distributed databases. Distributed databases have been attempted in a variety of implementations, where the central goal is to distribute data among multiple locations in order to provide 30 load-balancing. Databases are composed of tables of records, where each record (e.g., row) includes a number of elements. An element, referred to as a field or column (i.e., a column of a row includes an element), is the quantum of data that is stored (as an alphanumeric value) with other elements of the 35 record (row) based on an established relationship; hence, a record can be illustrated as a row, and each column has corresponding field of the record.
Conventional attempts in implementing distributed databases have involved database tables being split, where table 40 records (i.e., rows) are arbitrarily split across storage device boundaries. For example, a distributed database may store records 1-50 on a first storage device and records 51-100 on a second storage device, for example as implemented as early as the 1970s using the Honeycomb Drive technology devel- 45 oped by IBM. Hence, the database records are split according to a physical hierarchy of the storage devices: the database server is aware of the multiple storage devices storing the respective groups of database records, and accesses the appropriate storage device to obtain the database record as 50 required. Hence, databases were designed (i.e. structured) such that distribution of the contents of a database were split among a number of storage devices, such as the cylinder drives used by the Honeycomb Drive technology.
Another technique of splitting contents of databases, par- 55 ticularly relevant to relational databases, involves dividing tables and then distributing the tables as divided across different machines based on content. For example, if a database has a number of records that exceeds the capacity for a single storage device, or if the received database queries exceed the 60 capacity of a single database server machine, the database tables may be split among multiple database server machines, where the first machine has tables associated with a first attribute (e.g., employee identifier attributes), and the second machine has tables associated with a second attribute (e.g., 65 salary information). One of the columns will specify which storage device stores the associated data from that table,
enabling a machine executing a database server to locate the data associated with the record. Hence, a SQL query can be sent to multiple machines in order to reconstruct the requested data according to the SQL query. In addition, two tables can be created separately, and then joined at runtime in response to a SQL query.
In addition, distributed database technology includes optimizing queries (e.g., relational queries) requiring joining tables from different storage devices, where the processing of the query is optimized based on different portions of the query being processed on different machines, wherein those machines transfer their results of their respective portions of the query to another machine that compiles the respective results.
However, the necessity of performing queries by accessing tables on multiple remote storage devices, and forwarding the data to different machines, can result in costly procedures in terms of network bandwidth and latency, especially in the case of high-frequency queries that are performed repeatedly to access remote data. Moreover, as data records become more distributed based on storage across a larger number of respective storage locations, global queries suffer a decrease in performance due to the database server requiring to access each and every one of the storage locations in order to complete a global query, even if the global query specifies a substantially small subset of a data record, for example selecting all records where a specified element has a specific value (or range of values) specified in the global query.
SUMMARY OF THE INVENTION
Despite the numerous attempts at optimizing distributed database technology, there has been no attempt to date to optimize a distributed database such that the data records are moved for storage as close as possible to the consumers of the data records. In particular, it is highly desirable that data records that match a predominantly large number of data queries be stored in proximity to the database server device performing the queries on the data records. In other words, there is a need for data records to be moved as close as possible to their center of activity.
There also is a need for an arrangement that enables the data records to be moved toward a new storage location based on identifying a center of query activity for the data records at or near the new storage location.
There also is a need for an arrangement that enables database records to be replicated across multiple storage locations, enabling a database server to perform a database query based on accessing one of the multiple storage locations identified as being closest to the database server.
There also is a need for an arrangement that enables a new computing node, configured for providing distributed computing services, to join a resource group composed of other computing nodes participating in providing distributed services for resource group data, based on the new computing node determining that the resource group data has a center of query activity that is proximally located to the new computing node.
There also is a need for an arrangement that enables services to be provided in a network by multiple computing nodes, where data elements of a distributed database can be moved to a new storage location that is closer to a center of activity for the data, without the necessity of complex transfer and data integrity verification procedures that impose additional processing requirements on the system.
These and other needs are attained by the president invention, where a network provides distributed database services
based on replicating data records to a new storage location that is closest to consumers of the data records, based on comparing a network distance between the new storage location and an attribute identifying a center of activity of at least one data element, relative to the network distance between any existing storage location in the center of activity. If the new storage location is closer to the center of activity relative to any of the other existing stores locations, than at least the one data element associated with the center of activity is copied to the new storage location. Multiple centers of activity may be utilized for respective data records, enabling data records to be copied on a per-record basis, or grouped as needed.
Hence, data records are stored at identified storage locations based on network topology, and query attributes that identified the center of activity from which the record accesses are deemed to have originated. Consequently, data records can be copied to storage locations that are closest to the source of the data record requests, for example a remote database server having generated a substantially large percentage of the database queries for the data records associated with the attribute identifying the center of activity.
One aspect of the present invention provides a method in a network. The method includes storing a database of records at a first storage location according to a topology of the network. Each record has at least first and second elements. The database further includes at least one data attribute element that specifies a corresponding center of activity of at least one of the elements of the database. The method also includes determining whether a second storage location has a first network distance to the center of activity that is less than a second network distance of the first storage location to the center of activity. The at least one element associated with the center of activity is copied to the second storage location based on determining the corresponding first network distance is less than the second network distance.
Another aspect of the present invention provides a network having a plurality of storage locations and a plurality of database server resources. The storage locations are identifiable according to a topology of the network, and at least a first of the storage locations stores a database of records. Each record has at least first and second elements. The database also includes at least one data attribute element that specifies a corresponding center of activity of at least one of the elements of the database. The database server resources are at respective access locations in the topology, each configured for performing a database query on the database, wherein the center of activity is updated based on each access from the corresponding access location. Further, the at least one element associated with the center of activity is copied to a second of the storage locations, in response to a determination in the network that the second of the storage locations has a first network distance to the center of activity that is less than a second network distance of the first of the storage locations to the center of activity.
Still another aspect of the present invention provides a method in a computing node. The method includes identifying available resource groups within a network, each resource group having corresponding resource group attributes including a data attribute and a resource requirement for providing a corresponding distributed computing service within a network. The data attribute specifies at least one center of activity within a topology of the network for a corresponding collection of data elements associated with the distributed computing service. The method also includes identifying one of the available resource groups having the corresponding at least one center of activity that is closest to a network location of
the computing node. The method also includes selectively joining the one available resource group based on determining the corresponding resource requirement is less than an available node capacity, and based on a determined network
5 distance between the network location and the at least one center of activity.
Additional advantages and novel features of the invention will be set forth in part in the description which follows and in part will become apparent to those skilled in the art upon
l o examination of the following or may be learned by practice of the invention. The advantages of the present invention may be realized and attained by means of instrumentalities and combinations particularly pointed out in the appended claims.
15 BRIEF DESCRIPTION OF THE DRAWINGS
Reference is made to the attached drawings, wherein elements having the same reference numeral designations represent like elements throughout and wherein: 20 FIG. 1 is a diagram illustrating a distributed network having multiple computing nodes configured for providing distributed database services throughout multiple storage locations, according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating a database table having 25 records including data attributes for identifying access attributes of the respective records.
FIG. 3 is a diagram illustrating one of the computing nodes, according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating computing nodes of a 30 resource group migrating data based on arbitrating for dividing the resource group into divided resource groups, and arbitrating for merging resource groups into a merged resource group, according to an embodiment of the present invention.
35 FIG. 5 is a diagram illustrating the method of migrating data based on joining of resource groups based on a center of data activity, according to an embodiment of the present invention.
40 BEST MODE FOR CARRYING OUT THE
FIG. 1 is a diagram illustrating a network 10 having an identified topology, illustrated by a topology map 12, and
45 having multiple storage locations 14 and computing nodes 16 for distributed storage of database records, according to an embodiment of the present invention. As illustrated in FIG. 1, each computing node 16 has an associated storage location 14, either internal to the computing node 16 or externally
50 local to the computing node, as shown in FIG. 1. In addition, a mass storage device may be implemented within the network 10 independent of any computing node 16, as illustrated by the storage location 14/.
As described below, the topology 12 illustrates a cost
55 based multidimensional space that positions storage locations 14 and computing nodes 16 relative to cost-based dimensions that identify a "network distance" between different locations within the topology. For example, the nodes 16a, 16b, 16c, and 16d are positioned within the topology 12
60 based on their connectivity within a first local area network 18a of the network 10, and the nodes 16e, 16f, 16g, and 16/z are positioned within the topology 12 based on their connectivity within a second local area network 18b, the nodes 16d and 16e being interconnected via a wide area communica
65 tions link 20.
As described below, each storage location 14 has at least one corresponding center of activity 22 that identifies for the
associated records at the corresponding storage location 14 a logical origin of all access attempts to the associated records within the storage location 14. For example, the storage location 14d has a center of activity 22 that is located within the computing node 16c?, indicating that all access attempts for 5 the data records within the storage location 14d that are associated with that center of activity within the computing node 16d originate exclusively from the computing node 16c?. In contrast, the center of activity of the other storage locations 14a, 14b, 14c, 14e, 14/J 14g, 14h, 14/ are located at different 10 locations throughout the topology 12 based on an aggregation of requests having been received from various computing nodes throughout the network.
According to the disclosed embodiment, data records are replicated to a storage location 14 that is closest to the corre- 15 sponding center of activity 22 associated with the data records. As described below, query attributes are used to describe database queries that are performed on the data records within each of the storage locations 14, including identifying a logical center 22 of the query activity for the 20 data records relative to all queries that have been generated from different query servers (e.g., computing nodes 16) at respective locations.
Hence, the data records stored at the storage location 14h may be replicated to the storage location 14c, based on the 25 center of activity 22 of the storage location 14h having crossed a site boundary 26 into the site 24 encompassing the storage location 14c. Consequently, the network distance (i.e., vector) 44 between the storage location 14 (e.g., 14h) and the center of activity 22 can be reduced based on repli- 30 eating the data into a closer storage location 14c.
The disclosed embodiment is particularly effective among distributed servers in an enterprise-class network, for example a large-scale network deployed at a university or a multi-building office campus. In other words, a computing 35 node (e.g., 16a) within a particular site 24, defined for example as a collection of network nodes residing within prescribed cost boundaries 26 of the network 10, will execute a database server resource that executes a set of prescribed queries on a regular basis on a subset of the records of the 40 entire distributed database. For example, a multi-office corporation may employ an enterprise system composed of local area networks (LANs) 18a, 18b deployed in each group of locally connected offices, and WAN links 20 interconnecting the local area networks between offices. Hence, the network 45 nodes 16a, 16b, 16c, and 16d can be associated with respective offices at one geographic site (e.g., an office campus in North Carolina), and the network nodes 16e, 16f, 16g, and 16h can be associated with respective offices at another geographic site (e.g., an office campus in California). In this 50 example, most of the database queries for a first office will be for a first collection of records that are relevant to that first office; in contrast, only a few queries generated within the first office will tend to be for other records that are relevant to a second office that has little or no relationship with the first 55 office.
Hence, the disclosed embodiment enables data records to be replicated to a new location within a distributed network based on the query attributes, enabling the data records to be stored in proximity to the center of query activity 22. Conse- 60 quently, database access operations on the relevant data records are optimized based on moving the data records to the optimum location relative to the center of query activity that identifies the optimum location for storage of the data based on the database queries on the data records throughout the 65 network. As result, database queries on data records stored at distant locations can be minimized.
FIG. 2 is a diagram illustrating a table 30 of the distributed database deployed among the multiple storage locations 14a, 14b, etc., according to an embodiment of the present invention. The table 30 includes multiple records 32, also referred to as "rows", that include multiple data elements 34, for example a unique object identifier (OID) field 34a that uniquely identifies the corresponding record 32, a user field 34b that identifies a user name, and another data field 34c such as a salary field or other associated information relevant to the user name. Each record 32 also includes a corresponding data attribute element 36 that specifies attributes associated with the corresponding record 32.
In particular, the data attribute element 36 is created by the database server resource executed by the computing node 16 during storage of the associated elements 34a, 34b, and 34c of the record 32. The record attributes field 36 is configured for storing access attributes that identifies the source of data accesses, namely where the data is being accessed from. The access attributes 36 may include attribute components such as last accessed location (AL) 38, last accessed time 40, access frequency 42 that identifies how often the data is accessed (how "hot" is the data), etc. As illustrated in FIG. 2, the record attributes element 36 may store individual access statistics for multiple access locations 38.
As described above, the accessed location 38 is preferably identified using networked topology-based coordinates: the use of network topology-based coordinates enables the accumulation of all of the access locations (AL) 38 (i.e., the location of the device 16 having requested the data) relative to the number of requests N from those respective locations. In other words, each access location AL 38 that has issued a number of requests N 42 exhibits a tendency (i.e., "desire" or "want") for the data that can be expressed as a vector (V). The origin of the vector is the network-topology based coordinates of the storage location of the data at the local site, the direction of the vector is from the origin to the corresponding access location AL, the magnitude of the vector is based on the location that originated the request, and the "strength" of the vector is based on the corresponding number of access attempts (N). In particular, a given piece of data will have many vectors, each originating at the datum's current storage location, and pointing at the location of the requestor, where each vector has a "strength" or "thickness" corresponding to how much the data wants to move along that path. The summation of all these vectors produces an aggregate vector 44. Consequently, the vectors of all the access locations AL 38 that have accessed the corresponding data record can be accumulated to obtain a cumulative vector (i.e., network distance) 44, illustrated in FIG. 1, that identifies the center of query activity (C) 22 for either a given record, as illustrated in FIG. 2, or an aggregate center 22 of activity for all stored records as illustrated in FIGS. 1 and 4. The aggregate vector 44 can be used to determine whether a split or a merge needs to be performed, described below.
The network topology-based coordinates may include bandwidth, throughput, hop count, round-trip time delay, the cost of accessing data, etc., and any other attributes that quantify the relative ability (i.e., "network distance") to move data to and from another machine based on a comparison of the respective coordinates of the source and destination machines. Hence, the network distance between two nodes identifies the relative ability to move data between the two nodes. Additional details related to the use of topology-based coordinates to determine a network distance are described in commonly-assigned, copending application Ser. No. 11/049, 559, filed Feb. 2, 2005, entitled "Techniques For Locating Distributed Objects On A Network Based On Physical Com