US 7039638 B2 Abstract A distributed data clustering system having an integrator and at least two computing units. Each computing unit is loaded with common global parameter values and a particular local data set. Each computing unit then generates local sufficient statistics based on the local data set and global parameter values. The integrator employs the local sufficient statistics of all the computing units to update the global parameter values.
Claims(18) 1. A method for clustering data comprising:
(a) loading each of a plurality of computing units with common global parameter values and a corresponding local data set;
(b) receiving, by an integrator, from each computing unit, local sufficient statistics based on the local data set and global parameter values; and
(c) employing the local sufficient statistics of all the computing units to update the global parameter values,
wherein the local sufficient statistics and the global parameter values support implementation of a distributed K-Harmonic Means clustering algorithm or a distributed Expectation-Maximization clustering algorithm.
2. The method of
a
_{—}1) receiving a set of data points to be clustered; a
_{—}2) dividing the data points into at least two local data sets; a
_{—}3) sending common global parameter values to each of the computing units; and a
_{—}4) sending each local data sets to a designated computing unit. 3. The method of
c
_{—}1) the integrator determining global sufficient statistics based on the local sufficient statistics of all the computing units; and c
_{—}2) the integrator determining updated global parameter values based on the global sufficient statistics. 4. The method of
d) checking a convergence quality;
e) determining whether the convergence quality meets a predetermined quality; and
f) when the convergence meets a predetermined quality, stop processing; otherwise;
g) when the convergence fails to meet a predetermined quality, providing the updated global parameter values to the computing units and repeating steps (a) to (c).
5. The method of
broadcasting common global parameter values to each of the computing units.
6. The method of
initializing the common global parameter values before sending the common global parameter values to each of the computing units.
7. The method of
8. The method of
9. The method of
10. A distributed data clustering system comprising:
(a) a first computing unit that generates a first set of local sufficient statistics based on global parameter values and a first local data set that is a subset of data points to be clustered;
(b) a second computing unit that generates a second set of local sufficient statistics based on global parameter values and a second local data set that is a subset of the data points to be clustered; and
(c) an integrator unit that receives the first and second sets of local sufficient statistics from the first and second computing units, respectively, and that employs the first and second local sufficient statistics to update the global parameter values, wherein the global parameter values include centers, co-variance matrices, and mixing probabilities in accordance with an Expectation-Maximization (EM) clustering algorithm.
11. The distributed data clustering system of
12. The distributed data clustering system of
13. The distributed data clustering system of
wherein the integrator unit determines global sufficient statistics based on the local sufficient statistics of the first and second computing units; and
wherein the integrator unit determines updated global parameter values based on the global sufficient statistics.
14. The distributed data clustering system of
15. The distributed data clustering system of
16. The distributed data clustering system of
17. A distributed K Harmonic Means clustering system that comprises:
a plurality of computing units each configured to receive a set of centers, and each further configured to combine the set of centers with a local data set to obtain local sufficient statistics for updating the set of centers in accordance with the K Harmonic Means clustering algorithm;
at least one integrator unit configure to combine the local sufficient statistics from each of the plurality of computing units to obtain global sufficient statistics, and further configured to use the global sufficient statistics to update the set of centers in accordance with the K Harmonic Means clustering algorithm.
18. The system of
Description The present invention relates generally to data clustering and more specifically to a method and system for distributed data clustering. There has been the general notion of performing the data clustering in parallel by more than one computer to increase the efficiency of the data clustering. This is particularly important as the data sets increase in the number of data points that need to be clustered, and the case of naturally distributed data (e.g., data for an international company with offices in many different offices located different countries and locations). Unfortunately, the known data clustering techniques were developed for execution by a single processing unit. Furthermore, although there have been attempts to make these known data clustering techniques into parallel techniques, as described in greater detail hereinbelow, these prior art approaches to formulate a parallel data clustering technique offer only tolerable solutions, each with its own disadvantages, and leaves much to be desired. One prior art approach proposes a parallel version of the K-Means clustering algorithm. The publication entitled, “Parallel Implementation of Vision Algorithms on Workstation Clusters,” by D. Judd, N. K. Ratha, P. K. McKinley, J. Weng, and A. K. Jain, Proceedings of the 12 Unfortunately, these publications do not formalize the data clustering approach. Furthermore, the procedure for K-Means is described in a cursory fashion without explanation of how the procedure operates. Also, the publications are silent about whether the distributed clustering technique can be generalized, and if so, how the generalization can be performed, thereby limiting the applicability of the Judd approach to K-Means data clustering. Another prior art approach proposes non-approximated, parallel versions of K-Means. The publication, “Parallel K-means Clustering Algorithm on NOWs,” by Sanpawat Kantabutra and Alva L. Couch, NECTEC Technical Journal, Vol. 1, No. 1, March 1999 describes an example of this approach. Unfortunately, the Kantabutra and Couch algorithm requires re-broadcasting the entire data set to all computers for each iteration. Consequently, this approach may lead to heavy congestion in the network and may impose a communication overhead or penalty. Since the trend in technology is for the speed of processors to improve faster than the speed of networks, it is desirable for a distributed clustering method to reduce the amount of data that needs to be communicated between the computers in the network. Furthermore, the number of slave computing units in this algorithm is limited to the number of clusters to be found. Also, an analytical and empirical analysis of this approach estimates a 50% utilization of the processors. It would be desirable for a distributed clustering method that has a greater percentage of utilization of the processors. Accordingly, there remains a need for a method and system for data clustering that can utilize more than one computing unit for concurrently processing the clustering task and that overcomes the disadvantages set forth previously. According to one aspect of the present invention, a distributed data clustering method and system are provided for performing the data clustering in a parallel fashion instead of a sequential fashion. According to another aspect of the present invention, a distributed data clustering method and system are provided for utilizing a network of computing resources that are either homogenous computing resources or heterogeneous computing resources. According to yet another aspect of the present invention, a distributed data clustering method and system are provided for using a network of computers to efficiently process large data sets that need to be clustered. According to another aspect of the present invention, a distributed data clustering method and system are provided for using a network of computers to efficiently process non-distributed data that need to be clustered. According to yet another aspect of the present invention, a distributed data clustering method and system are provided for using a network of computers to efficiently process naturally distributed data that need to be clustered. A distributed data clustering system having an integrator and at least two computing units. Each computing unit is loaded with common global parameter values and a particular local data set. Each computing unit then generates local sufficient statistics based on the local data set and global parameter values. The integrator employs the local sufficient statistics of all the computing units to update the global parameter values. The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention. The following description and the drawings are illustrative of the invention and are not to be construed as limiting the invention. Homogenous Distributed Data Clustering System It is noted that the computing units Heterogeneous Distributed Data Clustering System It is noted that the computing components (e.g., computing units For example, more powerful computing units can be assigned a greater number of data points to process, and a less powerful computing units can be assigned a fewer number of data points to process so that all the computing units complete execution of their respective assigned tasks at about the same time. By so doing, the system can more efficiently manage the load of the clustering task and reduce the amount of time waiting for partial results that are needed to generate the final result. Integrator When executing on the processor The integration program A communication interface (I/F) Computing Unit When executing on the processor The computing program The GSSDU The distributed clustering technique of the present invention is especially suited to handle naturally distributed data or data where the number of data points is too large for a single processor or computing machine to handle. In the first case, distributed clustering technique of the present invention is well suited to process naturally distributed data. An example of naturally distributed data is customer data, sales data, and inventory data for an international company with offices in many different offices located different countries and locations. In such an example, the data splitter In this second case, the data is split into manageable tasks where each task can be adequately processed by the computing units. In this manner, a task that may be too big for any one computing unit can be split into sub-tasks that can be efficiently processed by each of the computing units. Distributed Clustering Method By finding K centers (local high densities of data), M={m Step In step As used herein the term “sufficient statistics” shall be defined as follows: quantities S can be called sufficient statistics of F, if the value of F can be uniquely calculated or determined from S without knowing any other information. In other words, S is sufficient for calculating F. For further information regarding sufficient statistics, please refer to Duda & Hart, “Pattern Classification and Scene Analysis,” John Wiley & Sons, pages 59-73. In step In step Timeline for Network without a Broadcast Feature The first computing unit Timeline for Network with Broadcast Feature For example, when the network of computers is Ethernet, the broadcasts can employ the broadcasting features of Ethernet. First, at a low level, there is a broadcast protocol for Ethernet. Second, Parallel Virtual Machine (PVM) and Message Passing Interface (MPI) (and their various implementations) typically have routines to broadcast messages that can be utilized. Third, shared memory protocols also support broadcasting. For example, a network interface card on each host may contain some memory, whose contents are replicated across all cards in the ring of hosts. In this regard, writing into the memory on one card replicates data to all other cards' memories. Exemplary Data Clustering Systems The system The system It is noted that other data clustering systems can be implemented in the distributed manner as set forth by the present invention. The distributed data clustering systems A Class of Center-Based Algorithms Let R What is essential here is that f The center-based algorithm, which minimizes the value of the performance function over M, is written as an iterative algorithm in the form of Q SS (I( ) stands for the iterative algorithm, and
This decomposition of center-based algorithms (and many other iterative parameter estimation algorithms) leads to a natural parallel structure with minimal need for communication. Let L be the number of computing units that has a CPU and local memory (e.g., personal computers (PCs), workstations or multi-processor computers). To utilize all L units for the calculation of (1)+(2), the data set is partitioned into L subsets, S=D This partition is arbitrary and has nothing to do with the clustering in the data. This partition is static. Data points in D The processing unit can be homogeneous processing units or heterogeneous processing units. The sizes of the partitions, |D The calculation is carried out on all L units in parallel. Each subset, D One of the computing units is chosen to be the Integrator. The integrator is responsible for summing up the SS from all partitions (4), obtaining the global SS (3); calculating the new parameter values, M; from the global SS; evaluating the performance function on the new parameter values, (2); checking the stopping conditions; and informing all units to stop or sending the new parameters to all computing units to start the next iteration. The duties of the Integrator may be assigned as a part time job to one of the regular units. There may also be more than one computer used as Integrators. The multiple integratos can be organized in a hierarchy if the degree of parallelism is sufficiently high. Special networking support is also an option. If broadcast is supported efficiently, it may be effective to have every node be an Integrator, thereby eliminating one direction of communication. The Parallel Clustering Algorithm Step 0: Initialization: Partition the data set and load the Step 1: Broadcast the integrated parameter values to all computing units. Step 2: Compute at each unit independently the SS of the local data based on (4). Step 3: Send SS from all units to the Integrator Step 4: Sum up the SS from each unit to get the global SS, calculate the new parameter values based on the global SS, and evaluate the performance function. If the Stopping condition is not met, goto Step 1 for the next iteration, else inform all computing units to stop. The stopping condition typically tests for sufficient convergence or the number of iterations. The application of the techniques of the present invention to three exemplary clustering algorithms: K-Means, K-Harmonic Means, and EM, is now described. K-Means is one of the most popular clustering algorithms. The algorithm partitions the data set into K clusters, S=(S The K-Means algorithm starts with an initial set of centers and then iterates through the following steps: For each data item, find the closest m -
- 1. Recalculate all the centers. The k
^{th }center becomes the centroid of the k^{th }cluster. This phase gives the optimal center locations for the given partition of data. - 2. Iterate through 1 & 2 until the clusters no longer change significantly.
- 1. Recalculate all the centers. The k
After each phase, the performance value never increases and the algorithm converges to a local optimum. More precisely, the algorithm reaches a stable partition in a finite number of steps for finite datasets. The cost per iteration is O(K·dim·N). The functions for calculating both global and local SS for K-Means are the 0 δ The set of SS presented here is more than sufficient for the simple version of K-Means algorithm. The aggregated quantity, Σ The l The leading cost of integration is O(K·dim·L), where L is the number of computing units. The new location of the k The parallel version of the K-Means algorithm gives exactly the same result as the original centralized K-Means because both the parallel version and the sequential version are based on the same global SS except on how the global SS are collected. K-Harrnonic Means is a clustering algorithm that features an insensitivity to the initialization of the centers. In contrast to the K-Means clustering algorithm whose results depend on finding good initializations, K-Harmonic Means provides good results that are not dependent on finding good initializations. The iteration step of the K-Harmonic Means algorithm adjusts the new center locations to be a weighted average of all x, where the weights are given by
(K-Means is similar, except its weights are the nearest-center membership functions, making its centers centroids of the cluster.) Overall then, the recursion equation is given by
on the data in its own memory, and then sends it to the Integrator. The size of the SS vector is K+K·dim (g which is the only information the units need to start the next iteration. This calculation costs O(K·dim·L). The updated global centers are sent to each unit for the next iteration. If broadcasting is an option, this is the total cost in time. If the Integrator finds the centers stop moving significantly, the clustering is considered to have converged to an optimum, and the units are stopped. In this example, the EM algorithm with linear mixing of K bell-shape (Gaussian) functions is described. Unlike K-Means and K-Harmonic Means in which only the centers are to be estimated, the EM algorithm estimates the centers, the co-variance matrices, Σ where the vector p=(p E Step: Estimating “the percentage of x belonging to the k where p(x|m) is the prior probability with Gaussian distribution, and p(m M-Step: With the fuzzy membership function from the E-Step, find the new center locations, new co-variance matrices, and new mixing probabilities that maximize the performance function.
The functions for calculating the SS are:
The vector length (in number of scalars) of the SS is 1+K+K·dim+K·dim There are numerous applications that can utilize the distributed clustering method and system of the present invention to cluster data. For example, these applications include, but are not limited to, data mining applications, customer segmentation applications, document categorization applications, scientific data analysis applications, data compression applications, vector quantization applications, and image processing applications. The foregoing description has provided examples of the present invention. It will be appreciated that various modifications and changes may be made thereto without departing from the broader scope of the invention as set forth in the appended claims. The distributed clustering method and system described hereinabove is not limited to data clustering algorithms, but can, for example, be applied to distributed parametric estimation applications (e.g., statistical estimation algorithms that use sufficient statistics). Furthermore, the distributed parameter estimation techniques of the present invention can be applied not only to large data sets, but also can be applied to naturally distributed data (e.g., environmental data, geological data, population data, governmental data on a global worldwide scale). Applications to collect, process and analyze the naturally distributed data, can advantageously utilize the distributed parameter estimations techniques of the present invention. These applications can include, for example, geological survey applications, environment monitoring applications, corporate management applications for an international company and economic forecast and monitoring applications. Patent Citations
Non-Patent Citations
Referenced by
Classifications
Legal Events
Rotate |