|Publication number||US20050086192 A1|
|Application number||US 10/688,287|
|Publication date||Apr 21, 2005|
|Filing date||Oct 16, 2003|
|Priority date||Oct 16, 2003|
|Also published as||US20090327248|
|Publication number||10688287, 688287, US 2005/0086192 A1, US 2005/086192 A1, US 20050086192 A1, US 20050086192A1, US 2005086192 A1, US 2005086192A1, US-A1-20050086192, US-A1-2005086192, US2005/0086192A1, US2005/086192A1, US20050086192 A1, US20050086192A1, US2005086192 A1, US2005086192A1|
|Original Assignee||Hitach, Ltd.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (9), Referenced by (27), Classifications (7), Legal Events (1)|
|External Links: USPTO, USPTO Assignment, Espacenet|
The present invention is related to computer file access and in particular to improving the performance of index maintenance in search engines.
The Internet is commonly associated with the world wide web (the “web”). The web has facilitated an explosive proliferation of information to the millions of users who access the web. This information is accessed in the form of files by web servers. However, the Internet has also provided access to files provided by file servers which pre-date the web, such as bulletin boards, ftp sites, and so on.
An intranet that is a private network of a company or any other organization is also used for sharing files. In this case, a file server or a NAS (Network Attached Storage) is common to store and get files. NFS and CIFS protocols are used for accessing files.
Search engines have become a valuable tool in navigating the Internet and/or file servers. Search engines are a commonly used tool to access the many millions of files on the Internet and/or file servers. Typically, the search engine accepts search requests from a user and sends a obtains a list of file names that match the search conditions.
An integral component of a search engine is its “index.” The index is a collection of information that is parsed or otherwise generated from an analysis of a file, and comprises keywords and related information used by the search engine to facilitate a file search. The specific information content and data structures of the index vary from one search engine to another, and is beyond the scope of the present invention.
However, common operations that are performed by typical search engines include the creation of the index and the subsequent maintenance or update of the index. The creation of the index typically involves the search engine checking updated dates of every files, reading every updated file on the Internet and/or file servers and parsing its contents to build up the index.
Invariably, file contents change over time. The search engine must therefore perform updates to the index in order that the index be current. This task typically involves once again crawling the web and/or file servers to access attributes of each file, and then determine whether the file has been updated since the last time the index was updated; or when the index was created, in the case of the very first index update. This determination can be made, for example, by accessing the modification date of the file and comparing it against the index. Making this check reduces the update effort and thus improves the update time; not every file will be re-indexed, only those that have changed relative to the time of the index.
Nevertheless, this update process remains a tedious task because modification date of every files need to be checked. This creates a large volume of traffic, just for the purpose of checking attributes of files. It is therefore very desirable to reduce Internet traffic and/or intranet traffic attributed to the indexing function. It is also desirable to further reduce the indexing effort to further increase the update time of an index.
In accordance with one aspect of the invention, an update list is maintained in a file server. Update information based on the update list is communicated to a search engine. The update information comprises only those files that have been modified during an previous update operation on an index in the search engine.
In accordance with another aspect of the invention, the file server presents a restricted directory listing to a search engine, as compared to a directory listing of the same directory to a client other than a search engine. A set of one or more filtering criteria can be used to limit the number of files presented to the search engine. This reduces the number of files the search engine must examine when performing an update of its search index.
In accordance with still another aspect of the invention, an update list is maintained in the file server. Files referenced in the update list are limited depending on one or more filtering criteria.
Aspects, advantages and novel features of the present invention will become apparent from the following description of the invention presented in conjunction with the accompanying drawings:
The files stored on a file server are organized into a system of files 010401. In one embodiment of the invention, the file server can access an update list 010402. In general, the update list can be contained in physical storage in a suitable location. In a more general sense, the file server element 0104 shown in the figure represents a plurality of file servers, each storing its own set of files. A typical protocol that file servers use is the network file system protocol (NFS). Another conventional protocol is the common internet file system (CIFS) protocol. Still other protocols such as HTTP can be used by a file server.
The architecture typically includes at least one NFS/CIFS clients 0101 who communicate with the file server(s) 0104 over the network 0103 via the NFS or CIFS protocol in order to read and write files in the file server. Clients include creators of the files, and users who can access the file to either read or modify files, or read and write files. In a more general sense, the client element 0101 of
A search engine server 0105 communicates via the network 0103. A file server controller 010502 provides the processing capability conventionally associated with a search engine. This may include a central processing unit (CPU), memory, and storage for program code to control the operation of the CPU. Although this embodiment of the invention is described using a search engine, it will be appreciated from the following description that aspects of the invention can be incorporated into any machine in a networked environment that tracks files and updates to the file. The search engine is merely a convenient example to use because search engines are well understood and familiar to most people who interact on a computer network.
A typical function of most search engines is the creation and maintenance of an index. The specific content and structure of the information comprising an index, and the specifics of the parsing function are beyond the scope of the present invention. For the purposes of discussion, it can be appreciated by those of ordinary skill that one can refer to an index on a particular file system, or an index associated with a file system. The index information can be represented generically as an index database 010501, without loss of generality.
The index is created and subsequently updated and otherwise maintained by the search engine. This activity includes parsing or otherwise generating information from files in the file server(s) 0104 in order to create the index database. It can be appreciated that the search engine can use the same NFS or CIFS protocol to access files in the file server(s).
The architecture shows at least one file search clients 0102. These are the users who access the search engine to submit file search requests. It can be appreciated that a “user” can be a human user or a machine user. An interface is understood to be provided by the search engine that is suitable to the kind of user being serviced. In a generalize sense, the file search client element 0102 shown in
The network 0103 is generally any suitable communication network that allows for communication among the various servers and clients mentioned above. The figure shows a local area network (LAN), but it can be appreciated that other communication networks are equally suitable. Connectivity to a LAN network is typically provided by the ethernet standard, using the TCP/IP protocol.
The file server 0104 and the search engine server 0105 each can be embodied in conventional computer hardware (e.g., comprising a suitable CPU, memory, storage devices, and so on). Conventional software platforms can be used to support the server; e.g., Unix or other UNIX-based OS's, Macintosh OS, various Microsoft OS's, and so on. It is also possible that the file server and the search engine server can run on same hardware and software platform. For example, NFS server and a search engine software can run on a Linux OS.
In a particular embodiment, the index may be one large database or some other single organization of data representing all file servers. However, logically, one can refer to each file server as having its own associated index; it being understood that reference is being made with respect to that portion of index structure associated with a file server.
Thus, when a search engine first comes online, an index is created for all of the files that can be accessed from a file server; this is done for every file server that is made known to the search engine. Also, if a search engine which is already online learns of a new file server, an index needs to be created for the accessible files contained in that file server. This is represented in
For a new file server, the search engine sends an initialization operation (see
Referring to the decision step 0201, if the index for a file server was previously created, then the search engine accesses update information contained in an update list 010402 associated with that file server (step 0203). Then in a step 0204, files referenced in the update information are accessed by the search engine (see
In one implementation, the update list can be accessed by the search engine, just like any other file. Thus, the file server creates a special file that contains a list of updated files and the search engine retrieves a copy of the file from the file server and stores it as a local copy. The search engine also deletes the contents of the special file. The search engine can then operate on the local copy; e.g., reading through the file to identify the files to parse. Alternatively, a protocol can be defined between the search engine and the file server to obtain the information contained in the update list. For example, the file server can communicate to the search engine each file name of the files in the updated file or a list of every file name of the files in the update list to be processed in the search engine. In accordance with another implementation the search engine can receive the actual files in the updated list instead of a list of file names from the file servers.
Thus, in a step 0301, the file server receives a file operation request from a client. In a determination step 0302, the request is handed off to an appropriate handler. For example, a file open request is handled by a file open handler 0303. A file read request is handled by a file read handler 0304. A file write request is handled by a file write handler 0305 in accordance with an embodiment of the present invention. This aspect of the invention will be discussed below. A directory listing request is handled by a directory listing handler 0306 in accordance with another aspect of the present invention. The directory listing request will be discussed further below. A “get update list” request is handled by the handler 0307. This function is provided in accordance with an embodiment of the present invention and is discussed below.
The purpose of checking for the first write operation in step 0501 is to avoid having multiple entries in the update list 010402 for the same file. One way to achieve this is as disclosed in step 0502. Alternatively, the update list can be inspected each time to determine whether the file is already in the list or not.
In the case of file creation, a created file initially contains no data. Therefore, it is not necessary that the file server make an entry in the update list to refer to a newly created file. When content is placed in the file, this will occur via a file write operation. However, in some file systems, the file create operation may leave the file in a state where subsequent write operations can be performed; thus obviating the need for a separate file open function call. Therefore with reference to the decision step 0501 in
The particular implementation shown in
Continuing with the figure, if the request is a get_file_list operation, then the file server will communicate the update list to the search engine (step 0404). A copy of the file can be communicated to the search engine, just like any other file. Alternatively, the file server can communicate the actual files to the search engine; either one at a time, or in groups, or in some other suitable manner. For each file in the update list, the search engine will analyze the file and update the index with information produce by the analysis, thereby updating the index.
When the update list is communicated to the search engine, the update list is cleared, in a step 0405. Thus, if the update list is communicated to the search engine as a single file, the update list can be cleared after the communication is complete. If the file server communicates files to the search engine instead, then each file that is referenced in the update list can be deleted from the update list after it is communicated to the search engine.
After the update list is cleared, the list is once again filled with references to files that are modified. The files referenced in the update list therefore represent those files that have been modified subsequent to a point in time when the update list was last cleared. Stated from a different point of view, the update list contains a list of file references that have been modified since the last time the update list was retrieved by the search engine.
From the point of view of the search engine, files referenced in the update list represent those files that have been modified subsequent to a point in time when the index was being updated. It can be appreciated that updating the index can be a time consuming operation. Thus, in practice, the clearing of the update list by the file server (by virtue of a get_file_list request) may very well occur before the completion of updating the index by the search engine.
The next time the search engine retrieves the update list to perform an update of the index, it will only need to parse through those files which were modified since the previous update operation on the index. The update list therefore avoids the search engine having to perform the brute force task of accessing and parsing every file on a given file server in order to update the index.
An index can be created for a file that does not have one. This situation may arise because the search engine was not previously aware of the file system, or for some reason it was decided to delete a previously existing index for the file system. When the search engine has completed the process of creating the index, it will send a get_file_list request for an initialization operation. This has the effect of creating the update list or of clearing an existing update list. If the file system was not previously known, then the file system may not likely to have an update list. In that case, an update list is created. If the file system already had an update list, then the initialization operation will serve to clear the list.
Based on the foregoing discussion, it can be appreciated that each file server has its own associated update list. However, as an alternative implementation, it is conceivable that an update list can be implemented that is accessible by two or more file servers that contains references to modified files from the two or more file servers. In the most general case, a global update list can be provided. However, this type of update list may or may not be preferable, depending on performance considerations, implementation considerations, and so on. In another alternative, one file server maintains multiple updated files. One update list is associated with one export point of the file server.
As illustrated in
In accordance with this embodiment, the search engine performs conventional processing to either create an index on the files on the file server, or to update the index. The search engine mounts the export that has been made available by the file server. An administrator of a file server creates an export for a search engine. An administrator of the search engine specifies a list of exports that the search engine needs to make an index. This can be done, for example, by editing a special file in the search engine. By using a directory service, this configuration can be done systematically. The search engine then makes one or more requests for directory listing(s) of files on the file server; for example, using the standard requests provided in the NFS and CIFS protocols.
In the case where the index is being created for the file system, each file identified in the directory listing(s) is parsed and indexed. In the case where the index is being updated, the search engine determines whether the file should be parsed for indexing based on the modification date (or some other similar information) of the file. If the file was modified since the last time the index for this file system was updated, then the file is parsed and indexed; otherwise it is not parsed.
In accordance with this aspect of the invention, the list of files made available via a directory listing by the file server to the search engine is less than the files that are available in a directory listing to other clients. This is made possible because the search engine mounts an export that is different than the export that is mounted by clients other than the search engine. As will be discussed now, the file server is configured to perform differently depending on which export the file service request is being made; e.g., a directory listing service request.
The file filter table embodiment show in
Typically, files that are indexed are those that contain text. Some search engines will also index files that have graphics or some kind of image data, if there is corresponding text in the file. The file filter table can reduce the set of files that the search engine must consider by filtering out executable files or other files which do not contain data that can be searched.
If the request originated from a search engine, then in a step 0702, the file server consults the file filtering table 0901 to determine (step 0703) for each file in that directory whether it will be contained in the directory listing information. If the file meets the criterion(a) set forth in the file filtering table, then a reference to the file is added to a temporary list (step 0704). The file server can determine whether the request came from a search engine or from a client by looking at which export the request has been issued or by looking at an IP address of the requester, or by some other suitable identification technique. Also, the file server can maintain a suitable list that identifies one or more computer systems (e.g., search engines) for which the file filtering table will be used to satisfy a directory request.
If the file does not match any of the criteria in the file filtering list, then it will not be added to the temporary list. In a step 0705, a check is made to determine whether all of the files have been checked against the file filtering table. If more files need to be checked, then processing continues with step 0702. Otherwise, the temporary list is further processed in a step 0706 to produce a suitable directory listing that can then be communicated back to the search engine. This might include adding a listing of the subdirectories to the temporary list. File attributes of the files contained in the temporary list may need to be supplied. This might include information such as file size, creation date, modify date, permission information, and so on. The directory information is then communicated to the search engine as a response to the directory listing request.
It can be appreciated that the directory listing that the search engine receives is filtered by the file filtering table, and thus can contain a subset of the files that a non-search engine client might receive. By virtue of this reduced file list, processing in the search engine to create an index for the file system or to update it index can be reduced, as compared to conventional processing where an unfiltered directory listing might include many more files.
If the write request is the first write request since the last file open operation, then processing proceeds to a decision step at step 0602. There, a file filtering table 0901 is consulted. This table is used in the same manner as discussed above. If the file that is the object of the write operation satisfies any of the criteria in the file table, then a reference to the file is added to an update list 010402, in a step 0603. If no criteria are satisfied, then the write operation is completed in a conventional manner in step 0604.
It was noted above in connection with
It can be appreciated that this aspect of the invention is similar to the aspect of the invention discussed in connection with update lists. The search engine will consult the update list associated with the file system when it is ready to perform an update of its index for that file system, as discussed above. Thus, the search engine need only access and parse those files referenced in the update list when performing an index update. However, with the use of the file filter table, the size of the update list can be reduced somewhat. This has the desired effect of potentially reducing the index update time.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US215601 *||Mar 28, 1879||May 20, 1879||Himself And Luther J||Improvement in child s chair and carriage|
|US5845273 *||Jun 27, 1996||Dec 1, 1998||Microsoft Corporation||Method and apparatus for integrating multiple indexed files|
|US6067541 *||Sep 17, 1997||May 23, 2000||Microsoft Corporation||Monitoring document changes in a file system of documents with the document change information stored in a persistent log|
|US6269362 *||Dec 19, 1997||Jul 31, 2001||Alta Vista Company||System and method for monitoring web pages by comparing generated abstracts|
|US6356863 *||Jun 1, 1999||Mar 12, 2002||Metaphorics Llc||Virtual network file server|
|US6418453 *||Nov 3, 1999||Jul 9, 2002||International Business Machines Corporation||Network repository service for efficient web crawling|
|US6636854 *||Dec 7, 2000||Oct 21, 2003||International Business Machines Corporation||Method and system for augmenting web-indexed search engine results with peer-to-peer search results|
|US7020658 *||Jun 4, 2001||Mar 28, 2006||Charles E. Hill & Associates||Data file management system and method for browsers|
|US7231382 *||Jun 1, 2001||Jun 12, 2007||Orbitz Llc||System and method for receiving and loading fare and schedule data|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7487138 *||Aug 25, 2004||Feb 3, 2009||Symantec Operating Corporation||System and method for chunk-based indexing of file system content|
|US7539702||Mar 12, 2004||May 26, 2009||Netapp, Inc.||Pre-summarization and analysis of results generated by an agent|
|US7603616 *||Nov 5, 2004||Oct 13, 2009||Microsoft Corporation||Proxy server using a statistical model|
|US7617255||Mar 8, 2006||Nov 10, 2009||Hitachi, Ltd.||Storage system, storage control device and recovery point detection method for storage control device|
|US7630994||Mar 12, 2004||Dec 8, 2009||Netapp, Inc.||On the fly summarization of file walk data|
|US7716198||Dec 21, 2004||May 11, 2010||Microsoft Corporation||Ranking search results using feature extraction|
|US7739277||Sep 30, 2004||Jun 15, 2010||Microsoft Corporation||System and method for incorporating anchor text into ranking search results|
|US7761448||Sep 30, 2004||Jul 20, 2010||Microsoft Corporation||System and method for ranking search results using click distance|
|US7792833||Apr 26, 2006||Sep 7, 2010||Microsoft Corporation||Ranking search results using language types|
|US7827181||Sep 29, 2005||Nov 2, 2010||Microsoft Corporation||Click distance determination|
|US7840569||Oct 18, 2007||Nov 23, 2010||Microsoft Corporation||Enterprise relevancy ranking using a neural network|
|US7844646 *||Mar 12, 2004||Nov 30, 2010||Netapp, Inc.||Method and apparatus for representing file system metadata within a database for efficient queries|
|US8024309||Aug 30, 2007||Sep 20, 2011||Netapp, Inc.||Storage resource management across multiple paths|
|US8037113 *||Jan 20, 2009||Oct 11, 2011||Novell, Inc.||Techniques for file system searching|
|US8095565||May 5, 2006||Jan 10, 2012||Microsoft Corporation||Metadata driven user interface|
|US8418258 *||Sep 23, 2010||Apr 9, 2013||Antenna Vaultus, Inc.||System for providing mobile data security|
|US8473636 *||Feb 5, 2008||Jun 25, 2013||Hitachi, Ltd.||Information processing system and data management method|
|US8595238||Jun 22, 2011||Nov 26, 2013||International Business Machines Corporation||Smart index creation and reconciliation in an interconnected network of systems|
|US8935789 *||Jul 17, 2009||Jan 13, 2015||Jayant Shukla||Fixing computer files infected by virus and other malware|
|US8959593 *||Dec 10, 2012||Feb 17, 2015||Antenna Vaultus, Inc.||System for providing mobile data security|
|US8990285||Feb 29, 2008||Mar 24, 2015||Netapp, Inc.||Pre-summarization and analysis of results generated by an agent|
|US20050086583 *||Nov 5, 2004||Apr 21, 2005||Microsoft Corporation||Proxy server using a statistical model|
|US20050203907 *||Mar 12, 2004||Sep 15, 2005||Vijay Deshmukh||Pre-summarization and analysis of results generated by an agent|
|US20050210006 *||Mar 18, 2004||Sep 22, 2005||Microsoft Corporation||Field weighting in text searching|
|US20100031361 *||Jul 17, 2009||Feb 4, 2010||Jayant Shukla||Fixing Computer Files Infected by Virus and Other Malware|
|US20110107437 *||May 5, 2011||Antenna Vaultus, Inc.||System for providing mobile data security|
|EP2069938A2 *||Sep 26, 2007||Jun 17, 2009||Sony Corporation||Providing a user access to data files distributed in a plurality of different types of user devices|
|U.S. Classification||1/1, 707/E17.01, 707/999.001|
|International Classification||G06F12/00, G06F17/30|
|Oct 16, 2003||AS||Assignment|
Owner name: HITACHI, LTD., JAPAN
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:KODAMA, SHOJI;REEL/FRAME:014626/0287
Effective date: 20031013