|Publication number||US7496636 B2|
|Application number||US 10/064,176|
|Publication date||Feb 24, 2009|
|Filing date||Jun 19, 2002|
|Priority date||Jun 19, 2002|
|Also published as||US20040143787|
|Publication number||064176, 10064176, US 7496636 B2, US 7496636B2, US-B2-7496636, US7496636 B2, US7496636B2|
|Inventors||Constantine Grancharov, Andrew Rolleston, Duncan Smith|
|Original Assignee||International Business Machines Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (19), Referenced by (1), Classifications (19), Legal Events (5)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This invention relates to a method and system for resolving Universal Resource Locators (URLs) from script code located in websites for the purpose of website crawling.
The World Wide Web available on the Internet provides a variety of specially formatted documents called web pages. The web pages are traditionally formatted in a language called HTML (HyperText Markup Language). Many web pages include links to other web pages which may reside in the same website or in a different website, and allow users to jump from one page to another simply by clicking on the links. The links use Universal Resource Locators (URLs) to jump to other web pages. URLs are the global addresses of web pages and other resources on the World Wide Web.
As web technology evolves, websites become more and more complex. The tendency in website development is to move from using purely static HTML to using HTML and script code to provide enhanced functionality. As a result, it is now common to use script code to construct web page links, i.e., to create URLs dynamically. Often the process of dynamically constructing URLs involves many variables and some rather complex script code. This makes it very difficult to resolve, i.e., extract and obtain, such URLs, when it comes to website crawling.
Website crawling or spidering is a process to automatically scan contents of websites by following links and fetching the web pages. Web crawling agents or “spiders” are software programs for performing the crawling over websites. Typically, existing web crawling agents are used to find specific information of interest in the Web.
Before the introduction of script code into Web pages, crawling agents could parse HTML code for standard URLs. Since all URLs had to be coded to the HTML specification, this task was relatively easy. However, as sites evolved they increasingly relied upon script code to provide more advanced functionality that standard HTML did not allow for. The format of the URLs in the script code varies widely from implementation to implementation. Unlike static HTML, there is no standard that the script code must follow for encoding URLs. Accordingly, script code presents problems for crawling agents that need to parse URLs. There is no longer a common syntax or format for the URLs and thus they are difficult to find consistently.
An existing approach to this problem is to use customizable pattern matching algorithms that statically read through the script code on a page or in a script file, and based on pattern matching try to “guess” what in that script code might be a URL. The pattern matching provides some utility but the use of the pattern matching algorithms has two basic problems: 1) the algorithms invariably miss URLs in the script code and 2) the algorithms do not always extract the entire URL correctly.
It is therefore desirable to provide a new mechanism that can more accurately resolve URLs from script code embedded in web pages.
It is an object of the invention to provide a novel system and method for more accurately resolving Universal Resource Locators (URLs) from script code.
The present invention examines the script code to obtain URLs from the output in the context of website crawling.
In accordance with an aspect of the present invention, a URL resolution system for resolving Universal Resource Locators (URLs) is provided. The URL resolution system comprises a website crawler for crawling a website and for locating script code which is used to dynamically create at least one script URL; and a script URL resolution component for causing examination of the script code located during the crawling to obtain the script URL.
In accordance with another aspect of the present invention, a method for resolving Universal Resource Locators (URLs) is provided. The method comprises steps of locating script code which creates at least one script URL while crawling a website; and examining the script code to obtain the script URL from the examination result.
In accordance with another aspect of the present invention, a computer readable medium storing the instructions and/or statements for use in the execution in a computer of the method for resolving Universal Resource Locators (URLs) is provided.
In accordance with another aspect of the present invention, there is provided electronic signals for use in the execution in a computer of the method for resolving Universal Resource Locators (URLs).
In accordance with another aspect of the present invention, there is provided a computer program product for use in the execution in a computer of a method for resolving Universal Resource Locators (URLs). The computer program product comprises a module for locating script code which creates at least one script URL while crawling a website; and a module for examining the script code to obtain the script URL from the examination result.
Other aspects and features of the present invention will be readily apparent to those skilled in the art from a review of the following detailed description of preferred embodiments in conjunction with the accompanying drawings.
The invention will be further understood from the following description with reference to the drawings in which:
The present invention is suitably used to check the integrity of links in a website. For example, a website 10 shown in
For example, in
The examination of script code at step 204 may be carried out by explicitly executing the script code. Alternatively, it may be done by examining the script code to obtain the script URLs without explicitly executing the script code. The script URL resolution component 140 may examine the script code or it may use another component to examine the script code, as described below in relation with other embodiments.
Therefore, the URL resolution system 100 allows automatic resolution of script URLs from embedded script code in websites in the context of website crawling, i.e., by locating script code while crawling a website or websites. Since the script code is examined to dynamically obtain the script URLs, complete URLs can be accurately obtained. Unlike the conventional pattern matching which resolve URLs statically, there are minimal possibilities that the URL resolution system 100 will miss script URLs in the website that is being crawled. Thus, the URL resolution system 100 produces accurate results of website crawling.
The URL resolution system 100 may have a function by which users can set the extent of the crawling, as described below.
Other embodiments of the present invention are described referring to
The website crawler 320 has script code detector 322 and crawling controller 324. OLE_LINK2The crawling controller 324 OLE_LINK2controls crawling carried out by the website crawler 320. The crawling controller 324 controls the website crawler 320 to crawl individual web pages included in website 310 or other websites to locate web pages that use script code to dynamically create script URLs. The crawling controller 324 receives output of the script URL resolution component 340 and uses the output to control the website crawler 320, as further described below.
To locate web pages that use script code to dynamically create script URLs, the website crawler 320 uses the script code detector 322 to determine if the script code contained in the web page should be executed by determining if it uses a specific part of the script code to dynamically create at least one script URL. The script code detector 322 issues a notification to the script URL resolution component 340 when a web page having such script code is found. The notification includes an identification of the web page.
The script URL resolution component 340 is activated in response to a notification generated by the script code detector 322 of the website crawler 320. The website crawler 320 crawls all web pages on the original website, but it only passes the web pages containing relevant script code to the script URL resolution component 340.
The script URL resolution component 340 controls a web page examiner 360. The web page examiner 360 is a component capable of loading the contents of web pages and executing the entire or a specific part of script code in the loaded web pages. The web page examiner 360 may be a web browser having these functions, or a combination of a web page parser and a script code examiner. The URL resolution system 300 uses an external web page examiner 360. Alternatively, as shown in
The script URL resolution component 340 has a web page loading controller 342 and a script code execution controller 344. The web page loading controller 342 notifies or instructs the web page examiner 360 to load relevant web pages. The script code execution controller 344 instructs the web page examiner 360 to execute specific parts of the script code that will result in dynamically created script URLs. For example, when the script URL resolution component 340 receives a notification from the script code detector 322, the web page loading controller 342 instructs the web page examiner 360 to load the contents of the web page identified in the notification. Then, the script code execution controller 344 executes the script code by interfacing with the web page examiner 360 and using its interface functions to force the execution of the specific parts of the script code in the loaded web pages. The web page examiner 360 captures the script URL(s) resulting from the script code execution and returns these script URLs to the script code execution controller 344. The script code execution controller 344 may instruct the web page examiner 360 to execute the entire script code, rather than only the specific parts thereof.
The script code execution controller 344 outputs the execution results to the website crawler 320. When the execution of the script code is successful, the execution result includes one or more resolved script URLs. When the execution of the script code is unsuccessful, the execution result includes a failure result.
The URL resolution systems 300, 400 may also have a presentation unit 480 or use an external presentation unit 380 to present to users the execution results. The presentation unit 380, 480 may be a user interface, a result log file, an email or other output unit or form. The execution results presented to users may include only the failure results or only resolved script URLs or both. Thus, an administrator of the website may attend to the failures.
Users may also use an input unit (not shown) to initiate or terminate the crawling, or set parameters of the crawling controller 324. For example, the crawling controller 324 may be set such that it crawls a website regularly in a predetermined interval and/or it may start crawling when the website is modified. Also, users may set the extent of the crawling, i.e., users may set the website crawler 320 to crawl only within the original website from which the crawling is initiated, or allow crawling of web pages residing in external websites when web pages in the external websites are linked. In the latter case, it is desirable to limit the extent or depth of the crawling of the external websites. For example, in
The website crawler 320 crawls a website 310 (500). Crawling of website 310 may start anywhere in the website 310. During the crawling, the script code detector 322 checks script code embedded in each web page in the website 310 to determine if the web page uses script code to dynamically create one or more script URLs. When the script code detector 322 locates a web page with script code that dynamically creates one or more script URLs (502), the script URL resolution component 340 is activated. The script code detector 322 sends a notification to the script URL resolution component 340 to this end.
In the script URL resolution component 340 the web page loading controller 342 instructs the web page examiner 360 to load the web page with the script code (504). The script code execution controller 344 then instructs the web page examiner 360 to execute the specific interface methods or functions that dynamically execute the script code and create one or more script URLs (506). The script code execution controller 344 may instruct the web page examiner 360 to execute the entire script code or only the relevant portions of the script code. Script URLs are thus resolved by the script code execution. The script code execution controller 344 receives the resolved script URLs from the web page examiner 360, and sends the received script URLs back to the crawling controller 324 (508).
The website crawler 320 continues the crawling (510). It may continue crawling on web pages identified by the resolved script URLs. The website crawler 320 may crawl those web pages immediately when the resolved script URLs are returned, or put them in a queue for crawling at a later time. The website crawler 320 may crawl multiple web pages in parallel.
The process of
The steps of crawling a website (500) to executing script code (506) are similar to those shown in
When the execution of the script code is unsuccessful, a failure result is output by the script URL resolution component 340 (530). The failure result is also returned to the website crawler 320. In the website crawler 320, the error result is logged (532), and the crawling of the current website is continued (534).
The process is repeated until crawling of the original website is completed.
The failure results logged at step 532 may be presented to users during and/or after the scanning.
Referring now to
The website crawler 820 has a script URL gatherer 822 for gathering script URLs from the advanced web page examiner 860. The advanced web page examiner 860 has a web page loader 862 for loading web pages, and a script code examiner 864 for executing script code in the loaded web pages. The advanced web page examiner 860 may be a part of the URL resolution system 800 or a component external to the system 800.
In operation, the website crawler 820 crawls a website 810. For each URL found on each of those web pages, the script URL gatherer 822 calls a function on the advanced web page examiner 860. It also calls the function for the URL of each web page on which the website crawler 820 crawls. The function takes the received URL as an input parameter and activates the web page loader 862 to load the contents of a web page identified by the received URL.
Then the function activates the script code examiner 864 to examine the loaded web page to obtain any script URLs created by script code in the web page. For example, during the examination of the loaded web page, the script code examiner 864 executes script code found in the loaded web page to obtain script URLs if any. The script code examiner 864 may execute all script code in the loaded web page or only script code that is used to create one or more script URLs. Also, the script code examiner 864 may execute the entire script code or only relevant portions of script code.
The function returns a collection of zero or more resolved script URLs as an output parameter to the script URL gatherer 822. The website crawler 820 may crawl web pages identified by the resolved script URLs. The crawling of those web pages may be carried out immediately or later. The website crawler 820 may crawl those pages in parallel with other web pages.
The website crawler 820 may have a crawling controller similar to crawling controller 324 shown in
The advanced web page examiner 860 may be a part of the URL resolution system 900 or a component external to the system 900.
In the embodiments shown in
In the above embodiments, the elements of the URL resolution system are described separately, however, two or more elements may be provided as a single element, or one or more elements may be shared with other components in a computer system in which the URL resolution system is installed. For example, in the embodiment shown in
The URL resolution system of the present invention may be implemented by any hardware, software or a combination of hardware and software having the above described functions. The software code, either in its entirety or a part thereof, may be stored in computer readable memory. Further, a computer data signal representing the software code which may be embedded in a carrier wave may be transmitted via a communication network. Such a computer readable memory and a computer data signal are also within the scope of the present invention, as well as the hardware, software and the combination thereof.
While particular embodiments of the present invention have been shown and described, changes and modifications may be made to such embodiments without departing from the true scope of the invention.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US5974572||Oct 14, 1997||Oct 26, 1999||Mercury Interactive Corporation||Software system and methods for generating a load test using a server access log|
|US5999929||Sep 29, 1997||Dec 7, 1999||Continuum Software, Inc||World wide web link referral system and method for generating and providing related links for links identified in web pages|
|US6253204||Dec 17, 1997||Jun 26, 2001||Sun Microsystems, Inc.||Restoring broken links utilizing a spider process|
|US6424966 *||Jun 30, 1998||Jul 23, 2002||Microsoft Corporation||Synchronizing crawler with notification source|
|US6665658 *||Jan 13, 2000||Dec 16, 2003||International Business Machines Corporation||System and method for automatically gathering dynamic content and resources on the world wide web by stimulating user interaction and managing session information|
|US6668369 *||Dec 29, 1999||Dec 23, 2003||International Business Machines Corporation||Software debugging tool for displaying dynamically written software code|
|US6701350 *||Dec 29, 1999||Mar 2, 2004||Nortel Networks Limited||System and method for web page filtering|
|US6857124 *||Jan 11, 2000||Feb 15, 2005||Eolas Technologies, Inc.||Method and system for hypermedia browser API simulation to enable use of browser plug-ins and applets as embedded widgets in script-language-based interactive programs|
|US20020013782||Feb 15, 2001||Jan 31, 2002||Daniel Ostroff||Software program for internet information retrieval, analysis and presentation|
|US20020052928 *||Mar 30, 2001||May 2, 2002||Eliyon Technologies Corporation||Computer method and apparatus for collecting people and organization information from Web sites|
|US20020078136 *||Dec 14, 2000||Jun 20, 2002||International Business Machines Corporation||Method, apparatus and computer program product to crawl a web site|
|US20020099723 *||Jan 16, 2001||Jul 25, 2002||Jorge Garcia-Chiesa||Apparatus and method to support management of uniform resource locators and/or contents of database servers|
|US20020107848 *||Nov 30, 2001||Aug 8, 2002||Ryan Brown||Search and index hosting system|
|US20020147637 *||Mar 7, 2001||Oct 10, 2002||International Business Machines Corporation||System and method for dynamically optimizing a banner advertisement to counter competing advertisements|
|US20020169801 *||Feb 16, 2001||Nov 14, 2002||Martin Barnes||Web site converter|
|US20020188717 *||Jun 8, 2001||Dec 12, 2002||International Business Machines Corporation||Method and apparatus for modeling the performance of Web page retrieval|
|US20020194161 *||Apr 12, 2002||Dec 19, 2002||Mcnamee J. Paul||Directed web crawler with machine learning|
|US20030084034 *||Nov 1, 2001||May 1, 2003||Richard Fannin||Web-based search system|
|US20070282818 *||Aug 21, 2007||Dec 6, 2007||Virage, Inc.||Network video guide and spidering|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US20110173178 *||Jul 14, 2011||Craig Conboy||Method and system for obtaining script related information for website crawling|
|U.S. Classification||709/217, 709/203, 709/218, 709/201, 709/200, 715/738, 707/999.003, 707/999.2, 707/999.202|
|International Classification||G06F12/00, G06F17/30, G06F7/00, G06F3/00, G06F15/00, G06F15/16|
|Cooperative Classification||Y10S707/99933, Y10S707/99953, G06F17/30864|
|Jun 19, 2002||AS||Assignment|
Owner name: WATCHFIRE CORPORATION, CANADA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GRANCHAROV, CONSTANTINE;ROLLESTON, ANDREW;SMITH, DUNCAN;REEL/FRAME:012809/0175
Effective date: 20020618
|Jan 25, 2008||AS||Assignment|
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:WATCHFIRE CORPORATION;REEL/FRAME:020403/0899
Effective date: 20080118
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION,NEW YO
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:WATCHFIRE CORPORATION;REEL/FRAME:020403/0899
Effective date: 20080118
|Oct 8, 2012||REMI||Maintenance fee reminder mailed|
|Feb 24, 2013||LAPS||Lapse for failure to pay maintenance fees|
|Apr 16, 2013||FP||Expired due to failure to pay maintenance fee|
Effective date: 20130224