Content Sources > Diagnostics > Crawl Queue

Use the Content Sources > Diagnostics > Crawl Queue page to define crawl queue snapshots and view information about the crawl queue. This help page contains information about the following topics:

• About the Crawl Queue and Crawl Queue Snapshots
• Before Starting these Tasks
• Creating Crawl Queue Snapshots
• Viewing Crawl Queue Snapshots

About the Crawl Queue and Crawl Queue Snapshots

The crawl queue is the set of URLs that the appliance is waiting to crawl or that are overdue to be crawled. Use this information to help you determine whether specific hosts are crawled at the right time and why information from certain documents is fresher than information from other documents.

Currently Queued Hosts shows a list of hosts that the search appliance is waiting to crawl and the associated host load. The queue continuously changes as the crawler processes the queued hosts. To ensure that the list of hosts is up-to-date, click Refresh.

The crawl queue is dynamic, because the queue continuously changes as the crawler processes new URLs. A crawl queue snapshot includes current information, but does not indicate what will happen in the future. The crawl queue changes constantly.

The table of crawl queue snapshots lists previously captured snapshots. You can view the contents of a snapshot, export it to a comma-separated value (.csv) file, or delete it.

A snapshot appears in the table as soon as you start the capture by clicking Capture Crawl Queue. When a snapshot is in progress, its status is Capturing, and it has a Cancel link. Once the snapshot is complete, the Cancel link becomes a Delete link.

Before Starting these Tasks

Before you start these tasks, ensure that crawl patterns are entered on the Admin Console and the crawler is running. Note that creating a crawl queue snapshot is resource-intensive and reduces the crawler's performance.

Creating Crawl Queue Snapshots

Use these instructions to create a crawl queue snapshot. Note that the search appliance can create only one crawl queue snapshot at a time. After you start creating a snapshot, wait to start another snapshot until the first is created.

To define a crawl queue snapshot:

1. Click Content Sources > Diagnostics > Crawl Queue.
2. In the Name field, type a report name of up to twenty characters, which must consist of ASCII or non-ASCII characters, hyphens, and underscores. The report name cannot start with a hyphen.
3. In the Number of URLs to include field, specify the number of queued URLs to include in the snapshot. The number can be from one to 100,000.
4. In the Forthcoming hours to include field, specify the number of future hours of scheduling to include in the snapshot, starting from the current time.
5. To limit the snapshot to a single host, click Include URLs from this host only and enter a hostname. By default, the crawl queue snapshot includes all hosts.
6. Click Get Snapshot.

Viewing Crawl Queue Snapshots

To display snapshots for a particular host, click View, then click the host name.

Each snapshot contains the following information.