Status and Reports > Real-time Diagnostics

Use the Status and Reports > Real-time Diagnostics page to retrieve and view real-time information for the search appliance, including HTTP headers for a specific URL and traffic on the network layer. This help page contains information about the following topics:

• Viewing HTTP Headers for a URL
• Viewing Crawl and Index Activity
• Running Network Capture
• Obtaining a Serving Timing Data Profile

Viewing HTTP Headers for a URL

Use the Manual fetch request section of this page to view HTTP headers for a specified URL and debug headers for SMB URLs. This feature uses the crawler to fetch a URL using all the crawler settings (for example, security and proxy). Fetching a URL does not affect the crawl and indexing of the URL nor does it use crawl pipeline information for the URL, such as If-Modified-Since.

To fetch a URL and view the HTTP headers:

1. Under Manual fetch request, type the URL in the Fetch headers for URL box.
2. Click Fetch.
   The headers for the URL appear in the Headers box.

Viewing Crawl and Index Activity

Use the View crawl and indexing activity section of this page to capture and view information about URLs (including content feeds) that the search appliance is currently crawling and indexing. For each URL that is captured, you can view information, such as HTTP headers.

Limit capture of URLs to documents of interest by specifiying a pattern for filtering URLs in the URL pattern to capture box. Only URLs that match the specified pattern appear in the list of captured URLs. For example, to capture only URLs in example.com/, enter this URL pattern in the box. You can specify any valid URL pattern. For information about valid URL patterns, see "Constructing URL Patterns" in "Administering Crawl," which is linked to the Google Search Appliance help center.

To capture and view information about URLs that the search appliance is currently crawling and indexing:

1. Under View crawl and indexing activity, type a URL pattern in the URL pattern to capture box.
2. Click Start Capture.
   The list of captured URLs appears in the Captured URLs box. To view the HTTP headers for a URL, expand the URL by clicking the expand icon.

You can end a capture at any time by clicking the Stop Capture button. Take note that capture automatically stops if 1 GB of data has been collected.

Also, you can export a text capture file that contains the detailed information about URLs.

To export a file:

1. Click the Export link next to the Start/Stop Capture button.
2. Browse to a location on the local computer for the file and click Save.

To request recrawl of a single URL, use the Status and Reports > Crawl Diagnostics page. To request recrawl for a set of patterns, use the Crawl and Index > Freshness Tuning page.

Running Network Capture

Use the Run packet capture section of this page to generate and download a file that contains information about current search appliance network traffic, such as crawling and serving traffic. The file is created by using tcpdump. You can view the downloaded captured file by using a network protocol analyzer tool such as Wireshark.

By default, all traffic is captured, but you can set a filter to limit the capture by hosts. To set a filter, type a comma-separated list of fully qualified hostnames or IP addesses in the box under Run packet capture and select one of the following options:

• Include--Show traffic going to the specified hosts only
• Exclude--Show all traffic, except that which is going to the specified hosts
  Note: If the box is empty and exclude is selected, then all traffic is captured (that is, Include is assumed).

To capture all traffic, leave the value empty.

To run network capture and download the captured file:

1. Under Run packet capture, enter filtering information, as described in the preceding paragraphs (optional).
2. Click Start Capture.
   You can monitor the capture by the checking the values that appear in the file size and packets captured fields next to the Export link. Take note that capture automatically stops if the output file size is greater than 1 GB.
   
3. Click the Export link.
   You can export the file while the capture is running.
4. Browse to a location on the local computer for the file and click Save.

Obtaining a Serve-Time Data Profile

A serve-time data profile captures information about serving and the speed of external content servers. You capture data for an arbitrary period of time that you determine, then export a file containing the data profile. When you open the file, you see a report similar to the following:

In-time: 1270054816189674496, Out-time: 1270054816428750336
1270054816363530752 verifiedUserId: test session id: d39146a218a503a61d67fef51f876303
1270054816363589632 groups: [] session id: d39146a218a503a61d67fef51f876303
1270054816364732416 This batch of requests completed. Elapsed time: 0 ms session id: d39146a218a503a61d67fef51f876303
1270054816364747520 Authorization results for session: d39146a218a503a61d67fef51f876303 decision [1/2] PERMIT by CACHE for URL http://mygsa.example.com:8080/secure
1270054816364804864 Authorization results for session: d39146a218a503a61d67fef51f876303 decision [2/2] DENY by CACHE for URL http://mygsa.example.com:8080/secure?param=806
In-time: 1270054823722820608, Out-time: 1270054823907411968
1270054823886574848 verifiedUserId: test session id: d39146a218a503a61d67fef51f876303
1270054823886611456 groups: [] session id: d39146a218a503a61d67fef51f876303
1270054823887193088 This batch of requests completed. Elapsed time: 0 ms session id: d39146a218a503a61d67fef51f876303
1270054823887865088 Authorization results for session: d39146a218a503a61d67fef51f876303 decision [1/2] PERMIT by CACHE for URL http://mygsa.example.com:8080/secure
1270054823887974144 Authorization results for session: d39146a218a503a61d67fef51f876303 decision [2/2] DENY by CACHE for URL http://mygsa.example.com:8080/secure?param=806
In-time: 1270054748705101056, Out-time: 1270054748736264704

To obtain a serving timing data profile:

1. Click Start Capture. You see a warning that starting a new data capture deletes previously captured data.
2. After a period of time elapses, click Stop Capture.
3. Click Export. The file Serve.log.html is exported.
4. Save or view the exported file.