Content Sources> Feeds

Use the Content Sources > Feeds page to perform the following tasks:

• Viewing Feed Statistics
• Viewing Information about Current Feeds
• Viewing Feed Data Source Logs
• Downloading a Feed
• Restarting Feeds
• Deleting a Feed
• Specifying Trusted IP Addresses for Feeds

You can feed, or push, documents to the search appliance. Use feeds when you have internal documents that cannot be found by the crawler or that do not lend themselves to HTTP crawling. The feeds are provided through an API and are displayed on the Content Sources > Feeds page.

The Google Search Appliance can also accept database feeds. For information on database feeds, see the help page for Database Feeds.

The search appliance does not support indexing compressed files sent in content feeds.

Before Starting these Tasks

Before you can view feed information or perform tasks on this page, the search appliance must be receiving feeds. Feeds can come from connectors, databases, or other data sources. The "Feeds Protocol Developer's Guide," which is linked to the Google Search Appliance help center, contains information on creating feeds. The connector documentation set is linked to the page Documentation for the Google Search Appliance Connectors.

URLs fed using feeds must match the entries in the patterns set in Content Sources > Web Crawl > Start and Block URLs > Follow and Crawl Only URLs with the Following Patterns field. Otherwise, they are discarded. For example if you want to feed the URL googledb://example.com/, add ^googledb:// to ensure that the URL is successfully parsed.

Viewing Feed Statistics

The Content Sources > Feeds page displays the statistical information about feeds in several sections.

Feeds Received - Current Day

The table displays the following information.

• The number of feeds submitted to the search appliance since midnight of the current day
• The average size of the feeds
• The number of bytes submitted

If feedergate is restarted, the values in the table reset to zero. Feedergate is restarted when you take any of the following actions:

• Toggling the setting for Enable HTTP (non-SSL) access for Feedergate on the Administration > SSL Settings page
• Enabling or disabling Client Certificate Authentication for Feedergate on the Administration > SSL Settings page
• If Client Certificate Authentication for Feedergate is enabled, removing all certificate authorities
• Changing the list of trusted IP addresses on the Content Sources > Feeds page
• Enabling or disabling the search appliance software

The graph displays additional information about feeds received by the appliance. The number of feeds per minute are plotted on the graph's Y axis and the times at which the feeds were received are plotted on the X axis. The type of connection over which the feed was received is indicated by the color of the lines used to plot the feed information.

• Standard (non-secure) connections are indicated by a yellow line.
• Unauthorized connections are indicated by a red line
• Secure connections are indicated by a green line.
• Feeds rejected by the search appliance are indicated by a purple line.

Records Processed - Current Day

The Records Processed table displays the number of feed records processed since midnight of the current day. The Records Processed graph plots the number of records processed and the time at which the feeds were processed. Each record corresponds to a feed record URL

Total Backlog Count

The Total Backlog Count table displays the number of unprocessed feeds. The Feeds Backlog graph plots the number of unprocessed feeds at the time the feed queue is polled.

Viewing Information about Current Feeds

The following table describes the information about the data source feeds that you can view on the Content Sources > Feeds page in the Current Feeds section. The Current Feeds section shows a maximum of 10 entries for each data source.

Viewing Feed Data Source Logs

You can view a log file that records information about the particular documents that have been fed to the search appliance.

To view a feed data source log:

1. In the Current Feeds table, locate the feed whose log you want to view.
2. Click the link in the Status column.
   1. The log for a Completed feed typically lists the records added and the document URL and file name of the record.
   2. The log for a Failed in error feed provides an error message about the document at which the feed failed and the reason for the failed.

Downloading a Feed

Use the Download link for a specific feed to download its source. Download links are available for the latest 5 feeds for a data source. If the Download link is not available, then it is probable that the feed was deleted or the data source was destroyed.

The downloaded feed file is compressed in .gzip format, so for you may need additional software to extract it.

Restarting Feeds

A feed that is in progress might stop unexpectedly. You can recognize a stopped feed because it retains the status In Progress longer than expected. To restart feeds that have stopped, click Restart Feeds.

Deleting a Feed

On this page, you can delete or destroy data sources that have content feeds. You cannot use the controls on this page to delete or destroy data sources that have web or metadata-and-url feeds.

When you delete a feed data source, all documents associated with the feed are removed from the index. For information about deleting URLs submitted by web or metadata-and-url feeds, see the "Feeds Protocol Developer's Guide: Removing Feed Content from the Index," which is linked to the Google Search Appliance help center.

To delete a feed:

1. Click the Delete link that corresponds to the feed source name.
2. Click OK to confirm the deletion.

Once you have deleted a data source, you can destroy it by clicking the Destroy link. Destoying a data source removes the feed data source from this page.

Specifying Trusted IP Addresses for Feeds

You can specify that the search appliance can either trust feeds from all IP addresses or only trust feeds from specific IP addresses.

Trust Feeds from All IP Addresses

To trust feeds from all IP addresses:

1. Under List of Trusted IP Addresses, select Trust feeds from all IP addresses.
2. Click Save.

Only Trust Feeds from Specific IP Addresses

When you choose this option, list the IP addresses to trust. Enter the IP addresses of servers that the search appliance trusts to supply feed sources. The search appliance accepts individual IPv4 addresses in the format n.n.n.n, where n represents one octet of a value from 0 to 255. The system also accepts subnet IP addresses in the format n.n.n.n/Y where Y represents the subnet mask in a range of 0 to 32.

To trust feeds from specific IP addresses only:

1. Under List of Trusted IP Addresses, select Only trust feeds from these IP addresses.
   
2. Type the IP addresses.
   If you need to add more addresses, click the Add More Rows button.
3. Click Save.

For More Information

For more information about feeds, see the following topics:

• For database feed information, see Content Sources > Databases.
• To create a test feed, see the section "Quickstart" in the "Feeds Protocol Developer's Guide," which is linked to the Google Search Appliance help center.
• For connector feed information, see the "Connector Developer's Guide," which is linked to Documentation for the Google Enterprise Connectors.