|
|
Can I restrict Google from crawling my site on a specific day of the week?
|
Report abuse
|
|
Is it possible to restrict Google from crawling my website on Saturdays or Sundays routinely? I'd like it to crawl my site on weekdays only. Thanks
Replies:
1 of 9
All replies
|
Phil Payne
Top Contributor
Webmaster Help Bionic Poster
9/8/09
|
There is a defined interface - sniff out the Googlebot user-agent in a server-side script and serve it a 503 response code with a Retry-After header set for Monday.
The Googlebot is very, very likely to ignore the Retry-After - some years ago people were routinely putting up certain pages for s short window and getting the Googlebot to crawl them, then blocking them with such a 503 and Retry-After so that the Googlebot saw and indexed one lot of content but users generally saw another. So the Googlebot started ignoring Retry-After.
You can still block for the weekend and allow it in on weekdays, but beware that it might become suspicious and think you're trying this exploit.
Thanks for your answer. I'm not trying to trick Googlebot (but try explain that to a bot). Since our site is geared towards Sabbath observing people (Orthodox Jews), there's a demand that the site will be down on Saturday, because we are not allowed to conduct any business on this day. There's an application that provides this service (by figuring out if it's Saturday in the visitors region (by IP number) and puts a notice that the site is closed on Saturday), but before using them, I want to understand what effect it will have on Googlebot and other crawlers. Thank you, your answer was very helpful.
|
|
Phil Payne
Top Contributor
Webmaster Help Bionic Poster
9/8/09
|
Best answer
-
JohnMu
(Google Employee)
Yes - a 503 is the correct server response for "We're closed". If you substitute a normal HTML page saying "We're closed" and serve a 200 it's very likely to get indexed by Google.
If you give the Googlebot a 503, it will just go away and come back later without indexing what you give it.
For humans, you can serve a custom 503 page that explains the situation. Are there no other Orthodox sites you can ask, to see how they do it?
That's an interesting problem....
think I would consider an image with a phrase you consider appropriate that indicates you are closed for business and only serve that image up at the top of the page to visitors on a Saturday for instance... any other method could mean you don't get any "business" on other days....as per Phil's post....
I assume you are really wanting to inform human visitors and crawlers aren't the real problem.
p.s. BTW you would still deliver the usual page too, but insert the "closed" image using the s/w you have identified! Robots can't read images so should be a safe method.
|
|
Phil Payne
Top Contributor
Webmaster Help Bionic Poster
9/8/09
|
> BTW you would still deliver the usual page too, but insert the "closed" image using the s/w you have identified!
I don't think you can do that - I don't think you can show any business information at all. The 503 custom page, as I understand Orthodox rules, may not even say what the business does.
Some of these people are very strict. There was a case of a couple feeling trapped in their own home by an automatic light outside their door.
WOW that's going to be tough .... unless a new code is defined for this type of thing... but no doubt someone would find a way to abuse it! From what you are saying... a physical shop would need shutters over windows & signs etc .... The only methods I can think of to emulate, you would need to present the usual html but not show it to the visitor on the relevant day... but that could be disasterous for serps... I think the 503 method would probably have severe implications over time based on the definition below (as it could be seen that your service is unreliable):- "Service Unavailable
The server is currently unable to handle the request due to a
temporary overloading or maintenance of the server. The implication
is that this is a temporary condition which will be alleviated after
some delay. If known, the length of the delay MAY be indicated in a
Retry-After header. If no Retry-After is given, the client SHOULD
handle the response as it would for a 500 response."
I understood that you've a tool in place to 503 visitors based on their IPs. Why not just perform crawler detection using the recommended methods to exclude bots? Is having computers running unattended against the rules? If so, why is it ok to run a Web server? I mean even 503ing HTTP requests means that the machine works. Also, bots requesting pages that didn't change on Sabbath wouldn't trigger a change in a search engine's index to your advantage, and do so in favor of previously changed contents just delayed for technical reasons. Another way to handle bots during Sabbath would be to serve crawlers a 304 (not modified) but that would create a problem with newly discovered URIs because for those requests crawlers don't use the "if modified since" header. Anyways, from a technical POV I strongly suggest not to install any crawling barriers. Please forgive my ignorance regarding the religious aspects.
Sebastian, I think it depends on the Rabbi you ask.
rcneeman, is it about making money on Shabbat or about the servers working on Shabbat? The easy solution, if it was money making is to just drop the "add to cart" and shopping cart feature.
The loophole, if you want to call it that, is that most payment gateways do not transfer money made on a web site on the same day. i..e it takes a few days from the money to go from the merchant account to the bank account.
|
|
|
Feed
