Backoff exponentially if google scholar blocks us

[liuml07]

Currently we sleep 60 seconds. We may need to re-retry after 120 seconds, 240 seconds, 480 seconds..., instead of exiting immediately.

This way, the user can run the program as a long job without retrying manually.

[ruudkalis]

It seems to me that Google blocks for a whole day once it detected you are using a script / robot to get the citation data. Performing the captcha verification in my browser does not make any difference. I wrote a similar script in R that catches all citations for a certain publication but I keep running into the robot issue. I used between 10 - 30 seconds sleep time.

[liuml07]

Thanks @ruudkalis for sharing your experience. It is also true for me that the block time is longer than a few hours. A whole day might be. Performing manual captcha verification seems not helpful either here. Using cookie is not helpful.

Is 5 mins sleep time tolerable? We should be very patient to only handle ~10 citations per hours.

[yilihong]

@liuml07 This is tough. The block seems to be specific to the author page rather than the IP address? Even changing a machine doesn't help once blocked (on that specific citation). I will do some research into how google blocks robot scrapers. Not surprising Google is so good at detecting because the idea of Google was built on crawling the web.

[liuml07]

@yilihong Thanks for the observation. I don't know if there is any IP+cookie+URL pattern that is banned, but the author page should not be alone (or the IP address).

~200 citations / day seems OK to most people. So our last resort is to sleep longer, say 100 seconds as the code now. Anyway it merits to make the program sleep (even a day) instead of exiting in case of being blocked, so that users don't have to monitor the program and manually restart the program.