google.com/goto: Google's anti-scraping update
google.com/goto: Google's anti-scraping update
Google is rolling out google.com/goto redirect URLs on SERPs. Here is what changed, why it targets scrapers, and how Autom updated its Google Search API.
…but that’s how you follow a redirect?
You’re right, and this article is spouting nonsense. If you have to request the
/gotolink in order to get the URL then you’ve already been tracked by Google (or blocked if they think you’re a scraper), and following theLocationheader is exactly what your browser would do anyway. That’s how HTTP redirects work.This article is from a company that offers APIs to just proxy Google search results, so they’re exactly the kind of people that Google is trying to block with this change. Reading some other posts on their site (eg. the one that explains how to follow the Location header) there are obvious signs of slop, so this whole article is probably an AI hallucination.
One is under your control. The other is being passed through a script on the server that may be doing who-knows-what in addition to redirecting you.
(Better if you just drop Google altogether.)
No? If you want to resolve this URI, you have to follow the redirect, there’s no way around it. Requesting it from google and looking at the
Locationheader is following the redirect.I mean, I agree that one should drop google and that this is probably just another thing they do to track people, but that’s not what I meant with that comment.
Can someone just “distillation attack” google search so I can just use the derived data instead of giving them all my click tracking?
No. But you can use alternative search engines, or frontends like searxng that won’t send your clicks to Google (while they still work, anyway, this change might break them)
I’d rather just have the original dataset.
That’s the thing they’ve built the moat around.
You could build a browser extension to extract search results but the infrastructure will probably be expensive.