An update to Google’s privacy policy suggests that the entire public internet is fair game for it’s AI projects. If Google can read your words, assume they belong to the company now, and expect that they’re nesting somewhere in the bowels of a chatbot.
But when the world needed them most the robots(.txt) were no where to be found
It’s kind of you to assume that they respect those, lol.
Oh and they won’t. We had a lengthy verification process where we proved we owned these sites we didn’t want aggregated and they still refused to honor robots.txt.