• tal@lemmy.today
    link
    fedilink
    English
    arrow-up
    28
    arrow-down
    3
    ·
    edit-2
    3 months ago

    Robots.txt has been around for a long time, and all the major search engines will honor it. Not having a full index of the Web is the norm.

    That isn’t to say that the practice of signing agreements isn’t potentially a concern. Not sure that I like the idea of search engines paying sites money to degrade search results of competitors.