Créer un compteSe connecter

Recherches récentes

Aucune recherche récente

Options de recherche

Disponible uniquement lorsque vous êtes connecté.
mastouille.fr est l'un des nombreux serveurs Mastodon indépendants que vous pouvez utiliser pour participer au fédiverse.
Mastouille est une instance Mastodon durable, ouverte, et hébergée en France.

Administré par :

Statistiques du serveur :

640
comptes actifs

mastouille.fr: À proposAnnuaire des profilsPolitique de confidentialité

Mastodon: À proposTélécharger l’applicationRaccourcis clavierVoir le code sourcev4.3.4

#robotstxt

0 message0 participant0 message aujourd’hui
Suite du fil
Public
George E. (he/him/his)

Here's #Cloudflare's #robots-txt file:

# Cloudflare Managed Robots.txt to block AI related bots.

User-agent: AI2Bot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: amazon-kendra
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Applebot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: AwarioRssBot
Disallow: /

User-agent: AwarioSmartBot
Disallow: /

User-agent: bigsur.ai
Disallow: /

User-agent: Brightbot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: DigitalOceanGenAICrawler
Disallow: /

User-agent: DuckAssistBot
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: FriendlyCrawler
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: iaskspider/2.0
Disallow: /

User-agent: ICC-Crawler
Disallow: /

User-agent: img2dataset
Disallow: /

User-agent: Kangaroo Bot
Disallow: /

User-agent: LinerBot
Disallow: /

User-agent: MachineLearningForPeaceBot
Disallow: /

User-agent: Meltwater
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: meta-externalfetcher
Disallow: /

User-agent: Nicecrawler
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: omgili
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: PanguBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: PiplBot
Disallow: /

User-agent: QualifiedBot
Disallow: /

User-agent: Scoop.it
Disallow: /

User-agent: Seekr
Disallow: /

User-agent: SemrushBot-OCOB
Disallow: /

User-agent: Sidetrade indexer bot
Disallow: /

User-agent: Timpibot
Disallow: /

User-agent: VelenPublicWebCrawler
Disallow: /

User-agent: Webzio-Extended
Disallow: /

User-agent: YouBot
Disallow: /
#robotstxt
Public
Inautilo

#Business #Explorations
What would happen if I blocked big search? · Pros and cons of blocking major search engines ilo.im/163yb3

_____
#SearchEngine #SEO #AI #Website #Blog #RobotsTxt #Development #WebDev #Frontend #Backend

Luke's Wild Website · What would happen if I blocked big search?The major search engines are showing slop summaries first and websites second. They link to sites sometimes, but they’d much prefer searchers stay on the results page and ask followup questions. Engagement numbers and all that. These companies run mass content scraping campaigns to train their own slop models and provide servers to other slop companies in the same data centers their crawlers operate from. Every word I’ve thrown into this illustrious publication—the reading of which a total of five people have, at various times, expressed mild enjoyment—has been both silently and explicitly declared fair game by the slop companies. The only positive thing I can egotistically hope to grasp about this development is that a distant and diluted abstraction of my words and thoughts could be subtly influencing the thoughts and decisions of people around the world—the same way a pinch of salt would affect the flavor profile of a large vat at the soup factory, or like the breeze under the wings of chaos theory’s cute and frighteningly powerful butterfly mascot.
A répondu dans un fil de discussion
Public
jeSuisatire …ᘛ⁐̤ᕐᐷ

@ErikUden

Worrying is their self centered megalomanic ego trip, not realizing that they are the remaining world power, armed to their teeth with weapons of all kind, and with all the private data of the worlds population.

That said, having in mind that apparently you, being in charge of several #mastodon instances in the #fediVerse, are not able to fix the #robotsTxt of them while wasting time about talking of other countries internal affairs is kinda embarrassing.
sry

#justSayin#fediAdmin
Suite du fil
Public
Konstantin Weddige

Extending the meta tags would be fairly straightforward. In addition to the existing "INDEX", "NOINDEX", "FOLLOW", "NOFOLLOW", we could introduce "MODELTRAINING" and "NOMODELTRAINING".

Of course, just because there is an RfC does not mean that anyone will follow it. But it would be a start, and something to push for. I would love to hear your opinion.

3/3

#robotstxt#rfc
Suite du fil
Public
Konstantin Weddige

This is not an acceptable situation and therefore I propose to extend the robots.txt standard and the corresponding HTML meta tags.

For robots.txt, I see two ways to approach this:

The first option would be to introduce a meta-user-agent that can be used to define rules for all AI bots, e.g. "User-agent: §MODELTRAINIGN§".

The second option would be a directive like "Crawl-delay" that indicates how to use the data. For example, "Model-training: disallow".

2/3

· 3 personnes · Fermé
#robotstxt#rfc
Public
Konstantin Weddige

I was just confronted with the question of how to prevent a website being used to train AI models. Using robots.txt, I only see two options, both of which are bad in different ways:

I can either disallow all known AI bots while still being guaranteed to miss some bots.

Or I can disallow all bots and explicitly allow known search engines. This way I will overblock and favour the big search engines over yet unknown challengers.

1/3

#robotstxt#AI#LLM
A répondu dans un fil de discussion
Public *
David Sardari

@Codeberg Thanks for the inspiration. After checking it, I'll use the output of following command in the robots.txt of my doc:

❯ bash -c '(
    curl -fsS --tlsv1.3 https://codeberg.org/robots.txt | \
      tac | \
      grep -A999 "^Disallow: /$" | \
      grep -m1 -B999 "^[[:space:]]*$" | \
      tac
    curl -fsS --tlsv1.3 https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/main/robots.txt
  ) | sort -ru'
User-agent: omgilibot
User-agent: omgili
User-agent: meta-externalagent
User-agent: img2dataset
User-agent: facebookexternalhit
User-agent: cohere-ai
User-agent: anthropic-ai
User-agent: YouBot
User-agent: Webzio-Extended
User-agent: VelenPublicWebCrawler
User-agent: Timpibot
User-agent: Scrapy
User-agent: PetalBot
User-agent: PerplexityBot
User-agent: Omgilibot
User-agent: Omgili
User-agent: OAI-SearchBot
User-agent: Meta-ExternalFetcher
User-agent: Meta-ExternalAgent
User-agent: ImagesiftBot
User-agent: ICC-Crawler
User-agent: GoogleOther-Video
User-agent: GoogleOther-Image
User-agent: GoogleOther
User-agent: Google-Extended
User-agent: GPTBot
User-agent: FriendlyCrawler
User-agent: FacebookBot
User-agent: Diffbot
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: ChatGPT-User
User-agent: CCBot
User-agent: Bytespider
User-agent: Applebot-Extended
User-agent: Applebot
User-agent: Amazonbot
User-agent: Ai2Bot-Dolma
User-agent: AI2Bot
Disallow: /
#ai#seo#robotstxt
Public *
Webrocker

A(I)le bekloppt

Drüben im Blog der Uberspace-Betreiber findet sich ein sehr interessanter Artikel dazu, was die (mittlerweile anscheinend komplett hohldrehenden) Bots der AI-Firmen ohne Rücksicht auf Verluste so auslösen:

(…) Zusammenfassend lässt sich sagen, dass nach unserer Beobachtung rund 30 %-50 % aller Anfragen für kleine Seiten inzwischen von Bots generiert werden. Für große Seiten schwankt diese Zahl sogar zwischen 20 % und 75 %. In unseren Augen und mit Ignorieren der robots.txt ist damit inzwischen ein Punkt erreicht, an dem dieses Verhalten von Bots nicht mehr akzeptabel ist und unserem Betrieb schadet.

blog.uberspace.de

Bei meinen unregelmässigen Ausflügen in die Serverlogs meiner eigenen Seiten, aber auch von Auftritten meiner Kunden ist das genauso: Die bot-Zugriffe haben überproportional zugenommen und es ist teilweise wirklich heftig, mit welcher Frequenz und mit wieviel wechselnden IPs die Dinger auf die Site hämmern. >:-(

#Bots #DigitaleSelbstVerteidigung #robotsTxt #Scraper #WildWest

https://webrocker.de/?p=29216

blog.uberspace.deBad Robots
Public *
Ecologia Digital

#Robotstxt #CrawlerBacklash Trickle-down effects: "people start blocking all crawlers, and some crawlers are very important, for search indexing, internet archiving, some are used for academic research, and so the bad behaviours of all these #AIcompanies, and the backlash to it, is kind of fundamentally changing how the Internet works, how it is remembered and indexed..."
pca.st/yto6v3il?t=11m34s

Pocket CastsGoogle, Reddit, and the Robots.txt Rebellion - The 404 Media PodcastWelcome to the podcast from 404 Media where Joseph, Sam, Emanuel, and Jason catch you up on the stories we published this week. 404 Media is a journalist-owned digital media company exploring the way technology is shaping–and is shaped by–our world. We bring you unparalleled access to hidden worlds both online and IRL through investigative reporting, smart blogging, and breaking news. At 404 Media you’ll read, and hear, stories you can’t find anywhere else written by journalists who are leading experts on their beats. Subscribe to 404 Media at 404media.co to gain access to an ad-free version of this podcast, as well as a bonus podcast episodes. Subscribers are the bedrock of building a sustainable business for our journalism. Hosted on Acast. See acast.com/privacy for more information.
Public
JdeBP

@ianb @Meyerweb

I was about to ask, myself, where that bloke was who says "Hah! No." to such questions. (-:

That said, if the argument was (and is) that 35 years was an egregious and unjust suggestion by prosecutors, it is *still* surely egregious and unjust in *other* cases if one contends that they are alike.

*That* said, I wonder how that "Hah! No." bloke answers the question: Is ignoring robots.txt illegal?

(-:

law.stackexchange.com/q/77755/

Law Stack ExchangeDoes the robots exclusion standard have any legal weight?There is a standard through which websites communicate to the web crawlers upon which search engines are based which pages should be indexed and included in the search results. This is called the ...
#USLaw#RobotsTxt#Anthropic
Public
Seirdy

New scraper just dropped (well, an old scraper was renamed):

Facebook/Meta updated its robots.txt entry for opting out of GenAI data scraping. If you blocked FacebookBot before, you should block meta-externalagent now:

User-Agent: meta-externalagent
Disallow: /

Official references:

Meta for DevelopersAbout FacebookBot - Partage - Documentation - Meta for Developers
#RobotsTxt
Public *
Stefan Bohacek

Just throwing out a thought before I do some research on this, but I think robots.txt needs an update.

Ideally I'd like to define an "allow list" that tells web scrapers how my content can be used. Eg.:

- monetizable: false
- fediverse: true
- nonfediverse: false
- ai: false

Etc. And I'd like to apply this to my social media profile and any other web presence, not just my personal website.

#internet#fediverse#SocialMedia
ExplorerFlux en direct
À propos