Recherches récentes
Options de recherche
#scraper
»Cloudflare Introduces Default Blocking of A.I. Data Scrapers«
Nett, wird aber kaum funktionieren. Weil: Fortgeschrittene Scraper nutzen Browser-Emulation und rotierende IPs, um sich als echte Nutzer auszugeben und technische Erkennung zu umgehen. Da es sich nur um eine serverseitige Maßnahme ohne rechtliche Bindung handelt, können solche Akteure die Sperren leicht und folgenlos ignorieren.
https://www.nytimes.com/2025/07/01/technology/cloudflare-ai-data.html
/kuk
The most disgusting feature of this relatively new #AI #scraper |s plague is that they are about to defile everything we like in the *good* internet.
Images with relevant #AltText? Perfect training materials for text-to-image generative models.
Static webpages? No #Anubis - no problem to scrape.
#Anubis uses proof-of-work ( #PoW ), which implies either #JavaScript or manual instructions. No, it is a good solution... Best of the worst (as if there were any good ones...)
Last days I learned that (1) #Tor has a #PoW mechanism (2) Anubis seems to somehow whitelist #lynx browser, allowing no-JS Lynx users in (a big favour for #accessibility and #smolweb ). Good (let's hope all these will persist).
Update: I reported the bot. Thanks.
A Mastodon bot account at mastodon.cloud scans the fediverse, scrapes selected web pages shared there, rewrites them with AI, posts them to its own site, and shares on Mastodon as tech news the rewritten AI slop. The bot scraped a post of mine (including the attached image) within minutes of my federated blog publishing it.
Is it worth flagging the bot and reporting it to its instance? Are the mods likely to take action?
@khobochka guess why I maintain a #Scraper #blocklist?
- In fact I know multiple people and organizations that decide to basically redirect #ValueRemoving #Scrapers like #GPTbot, #ByteSpider (which literally #DDoS'd #MattKC because #ClownFlare are a criminally incompetent #RogueISP!) to #Hetzner's 10GB Speedtest file which can be found at
http://hil-speed.hetzner.com/10GB.binas an extra middlefinger!
A(I)le bekloppt
Drüben im Blog der Uberspace-Betreiber findet sich ein sehr interessanter Artikel dazu, was die (mittlerweile anscheinend komplett hohldrehenden) Bots der AI-Firmen ohne Rücksicht auf Verluste so auslösen:
(…) Zusammenfassend lässt sich sagen, dass nach unserer Beobachtung rund 30 %-50 % aller Anfragen für kleine Seiten inzwischen von Bots generiert werden. Für große Seiten schwankt diese Zahl sogar zwischen 20 % und 75 %. In unseren Augen und mit Ignorieren der robots.txt ist damit inzwischen ein Punkt erreicht, an dem dieses Verhalten von Bots nicht mehr akzeptabel ist und unserem Betrieb schadet.
blog.uberspace.de
Bei meinen unregelmässigen Ausflügen in die Serverlogs meiner eigenen Seiten, aber auch von Auftritten meiner Kunden ist das genauso: Die bot-Zugriffe haben überproportional zugenommen und es ist teilweise wirklich heftig, mit welcher Frequenz und mit wieviel wechselnden IPs die Dinger auf die Site hämmern. >:-(
#Bots #DigitaleSelbstVerteidigung #robotsTxt #Scraper #WildWest
Another new LLM scraper just dropped: AI2 Bot.
First-party documentation does not list any way to opt-out except filtering the user-agent on your server/firewall. The docs list the following User-Agent to filter:
Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)
My server logs contained the following string:
Mozilla/5.0 (compatible) Ai2Bot-Dolma (+https://www.allenai.org/crawler)
That appears to be for Ai2’s Dolma product.
159 hits came from 174.174.51.252, a Comcast-owned IP in Oregon.
I recommend adding ai2bot to your server’s user-agent matching rules if you don’t want to be in the Dolma dataset; unlike Common Crawl, this seems tailored specifically for training LLMs with few other users.
And another AI scraping case (also see my previous post)…
AI video startup Runway reportedly trained on ‘thousands’ of YouTube videos without permission
Noo… Really?! 
Anthropic’s crawler is ignoring websites’ anti-AI scraping policies…
With regards to the utoots.com #scraper:
1. It currently depends on a Mastodon instance flashist[.]video; it is recommended to block the instance. flashist.(me|health) and previously flashist.(org|vip|live) is also operated by the same person. Ban evasion is to be expected.
2. I wrote a GitHub issue about it, archived at https://archive.ph/8ynKh. However he has chosen to cover up his GitHub profile instead.
Update: https://cyberpunk.lol/@vantablack/112849043193285926 (tldr: it's gone)
okay yeah https://utoots.com is DEFINITELY a scraper
i've updated the original post, making a reply too since edits don't always federate cleanly
just found another scraper indexer thingy
Ping @s0, I guess it’s time to reset the counter. https://www.reddit.com/r/Mastodon/s/TKQ5lMdIoY
Now I have a 
You may want to use:
[Dark Visitor](https://darkvisitors.com/)
And:
[ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt)
Adding to the list the `robots.txt` by #VLC:
[VLC robots.txt](https://www.videolan.org/robots.txt)
Source: **Bloquer les gaveurs d'IA** by @lord (in French)
https://lord.re/fast-posts/76-bloquer-les-gaveurs-dia/
#ChatGPT #IA #Robots #Scraper
**Bloquer les gaveurs d'IA**
https://lord.re/fast-posts/76-bloquer-les-gaveurs-dia/
Utiliser :
[Dark Visitor](https://darkvisitors.com/)
[ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt)
J'ajoute à la liste le `robots.txt` de #VLC également : [VLC robots.txt](https://www.videolan.org/robots.txt)
#ChatGPT #IA #Robots #Scraper
FEDI SCRAPER AND INDEXER
okay according to multiple peeps in the replies of the original post, this is indeed in fact a fedi scraper and indexer i found