Anyone else getting these ridiculous repo scraping spikes? A clean checkout of the https://thi.ng/umbrella monorepo is ~370MB. Over the past 14 days there were 222k clones (only 117 unique) of this repo which have caused downloads of a whopping ~78TB. WTF!

Website owner? Not keen on the Mellowtel browser library building a botnet of untraceable scrapers from unwitting users who are using a browser plugin that contains Mellowtel? I've raised a GitHub issue for them to explain how much contempt they have for our consent. Join in, politely, make them look like the jerks they are. https://github.com/mellowtel-inc/mellowtel-js/issues/41

List of AI bots to add to robots.txt (although they may not obey -- may need to throw them in the bitbucket and 404 or 444 them). In addition to these, you may have to block specific random browser versions for the most aggressive bots who ignore robots.txt.

https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt

Made a little Astro integration to easily disallow known AI scrapers in your site’s `robots.txt`

https://delucis.github.io/astro-ai-robots-txt/

You might have heard this already, but if you haven’t it’s important:

Russian #propaganda has been targeting #ai data #scrapers for intentionally with disinformation for some time.

”Despite having minimal organic reach among human audiences, the network’s focus on SEO ensures its content ranks highly in search results. This makes it more likely for AI chatbots ..”

What US oligarchs think makes them rich, #Russia uses to make itself more powerful.

https://disa.org/russian-propaganda-infiltrates-global-ai-tools-via-moscow-backed-network/

I’ve said this before, but the fact that pretty much every fedi scraper/bot/etc. developer gets immediately shot with a gun is bad, and seriously misaligns incentives.

A strange developer that announces their obvious scraper bot (that individuals and instances can block) is bleeding out, while the evil ones with secret bots sip their tea contentedly. People need to be okay with opt-out here, or we just functionally encourage the evil ones.

Posts on fedi are literally public, in any case. This isn’t a private chat-room, I don’t think it’s reasonable to expect others to treat it that way.

#lang_en #fedi #fediverse #privacy #scrapers #bots [Re: ]

#AI haters build #tarpits to trap and trick #AI #scrapers that ignore #robots.txt
#tarpit #security #privacy #robotstxt

https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/

If you use #cloudflare I just noticed a new option

not sure how effective it is but it's part of the free plan might as well

If #Cloudflare is to be believed, #Lemmy instances have a built-in AI scraping bot operating beneath the covers. Do you think the developers have snuck it in?

Looking through my logs, these requests have all been blocked by Cloudflare because they are identified as "AI Bots". There are many more requests by Lemmy instances blocked in the logs. This is just a sample. Other Lemmy requests from these servers get through. Only a few are blocked as AI Bots.

Cloudflare says they use AI to determine if a request is a legitimate request or an AI bot trying to scrape.

207.204.58.144
AS19045 DIRECTCOM
United States
User agent: Lemmy/0.19.5; +https://lemmy.cryonex.net

23.127.223.238
AS7018 ATT-INTERNET4
United States
User agent: Lemmy/0.19.3; +https://lemux.minnix.dev

2a01:cb19:f85:ec00:82fa:5bff:fe51:ed4a
AS3215 France Telecom - Orange
France
User agent: Lemmy/0.19.5; +https://lemmy.sidh.bzh

50.247.53.42
AS7922 COMCAST-7922
United States
User agent: Lemmy/0.19.5; +https://toast.ooo

69.42.19.234
AS11404 AS-WAVE-1
United States
User agent: Lemmy/0.19.5; +https://lemmy.schlunker.com

155.138.226.183
AS20473 AS-CHOOPA
United States
User agent: Lemmy/0.19.5; +https://lemmy.mbl.social

#konterfAI, the #AI #modelpoisoner for unfriendly, respectless and malicious AI #scrapers / #crawlers has now an interesting statistics function that collects data about the nasty "guests" that don't respect your robots.txt.... Enjoy!
#robots_txt
See update on
https://korium.org/2024/08/02/konterfai/
and version 0.2.0 on
https://codeberg.org/konterfai/konterfai/releases/tag/v0.2.0

New #blog: Autodetecting and Announcing #Mastodon Scrapers and Crawlers

There've been quite a few #fedisearch issues recently, but the common thread is that there's usually a gap in reporting - they're often live for weeks before people are made aware.

It's not just people's pet projects either, there are other #scrapers active, quietly consuming posts

So, I built a bot to detect and out them so that fedi admins can block as necessary

https://www.bentasker.co.uk/posts/blog/security/autodetecting-and-outing-mastodon-scrapers-with-scrapersnitchbot.html

New #blog: Tightening #security control over #mastodon public #api endpoints

The concern in fediblock around @cloy's #fedisearch plans earlier in the week prompted me to put my #infosec hat on and look into ways to make it harder for external #scrapers to hit Mastodon's API feeds.

This post suggests a possible solution for concerned instance admins as well as details of some #crawlers I spotted.

https://www.bentasker.co.uk/posts/blog/security/restricting-unauthenticated-access-to-mastodons-public-feeds.html