Mastouille

0 message0 participant0 message aujourd’hui

Bob Carver<a href="https://www.the-independent.com/tech/bots-internet-traffic-ai-chatgpt-b2733450.html" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.the-independent.com/tech/bots-internet-traffic-ai-chatgpt-b2733450.html</a> <a href="https://infosec.exchange/tags/cybersecurity" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#cybersecurity</a> <a href="https://infosec.exchange/tags/bots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#bots</a> <a href="https://infosec.exchange/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://infosec.exchange/tags/LLMs" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LLMs</a> <a href="https://infosec.exchange/tags/Bytedance" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Bytedance</a> <a href="https://infosec.exchange/tags/ByteSpider" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ByteSpider</a>

Kevin Karhan :verified:<a href="https://mastodon.social/@khobochka" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@khobochka</a> guess why I <a href="https://github.com/greyhat-academy/lists.d/blob/main/scrapers.ipv4.block.list.tsv" rel="nofollow noopener noreferrer" target="_blank">maintain</a> a <a href="https://infosec.space/tags/Scraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraper</a> <a href="https://infosec.space/tags/blocklist" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#blocklist</a>?<ul><li>In fact I know multiple people and organizations that decide to basically redirect <a href="https://infosec.space/tags/ValueRemoving" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ValueRemoving</a> <a href="https://infosec.space/tags/Scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scrapers</a> like <a href="https://infosec.space/tags/GPTbot" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#GPTbot</a>, <a href="https://infosec.space/tags/ByteSpider" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ByteSpider</a> (which <a href="https://www.youtube.com/watch?v=Hi5sd3WEh0c" rel="nofollow noopener noreferrer" target="_blank">literally</a> <a href="https://infosec.space/tags/DDoS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DDoS</a>'d <a href="https://infosec.space/tags/MattKC" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#MattKC</a> because <a href="https://infosec.space/tags/ClownFlare" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ClownFlare</a> are a criminally incompetent <a href="https://infosec.space/tags/RogueISP" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RogueISP</a>!) to <a href="https://infosec.space/tags/Hetzner" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Hetzner</a>'s <a href="http://hil-speed.hetzner.com/" rel="nofollow noopener noreferrer" target="_blank">10GB Speedtest file</a> which can be found at <code>http://hil-speed.hetzner.com/10GB.bin</code> as an extra middlefinger!</li></ul><a href="https://infosec.space/tags/Cloudflare" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Cloudflare</a> <a href="https://infosec.space/tags/hetznered" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hetznered</a> <a href="https://infosec.space/tags/ByteDance" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ByteDance</a> <a href="https://infosec.space/tags/ChatGPT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ChatGPT</a>

Neil CraigThe response we got from ByteDance on what ByteSpider does has not added any detail "is used in ByteDance products or future products" though it does specifically say it's for "search". I don't think we're going to get anything helpful from them. <a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SEO</a> <a href="https://mastodon.social/tags/byteSpider" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#byteSpider</a>

Neil CraigWe recently noticed a fair bit of traffic on www.bbc.co.uk & www.bbc.com from a User Agent which identifies itself as "ByteSpider" (& has a @bytedance.com email address).Lots of docs on the web state it doesn't obey robots.txt but ByteDance have told us it *does*:> *...in the robots.txt files* *> user-agent:Bytespider* *> Disallow:/*Thought that might be worth documenting as it might be a recent change & several of us searched but found zero docs from ByteDance<a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SEO</a> <a href="https://mastodon.social/tags/ByteSpider" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ByteSpider</a> <a href="https://mastodon.social/tags/RobotsTXT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTXT</a>

Recherches récentes

Options de recherche

Administré par :

Statistiques du serveur :

#bytespider