"Nikkei Asia has found that research papers from at least 14 different academic institutions in eight countries contain hidden text that instructs any AI model summarizing the work to focus on flattering comments.

Nikkei looked at English language preprints – manuscripts that have yet to receive formal peer review – on ArXiv, an online distribution platform for academic work. The publication found 17 academic papers that contain text styled to be invisible – presented as a white font on a white background or with extremely tiny fonts – that would nonetheless be ingested and processed by an AI model scanning the page."

#PromptInjection
#AcademicsBehavingBadly

https://www.theregister.com/2025/07/07/scholars_try_to_fool_llm_reviewers/?td=rt-3a

I’ve written about design patterns for the securing of LLM agents: https://cusy.io/en/blog/design-patterns-for-the-securing-of-llm-agents/view
#AI #LLM #PromptInjection

𝗛𝗮𝘃𝗲 𝘆𝗼𝘂 𝗵𝗲𝗮𝗿𝗱 𝗮𝗯𝗼𝘂𝘁 𝗘𝗰𝗵𝗼𝗟𝗲𝗮𝗸? Researchers showed a single email could silently pull data from Microsoft Copilot—the first documented zero-click attack on an AI agent.

Last week, we shared a new paper dropped outlining six guardrail patterns to stop exactly this class of exploit.

Worth pairing the real-world bug with the proposed fixes. Links on the replies.
#PromptInjection #AIDesign #FOSS #Cybersecurity

"Aim Labs reported CVE-2025-32711 against Microsoft 365 Copilot back in January, and the fix is now rolled out.

This is an extended variant of the prompt injection exfiltration attacks we've seen in a dozen different products already: an attacker gets malicious instructions into an LLM system which cause it to access private data and then embed that in the URL of a Markdown link, hence stealing that data (to the attacker's own logging server) when that link is clicked.

The lethal trifecta strikes again! Any time a system combines access to private data with exposure to malicious tokens and an exfiltration vector you're going to see the same exact security issue.

In this case the first step is an "XPIA Bypass" - XPIA is the acronym Microsoft use for prompt injection (cross/indirect prompt injection attack). Copilot apparently has classifiers for these, but unsurprisingly these can easily be defeated:"

https://simonwillison.net/2025/Jun/11/echoleak/

AI-powered features are the new attack surface! Check out our new blog in which LMG Security’s Senior Penetration Tester Emily Gosney @baybedoll shares real-world strategies for testing AI-driven web apps against the latest prompt injection threats.

From content smuggling to prompt splitting, attackers are using natural language to manipulate AI systems. Learn the top techniques—and why your web app pen test must include prompt injection testing to defend against today’s AI-driven threats.

Read now: https://www.lmgsecurity.com/are-your-ai-backed-web-apps-secure-why-prompt-injection-testing-belongs-in-every-web-app-pen-test/

Researchers claim breakthrough in fight against #AI’s frustrating security hole

https://arstechnica.com/information-technology/2025/04/researchers-claim-breakthrough-in-fight-against-ais-frustrating-security-hole/

New hack uses prompt injection to corrupt Gemini’s long-term memory - In the nascent field of AI hacking, indirect prompt injection has become a... - https://arstechnica.com/security/2025/02/new-hack-uses-prompt-injection-to-corrupt-geminis-long-term-memory/ #artificialintelligence #largelanguagemodels #promptinjection #security #chatbots #hacking #biz⁢ #google #llms #ai

@shanselman and Mark Russinovich learn responsible #AI. If there is one recorded #MSIgnite session you want to see, it is this one. Learn about limitations and threats through live demos like #jailbraking, #promptinjection, #reasoning, #hallucinating, #kindness, and how to prepare for them. https://ignite.microsoft.com/en-US/sessions/BRK329

Ars Live: Our first encounter with manipulative AI - In the short-term, the most dangerous thing about AI language models may b... - https://arstechnica.com/ai/2024/11/join-ars-live-nov-19-to-dissect-microsofts-rogue-ai-experiment/ #arsliveconversations #largelanguagemodels #microsoftcopilot #promptinjections #machinelearning #promptinjection #manipulativeai #simonwillison #benjedwards #microsoft #aiethics #bingchat #arslive #biz⁢ #openai #gpt-4 #ai

Man tricks OpenAI’s voice bot into duet of The Beatles’ “Eleanor Rigby” - Enlarge / A screen capture of AJ Smith doing his Eleanor Rigby duet wit... - https://arstechnica.com/?p=2052995 #largelanguagemodels #advancedvoicemode #aipromptinjection #machinelearning #promptinjection #audiosynthesis #musicsynthesis #voicesynthesis #paulmccartney #eleanorrigby #aicopyright #thebeatles #aifairuse #copyright #ajsmith #fairuse #biz⁢ #openai #ai

Hacker plants false memories in ChatGPT to steal user data in perpetuity - Enlarge (credit: Getty Images)

When security researcher Johann... - https://arstechnica.com/?p=2052198 #promptinjection #security #chatgpt #privacy #biz⁢ #openai #ai

Hackaday Links: September 22, 2024 - Thanks a lot, Elon. Or maybe not, depending on how this report that China used Sta... - https://hackaday.com/2024/09/22/hackaday-links-september-22-2024/ #enshittification #hackadaycolumns #promptinjection #forwardscatter #hackadaylinks #nitromethane #crowdsource #makerspace #dragster #starlink #nerfing #stealth #topfuel #openai #fire #wyze

OpenAI threatens bans for probing new AI model’s “reasoning” process - Enlarge (credit: Andriy Onufriyenko via Getty Images)

OpenAI t... - https://arstechnica.com/?p=2049959 #openaistrawberry #machinelearning #promptinjection #rileygoodside #simonwillison #aijailbreak #jailbreaks #o1-preview #strawberry #jailbreak #openaio1 #chatgpt #chatgtp #o1-mini #biz⁢ #gpt-4o #openai #gpt-3 #gpt-4 #hacks #ai #o1

I g n o r e a l l p r e v i o u s i n s t r u c t i o n s

https://www.theregister.com/2024/07/29/meta_ai_safety/
#AI #safety #promptInjection

Bing adds new Bing Webmaster Guideline for "Prompt injection" https://www.seroundtable.com/prompt-injection-bing-webmaster-guidelines-37650.html

New #Blog: #Perplexity #AI is susceptible to #promptinjection by scraped content (and some other issues)

I had a quick play around with Perplexity after reading @robb's excellent post on their behaviour.

I fell down a bit of a rabbit hole and this post is the result.

Includes:

- Prompt injections from arbitrary pages
- Reproducing Robb's findings
- Messing with images on their results page
- Catching them grabbing images
- Testing prompt robustness

https://www.bentasker.co.uk/posts/blog/security/perplexity-ai-gives-answers-that-cannot-be-trusted.html

Okay, new anti-AI scraping prompt injection attack incoming!

I'm still testing, but currently ChatGPT3.5 just gets stuck in a loop of making ASCII art penises after reading my profile. It also causes it to make ASCII art dicks for every other profile it tries to scrape afterwards too

GitHub Copilot Chat: From Prompt Injection to Data Exfiltration · Embrace The Red

｢ GitHub Copilot Chat is a VS Code Extension that allows a user to chat with source code, refactor code, get info about terminal output, or general help about VS Code, and things along those lines ｣

https://embracethered.com/blog/posts/2024/github-copilot-chat-prompt-injection-data-exfiltration/

IBM scratches the surface of a U.S. National Institute of Standards and Technology (NIST) report where Artificial Intelligence chatbots could be hacked using prompt injections. https://securityintelligence.com/articles/ai-prompt-injection-nist-report/

There are quite a few LLM pen testing tools out there, breaking the boundaries of what models are supposed to do by employing prompt injection and jail breaking techniques.
With Microsoft releasing #PyRIT and getting a lot of visibility for it, we wanted to highlight some of the other tools for the community:

- garak
https://github.com/leondz/garak

- HouYi
https://github.com/LLMSecurity/HouYi

- JailbreakingLLMs
https://github.com/patrickrchao/JailbreakingLLMs

- llm-attacks
https://github.com/llm-attacks/llm-attacks

- PromptInject
https://github.com/agencyenterprise/PromptInject

- LLM-Canary
https://github.com/LLM-Canary/LLM-Canary

- And now, of course, PyRIT
https://github.com/Azure/PyRIT

With thanks to Idan Gelbourt and Simo Jaanus for the research!

More AI security related posts and research will be published in the future from Knostic even while we’re still in stealth. Follow us to stay in the loop.