OWASP AI Exchange

Years of work resulted in this picture: the essentials of AI security. It looks so simple, because it is. I've been on stages around the world, work with the best brains, built OpenCRE, OWASP AI Exchange, MOSAIC standards - all to make AI security easier to understand. It took a while, but this finally IS the big picture of threats. It is part of a larger release that we have developed with SANS Institute - to be announced soon. It's also central in my training on June 24. For details and more content, see the links in the visual. It comes down to this: 1️⃣ AI introduces new assets that face conventional threats. AI systems are still IT systems, so they remain exposed to familiar security threats such as SQL injection and password theft. The difference is that this now also applies to AI-specific assets like training data, models, augmentation data, model input, and model output – the blue containers. Augmentation data is everything that is added to the model input, e.g. retrieved data, context, system prompts. The assets can leak and can be manipulated - often leading to changed model behaviour, which is referred to as poisoning. 👉 Call to action: Add the assets to your ISMS and understand the AI attention points (see the AI exchange). The rest is standard security. 2️⃣ AI introduces new suppliers. Organizations may rely on suppliers for training data and ready-made models, which may be manipulated. In addition, when a model is hosted externally, sensitive data may travel to the hosting provider. The provider must therefore protect confidentiality, integrity, logging, retention, and operational security. 👉 Call to action: Include the new suppliers in your existing supply chain management (e.g., provenance, lineage, integrity, contracts, audits). 3️⃣ The big one: input threats. Input threats represent a new attack surface: attackers using the AI system through its normal interface: sending input and receiving output. They can: 🔓 Mislead the model Evasion attacks make a model misclassify input (e.g. circumvent spam detection). Prompt injection makes a generative model follow malicious instructions, including instructions hidden in retrieved or user-provided data. 🔓 Exhaust resources Attackers can use input to consume excessive compute, tokens, or other resources, causing service disruption or unnecessary cost. 🔓 Extract data or models Input attacks can also exfiltrate the model, reconstruct sensitive information(model inversion/membership inference), or use prompt injection to disclose input, training, or augmentation data. 👉 Call to action: build or buy models with some resilience, use detection techniques, and mostly: apply zero model trust, which means: invest in monitoring, incident response, minimizing/obfuscating data, and mostly: limiting model behaviour through agent privilege control, human in the loop, and automated oversight. Blast radius control is key! That's it. Easy does it 🙂

𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗦𝘂𝗽𝗽𝗹𝘆 𝗖𝗵𝗮𝗶𝗻 𝗔𝘀𝘀𝘂𝗿𝗮𝗻𝗰𝗲 𝗙𝗼𝗿𝘂𝗺 𝟮𝟬𝟮𝟲, 𝗠𝗰𝗟𝗲𝗮𝗻 𝗩𝗔 Let me start with the journey, because it earned its own paragraph. The MTA strike hit right before the trip, turning what should have been a straightforward ride into a full logistical puzzle. Roads out of New York were a mess, bumper to bumper most of the way down to Virginia. Somewhere in all of that, my back decided to join the chaos. Quick stop at the ER to get it under control. 𝗧𝗵𝗲𝗻, 𝗸𝗲𝘆𝗻𝗼𝘁𝗲 𝘁𝗶𝗺𝗲. 𝗦𝗼𝗺𝗲 𝘁𝗵𝗶𝗻𝗴𝘀 𝗮𝗿𝗲 𝗷𝘂𝘀𝘁 𝘄𝗼𝗿𝘁𝗵 𝘀𝗵𝗼𝘄𝗶𝗻𝗴 𝘂𝗽 𝗳𝗼𝗿. Last week I had the privilege of 𝗱𝗲𝗹𝗶𝘃𝗲𝗿𝗶𝗻𝗴 𝘁𝗵𝗲 𝗼𝗽𝗲𝗻𝗶𝗻𝗴 𝗸𝗲𝘆𝗻𝗼𝘁𝗲 𝗮𝘁 𝘁𝗵𝗲 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗮𝗻𝗱 𝗦𝘂𝗽𝗽𝗹𝘆 𝗖𝗵𝗮𝗶𝗻 𝗔𝘀𝘀𝘂𝗿𝗮𝗻𝗰𝗲 𝗦𝗽𝗿𝗶𝗻𝗴 𝗙𝗼𝗿𝘂𝗺 𝟮𝟬𝟮𝟲 𝗮𝘁 𝗠𝗜𝗧𝗥𝗘 in McLean, Virginia. The title: 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝗮𝗹𝗶𝘇𝗶𝗻𝗴 𝗧𝗿𝗮𝗻𝘀𝗽𝗮𝗿𝗲𝗻𝗰𝘆: 𝗡𝗮𝘃𝗶𝗴𝗮𝘁𝗶𝗻𝗴 𝘁𝗵𝗲 𝗔𝗜 𝗦𝘂𝗽𝗽𝗹𝘆 𝗖𝗵𝗮𝗶𝗻 𝘄𝗶𝘁𝗵 𝗢𝗪𝗔𝗦𝗣 𝗔𝗜 𝗘𝘅𝗰𝗵𝗮𝗻𝗴𝗲 𝗮𝗻𝗱 𝗔𝗜𝗕𝗢𝗠. This was one of those rooms where you feel the weight of the work. NIST, CISA, MITRE, GSA, CIS, Idaho National Lab, the Department of Defense community, all in one place, 𝗮𝗹𝗹 𝘄𝗿𝗲𝘀𝘁𝗹𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝘁𝗵𝗲 𝘀𝗮𝗺𝗲 𝗵𝗮𝗿𝗱 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀 𝗮𝗯𝗼𝘂𝘁 𝘀𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗮𝗻𝗱 𝗔𝗜 𝘀𝘂𝗽𝗽𝗹𝘆 𝗰𝗵𝗮𝗶𝗻 𝗿𝗶𝘀𝗸. We are past the point of debating whether AI supply chain transparency matters. 𝗧𝗵𝗲 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻 𝗻𝗼𝘄 𝗶𝘀 𝗵𝗼𝘄 𝗳𝗮𝘀𝘁 𝘄𝗲 𝗰𝗮𝗻 𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝗮𝗹𝗶𝘇𝗲 𝗶𝘁, 𝗮𝗻𝗱 𝘄𝗵𝗲𝘁𝗵𝗲𝗿 𝗼𝘂𝗿 𝘀𝘁𝗮𝗻𝗱𝗮𝗿𝗱𝘀 𝗰𝗮𝗻 𝗸𝗲𝗲𝗽 𝗽𝗮𝗰𝗲 𝘄𝗶𝘁𝗵 𝗵𝗼𝘄 𝗳𝗮𝘀𝘁 𝘁𝗵𝗲 𝗮𝘁𝘁𝗮𝗰𝗸 𝘀𝘂𝗿𝗳𝗮𝗰𝗲 𝗶𝘀 𝗲𝘃𝗼𝗹𝘃𝗶𝗻𝗴. 𝗔 𝗳𝗲𝘄 𝘁𝗿𝗮𝗰𝗸𝘀 𝗳𝗿𝗼𝗺 𝘁𝗵𝗲 𝘁𝘄𝗼-𝗱𝗮𝘆 𝗮𝗴𝗲𝗻𝗱𝗮 𝘁𝗵𝗮𝘁 𝘀𝗵𝗮𝗽𝗲𝗱 𝘁𝗵𝗲 𝗰𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻:  • BOM Ecosystem evolution  • FAR overhaul from GSA  • NIST Secure Software Development Framework updates and DevSecOps  • Secure by Design guidance from the CIS  • AI and automation for risk mediation in the software supply chain  • Critical manufacturing and semiconductor supply chain traceability  • ICS risk through Cyber-Informed Engineering from Idaho National Lab A special thank you to the OWASP AI Exchange, particularly Rob van der Veer for the foundational work that made this conversation possible. The OWASP AIBOM project continues to build the scaffolding the industry needs to move from policy intent to real operational transparency. 𝗧𝗵𝗲 𝗺𝗼𝗺𝗲𝗻𝘁𝘂𝗺 𝗮𝘁 𝘁𝗵𝗲 𝗘𝘅𝗰𝗵𝗮𝗻𝗴𝗲 𝗿𝗶𝗴𝗵𝘁 𝗻𝗼𝘄 𝗶𝘀 𝗿𝗲𝗮𝗹. If you are working on AI governance, supply chain security, or SBOM/AIBOM, get involved. The community is where the standards are being written. To everyone at NIST, MITRE, and the SSCA community 𝘄𝗵𝗼 𝗺𝗮𝗱𝗲 𝘁𝗵𝗶𝘀 𝗳𝗼𝗿𝘂𝗺 𝘄𝗵𝗮𝘁 𝗶𝘁 𝘄𝗮𝘀, 𝘁𝗵𝗮𝗻𝗸 𝘆𝗼𝘂. These are exactly the conversations that move the needle. 𝗪𝗼𝗿𝘁𝗵 𝗲𝘃𝗲𝗿𝘆 𝗺𝗶𝗹𝗲 𝗼𝗳 𝘁𝗵𝗮𝘁 𝗱𝗿𝗶𝘃𝗲. 𝗕𝗮𝗰𝗸 𝗮𝗻𝗱 𝗮𝗹𝗹.

Most people discovered AI in 2022 when ChatGPT launched. Rob van der Veer has been working in AI since 1992. That 34-year vantage point is exactly why this conversation is going to be different. Rob is Chief AI Officer at @Software Improvement Group and Project Lead for OWASP AI Exchange, one of the most important open-source AI security initiatives in the world and a global OWASP flagship project. He also leads Open CRE and Project CRE, frameworks that are quietly doing some of the most important work in connecting security standards across the industry. I have had the privilege of working alongside Rob in the OWASP Community as Project Co-Lead since early 2024. His thinking is sharp, his perspective is long, and he pulls no punches on where AI is headed and what it means for all of us. On May 11th at 11am EST ET, Rob joins me on CyberFront Live and we are going deep. His 34-year journey in AI, what he saw coming and what surprised even him. The founding of OWASP AI Exchange, why it was needed, and what it has become. His projections on AI adoption, where it is going, and what organizations are fundamentally unprepared for. The pace of AI innovation versus the pace of AI security. Why the gap between the two is widening, not closing. And the growing urgency for regulation. What is working, what is performative, and what needs to happen before the next major incident forces the issue. This is not a conversation about hype. It is a conversation about what 34 years of watching this technology evolve actually teaches you. You do not want to miss this one. Link in the comments. #OWASP #AIExchange #AISecurity #AIGovernance #OpenCRE #CyberFrontLive #AIRegulation #ResponsibleAI #CyberSecurity 🎙️ New to streaming or looking to level up? Check out StreamYard and get $10 discount! 😍 https://lnkd.in/eg9zUPJw

🔐 How to prepare for the AI Act, now that its harmonized cybersecurity standard has been published for review? prEN 18282 has just entered Public Enquiry. I have been working on it for almost three years with CEN and CENELEC JTC21/WG5 and reviewers, involving about a hundred meetings, as co-editor, expert, and liaison of the OWASP AI Exchange - through which we contributed a large part of the material. What many people don’t know: 1️⃣ The standard is outcome-based, not control-based. 2️⃣ This standard can only give presumption of conformity once it is published in the Official Journal of the European Union. The standard is roughly scheduled for the beginning of next year. Publication in the OJEU is not guaranteed. 3️⃣ Conformity is by self-assessment, except for some systems from Annexes I and III. 4️⃣ The standard’s coverage is not yet complete. 5️⃣ Details for conventional cybersecurity are out of scope and covered in e.g., DORA, NIS2, and CRA. Let me zoom into points 1 and 4. Ad 1️⃣: The great strength, and also the big challenge, of the standard is that it is outcome-based. It requires assurance of sufficient risk mitigation, whereas in ISO/IEC 27001 you can often suffice by showing that you implemented controls. Less checklist, more proof. Ad 4️⃣: prEN 18282 was 107 pages half a year ago and, for various reasons had to be restructured and reduced. This included replacing most of the generative AI coverage with a summary and a placeholder for input. The plan is to make up for that during the Enquiry phase, based on the comments. It is not ideal, but nothing in high-pressure standardization on emerging technology is ideal. Thankfully, WG5 still has the previous material that was set aside to work from. 👉 What to do? Organizations need to Act now on AI security. Don’t wait until standards are final. If your AI can make a mess now, it is already a liability. Not just for safety, health, and fundamental rights, but also for fines, business continuity, and reputation. The risk is real, and the risk is now. The standard applies to high-risk AI systems on the EU market. But I recommend organizations to do their own additional risk assessment for deciding which other AI systems need security attention. If you want your organization to act on AI security, refer to the corresponding “Organize” section at the AI Exchange (link in the comments). Further details on the threats and controls can be found at the Exchange which lies at the foundation of the standard, and it links to other relevant sources. In the meantime, standard initiatives are collectively collaborating to attain more clarity, through MOSAIC standards. 👉 What else to do? Help improve the standard. Download the free draft from an EU national standards body website. Google “prEN 18282”, or see the comments for links. It may require free registration to gather your comments – which can be submitted until June 30th. Now is the time to act.

🍕🔥 The kitchen is open, join the OWASP PwnzzAI Team! PwnzzAI is building the future of practical AI security education, and we’re inviting developers, researchers, creators, and security enthusiasts from around the world to join the journey. OWASP PwnzzAI is an intentionally vulnerable, AI-powered pizza store designed to simulate realistic business environments and expose real-world AI and LLM security risks through hands-on learning. Built for education, experimentation, and community collaboration, PwnzzAI helps security professionals, developers, students, and researchers better understand how AI vulnerabilities can impact modern applications and businesses. In partnership with the OWASP AI Exchange, the project aims to make AI threats more practical, accessible, interactive, and engaging for education and training purposes. As the project continues to grow, we’re building an open-source community of people passionate about AI, security, education, and innovation. We’re currently looking for contributors in: • AI & LLM Security • Web Application Security • Backend & Frontend Development • DevOps & Infrastructure • Vulnerability Research • Security Testing & Challenge Creation • Technical Writing & Educational Content • UI/UX & Creative Design • Community & Open-Source Collaboration 💡 Why join PwnzzAI? • Contribute to a real OWASP AI security initiative • Work on realistic AI vulnerability scenarios and attack simulations • Collaborate with an international open-source community, united by security and pizza 🍕 • Help shape the future of AI security education • Learn, build, experiment, teach, and grow alongside passionate contributors Whether you’re an experienced security professional, developer, researcher, student, technical writer, designer, or simply curious about AI security, there’s a place for you in this kitchen. 🔗 Project Repository: https://lnkd.in/dznJyvmU 📩 Interested in joining the team? Send your CV to: maryam.mouzarani@owasp.org 🥘 The kitchen is open. Come cook security with us. #OWASP #OpenSource #AISecurity #LLMSecurity #CyberSecurity #AppSec #AI #SecurityResearch #DevSecOps #AIEducation #TechnicalWriting

Many people assume that AI security is mostly about the AI model. It is not. Especially not when we talk about prompt injection and other forms of manipulation. AI models are powerful, but also fundamentally fragile. That means our attention should be at least as much on how solution builders use the model: how they integrate it, what agency they give it, what data it can access, what tools it can call, how it is monitored, and what controls surround it. That was my main point in Fortune today(link in the comments), in an article about U.S. plans to evaluate AI models before vendors are allowed to release them. And yes: we need more resilient models. It is good that model vendors invest in this, and: 👉 AI models will remain fragile, no matter how much we test them. This is an intrinsic property of AI, just as it is of humans. 👉 So model vetting can easily create a false sense of security. 👉 The state of the art in AI model testing cannot give us the level of assurance we need. In practice, testing often means running a set of standard tests and then choosing a threshold where we say: this is resilient enough. But without a specific use case, that result has limited meaning. 👉 There is also a deeper issue: some risks cannot be assessed properly from the outside. For example, you cannot meaningfully test a general-purpose AI model for backdoor data poisoning without understanding the data, processing steps, and training methods behind it. At some point, you need to open the black box - a point made by Gary McGraw from BIML (Berryville Institute of Machine Learning) the article. So the bigger challenge is not making AI models perfectly robust. It is how you handle models that remain fallible. That is where standardization and guidance are needed most. And that is where Software Improvement Group helps clients, where I provide training, and why OWASP AI Exchange brought together AI security standard makers in the new MOSAIC initiative. It is time to coordinate on how to build secure AI systems, not just how to test AI models.