Welcome to Jailbroken. This is a place to learn AI red teaming by doing it, safely, against practice agents that were built to be broken. This first lesson is short on hacking and long on mindset, because the mindset is what carries across every technique that follows.
What red teaming is
Red teaming is probing a system the way an attacker would, so its builders can find and fix the weaknesses before someone hostile does. For AI, that means finding inputs that make a model or agent do something its designers never intended: leak a secret, misuse a tool, or ignore its own rules. It is a defensive craft. You attack in order to defend, and the industry now treats it as a core part of shipping AI, formalized in resources like the OWASP Top 10 for LLM Applications.
The one weakness behind everything
Here is the idea that unlocks this entire course. A language model does not have a firm boundary between the instructions it was given and the data it is reading. Both arrive as text, in the same context window, and the model does its best to satisfy all of it at once.
Every attack you will learn is a variation on exploiting that blurry line:
- In a jailbreak, your message competes with the system's rules, and you make yours win.
- In prompt injection, instructions hidden in a document or web page the agent reads get followed as if they were commands.
- In memory poisoning, a planted "fact" becomes an instruction the agent trusts forever.
Once you see the seam, you stop guessing and start aiming.
The loop: how good red teamers actually work
Progress comes from a tight loop, not from one magic prompt:
- Observe. What is this agent for? What would it protect? What tools might it hold?
- Hypothesize. Pick one idea for why a defense might bend.
- Probe. Send a single, deliberate attempt.
- Read the response. This is the step beginners skip, and it is where the signal lives.
- Adjust. Change one thing and go again.
A flat refusal is not the end. A hedge, a partial answer, a "I can share some of that but not all," or the model explaining why it can't, all mean you are close. Push on exactly that thread instead of starting over.
How the game works
- The target is an AI agent with a hidden system prompt and, sometimes, hidden tools it can call.
- You send it messages, or in some scenarios plant content it will read.
- A judge (a separate model) scores each attempt from 0 to 100 against a rubric, and you win when you clear the pass threshold.
- Levels add defenses as you climb, and they mirror the real defense ladder: refusal training, then input filters, then output redaction, then a separate guardian model. The attack that beats level 1 rarely survives level 5, and learning why is the point.
- Each attempt costs a little energy, which refills over time, so aim before you fire.
Mistakes to skip
- Being too blunt. Asking directly for the forbidden thing trips the most obvious defense. Come in sideways.
- Quitting after one refusal. The first no is data, not a verdict.
- Changing five things at once. When something moves the needle, you want to know which thing did it. Vary one variable at a time.
- Ignoring the agent's role. A banking assistant, a clinic bot, and a support agent all have different soft spots. Use the persona against itself.
One rule: stay authorized
Everything here is sanctioned. That is the point of a practice range. Out in the world, only ever test systems you have explicit permission to test. If you want the bigger map of this field, skim the AI Red Teaming roadmap and Joseph Thacker's How to hack AI apps.
Ready? The next lesson gets your hands dirty.