AI Safety and Security MOC

AI safety and security studies how advanced models fail, how they can be misused, and how to deploy them responsibly.

This is not separate from capabilities. Frontier models are only useful if they are reliable enough to trust and constrained enough to deploy.

Why It Matters

Frontier labs need people who understand:

• Jailbreaks.
• Prompt injection.
• Misuse capabilities.
• Over-refusal.
• Sycophancy.
• Privacy leakage.
• Dangerous tool use.
• Model monitoring.
• Red-team methodology.
• Secure deployment.

Core Areas

Alignment Behavior

Learn:

• Helpful/harmless tradeoffs.
• Refusal behavior.
• Sycophancy.
• Reward hacking.
• Calibration.
• Capability elicitation.

Connected note:

• Post Training Alignment and Reasoning

Red Teaming

Red teaming tries to find model failures before deployment.

Primary source:

• Red Teaming Language Models to Reduce Harms

Jailbreaks And Adversarial Prompts

Learn:

• Universal adversarial suffixes.
• Prompt injection.
• Multi-turn jailbreaks.
• Tool-mediated attacks.
• Data exfiltration attempts.

Primary source:

• Universal and Transferable Adversarial Attacks on Aligned Language Models

Deployment Security

Learn:

• Tool permissioning.
• Sandboxing.
• Secrets handling.
• Logging and monitoring.
• Human approval for risky actions.
• Rate limits.
• Abuse detection.

Safety Evaluation

Safety evals should test:

• Harmful compliance.
• Over-refusal.
• Jailbreak robustness.
• Tool misuse.
• Privacy leakage.
• Bias/toxicity.
• Long-horizon agent risk.

Starter Project

Build a small safety eval set:

• 20 benign requests.
• 20 harmful requests.
• 20 ambiguous boundary requests.
• 20 prompt-injection/tool-use scenarios.
• Run a model or prompts through it.
• Record refusal, compliance, helpful redirection, and failure type.

Deliverable:

• A confusion matrix for safety behavior: helpful, refused, over-refused, unsafe, evasive.

Papers / Resources

• Concrete Problems in AI Safety
• AI Safety Gridworlds
• Red Teaming Language Models to Reduce Harms
• Universal and Transferable Adversarial Attacks
• Constitutional AI

What Good Looks Like

You can:

• Design safety evals with benign and adversarial cases.
• Distinguish over-refusal from real safety.
• Understand prompt injection in tool systems.
• Explain how post-training affects safety behavior.
• Build secure agent/tool scaffolds.

Related

• Frontier Lab Readiness MOC
• Post Training Alignment and Reasoning
• Evaluation and Benchmarking MOC
• Agents and Tool Use MOC
• Mechanistic Interpretability MOC