Domain 1 · 25–30%Plan and manage an Azure AI solution

Plan and manage

Choosing models and services, setting up Foundry, managing and securing it, and responsible AI. Read a lesson, then test it in the practice bank.

PracticeD1 practice bank →

34 items, all formats, practice or timed exam mode.

LabWeek 1 lab →

Keyless call to a Foundry model, with a break-it exercise.

RIdentity and access in Foundry

1.R1

The five Foundry roles, and the two Azure roles that can't build

Use agents → Agent Consumer. Build → Foundry User. Lead a team and publish → Project Manager. Set up resources, no building → Account Owner. Everything → Foundry Owner.

From CCA-F: New for you. CCA-F had API keys and workspaces, not role scopes. Think of it as Claude Console roles, but enforced by Azure RBAC at resource, project and agent level.

How it works

Decide what the principal must do: call, build, publish, set up
Pick the smallest role that does it
Pick the smallest scope: agent, project, or resource

Must know

Foundry Agent Consumer: call agent endpoints only. Assignable down to one agent.
Foundry User (was Azure AI User): build, test, evaluate in a project; read access. Least privilege for developers and for the project's managed identity.
Foundry Project Manager: build + manage projects + assign *only* Foundry User + publish agents (the minimum role to publish).
Foundry Account Owner: create resources and projects, manage models, assign Foundry User / ACR / monitoring roles. No data actions: can't build.
Foundry Owner: all of the above, building included.
Azure Owner and Contributor manage resources (Contributor can deploy models) but have no data actions.

Traps

Owner or Contributor on the subscription still can't create an agent: add Foundry User.
Azure AI Developer is for hubs and Azure Machine Learning, not Foundry projects.
Don't use *Cognitive Services …* roles for Foundry, except Cognitive Services Usages Reader for quota.
Renamed roles keep their role IDs; scripts should use the ID.

Check yourself: d1-r1 · d1-r2 · d1-r3 · Microsoft Learn

1.R2

Scopes and the two hops of identity

The caller needs a role on Foundry; the project's managed identity needs roles on everything Foundry reaches for it.

From CCA-F: Like an MCP server that calls another API with its own credentials: the user is allowed into the server, and the server separately needs rights downstream.

How it works

Hop 1: user or app → Foundry (Foundry roles, at resource, project or agent scope)
Hop 2: project managed identity → Search, Storage and other connected resources (their data roles)
Check both when you see a 403

Must know

Scopes: subscription → resource group → Foundry resource → project → agent.
Agent scope is evaluated only for calling that agent's endpoint.
The Azure portal assigns Agent Consumer only at account scope; use the Azure CLI for project or agent scope.
Project managed identity examples: Storage Blob Data Reader on a storage account; Search roles (Search Index Data Contributor, Search Service Contributor) on AI Search for the Search tool.
Quota and deployment eligibility: Cognitive Services Usages Reader at subscription scope; resource or project scope doesn't work.

Traps

Giving the *developer* a Search role doesn't fix the agent's Search tool: the call is made by the project's identity.
A project-scope Agent Consumer can call every agent in the project.

Check yourself: d1-r4 · d1-r5 · d1-r6 · Microsoft Learn

1.R3

Keys, tokens, and how Foundry differs from other AI services

Entra ID + Azure RBAC, like the rest of Azure; but keys skip RBAC entirely, and roles differ by resource type.

From CCA-F: Same split as Storage or Key Vault: owning the account isn't the same as reading the data. Control plane and data plane are separate permissions.

How it works

Prefer Entra ID; disable keys where possible
Use the token audience https://ai.azure.com/.default for the v1 API and projects
Match the role family to the resource type

Must know

An API key grants full access with no role checks. Agents and evaluations need Entra ID anyway.
Creating a resource or project in the portal auto-assigns Foundry User to you and the project identity, if you can assign roles; SDK or CLI deployments don't.
Fine-tuning needs both planes: Foundry Owner, or Foundry User + Foundry Account Owner.
Role families: Foundry resource and projects → Foundry roles; standalone Azure OpenAI or AI services resource → Cognitive Services roles (e.g. Cognitive Services OpenAI User); hub projects → Azure AI Developer and Azure ML roles.

Traps

Expecting RBAC to limit a key holder: it doesn't.
Using Cognitive Services OpenAI User for a Foundry project instead of Foundry User.

Check yourself: d1-r7 · Microsoft Learn

EEvaluators: which one? (with ELI5)

1.E0

What to learn for the exam: three tiers

Know the families and the look-alike pairs cold; recognise the rest by name; look up the rare ones on Microsoft Learn.

From CCA-F: Same exam tactic as CCA-F: learn the decision rules, not the catalogue. Here Learn is open, but a lookup costs a whole question's time (about 2 minutes).

How it works

Tier 1: learn until you can answer without looking
Tier 2: recognise the name and what it checks
Tier 3: skip; look up only if one appears

Must know

Most evaluator questions reduce to two calls: measure or block? and which family (RAG, agent, safety, ground-truth maths)?

Tier	What	Why
1 · Know cold	Evaluator vs guardrail vs red team vs continuous evaluation	Measure vs block vs attack: the most-tested idea
1 · Know cold	Groundedness vs Groundedness Pro	1–5 with your judge vs pass/fail with no deployment
1 · Know cold	Relevance · Retrieval · Groundedness	The RAG trio: answered the question? fetched useful text? stuck to it?
1 · Know cold	Intent Resolution · Task Adherence · Task Completion · Tool Call Accuracy	Understood? followed the rules? finished? used tools well?
1 · Know cold	Indirect Attack evaluator vs Prompt Shields	Measures vs blocks
1 · Know cold	Hate and Unfairness · Sexual · Violence · Self-Harm	One harm family
1 · Know cold	The needs rule	Ground truth → Similarity, F1/BLEU/ROUGE/METEOR · context → Groundedness, Retrieval · no deployment → maths or Microsoft-hosted
1 · Know cold	AI Red Teaming Agent → attack success rate	The pre-release safety answer
2 · Recognise	Ungrounded Attributes · Protected Materials · Code Vulnerability	Unsupported guesses about the user (emotions, protected traits) · copied text · insecure code
2 · Recognise	BLEU / METEOR · Document Retrieval · Response Completeness	Translation vs references · retrieval vs a known list · nothing important missing
2 · Recognise	Tool Selection · Tool Input Accuracy · Tool Output Utilization	Right tool · right details · used the result
2 · Recognise	Evaluation levels	Conversation level needs the messages format
3 · Skip, look up	GLEU, ROUGE vs F1; Customer Satisfaction; Task Navigation Efficiency; Tool Call Success; Prohibited Actions; Sensitive Data Leakage; bundles (Quality Grader is deprecated); Rubric; Azure OpenAI graders; custom	Unlikely as answers; the name is a strong hint

Traps

Memorising every evaluator: time better spent on the look-alike pairs.

Check yourself: d1-e4 · Microsoft Learn

1.E1

Evaluator, guardrail, red team, tracing: who does what

Evaluators mark the homework; guardrails stop bad answers at the door; the red-teaming agent tests the locks; tracing is the CCTV.

From CCA-F: Same split as CCA-F's evals vs runtime checks: an eval suite tells you how good the agent is; a hook or validator stops one bad action now.

How it works

Must something be stopped before a user sees it? → guardrail
Must something be measured? → evaluator (offline on a dataset, or continuous on live traffic)
Must you attack it before release? → AI Red Teaming Agent

Must know

Evaluators never block anything, even when they run on live traffic.
Where you set them: the portal's Evaluation page (pick a dataset or an agent, then evaluators); the agents playground (on by default, and billed); continuous evaluation on a deployed agent (with a sampling rate); or code: azure-ai-evaluation, cloud runs with builtin.* evaluators, in CI.
Three kinds: maths (F1, BLEU, ROUGE, METEOR, GLEU, String Checker: deterministic, no model); AI-assisted (quality, RAG, agent: an LLM judge on your deployment, so you pay its tokens); Microsoft-hosted (safety evaluators, Groundedness Pro: billed as a Foundry service).
Why not always on: every evaluator is extra calls per answer and can cost more than the app; many need inputs you may not have (ground truth, context, tool definitions); uncalibrated judges create noise; and they never block. Pick per app, sample live traffic, and keep cheap deterministic rules everywhere.
Evaluators score a single turn by default; scoring a whole conversation needs data in the messages format.
Combos from Learn: RAG → Retrieval + Groundedness + Relevance + safety; agents → Output Quality + Tool Use Quality + Rubric + safety; translation → BLEU + METEOR + Fluency + Coherence.

Feature	What it does	ELI5
Evaluator (offline)	Scores answers on a test dataset before release	A teacher marking homework before the report card goes out
Continuous evaluation	Scores a sample of live answers	A teacher spot-checking homework that was already handed back
Guardrail	Checks every input, output and tool call at runtime; can block	A security guard at the door who stops a bad answer leaving
AI Red Teaming Agent	Generates attacks before release; reports attack success rate	A friendly burglar paid to test your locks
Tracing	Records each step of every run	CCTV: it doesn't stop anything, but you can replay it

Traps

"Continuous evaluation" sounds live, but it scores answers users have already seen.
The Indirect Attack *evaluator* measures; *Prompt Shields* blocks.

Check yourself: d1-e4 · Microsoft Learn

1.E2

Quality and RAG evaluators

Did it read well, match the answer key, use the right books, stick to them, and answer the question?

From CCA-F: The RAG checks from CCA-F's context work, with Azure names: retrieval quality, groundedness, relevance.

How it works

Have a reference answer (ground truth)? → similarity and word-overlap metrics are open to you
Have the retrieved passages? → Retrieval, Groundedness, Groundedness Pro
Have neither? → Relevance, Coherence, Fluency

Must know

Need ground truth: Similarity, F1, BLEU, GLEU, ROUGE, METEOR, Document Retrieval, Response Completeness.
F1, BLEU, GLEU, ROUGE, METEOR are maths, not a model: same input, same score.
Groundedness = 1–5 from your own judge deployment; Groundedness Pro (preview) = pass/fail from Content Safety, no deployment.

Evaluator	Measures	ELI5	Needs	Cost
Coherence	Logical flow	Does the story make sense from start to end?	Judge model	Your judge's tokens
Fluency	Language quality	Does it sound like a fluent speaker wrote it?	Judge model	Your judge's tokens
Similarity	Meaning vs the reference	Does it mean the same as the answer key?	Ground truth + judge	Your judge's tokens
F1 · BLEU · GLEU · ROUGE · METEOR	Word overlap with the reference	Count the words it shares with the answer key; BLEU and METEOR are the translation ones	Ground truth, no model	Free
Retrieval	Usefulness of retrieved passages	Did the librarian bring useful books?	Retrieved context	Your judge's tokens
Document Retrieval	Retrieval accuracy against a known list	Did the librarian bring the books on the teacher's list?	Ground truth	Free
Groundedness	Answer supported by the context, 1–5	Is every sentence backed by the books on the desk? A teacher you hire gives a mark	Context + judge	Your judge's tokens
Groundedness Pro (preview)	Same, pass/fail	Same question, but the exam board stamps pass or fail; no teacher to hire	Context, no deployment	Foundry-billed
Relevance	Answers the question asked	Did it answer what you actually asked?	Judge model	Your judge's tokens
Response Completeness (preview)	Nothing important missing	Did it leave out anything from the answer key?	Ground truth	Your judge's tokens

Traps

Picking BLEU when there are no reference answers.
Picking Groundedness when the question says "no model deployment": that's Groundedness Pro.

Check yourself: d1-e1 · d1-e2 · d1-e4 · Microsoft Learn

1.E3

Safety and agent evaluators

Safety: is anything harmful, copied, leaked or made up about the user? Agent: did it understand, follow the rules, use its tools well, and finish?

From CCA-F: CCA-F's agent evaluation (task success, tool-use correctness) split into separately named checks; read the question for which part failed.

How it works

Find which part failed: understanding, rules, tool choice, tool arguments, using tool results, or finishing
Pick the evaluator named for that part
Use a bundle (Output Quality, Tool Use Quality) to run several in one judge call

Must know

Risk and safety evaluators run on Microsoft's side; you don't deploy a judge.
Several agent evaluators are preview: Task Adherence, Task Completion, Intent Resolution, Customer Satisfaction.
Ungrounded Attributes is about unsupported guesses about the user: their emotional state, protected traits (gender, ethnicity, religion, health) or circumstances. Not about the documents.

Evaluator	ELI5	Cost
Hate and Unfairness · Sexual · Violence · Self-Harm	Is there anything harmful in it?	Foundry-billed
Protected Materials	Did it copy song lyrics or a page from a book?	Foundry-billed
Indirect Attack (XPIA)	Did a note hidden in a document trick it? (measures only)	Foundry-billed
Code Vulnerability	Did the code it wrote leave a door unlocked?	Foundry-billed
Ungrounded Attributes	Did it guess things about the person with no evidence: how they feel, their gender, ethnicity or health, or their money ("you must be struggling financially")?	Foundry-billed
Prohibited Actions	Did it do something it was told never to do?	Foundry-billed
Sensitive Data Leakage	Did it spill a secret?	Foundry-billed
Intent Resolution (preview)	Did it understand what you actually wanted?	Your judge's tokens
Task Adherence (preview)	Did it follow the rules of the job?	Your judge's tokens
Task Completion (preview)	Did it finish the job?	Your judge's tokens
Customer Satisfaction (preview)	Would the customer leave happy?	Your judge's tokens
Task Navigation Efficiency	Did it take the shortest sensible route?	Free (no judge)
Tool Call Accuracy	Did it use its tools well, overall?	Your judge's tokens
Tool Selection	Did it pick the right tool from the toolbox?	Your judge's tokens
Tool Input Accuracy	Did it give the tool the right details?	Your judge's tokens
Tool Output Utilization	Did it actually use what the tool gave back?	Your judge's tokens
Tool Call Success	Did the tool calls work without errors?	Your judge's tokens
Output Quality · Tool Use Quality (preview)	One inspector checking several things in one visit: cheaper than separate visits. Quality Grader is deprecated	Your judge's tokens, one call for several
Rubric (preview)	Your own marking scheme, with weights	Your judge's tokens
Graders: String Checker · Model Labeler · Model Scorer · Text Similarity	An exact-match rule · sorting into labelled boxes · marking against your guideline · how close two texts are	String Checker free · Labeler, Scorer: judge tokens · Text Similarity: free for string metrics

Traps

Task Adherence (followed the rules) ≠ Task Completion (finished the job).
Tool Selection (right tool) ≠ Tool Input Accuracy (right arguments) ≠ Tool Output Utilization (used the result).
Limited support: avoid Tool Call Accuracy, Tool Input Accuracy, Tool Output Utilization, Tool Call Success and Groundedness when the agent uses Azure AI Search, Bing, SharePoint, Code Interpreter, Fabric Data Agent or Web Search tools. Function, MCP and File Search tools are supported.
Quality Grader is deprecated and can no longer run: use Output Quality and Tool Use Quality.

Check yourself: d1-e3 · d1-e5 · d1-e6 · Microsoft Learn

1.E4

When to use which evaluator, and what it costs

Start from the job: RAG, agent, reference-based text, or safety. Then check what you have (ground truth, context) and what you can pay for (a judge model).

From CCA-F: CCA-F's "cheapest check that answers the question": deterministic checks first, LLM judges where meaning matters, and never more judges than you'll act on.

How it works

Name the job you're evaluating
Check the inputs you have: reference answers? retrieved passages? tool definitions?
Pick the cheapest evaluator that answers the question; add LLM judges only where meaning matters

Must know

Free, deterministic: F1, BLEU, GLEU, ROUGE, METEOR, Document Retrieval, String Checker, and Task Navigation Efficiency (compares actions with expected actions; no judge).
Your judge's tokens: Coherence, Fluency, Similarity, Relevance, Retrieval, Groundedness, Response Completeness, every agent and tool evaluator except Task Navigation Efficiency, Rubric, Model Labeler, Model Scorer. About one judge call per evaluator per answer; bundles (Output Quality, Tool Use Quality) do several in one call.
Billed by Foundry, no deployment of yours: the risk and safety evaluators and Groundedness Pro. Check the pricing page for rates.

If you're evaluating…	Use	Needs	Cost
A RAG answer	Retrieval + Groundedness (or Groundedness Pro) + Relevance	Query, retrieved passages, response	Judge tokens (Groundedness Pro: Foundry-billed)
An agent	Intent Resolution + Task Adherence + Task Completion + Tool Call Accuracy, or the Output Quality and Tool Use Quality bundles	Conversation, tool definitions and calls	Judge tokens; bundles fewer calls
An agent's path against the expected one	Task Navigation Efficiency	Actions and expected actions	Free
A translation or summary with reference answers	BLEU / METEOR (translation), ROUGE (summaries), F1, Similarity	Ground truth	Free, except Similarity (judge)
Retrieval against a known answer list	Document Retrieval	Labelled relevant documents	Free
Harmful or risky output	Hate and Unfairness, Sexual, Violence, Self-Harm, Protected Materials, Indirect Attack, Code Vulnerability, Ungrounded Attributes	Query and response	Foundry-billed
Writing quality	Coherence, Fluency	Response	Judge tokens
A hard business rule	String Checker, or your own custom evaluator	The rule	Free
Attack resistance before release	AI Red Teaming Agent (not an evaluator)	The agent or model	Foundry-billed

Traps

Running every AI-assisted evaluator on every answer: the judge can cost more than the app.
Choosing an LLM judge where a deterministic check would do, such as "must mention clause WR-3" → String Checker.

Check yourself: d1-e1 · d1-e2 · d1-e5 · Microsoft Learn

1.E5

Agent evaluators: what each needs and returns

Each agent evaluator answers one question, needs specific fields, and returns pass/fail (some from a 1–5 score). Only Task Navigation Efficiency needs no judge.

From CCA-F: CCA-F's split between judging the outcome and judging the steps: Microsoft calls them system evaluation (did the job get done well?) and process evaluation (was each tool step right?).

How it works

Map the fields your data has: query and response, or messages; plus tool_definitions for tool evaluators
Pick turn level (one reply) or conversation level (the whole chat) where both are supported
Read pass/fail for the gate; read the score and reason to find the cause

Must know

System evaluation (the outcome): Task Completion, Customer Satisfaction, Task Adherence, Task Navigation Efficiency, Intent Resolution.
Process evaluation (the steps): Tool Call Accuracy, Tool Selection, Tool Input Accuracy, Tool Output Utilization, Tool Call Success.
Conversation level is supported by Task Completion, Customer Satisfaction and the two bundles; the rest are turn level only.
A bundle passes only if every applicable component passes; skipped components don't fail it. Learn recommends a GPT-5.6-family judge for bundles.

Evaluator	Question	Needs	Returns	Cost
Intent Resolution (preview)	Did it understand what the user wanted?	query + response, or messages	1–5, then pass/fail at a threshold	Your judge's tokens
Task Adherence (preview)	Did it follow its system instructions?	query + response, or messages (+ tool_definitions)	Pass/fail	Your judge's tokens
Task Completion (preview)	Did it finish the job with a usable result?	query + response, or messages	Pass/fail	Your judge's tokens
Customer Satisfaction (preview)	Would the user be happy?	query + response, or messages	1–5	Your judge's tokens
Tool Selection	Did it pick the right tools, and no unnecessary ones?	query + tool_definitions	Pass/fail	Your judge's tokens
Tool Input Accuracy	Did it pass the right arguments?	query + response + tool_definitions	Pass/fail	Your judge's tokens
Tool Output Utilization	Did it use what the tool returned?	query + response + tool_definitions	Pass/fail	Your judge's tokens
Tool Call Success	Did the tool calls run without errors?	response, or messages	Pass/fail	Your judge's tokens
Tool Call Accuracy	Right calls, right parameters, no redundancy, overall	query + tool_definitions	1–5, then pass/fail	Your judge's tokens
Task Navigation Efficiency	Did it take the expected path?	actions (or messages) + expected_actions	Maths, no judge: pass/fail plus precision, recall and F1. Match modes: exact, in order, any order	Free
Output Quality · Tool Use Quality (preview)	Several of the above in one judge call	query + response (+ tool_definitions), or messages	Passes only if every applicable component passes	Your judge's tokens, one call

Traps

Task Navigation Efficiency needs expected actions (ground truth): no expected path, no evaluator.
Tool Call Success still uses a judge model, even though it only reads the response.

Check yourself: d1-e5 · d1-e6 · Microsoft Learn

All D1 lessons

Choose the right model and Foundry service · Plan & manage · choose Foundry services

Start from the job, not the model. A small model for classification, a large or reasoning model for multi-step work, a multimodal model when images or audio are involved, and a Foundry Tool when a purpose-built service already does it. For knowledge, pick by where the content comes from.

Key facts

Foundry Models catalog has 10,000+ models. *Models sold by Azure* are billed by Microsoft; partner and community models are billed separately.
Model router picks a model per prompt. Modes: Balanced (default), Quality, Cost. Optional *model subset* limits the candidates.
Model router works only on Global Standard and Data Zone Standard deployments; its context window is the smallest underlying model's; it routes on text only.
Knowledge, by source: memory = what the agent learns about a user (preview) · Foundry IQ = curated organisational content (agentic retrieval over Azure AI Search) · file search = documents the user uploads in the conversation.
Foundry IQ status: API GA, portal still preview. Memory: preview, 100 scopes per store, 10,000 memories per scope, no VNet support.

Exam traps

Model router's context window is the smallest underlying model's, not the largest.
File search is for documents a user brings; organisational knowledge belongs in Foundry IQ or Azure AI Search.
Claude models must be deployed separately before the router can use them.

Hands-on: Deploy model router in a Foundry project, send five prompts of different difficulty, and read which model served each in the response.

Source: learn: azure/foundry/openai/concepts/model-router · learn: azure/foundry/agents/concepts/what-is-foundry-iq

Set up Foundry: resources, projects and deployment types · Plan & manage · set up AI solutions

One Foundry resource is the security, networking and billing boundary, and holds the model deployments. Projects inside it are where teams build; agents in one project share files, conversations and indexes. How you deploy a model decides where data is processed and how you pay.

Key facts

Project endpoint: https://<resource>.services.ai.azure.com/api/projects/<project>.
Hub-based projects and standalone Azure OpenAI resources are *classic*; new work uses Foundry projects (an Azure OpenAI resource can be upgraded and keep its endpoint).
Deployment types: Global Standard (default, newest models first) · Data Zone (US, EU or APAC residency) · Standard (one geography) · Provisioned / PTU (predictable latency) · Global/Data Zone Batch (50% off, 24-hour target, separate quota).
Agent environment: Basic (platform storage) · Standard (your own Storage, AI Search, Cosmos DB; customer-managed keys) · Standard + your VNet (private outbound).
Private MCP servers and the Azure Functions tool require the Standard setup.
CI/CD: define agents with the SDK or REST in the pipeline (versions are snapshotted), use azd ai commands, and gate releases on evaluations over versioned datasets.

Exam traps

Batch is for offline bulk work, never for a live conversation.
Data Zone is the answer for 'data must stay in the EU'; Global Standard may process anywhere.
Limits worth knowing: 250 projects per resource, 32 deployments per resource.

Hands-on: Create a Foundry resource and two projects; deploy one model as Global Standard and one as Data Zone Standard; compare where each can process data.

Source: learn: azure/foundry/foundry-models/concepts/deployment-types · learn: azure/foundry/agents/environment-setup

Manage, monitor and secure AI systems · Plan & manage · manage, monitor, secure

Quotas protect everyone's capacity; roles decide who may build or only call; keyless identity removes secrets; private networking keeps traffic off the internet. Monitoring watches quality and safety as well as uptime.

Key facts

Quota (TPM/RPM) is per subscription, region, model and deployment type. A 429 means the limit was hit: back off and honour Retry-After.
Reasoning tokens are billed as output tokens. Playground evaluations are on by default and billed.
Keyless: Entra ID with DefaultAzureCredential, scope https://ai.azure.com/.default. An API key grants full access with no role granularity.
Agents, evaluations, datasets, workflows and Content Understanding require Entra ID; keys don't work there.
Roles: Foundry Agent Consumer (call agents only) · Foundry User (build in a project; least privilege for developers) · Foundry Project Manager · Foundry Account Owner (no data actions) · Foundry Owner.
Inbound privacy: private endpoint with public access disabled. Outbound agent isolation: Standard setup with your own VNet.

In code

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

project = AIProjectClient(
    endpoint="https://<res>.services.ai.azure.com/api/projects/<proj>",
    credential=DefaultAzureCredential())   # keyless: no API key anywhere

Exam traps

*Azure AI User* is now Foundry User (same role ID).
*Azure AI Developer* is scoped to Azure ML hubs; it's the wrong answer for Foundry projects.
Viewing quota needs Cognitive Services Usages Reader at subscription scope.
For a standalone Azure OpenAI resource the role is *Cognitive Services OpenAI User*; for a Foundry project, *Foundry User*.

Hands-on: Assign yourself Foundry User on a project, remove key access, and call a model with DefaultAzureCredential only.

Source: learn: azure/foundry/concepts/rbac-foundry · learn: azure/foundry/foundry-models/quotas-limits

Responsible AI: guardrails, evaluators and red teaming · Plan & manage · responsible AI

A guardrail is a named set of controls: which risk, where it's checked (input, output, tool call, tool response) and what happens (annotate, or annotate and block). Evaluators score quality and safety; the AI Red Teaming Agent attacks the app and reports how often attacks succeed.

Key facts

Risks: hate, sexual, self-harm, violence (severity threshold) · Prompt Shields (direct and indirect attacks) · protected material · groundedness (preview, models only) · PII (preview) · task adherence (preview).
Guardrails for models are GA; for agents, preview. Tool-call and tool-response checkpoints apply to agents only.
Default guardrail: Microsoft.DefaultV2. An agent without its own guardrail inherits its model deployment's; one with its own fully overrides it.
Evaluators: quality (coherence, fluency), RAG (retrieval, groundedness 1–5, relevance), safety (violence, protected material, indirect attack…), agent (task adherence, intent resolution, tool call accuracy…).
Groundedness Pro is pass/fail via Content Safety and needs no model deployment.
AI Red Teaming Agent: built on PyRIT, metric is Attack Success Rate (ASR), strategies such as Base64 and Flip.

Exam traps

Three different 'groundedness' things: the guardrail (blocks), the evaluator (scores 1–5) and Groundedness Pro (pass/fail).
Annotate-only is not available for agents.
An agent's own guardrail replaces the model's entirely; it doesn't merge.

Hands-on: Create a guardrail with Prompt Shields for indirect attacks on tool responses, attach it to an agent, and send a document with a hidden instruction.

Source: learn: azure/foundry/guardrails/guardrails-overview · learn: azure/foundry/concepts/built-in-evaluators