Skip to content
Skills Library
AI & agents · Testing

testing-course-samples

/testing-course-samples

Use when asked to validate, test, smoke-test, or run the course's notebook and code samples against a live Microsoft Foundry / Azure OpenAI configuration. Covers environment setup (.env, az login, packages), the scripts/validate-notebooks.ps1 runner, interpreting PASS/FAIL results, and which lessons need extra resources (Azure AI Search, GitHub MCP, Foundry Local, Playwright).

Licence
MIT
Last repository change
24.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
2
Deprecation

No signals detected. Repository changed on 24.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Use when asked to validate, test, smoke-test, or run the course's notebook and code samples against a live Microsoft Foundry / Azure OpenAI configuration. Covers environment setup (.env, az login, packages), the scripts/validate-notebooks.ps1 runner, interpreting PASS/FAIL results, and which lessons need extra resources (Azure AI Search, GitHub MCP, Foundry Local, Playwright).

Original author description

Use when asked to validate, test, smoke-test, or run the course's notebook and code samples against a live Microsoft Foundry / Azure OpenAI configuration. Covers environment setup (.env, az login, packages), the scripts/validate-notebooks.ps1 runner, interpreting PASS/FAIL results, and which lessons need extra resources (Azure AI Search, GitHub MCP, Foundry Local, Playwright).

How it works

  1. When to use
  2. Prerequisites (check first)
  3. Running the validation
  4. Interpreting results
  5. Lessons that need extra resources (expected to fail without them)
  6. Reporting back
Original SKILL.md headings · 76 lines, 530 words
SKILL.md preview
---
name: "testing-course-samples"
description: "Use when asked to validate, test, smoke-test, or run the course's notebook and code samples against a live Microsoft Foundry / Azure OpenAI configuration. Covers environment setup (.env, az login, packages), the scripts/validate-notebooks.ps1 runner, interpreting PASS/FAIL results, and which lessons need extra resources (Azure AI Search, GitHub MCP, Foundry Local, Playwright)."
---

# Testing the Course Samples

Validate that the lesson notebooks and code samples run against a live
Microsoft Foundry / Azure OpenAI setup. The repo ships a runner at
[`scripts/validate-notebooks.ps1`](../../../scripts/validate-notebooks.ps1) that
executes every Python notebook headlessly and prints a PASS/FAIL matrix.

## When to use
- "Validate all the notebooks / samples against my Azure subscription."
- "Smoke-test the course after upgrading packages or changing models."
- "Which lessons still pass / fail live?"

Do **not** use this for the AI Smoke Test GitHub Action (that validates *deployed*
hosted agents — see [`tests/README.md`](../../../tests/README.md)). This skill
runs the notebooks locally.

## Prerequisites (check first)
1. **Python 3.12+** with course deps: `python -m pip install -r requirements.txt`
   plus the executor: `python -m pip install nbconvert ipykernel`.
2. **`.env` at the repo root** (copy from [`.env.example`](../../../.env.example)) with at least:
   - `AZURE_AI_PROJECT_ENDPOINT` — Foundry project endpoint
     (`https://<account>.services.ai.azure.com/api/projects/<project>`)
   - `AZURE_AI_MODEL_DEPLOYMENT_NAME` — a non-deprecated deployment (e.g. `gpt-5-mini`)
   - `AZURE_OPENAI_ENDPOINT` (`https://<account>.openai.azure.com`) and `AZURE_OPENAI_DEPLOYMENT`
     for lessons that call Azure OpenAI directly (Lesson 06, 02-azure-openai, 14 handoff/human-loop).
3. **`az login`** completed — samples authenticate with `AzureCliCredential` (Entra ID, keyless).
4. Verify the model deployment exists:
   `az cognitiveservices account deployment list -g <rg> -n <account> -o table`.

## Running the validation
```powershell
# All Python notebooks (skips .NET, .venv, site-packages, translations, skill assets)
pwsh scripts/validate-notebooks.ps1

# A single lesson, with a longer per-cell timeout
pwsh scripts/validate-notebooks.ps1 -Filter '08-*' -Timeout 600

# Just list what would run (no execution)
pwsh scripts/validate-notebooks.ps1 -List

# Explicit interpreter (if `python` is not on PATH, e.g. Windows Store alias)
pwsh scripts/validate-notebooks.ps1 -Python "C:/path/to/python.exe"
```
The script writes executed copies, per-notebook logs, and `results.json` to
`$env:TEMP\aiab-nbval` and exits with the number of failures.

Transient failures (shared-subscription HTTP 429 rate limits, an occasional
`AzureCliCredential` token hiccup, or a timeout) are retried automatically
(`-Retries`, default 2, with `-RetryDelaySeconds` backoff, default 20). If a
model deployment is regularly 429-ing, check the subscription's GlobalStandard
TPM quota (`az cognitiveservices usage list -l <region>`) — raising a single
deployment's capacity does not help when the *subscription* quota is exhausted.

## Interpreting results
- `PASS` — the notebook ran end-to-end with no cell error.
- `FAIL` — the first `*Error` / `*Exception` line is shown; open the matching
  `log_*.txt` in the output dir for the full traceback.
- A single notebook's failure is bounded by `-Timeout` (per cell), so a hung
  human-in-the-loop cell surfaces as `StdinNotImplementedError` rather than hanging.

## Lessons that need extra resources (expected to fail without them)
| Lesson | Extra requirement |
|--------|-------------------|
| 05 Agentic RAG | Azure AI Search (`AZURE_SEARCH_SERVICE_ENDPOINT`, key) — has an in-memory fallback path |
| 11 MCP / GitHub | GitHub MCP server + PAT |
| 13 memory (cognee) | `cognee` configured with a model provider |
| 15 browser-use | Playwright browsers installed (`playwright install`) + `AZURE_OPENAI_CHAT_DEPLOYMENT_NAME` |
| 17 local agent | Foundry Local runtime + a downloaded Qwen model (on-device, no cloud) |
| `*-dotnet-*` notebooks | .NET Interactive kernel (excluded by default; use `-IncludeDotnet`) |

## Reporting back
Summarise as a PASS/FAIL table grouped by lesson. Separate genuine regressions
(code/config bugs to fix) from environment gaps (missing Search/Foundry Local/PAT),
and cite the failing `log_*.txt` for each real failure.

Versions

RepositoryScoreLicence
This page · microsoft/ai-agents-for-beginners100MIT
microsoft/ai-agents-for-beginners100MIT
microsoft/ai-agents-for-beginners100MIT
microsoft/ai-agents-for-beginners100MIT
microsoft/ai-agents-for-beginners100MIT

Identical content appears in 1 other GitHub folders. We show one copy with the highest score.

Database snapshot: 13.11.2026 · package version 2e161421d32b (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 85microsoft-foundry-local
    Expert knowledge for Microsoft Foundry Local (aka Azure AI Foundry Local) development including best practices, decision making, configuration, and integrations & coding patterns. Use when compiling HF models with Olive, using Foundry Local CLI, embeddings/chat/audio APIs, or…
  • 85microsoft-foundry-local
    Expert knowledge for Microsoft Foundry Local (aka Azure AI Foundry Local) development including best practices, decision making, configuration, and integrations & coding patterns. Use when compiling HF models with Olive, using Foundry Local CLI, embeddings/chat/audio APIs, or…
  • 95microsoft-foundry-classic
    Expert knowledge for Microsoft Foundry Classic (aka Azure AI Foundry classic) development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use…
  • 95microsoft-foundry
    Expert knowledge for Microsoft Foundry (aka Azure AI Foundry) development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when building…
  • 95microsoft-foundry
    Use this skill to work with Microsoft Foundry (Azure AI Foundry): deploy AI models from catalog, build RAG applications with knowledge indexes, create and evaluate AI agents. USE FOR: Microsoft Foundry, AI Foundry, deploy model, model catalog, RAG, knowledge index, create agent,…
  • 98foundry-hosted-agent-validation
    Step-by-step process for validating a Python Foundry hosted agent sample (under python/samples/04-hosting/foundry-hosted-agents/) end to end — running it locally (native runtime and `azd ai agent run`) and after deploying it to an Azure AI Foundry project with `azd`. Use this…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.