Run Live Provider Tests

Prompty’s unit tests mock provider SDKs. Live integration tests use real endpoints to catch wire-format and authentication issues that mocks can miss, such as unsupported empty stop arrays, deployment/model differences, and SDK response-shape differences.

The shared live-provider conformance vectors use the same rule in every runtime: missing environment variables or tenant-compatible identity skip the vector, while present capability plus provider rejection is a failure. The vectors assert structural acceptance (accepted: true and non-empty content) rather than provider-specific wording, so runtimes can verify live wire compatibility without making brittle assertions about model output.

Caution

Live tests call paid services. Use test resources, set conservative model options, and never commit .env files.

Environment variables

Create a local .env file for the runtime you are testing.

Variable	Used by	Notes
OPENAI_API_KEY	OpenAI-compatible proxy tests	Optional OPENAI_BASE_URL may point at a proxy endpoint.
DIRECT_OPENAI_API_KEY	Direct OpenAI tests	Uses https://api.openai.com/v1.
ANTHROPIC_API_KEY	Anthropic tests	Used by Anthropic provider tests.
AZURE_OPENAI_ENDPOINT	Foundry/Azure OpenAI tests	Azure OpenAI endpoint or compatible Foundry endpoint.
AZURE_OPENAI_API_KEY	Foundry/Azure OpenAI tests	API key auth.
AZURE_OPENAI_CHAT_DEPLOYMENT	Chat/agent tests	Deployment id, not base model name.
AZURE_OPENAI_EMBEDDING_DEPLOYMENT	Embedding tests	Deployment id for embedding model.

Entra ID tests also require a local Azure identity that can access the target resource. If the token tenant does not match the resource tenant, tests should skip rather than report a runtime failure.

Commands

Python

cd runtime/python/prompty
uv run pytest tests/integration/ -v -o "addopts="

Integration tests are excluded from the normal unit-test run and skip automatically when required environment variables are missing.

TypeScript

cd runtime/typescript
npm run build
npm run test --workspaces

Provider tests import the built packages from dist, so build before relying on integration behavior.

Rust

cd runtime/rust
cargo test --workspace -- --ignored --test-threads=1

Rust live tests are ignored by default. Use one test thread when tests mutate environment variables.

C#

cd runtime/csharp
dotnet test prompty.sln --filter "Category=Integration"

C# integration tests use SkippableFact for missing credentials or tenant-specific auth conditions.

What live tests cover

• Chat completions and provider-specific wire options
• Streaming wrappers and trace accumulation
• Structured output and response-format mapping
• Embeddings and image generation where supported
• Tool-calling agent loops
• Provider model listing/discovery
• API-key auth and, where configured, Entra ID auth

Interpreting skips

A skip usually means the local machine is missing a provider key, deployment name, or tenant-compatible identity. A failure means the runtime likely sent an invalid request, parsed a response incorrectly, or used an endpoint with incompatible options.

Run the smallest provider-specific test first when debugging, then rerun the full integration suite once fixed.