Policy-Driven AI: Designing Configuration-Driven Model Selection for Enterprise Systems

Every enterprise AI integration starts the same way. A developer writes client.chat("gpt-4", ...) somewhere in the codebase, ships it, and moves on. client.chat("gpt-4", ...) Six months later, the organization is running 14 different AI-powered features across 6 services, each with hardcoded model references, scattered API keys, and zero visibility into which model is handling what data. The finance team wants cost controls. Legal wants to know if customer PII is hitting external APIs. Platform engineering wants to consolidate providers. And every single change requires a code deploy. This is the model selection problem, and most teams don't realize they have it until it's already expensive. The solution isn't another wrapper library. It's a policy layer — a configuration-driven system that decouples what model to use from where the call happens, governed by rules that non-engineers can reason about. what where Why Hardcoded Model Selection Breaks at Scale Before diving into architecture, it's worth understanding why direct model calls fail in enterprise environments. The failure modes are predictable. Provider lock-in accelerates silently. When model identifiers are scattered across services, switching from OpenAI to Anthropic or a self-hosted model becomes a multi-sprint migration. Each integration point is a separate refactor. Teams that planned for "easy swapping" discover that prompt formats, token limits, and response structures differ enough to make swaps non-trivial even with an abstraction layer. Provider lock-in accelerates silently. Cost visibility disappears. Without centralized routing, there's no single point where you can answer "how much are we spending on GPT-4 vs. Claude Sonnet vs. Llama for classification tasks?" Chargebacks to product teams become guesswork. Optimization is impossible without measurement. Cost visibility disappears. Compliance becomes a game of whack-a-mole. GDPR, HIPAA, SOC 2, and internal data classification policies all impose constraints on where data can go. If model selection lives in application code, enforcing data routing policies means auditing every service. One missed integration and sensitive data flows to an external provider it shouldn't. Compliance becomes a game of whack-a-mole. Experimentation requires deployment. Want to A/B test whether Claude Sonnet handles customer support summarization better than GPT-4o-mini? With hardcoded calls, that's a code change, a PR, a review, a deploy, and monitoring — for each experiment. Experimentation requires deployment. These aren't theoretical problems. They're the reality at any organization running AI at scale. The Architecture: A Policy Engine for Model Selection The core idea is straightforward: introduce a layer between your application code and model providers that selects models based on declarative policies rather than imperative code. Here's the high-level architecture: ┌─────────────────────────────────────────────────────┐ │ Application Code │ │ │ │ ai.complete(task="summarize", input=doc, meta={}) │ └──────────────────────┬──────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────┐ │ Policy Resolution Engine │ │ │ │ 1. Match request to policy rules │ │ 2. Evaluate constraints (cost, latency, compliance) │ │ 3. Select model + provider │ │ 4. Apply transformations (prompt template, params) │ └──────────────────────┬───────────────────────────────┘ │ ┌───────────┼───────────┐ ▼ ▼ ▼ ┌─────────┐ ┌─────────┐ ┌──────────┐ │ OpenAI │ │Anthropic│ │ Self-Host │ │ Adapter │ │ Adapter │ │ Adapter │ └─────────┘ └─────────┘ └──────────┘ ┌─────────────────────────────────────────────────────┐ │ Application Code │ │ │ │ ai.complete(task="summarize", input=doc, meta={}) │ └──────────────────────┬──────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────┐ │ Policy Resolution Engine │ │ │ │ 1. Match request to policy rules │ │ 2. Evaluate constraints (cost, latency, compliance) │ │ 3. Select model + provider │ │ 4. Apply transformations (prompt template, params) │ └──────────────────────┬───────────────────────────────┘ │ ┌───────────┼───────────┐ ▼ ▼ ▼ ┌─────────┐ ┌─────────┐ ┌──────────┐ │ OpenAI │ │Anthropic│ │ Self-Host │ │ Adapter │ │ Adapter │ │ Adapter │ └─────────┘ └─────────┘ └──────────┘ The application code never references a specific model. It declares intent — what task it's performing, what data classification it's working with, what quality tier it needs. The policy engine resolves that intent to a concrete model, provider, and configuration. intent Designing the Policy Schema The policy schema is the heart of the system. It needs to be expressive enough to capture real enterprise constraints while staying readable by non-engineers. YAML is the pragmatic choice — it's version-controllable, diff-friendly, and most ops teams already know it. Here's a schema that covers the critical dimensions: yaml # model-policies.yaml version: "2.0" defaults: max_latency_ms: 5000 max_cost_per_request: 0.05 fallback_strategy: "cascade" retry: max_attempts: 3 backoff_multiplier: 1.5 data_classifications: public: allowed_providers: ["openai", "anthropic", "google", "self-hosted"] internal: allowed_providers: ["anthropic", "openai", "self-hosted"] require_encryption_at_rest: true confidential: allowed_providers: ["self-hosted", "anthropic"] require_encryption_at_rest: true require_dpa: true restricted: allowed_providers: ["self-hosted"] require_encryption_at_rest: true require_audit_log: true require_vpc: true policies: - name: "customer-support-summarization" match: task: "summarize" domain: "customer-support" data_classification: "confidential" routing: primary: provider: "anthropic" model: "claude-sonnet-4-20250514" max_tokens: 1024 temperature: 0.3 fallback: - provider: "self-hosted" model: "llama-3.1-70b" max_tokens: 1024 temperature: 0.3 constraints: max_latency_ms: 3000 max_input_tokens: 8000 cost_tier: "standard" - name: "internal-code-review" match: task: "code-review" domain: "engineering" data_classification: "internal" routing: primary: provider: "anthropic" model: "claude-sonnet-4-20250514" temperature: 0.2 fallback: - provider: "openai" model: "gpt-4o" temperature: 0.2 constraints: max_latency_ms: 10000 cost_tier: "premium" - name: "bulk-classification" match: task: "classify" priority: "batch" data_classification: "internal" routing: primary: provider: "openai" model: "gpt-4o-mini" temperature: 0.0 fallback: - provider: "self-hosted" model: "llama-3.1-8b" constraints: max_cost_per_request: 0.002 cost_tier: "economy" - name: "default-catch-all" match: task: "*" data_classification: "public" routing: primary: provider: "openai" model: "gpt-4o-mini" constraints: cost_tier: "economy" # model-policies.yaml version: "2.0" defaults: max_latency_ms: 5000 max_cost_per_request: 0.05 fallback_strategy: "cascade" retry: max_attempts: 3 backoff_multiplier: 1.5 data_classifications: public: allowed_providers: ["openai", "anthropic", "google", "self-hosted"] internal: allowed_providers: ["anthropic", "openai", "self-hosted"] require_encryption_at_rest: true confidential: allowed_providers: ["self-hosted", "anthropic"] require_encryption_at_rest: true require_dpa: true restricted: allowed_providers: ["self-hosted"] require_encryption_at_rest: true require_audit_log: true require_vpc: true policies: - name: "customer-support-summarization" match: task: "summarize" domain: "customer-support" data_classification: "confidential" routing: primary: provider: "anthropic" model: "claude-sonnet-4-20250514" max_tokens: 1024 temperature: 0.3 fallback: - provider: "self-hosted" model: "llama-3.1-70b" max_tokens: 1024 temperature: 0.3 constraints: max_latency_ms: 3000 max_input_tokens: 8000 cost_tier: "standard" - name: "internal-code-review" match: task: "code-review" domain: "engineering" data_classification: "internal" routing: primary: provider: "anthropic" model: "claude-sonnet-4-20250514" temperature: 0.2 fallback: - provider: "openai" model: "gpt-4o" temperature: 0.2 constraints: max_latency_ms: 10000 cost_tier: "premium" - name: "bulk-classification" match: task: "classify" priority: "batch" data_classification: "internal" routing: primary: provider: "openai" model: "gpt-4o-mini" temperature: 0.0 fallback: - provider: "self-hosted" model: "llama-3.1-8b" constraints: max_cost_per_request: 0.002 cost_tier: "economy" - name: "default-catch-all" match: task: "*" data_classification: "public" routing: primary: provider: "openai" model: "gpt-4o-mini" constraints: cost_tier: "economy" Several design decisions here are worth calling out. Match rules use task + domain, not service names. This is intentional. If you route based on which microservice is calling, you couple policies to your service topology. When services get refactored or merged, policies break. Routing based on task semantics ("summarize," "classify," "generate") and business domain ("customer-support," "engineering") is stable across architectural changes. Match rules use task + domain, not service names. Data classification drives provider constraints. The data_classifications block acts as a hard gate. Even if a policy says to use OpenAI, if the request is tagged as restricted, the classification rules override it. This creates a layered enforcement model where compliance rules are structurally impossible to bypass through policy misconfiguration. Data classification drives provider constraints. data_classifications restricted Fallback chains are explicit. Rather than retry the same model, the cascade strategy tries different providers. If Anthropic is down, the system falls back to the self-hosted Llama instance. This is both a reliability pattern and a cost optimization — fallback models can be cheaper alternatives that are acceptable for degraded operation. Fallback chains are explicit. The Policy Resolution Algorithm The resolution engine needs to be fast (it's in the hot path of every AI call) and deterministic (same inputs always produce the same routing decision). Here's the logic: RESOLVE(request): 1. Extract metadata: task, domain, priority, data_classification 2. Find matching policies (ordered by specificity) 3. For each matched policy: a. Check data_classification constraints b. Evaluate cost/latency constraints against current metrics c. Check provider health status d. If all pass → return routing config 4. If primary fails constraints → walk fallback chain 5. If all fail → return default policy or reject RESOLVE(request): 1. Extract metadata: task, domain, priority, data_classification 2. Find matching policies (ordered by specificity) 3. For each matched policy: a. Check data_classification constraints b. Evaluate cost/latency constraints against current metrics c. Check provider health status d. If all pass → return routing config 4. If primary fails constraints → walk fallback chain 5. If all fail → return default policy or reject Specificity ordering matters. A request with task=summarize, domain=customer-support should match the customer-support-summarization policy over the default-catch-all. The resolution engine scores matches by the number of matched fields, with exact matches ranking above wildcards. task=summarize, domain=customer-support customer-support-summarization default-catch-all The critical implementation detail is step 2b: evaluating constraints against current metrics. This means the policy engine needs access to real-time (or near-real-time) data about provider latency, costs, and availability. This creates a feedback loop: current metrics Request → Policy Engine → Provider → Response ↑ │ └── Metrics Collector ◄──┘ Request → Policy Engine → Provider → Response ↑ │ └── Metrics Collector ◄──┘ The metrics collector feeds back p50/p95 latency, error rates, and per-request costs. The policy engine uses these to make dynamic routing decisions. If Anthropic's p95 latency spikes above the policy's max_latency_ms, the engine automatically routes to the fallback without any human intervention. max_latency_ms Implementing the Abstraction Layer The application-facing API should be minimal. Developers shouldn't need to understand the policy system to use it. Here's what the interface looks like: python from ai_gateway import complete, classify, summarize # Option 1: Task-specific functions result = summarize( input=document_text, domain="customer-support", data_classification="confidential", metadata={"ticket_id": "TICK-4521"} ) # Option 2: Generic interface with task declaration result = complete( task="summarize", input=document_text, context={ "domain": "customer-support", "data_classification": "confidential", } ) # The developer never specifies a model. # The policy engine handles selection, fallback, retries, and logging. from ai_gateway import complete, classify, summarize # Option 1: Task-specific functions result = summarize( input=document_text, domain="customer-support", data_classification="confidential", metadata={"ticket_id": "TICK-4521"} ) # Option 2: Generic interface with task declaration result = complete( task="summarize", input=document_text, context={ "domain": "customer-support", "data_classification": "confidential", } ) # The developer never specifies a model. # The policy engine handles selection, fallback, retries, and logging. Under the hood, this client library does the following: Packages the request with metadata Sends it to the policy resolution engine (either in-process or via a gateway service) The engine resolves the policy, selects the model, and formats the provider-specific request Executes the call with retry/fallback logic Returns a normalized response regardless of which provider handled it Packages the request with metadata Sends it to the policy resolution engine (either in-process or via a gateway service) The engine resolves the policy, selects the model, and formats the provider-specific request Executes the call with retry/fallback logic Returns a normalized response regardless of which provider handled it The response normalization is subtle but critical. Each provider returns slightly different structures. The gateway abstracts this so downstream code doesn't need provider-specific parsing. python @dataclass class AIResponse: content: str model_used: str # Which model actually handled this provider: str # Which provider latency_ms: int # End-to-end latency input_tokens: int # Normalized token counts output_tokens: int cost_usd: float # Estimated cost policy_name: str # Which policy was applied fallback_used: bool # Whether primary failed request_id: str # For tracing @dataclass class AIResponse: content: str model_used: str # Which model actually handled this provider: str # Which provider latency_ms: int # End-to-end latency input_tokens: int # Normalized token counts output_tokens: int cost_usd: float # Estimated cost policy_name: str # Which policy was applied fallback_used: bool # Whether primary failed request_id: str # For tracing This response structure gives observability teams everything they need. You can now answer questions like "what percentage of customer-support requests fell back to the self-hosted model last week?" directly from your metrics pipeline. Handling the Hard Parts Prompt Compatibility Across Models Models aren't interchangeable. A prompt optimized for Claude might perform differently on GPT-4o or Llama. The policy system needs to handle this. The cleanest approach is prompt templates keyed by model family: yaml prompt_templates: summarize: anthropic: system: "You are a precise summarizer for customer support tickets. Extract the core issue, customer sentiment, and any action items." template: "Summarize the following support ticket:\n\n{input}\n\nProvide: 1) Core issue 2) Sentiment 3) Action items" openai: system: "Summarize customer support tickets concisely." template: "Ticket content:\n{input}\n\nSummarize with: core issue, sentiment, action items." self-hosted: template: "[INST] Summarize this support ticket. Include the core issue, customer sentiment, and action items.\n\n{input} [/INST]" prompt_templates: summarize: anthropic: system: "You are a precise summarizer for customer support tickets. Extract the core issue, customer sentiment, and any action items." template: "Summarize the following support ticket:\n\n{input}\n\nProvide: 1) Core issue 2) Sentiment 3) Action items" openai: system: "Summarize customer support tickets concisely." template: "Ticket content:\n{input}\n\nSummarize with: core issue, sentiment, action items." self-hosted: template: "[INST] Summarize this support ticket. Include the core issue, customer sentiment, and action items.\n\n{input} [/INST]" This adds maintenance overhead, but it's the honest solution. The alternative — pretending all models respond identically to the same prompt — leads to subtle quality degradation that's hard to debug. Cost Budgets and Rate Limiting Enterprise systems need spending guardrails. The policy layer is the natural place for this: yaml budgets: global: daily_limit_usd: 5000 alert_threshold: 0.8 # Alert at 80% per_team: engineering: monthly_limit_usd: 15000 customer-support: monthly_limit_usd: 8000 marketing: monthly_limit_usd: 3000 per_policy: customer-support-summarization: daily_limit_usd: 500 rate_limit: 1000/hour budgets: global: daily_limit_usd: 5000 alert_threshold: 0.8 # Alert at 80% per_team: engineering: monthly_limit_usd: 15000 customer-support: monthly_limit_usd: 8000 marketing: monthly_limit_usd: 3000 per_policy: customer-support-summarization: daily_limit_usd: 500 rate_limit: 1000/hour When a budget is exhausted, the policy engine can either reject requests, downgrade to a cheaper model automatically, or queue requests for batch processing. The behavior is — you guessed it — configurable via policy. Model Versioning and Canary Deployments When OpenAI releases a new model or Anthropic updates Claude, you don't want to switch all traffic instantly. The policy layer supports canary-style rollouts: yaml policies: - name: "customer-support-summarization" match: task: "summarize" domain: "customer-support" routing: primary: provider: "anthropic" model: "claude-sonnet-4-20250514" weight: 90 canary: provider: "anthropic" model: "claude-sonnet-4-20250514" # newer version weight: 10 evaluation: metrics: ["quality_score", "latency", "cost"] min_sample_size: 500 auto_promote_threshold: 0.95 policies: - name: "customer-support-summarization" match: task: "summarize" domain: "customer-support" routing: primary: provider: "anthropic" model: "claude-sonnet-4-20250514" weight: 90 canary: provider: "anthropic" model: "claude-sonnet-4-20250514" # newer version weight: 10 evaluation: metrics: ["quality_score", "latency", "cost"] min_sample_size: 500 auto_promote_threshold: 0.95 Ten percent of traffic goes to the canary. Once the system collects 500 samples and the quality score exceeds the threshold, it can auto-promote the canary to primary. This is the same pattern used for service deployments, applied to model versions. The Governance Layer For larger organizations, the policy files themselves need governance. Who can change routing for PII-handling tasks? Who approves adding a new provider? This is where GitOps shines. Policy files live in a dedicated repository with branch protection: ai-policies/ ├── policies/ │ ├── customer-support.yaml │ ├── engineering.yaml │ ├── marketing.yaml │ └── defaults.yaml ├── classifications/ │ └── data-classes.yaml ├── budgets/ │ └── team-budgets.yaml ├── tests/ │ ├── test_policy_resolution.py │ └── test_classification_enforcement.py └── CODEOWNERS ai-policies/ ├── policies/ │ ├── customer-support.yaml │ ├── engineering.yaml │ ├── marketing.yaml │ └── defaults.yaml ├── classifications/ │ └── data-classes.yaml ├── budgets/ │ └── team-budgets.yaml ├── tests/ │ ├── test_policy_resolution.py │ └── test_classification_enforcement.py └── CODEOWNERS The CODEOWNERS file enforces review requirements: CODEOWNERS # Data classification changes require security team approval classifications/ @security-team # Budget changes require finance approval budgets/ @finance-team @platform-team # Policy changes require domain team + platform review policies/customer-support.yaml @cs-team @platform-team policies/engineering.yaml @eng-leads @platform-team # Data classification changes require security team approval classifications/ @security-team # Budget changes require finance approval budgets/ @finance-team @platform-team # Policy changes require domain team + platform review policies/customer-support.yaml @cs-team @platform-team policies/engineering.yaml @eng-leads @platform-team Changes are validated by CI before merge. Tests verify that no policy routes restricted data to external providers, that budget limits are within organizational bounds, and that every task has at least one valid routing path. restricted Observability: The Payoff Once all AI traffic flows through the policy layer, you get centralized observability for free. Every request generates a structured event: json { "timestamp": "2026-02-16T14:30:00Z", "request_id": "req_abc123", "task": "summarize", "domain": "customer-support", "data_classification": "confidential", "policy_applied": "customer-support-summarization", "provider": "anthropic", "model": "claude-sonnet-4-20250514", "fallback_used": false, "input_tokens": 2847, "output_tokens": 312, "latency_ms": 1243, "estimated_cost_usd": 0.0089, "team": "cs-operations", "quality_score": null } { "timestamp": "2026-02-16T14:30:00Z", "request_id": "req_abc123", "task": "summarize", "domain": "customer-support", "data_classification": "confidential", "policy_applied": "customer-support-summarization", "provider": "anthropic", "model": "claude-sonnet-4-20250514", "fallback_used": false, "input_tokens": 2847, "output_tokens": 312, "latency_ms": 1243, "estimated_cost_usd": 0.0089, "team": "cs-operations", "quality_score": null } From this, you can build dashboards that answer the questions executives actually ask: "How much are we spending on AI per team?" "Which tasks have the highest error rates?" "Are we compliant with our data handling policies?" "What would happen to our costs if we moved all classification tasks to the self-hosted model?" That last question is particularly powerful. Because the policy layer is declarative, you can run simulations against historical traffic to model the impact of policy changes before deploying them. What This Is Not This architecture is not a universal AI gateway product. It's a pattern for teams building AI into enterprise systems that need control and visibility without sacrificing development velocity. It doesn't solve prompt engineering. It doesn't solve evaluation. It doesn't replace your ML platform. What it does is give you a single, auditable, policy-driven control plane for the question of "which model handles this request and why." For teams running fewer than five AI-powered features on a single provider, this is overkill. For everyone else — especially in regulated industries, multi-provider environments, or organizations where AI costs are becoming a line item worth managing — this pattern pays for itself the first time someone asks, "Can we switch providers for this workload?" and the answer is a YAML change instead of a sprint. Getting Started If you're building this from scratch, here's the pragmatic sequence: Start with the abstraction layer. Get your application code calling complete(task=..., domain=...) instead of provider-specific clients, even if the "policy engine" is initially just a hardcoded mapping. This decoupling is the most valuable change and costs the least. complete(task=..., domain=...) Add the policy file next. Replace the hardcoded mapping with YAML-driven configuration. Keep it simple — task-to-model mappings with a default fallback. Layer in data classification enforcement. This is where you get compliance value. Hard-gate provider access based on data sensitivity. Build the metrics feedback loop. Collect latency, cost, and error data per policy. Use it for reporting first, then for dynamic routing decisions. Add cost controls and canary support as you need them. These are refinements, not foundations. The goal isn't to build the full system on day one. It's to establish the pattern early so that every new AI integration automatically inherits routing policies, cost controls, compliance enforcement, and observability without any additional work from the feature team.