I have been building microservices in distributed systems for over a decode now. Everything has been predictable and controllable—I never had any big issue when it comes to the behavior of these systems. This changed when I introduced AI: The system started being unpredictableDebugging became harderThe system became unreliable The system started being unpredictable Debugging became harder The system became unreliable That's when I realized: Adding AI to microservices is not straightforward and can lead to serious problems if not handled in the right way. What Caused These Changes? What Caused These Changes? In every system I have built before, I always had one assumption that a microservice produces the same output on the same input. And this has always been true and is the foundation of everything: DebuggingTesting,CachingAnd most importantly, system reliability. Debugging Testing, Caching And most importantly, system reliability. On adding AI, this assumption turned false—same input produced different outputs. The system shifted from being deterministic to probabilistic. deterministic probabilistic At first, I thought it was a normal bug. It wasn't, It was the model. The Problematic system The Problematic system I was working on a fraud detection system that analyzed users transactions and flagged suspicious activities. It was a backend setup with: API gatewayDatabaseTransaction serviceAnd the AI service for fraud detection API gateway Database Transaction service And the AI service for fraud detection The functions of the AI service were: To look at a user's transactionsAssign risk scoreAnd to decide whether the transaction is fraudulent or not, explaining the reason behind that decision To look at a user's transactions Assign risk score And to decide whether the transaction is fraudulent or not, explaining the reason behind that decision At first, it worked as expected. The API request looked like this: {
    "amount": 1000,
    "location": "California",
    "merchant": "xyz online store",
    "user_history": [...]
} {
    "amount": 1000,
    "location": "California",
    "merchant": "xyz online store",
    "user_history": [...]
} The response was predictable and clean: {
    "risk_score": 0.68,
    "is_fraud": true,
    "reason": "Transaction is unusual based on the user's past spending pattern"
} {
    "risk_score": 0.68,
    "is_fraud": true,
    "reason": "Transaction is unusual based on the user's past spending pattern"
} I ran the same request multiple times, and the responses were different. Sometimes it flagged the transaction as fraudulent, other times it didn't. Keep in mind this was one system, same input, different outputs. same input, different outputs That's a very serious problem as far as financial systems are concerned. Breakdown Breakdown The system had the following trade-offs: It lost determinism It lost determinism Microservices are deterministic in nature, whereas AI models are non-deterministic and work on the concept of probability. When I introduced AI to the system, I lost this determinism and so the idempotency I initially relied on became useless. Retries became unsafe and caching didn’t make sense anymore. This broke: My retry logic and assumptionsMy caching strategiesAnd most importantly, the confidence I had in the system—it became unreliable My retry logic and assumptions My caching strategies And most importantly, the confidence I had in the system—it became unreliable Debugging became harder Debugging became harder Before I added AI, debugging was relatively easier and straightforward. I could trace bugs by simply checking requests and logs, then reproduce issues. With AI, debugging became a nightmare. I couldn’t reproduce any meaningful issues. Logs didn’t help because there was no clear execution path to inspect: Same requestSame inputDifferent response Same request Same input Different response I found debugging draining and almost impossible. No explicit errors No explicit errors Microservices are designed to produce and handle explicit errors. This is contrary to AI models because they don’t produce explicit errors, but instead deliver silent failures in the form of false statements—hallucination. hallucination. No error, no warning—just confident incorrect statements {
    “is_fraud”: false
    “reason”: “User frequently shops at this merchant”
} {
    “is_fraud”: false
    “reason”: “User frequently shops at this merchant”
} This was produced even though the user had never shopped at that merchant before. This is extremely dangerous. My retry logic, circuit breakers, and idempotency keys didn’t help—the system thought everything was perfect but in reality, catastrophic failures were awaiting. API interfaces became inconsistent API interfaces became inconsistent Before AI, types were strict and interfaces were consistent. With AI, interfaces lost strictness and became dangerously flexible: Interface fields changed or missed at timesFormat driftedAnd funny enough, reasons varied wildly. Interface fields changed or missed at times Format drifted And funny enough, reasons varied wildly. I was no longer dealing with strict interfaces. Mistakes Most Developers Do Mistakes Most Developers Do Blindly trusting AI output without manual review Blindly trusting AI output without manual review AI responses at times can be incorrect. Trusting it and taking their responses as final judgement can lead to: Miscalculated fraudLegal exposure, mainly from misjudgment Miscalculated fraud Legal exposure, mainly from misjudgment And bad financial decisions that can be costly Treating AI model as an ordinary microservice Treating AI model as an ordinary microservice Most developers wrap LLM models in APIs thinking it will work just like any other service. It won’t since AI is probabilistic—not deterministic as a normal microservice. Putting AI in critical paths Putting AI in critical paths Putting AI directly in critical paths such as core transaction flow or payment authorization, or just directly influencing transaction handling can lead to: Latency spikesHard to explain inconsistencies Latency spikes Hard to explain inconsistencies And even cascading failures—which can render a system unavailable Not validating AI output Not validating AI output AI responses are always unpredictable. Their outputs can drift from expected formats, thus validation is not only necessary—but a must. Accepting these outputs without validation is the same as accepting user inputs without validation. Solutions To The Above Mistakes Solutions To The Above Mistakes Most of the problems faced when adding AI models to microservices are caused by our design choices. We tend to use AI in a deterministic way contrary to its philosophy. AI is probabilistic. You can’t fix this—but you can contain it. The following are the solutions to most of these problems: Enforcing strict validation Enforcing strict validation Always treat AI output as untrusted information and validate it first. from pydantic import BaseModel, Field, ValidationError
from typing import Optional, Any
import re
import logging

logger = logging.getLogger(__name__)

class RiskAssessment(BaseModel):
    risk_score: float = Field(ge=0, le=1, description="Must be between 0 and 1")
    is_fraud: bool = Field(description="Must be boolean")
    reason: str = Field(max_length=500, description="Explanation, max 500 chars")

def normalize_fields(raw: dict) -> dict:
    normalized = {}
    for key, value in raw.items():
        snake_key = re.sub(r'(?<!^)(?=[A-Z])', '_', key).lower()
        if snake_key in ["risk_score", "riskscore"]:
            normalized["risk_score"] = value
        elif snake_key in ["is_fraud", "fraud"]:
            normalized["is_fraud"] = value
        elif snake_key in ["reason", "explanation", "message"]:
            normalized["reason"] = value
        else:
            normalized[snake_key] = value
    return normalized

def coerce_types(raw: dict) -> dict:
    coerced = raw.copy()
    if "risk_score" in coerced:
        try:
            coerced["risk_score"] = float(coerced["risk_score"])
        except (TypeError, ValueError):
            coerced["risk_score"] = 0.5
    if "is_fraud" in coerced:
        if isinstance(coerced["is_fraud"], str):
            coerced["is_fraud"] = coerced["is_fraud"].lower() in ["true", "yes", "1", "fraud"]
        else:
            coerced["is_fraud"] = bool(coerced["is_fraud"])
    if "reason" in coerced and not isinstance(coerced["reason"], str):
        coerced["reason"] = str(coerced["reason"])
    return coerced

def validate_ai_output(raw_output: Any) -> Optional[dict]:
    if not isinstance(raw_output, dict):
        logger.error(f"AI output is not a dict: {type(raw_output)}")
        return None
    normalized = normalize_fields(raw_output)
    coerced = coerce_types(normalized)
    try:
        validated = RiskAssessment(**coerced)
        return validated.dict()
    except ValidationError as e:
        logger.error(f"Validation failed: {e}")
        logger.error(f"Coerced data was: {coerced}")
        return None

async def get_ai_with_validation(transaction):
    raw = await call_ai_service(transaction)
    if not raw:
        return None
    validated = validate_ai_output(raw)
    if not validated:
        return rule_based_fallback(transaction)
    return validated from pydantic import BaseModel, Field, ValidationError
from typing import Optional, Any
import re
import logging

logger = logging.getLogger(__name__)

class RiskAssessment(BaseModel):
    risk_score: float = Field(ge=0, le=1, description="Must be between 0 and 1")
    is_fraud: bool = Field(description="Must be boolean")
    reason: str = Field(max_length=500, description="Explanation, max 500 chars")

def normalize_fields(raw: dict) -> dict:
    normalized = {}
    for key, value in raw.items():
        snake_key = re.sub(r'(?<!^)(?=[A-Z])', '_', key).lower()
        if snake_key in ["risk_score", "riskscore"]:
            normalized["risk_score"] = value
        elif snake_key in ["is_fraud", "fraud"]:
            normalized["is_fraud"] = value
        elif snake_key in ["reason", "explanation", "message"]:
            normalized["reason"] = value
        else:
            normalized[snake_key] = value
    return normalized

def coerce_types(raw: dict) -> dict:
    coerced = raw.copy()
    if "risk_score" in coerced:
        try:
            coerced["risk_score"] = float(coerced["risk_score"])
        except (TypeError, ValueError):
            coerced["risk_score"] = 0.5
    if "is_fraud" in coerced:
        if isinstance(coerced["is_fraud"], str):
            coerced["is_fraud"] = coerced["is_fraud"].lower() in ["true", "yes", "1", "fraud"]
        else:
            coerced["is_fraud"] = bool(coerced["is_fraud"])
    if "reason" in coerced and not isinstance(coerced["reason"], str):
        coerced["reason"] = str(coerced["reason"])
    return coerced

def validate_ai_output(raw_output: Any) -> Optional[dict]:
    if not isinstance(raw_output, dict):
        logger.error(f"AI output is not a dict: {type(raw_output)}")
        return None
    normalized = normalize_fields(raw_output)
    coerced = coerce_types(normalized)
    try:
        validated = RiskAssessment(**coerced)
        return validated.dict()
    except ValidationError as e:
        logger.error(f"Validation failed: {e}")
        logger.error(f"Coerced data was: {coerced}")
        return None

async def get_ai_with_validation(transaction):
    raw = await call_ai_service(transaction)
    if not raw:
        return None
    validated = validate_ai_output(raw)
    if not validated:
        return rule_based_fallback(transaction)
    return validated Making AI advisory Making AI advisory Never allow AI to decide for your system. The workflow should be as follows: AI suggests → the system verifies → and finally, the system decides By combining AI with rule-based fallback, AI becomes input—not authority. Versioning prompts Versioning prompts AI prompts are like code changes. Versioning prompts help to trace change in behaviour of the model with time. Adding a control layer Adding a control layer Instead of calling AI service directly from core logic, wrap it with a layer responsible for: ValidationNormalizationFallback coordinationPrompt versioningAnd safety checks Validation Normalization Fallback coordination Prompt versioning And safety checks This layer sits between the core logic and AI service. Adding rule-based fallbacks Adding rule-based fallbacks Despite having AI, the system should still be able to perform its operations without fully relying on AI as illustrated by the following FastAPI code snippet: from fastapi import FastAPI
from pydantic import BaseModel, Field, ValidationError
import httpx
import logging

app = FastAPI()
logger = logging.getLogger(__name__)

class TransactionRequest(BaseModel):
    amount: float
    location: str
    merchant: str
    user_history: list = []

class RiskAssessment(BaseModel):
    risk_score: float = Field(ge=0, le=1)
    is_fraud: bool
    reason: str = Field(max_length=500)

def rule_based_fallback(transaction: TransactionRequest):
    risk_score = 0.0
    if transaction.amount > 50000:
        risk_score += 0.5
    if transaction.location in ["California", "Dallas", "Columbus"]:
        risk_score += 0.3
    if transaction.user_history and transaction.merchant not in transaction.user_history:
        risk_score += 0.2

    risk_score = min(risk_score, 1.0)

    return {
        "risk_score": risk_score,
        "is_fraud": risk_score > 0.6,
        "reason": "Fallback: AI unavailable, using rule-based decision"
    }

async def get_risk_assessment(transaction: TransactionRequest):
    try:
        async with httpx.AsyncClient(timeout=5.0) as client:
            response = await client.post(
                "http://ai-service/predict",
                json=transaction.dict()
            )

        if response.status_code != 200:
            logger.warning(f"AI returned {response.status_code}, using fallback")
            return rule_based_fallback(transaction)

        raw = response.json()

        try:
            validated = RiskAssessment(**raw)
            return validated.dict()
        except ValidationError as e:
            logger.error(f"AI validation failed: {e}")
            return rule_based_fallback(transaction)

    except httpx.TimeoutException:
        logger.warning("AI timeout, using fallback")
        return rule_based_fallback(transaction)
    except Exception as e:
        logger.error(f"AI error: {e}, using fallback")
        return rule_based_fallback(transaction)

@app.post("/analyze")
async def analyze(transaction: TransactionRequest):
    return await get_risk_assessment(transaction) from fastapi import FastAPI
from pydantic import BaseModel, Field, ValidationError
import httpx
import logging

app = FastAPI()
logger = logging.getLogger(__name__)

class TransactionRequest(BaseModel):
    amount: float
    location: str
    merchant: str
    user_history: list = []

class RiskAssessment(BaseModel):
    risk_score: float = Field(ge=0, le=1)
    is_fraud: bool
    reason: str = Field(max_length=500)

def rule_based_fallback(transaction: TransactionRequest):
    risk_score = 0.0
    if transaction.amount > 50000:
        risk_score += 0.5
    if transaction.location in ["California", "Dallas", "Columbus"]:
        risk_score += 0.3
    if transaction.user_history and transaction.merchant not in transaction.user_history:
        risk_score += 0.2

    risk_score = min(risk_score, 1.0)

    return {
        "risk_score": risk_score,
        "is_fraud": risk_score > 0.6,
        "reason": "Fallback: AI unavailable, using rule-based decision"
    }

async def get_risk_assessment(transaction: TransactionRequest):
    try:
        async with httpx.AsyncClient(timeout=5.0) as client:
            response = await client.post(
                "http://ai-service/predict",
                json=transaction.dict()
            )

        if response.status_code != 200:
            logger.warning(f"AI returned {response.status_code}, using fallback")
            return rule_based_fallback(transaction)

        raw = response.json()

        try:
            validated = RiskAssessment(**raw)
            return validated.dict()
        except ValidationError as e:
            logger.error(f"AI validation failed: {e}")
            return rule_based_fallback(transaction)

    except httpx.TimeoutException:
        logger.warning("AI timeout, using fallback")
        return rule_based_fallback(transaction)
    except Exception as e:
        logger.error(f"AI error: {e}, using fallback")
        return rule_based_fallback(transaction)

@app.post("/analyze")
async def analyze(transaction: TransactionRequest):
    return await get_risk_assessment(transaction) Important Checklist Important Checklist To summarize, before you deploy any system that has AI model, ask yourself the following critical questions: Did I validate every AI output?Does the system have a control layer for AI?Does it have a fallback?Can the system run if AI fails?Is the AI authority or advisory? Did I validate every AI output? Does the system have a control layer for AI? Does it have a fallback? Can the system run if AI fails? Is the AI authority or advisory? If you can’t answer all those questions clearly, then the system is not ready for production. If you can’t answer all those questions clearly, then the system is not ready for production Conclusion Conclusion Integrating AI with microservices introduces inconsistencies, makes debugging hard, and makes the system unreliable. By applying the right measures, you can contain these problems and design reliable microservices that work seamlessly with AI models. AI won’t crash your system, it will quietly break. Validate everything. Never let it decide alone. And always have a fallback. That’s how your system will survive.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

The is an opinion piece based on the author’s POV and does not necessarily reflect the views of HackerNoon.

Why Microservices Struggle With AI Systems

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Why Your Retry Logic Is Taking Down Your System (And How to Fix It)

The Noonification: Use This 7-Step McKinsey Framework to Solve Any Problem (1/10/2023)

The Noonification: A Taxonomy of Inclusiveness (1/11/2024)

The Noonification: What is the InfiniteNature-Zero AI Model? (11/19/2022)

10 Ways AI Has Changed Our Lives

100 Days of AI, Day 8: Experimenting With Microsoft's Semantic Kernel Using GPT-4

Why Your Retry Logic Is Taking Down Your System (And How to Fix It)

The Noonification: Use This 7-Step McKinsey Framework to Solve Any Problem (1/10/2023)

The Noonification: A Taxonomy of Inclusiveness (1/11/2024)

The Noonification: What is the InfiniteNature-Zero AI Model? (11/19/2022)

10 Ways AI Has Changed Our Lives

100 Days of AI, Day 8: Experimenting With Microsoft's Semantic Kernel Using GPT-4

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps