So this started because of a Slack argument. Someone on my team pasted a SQL query from ChatGPT straight into a pull request. No testing. No checking. And the query looked fine. Clean formatting, proper aliases, and even had a comment explaining the logic. looked One problem: it was returning wrong numbers. Off by about 15%. The join logic was subtly broken in a way that silently dropped rows instead of throwing an error. This happens at least once a week now. And every time, the same debate kicks off: "AI-generated SQL is fine, you just need to check it." Cool. But if you need to fully understand and verify every query anyway, what exactly is the AI saving you? I decided to stop arguing about it and actually test it. What I Did I took 10 SQL problems, real ones, the kind that show up in actual data engineering work, and ran each one through five LLMs. Same prompt. Same schema. Same expected answer. Then I graded every single response against the correct output. Not "does it look right." Does it return the right numbers? The models: The models GPT-5.2 Thinking
Claude Sonnet 4.6
Gemini 3.1 Pro
Llama 4 Maverick
Mistral Large GPT-5.2 Thinking Claude Sonnet 4.6 Gemini 3.1 Pro Llama 4 Maverick Mistral Large No fancy system prompts. No few-shot examples. No "act as a senior data engineer" preamble. I just asked the question the way a normal person would at 2 pm when they're stuck on a query. The schema: A boring e-commerce dataset. Orders, order items, customers, products. 500K orders, 1.2M line items, 80K customers. The kind of tables every data person has seen a thousand times. The schema: CREATE TABLE orders (
    order_id INT PRIMARY KEY,
    customer_id INT,
    order_date DATE,
    status VARCHAR(20),  -- 'completed', 'cancelled', 'refunded', 'pending'
    total_amount DECIMAL(10,2),
    discount_amount DECIMAL(10,2),
    region VARCHAR(50)
);

CREATE TABLE order_items (
    item_id INT PRIMARY KEY,
    order_id INT,
    product_id INT,
    quantity INT,
    unit_price DECIMAL(10,2),
    returned BOOLEAN
);

CREATE TABLE customers (
    customer_id INT PRIMARY KEY,
    signup_date DATE,
    plan_type VARCHAR(20),  -- 'free', 'basic', 'premium'
    country VARCHAR(50),
    is_active BOOLEAN
);

CREATE TABLE products (
    product_id INT PRIMARY KEY,
    category VARCHAR(50),
    subcategory VARCHAR(50),
    product_name VARCHAR(200),
    cost_price DECIMAL(10,2)
); CREATE TABLE orders (
    order_id INT PRIMARY KEY,
    customer_id INT,
    order_date DATE,
    status VARCHAR(20),  -- 'completed', 'cancelled', 'refunded', 'pending'
    total_amount DECIMAL(10,2),
    discount_amount DECIMAL(10,2),
    region VARCHAR(50)
);

CREATE TABLE order_items (
    item_id INT PRIMARY KEY,
    order_id INT,
    product_id INT,
    quantity INT,
    unit_price DECIMAL(10,2),
    returned BOOLEAN
);

CREATE TABLE customers (
    customer_id INT PRIMARY KEY,
    signup_date DATE,
    plan_type VARCHAR(20),  -- 'free', 'basic', 'premium'
    country VARCHAR(50),
    is_active BOOLEAN
);

CREATE TABLE products (
    product_id INT PRIMARY KEY,
    category VARCHAR(50),
    subcategory VARCHAR(50),
    product_name VARCHAR(200),
    cost_price DECIMAL(10,2)
); Every "correct" answer was written by me and verified against the actual data. If a model's query returned different numbers, it failed. Simple as that. The 10 Queries (and Where Things Got Ugly) Q1: Basic aggregation Revenue by region for completed orders in 2025. All five got it right. Honestly, if any model had whiffed on this, I would've just closed my laptop and gone outside. Moving on. Q2: Customer retention cohort For each monthly signup cohort, what % placed at least one order in each of the next 3 months? 2 out of 5. 2 out of 5. This is where I started taking notes. GPT-5.2 and Claude nailed it. Gemini's version ran fine, but used order_date instead of signup_date for the cohort definition. Completely different question. Llama had a syntax error in the generate_series call (come on). And Mistral? Mistral gave me a cumulative retention curve instead of a month-by-month retention. ran generate_series Here's what bugged me most about Mistral's answer: the output table looked totally reasonable. Nice clean percentages, declining over time as you'd expect. If you didn't already know the right answer, you'd ship it. That scared me a little. Q3: Running total with yearly reset Running sum of revenue per customer, resetting every January 1st. 4 out of 5. Window functions are well-represented in training data, I guess. Llama forgot to partition by year, so the total just... never reset. One of those bugs where you'd stare at the output for 10 minutes before you noticed. 4 out of 5. Q4: Gaps and islands Find consecutive days where daily revenue exceeded $50K. Return the start date, end date, and streak length. 1 out of 5. Yeah. 1 out of 5. This is the classic date - row_number() trick, and apparently only Claude knows it. GPT-5.2 tried a self-join approach that almost worked but miscounted streaks crossing month boundaries by one day. The other three produced queries that ran, returned data, and were wrong in three completely different ways. date - row_number() almost I'm starting to see a pattern here: if a problem requires knowing a specific algorithmic technique, most LLMs just... don't. They generate something that structurally resembles SQL for that type of problem, but it doesn't actually solve it. Q5: Sessionization Group orders into "sessions" where a 30-day gap between orders starts a new session. Return session numbers and order counts. 2 out of 5. GPT-5.2 and Claude got it. Gemini went the recursive CTE route, which was technically correct but timed out on 500K rows. So, no. Llama and Mistral both broke on customers with exactly one order. Edge cases, man. 2 out of 5. Q6: Percentage of total within groups Each product's revenue as a % of its category total, excluding returned items. 4 out of 5. Straightforward window function stuff. Llama applied the returned = false filter in the wrong place, after the percentage calc instead of before, so the denominator was inflated. This is exactly the kind of bug that makes it past code review because the query looks logically structured. 4 out of 5. returned = false Q7: Time-constrained funnel Of Q1 2025 signups, what % ordered within 7 days? 30 days? 90 days? 3 out of 5. GPT-5.2, Claude, and Gemini all got it (different approaches, same answer, love to see it). Llama forgot to find the first order date per customer, so repeat buyers threw off the funnel numbers. Mistral made the buckets exclusive instead of cumulative. These are the kind of mistakes a junior analyst would make. And honestly, I've made both of them at various points in my career. But that's the thing. I learned from making them. The model just... makes them again next time. 3 out of 5. first Q8: Year-over-year with NULL handling Monthly revenue 2024 vs 2025, side by side, with YoY growth %. Don't divide by zero. 3 out of 5. The NULL/zero trap got Gemini (used COALESCE(prior_year, 0) in the denominator, which gives you a meaningless growth rate) and Llama (jumbled the FULL OUTER JOIN in a way I honestly couldn't follow even after reading it three times). 3 out of 5. COALESCE(prior_year, 0) Q9: Recursive hierarchical query Build the full category path for each product using a self-referencing parent_category_id column. Like "Electronics > Computers > Laptops." 1 out of 5. 1 out of 5. Recursive CTEs are apparently where LLMs go to die. Only GPT-5.2 got it right. Claude walked the tree in the wrong direction and got reversed paths. The other three had various issues: wrong base cases, infinite loops, and missing anchor members. I was genuinely surprised by this one because recursive CTEs aren't that exotic. But I guess they're underrepresented enough in training data that the models can't reliably generate them. that Q10: The Friday afternoon special "Find customers who purchased in 3+ categories, spent over $1K total, but whose average order value is declining. Compare their last 3 orders to their first 3. Only include customers active for 6+ months." 0 out of 5. 0 out of 5. Zero. None. Not a single model got this fully right. And look, this is a hard query. Multiple joins, multiple conditions that interact with each other, edge cases around customers with fewer than 6 orders, potential date ties when determining "first 3" and "last 3." I get it. But this is also exactly the kind of question a stakeholder asks you on a Friday at 4 pm. This is the real world. And right now, no LLM I tested can handle it. exactly the kind of question a stakeholder asks you on a Friday at 4 pm. The most common failure was the "declining AOV" comparison. Models either compared the wrong orders (didn't handle date ties), computed the average wrong (averaged order totals instead of per-order item spend), or just silently ignored the 6-month activity requirement. The Scoreboard Model

Got it Right

Partial Credit

Wrong

Score



GPT-5.2 Thinking

7

2

1

70%



Claude Sonnet 4.6

7

2

1

70%



Gemini 3.1 Pro

5

2

3

50%



Llama 4 Maverick

4

2

4

40%



Mistral Large

4

2

4

40% Model

Got it Right

Partial Credit

Wrong

Score



GPT-5.2 Thinking

7

2

1

70%



Claude Sonnet 4.6

7

2

1

70%



Gemini 3.1 Pro

5

2

3

50%



Llama 4 Maverick

4

2

4

40%



Mistral Large

4

2

4

40% Model

Got it Right

Partial Credit

Wrong

Score Model Model Got it Right Got it Right Partial Credit Partial Credit Wrong Wrong Score Score GPT-5.2 Thinking

7

2

1

70% GPT-5.2 Thinking GPT-5.2 Thinking 7 7 2 2 1 1 70% 70% Claude Sonnet 4.6

7

2

1

70% Claude Sonnet 4.6 Claude Sonnet 4.6 7 7 2 2 1 1 70% 70% Gemini 3.1 Pro

5

2

3

50% Gemini 3.1 Pro Gemini 3.1 Pro 5 5 2 2 3 3 50% 50% Llama 4 Maverick

4

2

4

40% Llama 4 Maverick Llama 4 Maverick 4 4 2 2 4 4 40% 40% Mistral Large

4

2

4

40% Mistral Large Mistral Large 4 4 2 2 4 4 40% 40% "Partial credit" = the approach was right, but edge cases or subtle logic errors produced wrong numbers. GPT-5.2 and Claude are basically tied. Both are  strong on standard patterns, both are weak on the genuinely hard stuff. Gemini is a tier below. Solid fundamentals, but trips up more often on logic. Llama and Mistral are fine for simple queries, but I wouldn't trust them on anything with real business logic. The Thing That Actually Worries Me It's not that LLMs get SQL wrong sometimes. Everything gets stuff wrong sometimes. I get stuff wrong sometimes. What worries me is how they get it wrong. how When I write a bad query, it usually errors out. Syntax error. Type mismatch. Column not found. Something loud and obvious. When an LLM writes a bad query, it runs perfectly, returns a clean result set with nice column names and reasonable-looking numbers, and is just... wrong. Silently. Confidently. I keep calling this the "looks right" problem, and I think it's the most dangerous thing about using LLMs for data work specifically. In software engineering, bad code tends to crash. In data engineering, bad queries tend to return plausible-looking garbage that nobody questions until someone makes a bad business decision based on it. What I Actually Think Now Look, I'm not anti-LLM for SQL. I use them myself literally every day. They're great for scaffolding queries when I know the pattern but forgot the syntax. Generating boilerplate CREATE TABLE statements. Explaining someone else's ugly query back to me in plain English. First drafts that I then rewrite. But after running this experiment, a few things crystallized for me. Verify against known output, not against vibes. Don't just eyeball the result and go "yeah, that looks about right." Run it against data where you already know the correct answer. If you can't do that, you shouldn't be shipping it. Verify against known output, not against vibes. Complexity kills accuracy. For simple stuff (Q1, Q3, Q6), LLMs are reliable. For anything with multiple interacting conditions (Q4, Q9, Q10), treat the output like an untested pull request from someone who doesn't understand your data. Complexity kills accuracy. You need to know SQL more now, not less. I've heard the take that LLMs mean analysts don't need to learn SQL deeply anymore. After this? I think it's the opposite. The only way to catch a confident wrong answer is to actually understand the logic yourself. If you can't read the query and spot the bug, the AI isn't helping you. It's creating a liability. You need to know SQL more now, not less. more Methodology Identical prompt to every model, verbatim
Default chat settings, no API tuning
First response only, no retries, no "actually can you try again"
Correct answers written by me, verified against actual data with known output
Tested over 10 days in Feb 2026. Models change fast. This is a snapshot
Schema DDL and all 10 prompts in a GitHub gist [link] so you can reproduce it Identical prompt to every model, verbatim Default chat settings, no API tuning First response only, no retries, no "actually can you try again" Correct answers written by me, verified against actual data with known output Tested over 10 days in Feb 2026. Models change fast. This is a snapshot Schema DDL and all 10 prompts in a GitHub gist [link] so you can reproduce it If you test this on a model I missed, or run it again in 6 months when these models are smarter, I want to see your results. Seriously. Comments or Twitter don't matter. These models are getting better, fast. A year ago, the scores would've been way worse. But right now, in February 2026, the answer to "can I trust LLM-generated SQL?" is: depends on the query, and you'd better check either way.

This story contains new, firsthand information uncovered by the writer.

The code in this story is for educational purposes. The readers are solely responsible for whatever they build with it.

The writer is smart, but don't just like, take their word for it. #DoYourOwnResearch before making any investment decisions or decisions regarding your health or security. (Do not regard any of this content as professional investment advice, or health advice)

Arweave

I Asked 5 LLMs to Write the Same SQL Query. Here's How Wrong They Got It

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

5 Data Pipeline Anti-Patterns That Silently Wreck Your Stack (And How I Fixed Them)

The Noonification: Use This 7-Step McKinsey Framework to Solve Any Problem (1/10/2023)

The Noonification: A Taxonomy of Inclusiveness (1/11/2024)

The Noonification: What is the InfiniteNature-Zero AI Model? (11/19/2022)

10 Ways AI Has Changed Our Lives

100 Days of AI, Day 8: Experimenting With Microsoft's Semantic Kernel Using GPT-4

5 Data Pipeline Anti-Patterns That Silently Wreck Your Stack (And How I Fixed Them)

The Noonification: Use This 7-Step McKinsey Framework to Solve Any Problem (1/10/2023)

The Noonification: A Taxonomy of Inclusiveness (1/11/2024)

The Noonification: What is the InfiniteNature-Zero AI Model? (11/19/2022)

10 Ways AI Has Changed Our Lives

100 Days of AI, Day 8: Experimenting With Microsoft's Semantic Kernel Using GPT-4

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps