Can AI truly reason, or is it just a fancy digital parrot? Recent experiments with popular AI models like ChatGPT, LLaMa, Gemini, and Grok have revealed some concerning truths about their problem-solving abilities – and their unexpected fondness for dessert. Late Addition: ChatGPT-o1 was revealed during this process and you can skip to that section for the latest. The Birthday Puzzle Challenge I set out to replicate and expand on experiments conducted by the Bank of International Settlements and journalist Tim Harford. The test? The infamous "Cheryl's Birthday" logic puzzle and a crafty variation. "Cheryl's Birthday" is a logic problem where Bernard and Albert must deduce Cheryl's birthday from a set of clues. It tests deductive reasoning and information processing. Here's what I found: The Original Puzzle: Most AIs solved it with ease. (Except you, Gemini. What happened there?) Name-Swapped Version: Nearly all AIs stumbled when we renamed the actors and swapped months and numbers to random words. The Cake Conundrum Now, here's where it gets interesting (and a little concerning). The variation replaced Bernard with Edgar and May 19th with “brinks cake.” I added one tiny, irrelevant detail: "Edgar has a sweet tooth" The results? Suddenly, our AI friends developed a serious cake obsession: Reasonable(?) Carrot Cake ChatGPT-o1’s advanced methods are a breakthrough. Its chain of reasoning sees past the obfuscation far more than any competitor. The breakthrough still stumbles on its sweet tooth though. Interestingly, it can rule out “cake” but then picks “Carrot” because that was the sweetest remaining (and yet wrong) option: Why This Matters (A Lot) Reasoning vs. Regurgitating: These experiments cast doubt on whether AI is truly "reasoning" or just really good at pattern matching. Easy to Manipulate: A single, irrelevant sentence dramatically shifts AI responses. Imagine the implications for more complex queries! RAG and Sensitive Data: If AI struggles with simple logic puzzles, how can we trust it to parse through our confidential documents and extract meaningful insights? Manufacturing "Truth": Systems that generate multiple AI responses and aggregate them for increased accuracy could be easily swayed by carefully placed suggestions. The Cake is a Lie (Portal Reference Intended) This isn't just about birthday puzzles and dessert preferences. It's a wake-up call for any organization considering AI for critical decision-making processes. We need: More rigorous testing Greater transparency in AI reasoning processes Robust safeguards against manipulation Until then, approach AI-generated insights with a healthy dose of skepticism. AI's promise is tantalizing, but we can't let it eat our cake and have it, too. Can AI truly reason, or is it just a fancy digital parrot ? Recent experiments with popular AI models like ChatGPT, LLaMa, Gemini, and Grok have revealed some concerning truths about their problem-solving abilities – and their unexpected fondness for desser t. digital parrot digital parrot t. Late Addition: ChatGPT-o1 was revealed during this process and you can skip to that section for the latest. Late Addition: ChatGPT-o1 was revealed during this process and you can skip to that section for the latest. The Birthday Puzzle Challenge The Birthday Puzzle Challenge I set out to replicate and expand on experiments conducted by the Bank of International Settlements and journalist Tim Harford . The test? The infamous "Cheryl's Birthday" logic puzzle and a crafty variation. Bank of International Settlements Bank of International Settlements Tim Harford Tim Harford "Cheryl's Birthday" "Cheryl's Birthday" "Cheryl's Birthday" is a logic problem where Bernard and Albert must deduce Cheryl's birthday from a set of clues. It tests deductive reasoning and information processing. Here's what I found: The Original Puzzle: Most AIs solved it with ease. (Except you, Gemini. What happened there?) Name-Swapped Version: Nearly all AIs stumbled when we renamed the actors and swapped months and numbers to random words. The Original Puzzle: Most AIs solved it with ease. (Except you, Gemini. What happened there?) The Original Puzzle : Most AIs solved it with ease. (Except you, Gemini. What happened there?) The Original Puzzle Name-Swapped Version: Nearly all AIs stumbled when we renamed the actors and swapped months and numbers to random words. Name-Swapped Version : Nearly all AIs stumbled when we renamed the actors and swapped months and numbers to random words. Name-Swapped Version The Cake Conundrum The Cake Conundrum Now, here's where it gets interesting (and a little concerning). The variation replaced Bernard with Edgar and May 19th with “brinks cake.” I added one tiny, irrelevant detail: "Edgar has a sweet tooth" "Edgar has a sweet tooth" The results? Suddenly, our AI friends developed a serious cake obsession: Reasonable(?) Carrot Cake Reasonable(?) Carrot Cake ChatGPT-o1’s advanced methods are a breakthrough. Its chain of reasoning sees past the obfuscation far more than any competitor. The breakthrough still stumbles on its sweet tooth though. Interestingly, it can rule out “cake” but then picks “Carrot” because that was the sweetest remaining (and yet wrong) option: Why This Matters (A Lot) Why This Matters (A Lot) Reasoning vs. Regurgitating: These experiments cast doubt on whether AI is truly "reasoning" or just really good at pattern matching. Easy to Manipulate: A single, irrelevant sentence dramatically shifts AI responses. Imagine the implications for more complex queries! RAG and Sensitive Data: If AI struggles with simple logic puzzles, how can we trust it to parse through our confidential documents and extract meaningful insights? Manufacturing "Truth": Systems that generate multiple AI responses and aggregate them for increased accuracy could be easily swayed by carefully placed suggestions. Reasoning vs. Regurgitating: These experiments cast doubt on whether AI is truly "reasoning" or just really good at pattern matching. Reasoning vs. Regurgitating : These experiments cast doubt on whether AI is truly "reasoning" or just really good at pattern matching. Reasoning vs. Regurgitating Easy to Manipulate: A single, irrelevant sentence dramatically shifts AI responses. Imagine the implications for more complex queries! Easy to Manipulate : A single, irrelevant sentence dramatically shifts AI responses. Imagine the implications for more complex queries! Easy to Manipulate RAG and Sensitive Data: If AI struggles with simple logic puzzles, how can we trust it to parse through our confidential documents and extract meaningful insights? RAG and Sensitive Data : If AI struggles with simple logic puzzles, how can we trust it to parse through our confidential documents and extract meaningful insights? RAG and Sensitive Data Manufacturing "Truth": Systems that generate multiple AI responses and aggregate them for increased accuracy could be easily swayed by carefully placed suggestions. Manufacturing "Truth" : Systems that generate multiple AI responses and aggregate them for increased accuracy could be easily swayed by carefully placed suggestions. Manufacturing "Truth" The Cake is a Lie (Portal Reference Intended) This isn't just about birthday puzzles and dessert preferences. It's a wake-up call for any organization considering AI for critical decision-making processes. We need: More rigorous testing Greater transparency in AI reasoning processes Robust safeguards against manipulation More rigorous testing More rigorous testing Greater transparency in AI reasoning processes Greater transparency in AI reasoning processes Robust safeguards against manipulation Robust safeguards against manipulation Until then, approach AI-generated insights with a healthy dose of skepticism. AI's promise is tantalizing, but we can't let it eat our cake and have it, too.
