How Search Engines Actually Answer Your Questions

In 2025, typing “best way to cancel a flight on X airline” into a browser rarely gives you just ten blue links anymore. You get: “best way to cancel a flight on X airline” a one‑sentence summary, a step‑by‑step list, maybe even a “people also asked” carousel that weirdly reads your mind. a one‑sentence summary, a step‑by‑step list, maybe even a “people also asked” carousel that weirdly reads your mind. Under the hood, that’s not “just a better search algorithm.” It’s a stack of question–answering (QA) systems: some reason over structured knowledge graphs, some run deep neural networks over raw web pages, and many glue the two together. stack of question–answering (QA) systems This piece breaks down how that stack actually works, based on a production‑grade design similar to QQ Browser’s intelligent Q&A system. We’ll walk through: Where QA shows up in real products The two core paradigms: KBQA and DeepQA + MRC How a knowledge‑graph Q&A system is wired How search‑based DeepQA handles noisy web data How long‑answer tasks and opinions are modeled A practical blueprint if you’re building your own stack Where QA shows up in real products The two core paradigms: KBQA and DeepQA + MRC KBQA DeepQA + MRC How a knowledge‑graph Q&A system is wired How search‑based DeepQA handles noisy web data How long‑answer tasks and opinions are modeled A practical blueprint if you’re building your own stack Grab a ☕ — this is more systems‑design deep dive than shiny demo. 1. Where QA Actually Lives in Products From a user’s point of view, QA shows up in lots of different skins: Natural‑language search – “Why is my phone battery draining overnight?” instead of phone battery drain overnight fix. Smart snippets – a short answer paragraph pinned at the top of results, often with a source link. Virtual assistants – Siri/Google Assistant/Alexa responding to “What’s traffic like on the way to work?” or “Remind me in 30 minutes.” Knowledge panels – right‑hand “cards” summarizing a person, company, movie, or recipe. Domain search – internal tools for docs, legal case retrieval, medical guidelines, etc. Smart customer support – bots that answer 80% of “where is my order”‑style questions before a human sees the ticket. Ed‑tech Q&A – “explain this theorem”, “walk me through this derivation step by step.” Natural‑language search – “Why is my phone battery draining overnight?” instead of phone battery drain overnight fix. Natural‑language search phone battery drain overnight fix Smart snippets – a short answer paragraph pinned at the top of results, often with a source link. Smart snippets Virtual assistants – Siri/Google Assistant/Alexa responding to “What’s traffic like on the way to work?” or “Remind me in 30 minutes.” Virtual assistants Knowledge panels – right‑hand “cards” summarizing a person, company, movie, or recipe. Knowledge panels Domain search – internal tools for docs, legal case retrieval, medical guidelines, etc. Domain search Smart customer support – bots that answer 80% of “where is my order”‑style questions before a human sees the ticket. Smart customer support Ed‑tech Q&A – “explain this theorem”, “walk me through this derivation step by step.” Ed‑tech Q&A The core task is always the same: Take a natural‑language question → understand intent + constraints → use knowledge → return an answer (not just a list of URLs). Take a natural‑language question → understand intent + constraints → use knowledge → return an answer (not just a list of URLs). The differences are in what knowledge you rely on and how structured that knowledge is. That’s where the split between KBQA and DeepQA comes from. what knowledge how structured KBQA DeepQA 2. Two Brains in One Search Engine: KBQA vs DeepQA Most modern search Q&A systems run both of these in parallel: both 2.1 KBQA – Question Answering over Knowledge Graphs Think of KBQA as your in‑house database nerd. in‑house database nerd Data lives as triples: (head_entity, relation, tail_value) e.g. (Paris, capital_of, France) or (iPhone 15, release_date, 2023-09-22). The graph is curated, structured, and schema‑driven: entities, types, attributes, relations. A KBQA system: Parses the question into a logical form – which entities, which relations? Translates that into graph queries (triple lookups, path queries). Runs them on the knowledge graph (via indices or a graph DB). Post‑processes and verbalizes the result. Data lives as triples: (head_entity, relation, tail_value) e.g. (Paris, capital_of, France) or (iPhone 15, release_date, 2023-09-22). Data lives as triples: (head_entity, relation, tail_value) e.g. (Paris, capital_of, France) or (iPhone 15, release_date, 2023-09-22). triples (head_entity, relation, tail_value) (Paris, capital_of, France) (iPhone 15, release_date, 2023-09-22) The graph is curated, structured, and schema‑driven: entities, types, attributes, relations. The graph is curated, structured, and schema‑driven: entities, types, attributes, relations. curated, structured, and schema‑driven A KBQA system: Parses the question into a logical form – which entities, which relations? Translates that into graph queries (triple lookups, path queries). Runs them on the knowledge graph (via indices or a graph DB). Post‑processes and verbalizes the result. A KBQA system: Parses the question into a logical form – which entities, which relations? Translates that into graph queries (triple lookups, path queries). Runs them on the knowledge graph (via indices or a graph DB). Post‑processes and verbalizes the result. Parses the question into a logical form – which entities, which relations? Parses the question into a logical form – which entities, which relations? logical form Translates that into graph queries (triple lookups, path queries). Translates that into graph queries (triple lookups, path queries). graph queries Runs them on the knowledge graph (via indices or a graph DB). Runs them on the knowledge graph (via indices or a graph DB). Post‑processes and verbalizes the result. Post‑processes and verbalizes the result. It’s perfect for hard factual questions: hard factual questions “What is the half‑life of Iodine‑131?” “Who directed the movie Dune (2021)?” “How many employees does Company X have?” “What is the half‑life of Iodine‑131?” “Who directed the movie Dune (2021)?” Dune (2021) “How many employees does Company X have?” If the fact is in the graph and your semantic parser doesn’t mess up, it’s fast and precise. 2.2 DeepQA – Search + Machine Reading Comprehension DeepQA is the chaotic genius that thrives on unstructured data: chaotic genius It works over web pages, docs, PDFs, UGC, forums, and FAQs. Pipeline (in a very simplified view): Use a search index to retrieve top‑N passages/pages. Feed them (plus the question) into a Machine Reading Comprehension (MRC) model. The model either extracts a span (short answer) or generates a natural sentence/paragraph. Score and calibrate confidence, then ship the best answer to the user. It works over web pages, docs, PDFs, UGC, forums, and FAQs. It works over web pages, docs, PDFs, UGC, forums, and FAQs. web pages, docs, PDFs, UGC, forums, and FAQs Pipeline (in a very simplified view): Use a search index to retrieve top‑N passages/pages. Feed them (plus the question) into a Machine Reading Comprehension (MRC) model. The model either extracts a span (short answer) or generates a natural sentence/paragraph. Score and calibrate confidence, then ship the best answer to the user. Pipeline (in a very simplified view): Use a search index to retrieve top‑N passages/pages. Feed them (plus the question) into a Machine Reading Comprehension (MRC) model. The model either extracts a span (short answer) or generates a natural sentence/paragraph. Score and calibrate confidence, then ship the best answer to the user. Use a search index to retrieve top‑N passages/pages. Use a search index to retrieve top‑N passages/pages. retrieve Feed them (plus the question) into a Machine Reading Comprehension (MRC) model. Feed them (plus the question) into a Machine Reading Comprehension (MRC) model. Machine Reading Comprehension (MRC) The model either extracts a span (short answer) or generates a natural sentence/paragraph. The model either extracts a span (short answer) or generates a natural sentence/paragraph. extracts generates Score and calibrate confidence, then ship the best answer to the user. Score and calibrate confidence, then ship the best answer to the user. Historically, this looked like IBM Watson: dozens of hand‑engineered features and brittle pipelines. Modern systems are closer to DrQA → BERT‑style readers → generative FiD‑style models, with much of the manual feature engineering replaced by deep models. DrQA → BERT‑style readers → generative FiD‑style models DeepQA is what you rely on when: The answer is buried in prose (“Explain why my dog’s vaccination schedule changed this year”). The answer involves opinions, arguments, pros/cons (“Is intermittent fasting safe?”). The knowledge graph simply doesn’t encode the nuance you need. The answer is buried in prose (“Explain why my dog’s vaccination schedule changed this year”). buried in prose The answer involves opinions, arguments, pros/cons (“Is intermittent fasting safe?”). opinions, arguments, pros/cons The knowledge graph simply doesn’t encode the nuance you need. The magic in production is not choosing one or the other, but blending them. blending them 3. System‑Level Architecture: Offline vs Online Brain A typical search QA stack is split into offline and online components. offline online Offline: Building and Understanding Knowledge Crawling & ingestion of web pages, docs, UGC, PGC. Quality and authority analysis – demoting spam, SEO junk, low‑trust sources. FAQ / QA pair mining – extracting question–answer pairs from forums, help centers, etc. Knowledge graph construction – entity extraction, linking, relation extraction, ontology maintenance. Pre‑training MRC & generative models on logs, QA pairs, and task‑specific objectives. Crawling & ingestion of web pages, docs, UGC, PGC. Crawling & ingestion Quality and authority analysis – demoting spam, SEO junk, low‑trust sources. Quality and authority analysis FAQ / QA pair mining – extracting question–answer pairs from forums, help centers, etc. FAQ / QA pair mining Knowledge graph construction – entity extraction, linking, relation extraction, ontology maintenance. Knowledge graph construction Pre‑training MRC & generative models on logs, QA pairs, and task‑specific objectives. Pre‑training MRC & generative models This is where you burn GPU hours and run large batch jobs. Latency doesn’t matter; coverage and robustness do. Online: Answering in ~100ms When a query hits the system: Query understanding: classification (is this QA‑intent?), domain detection, entity detection. Multi‑channel retrieval: KG candidate entities/relations. Web passages for DeepQA. High‑quality QA pairs (FAQs/community answers). Per‑channel answering: KBQA query execution and reasoning. Short‑ or long‑answer MRC. Fusion & decision: Compare candidates: score by relevance, trust, freshness, and presentation quality. Decide: graph card? snippet? long answer? multiple options? Query understanding: classification (is this QA‑intent?), domain detection, entity detection. Query understanding Multi‑channel retrieval: KG candidate entities/relations. Web passages for DeepQA. High‑quality QA pairs (FAQs/community answers). Multi‑channel retrieval KG candidate entities/relations. Web passages for DeepQA. High‑quality QA pairs (FAQs/community answers). KG candidate entities/relations. Web passages for DeepQA. High‑quality QA pairs (FAQs/community answers). Per‑channel answering: KBQA query execution and reasoning. Short‑ or long‑answer MRC. Per‑channel answering KBQA query execution and reasoning. Short‑ or long‑answer MRC. KBQA query execution and reasoning. Short‑ or long‑answer MRC. Fusion & decision: Compare candidates: score by relevance, trust, freshness, and presentation quality. Decide: graph card? snippet? long answer? multiple options? Fusion & decision Compare candidates: score by relevance, trust, freshness, and presentation quality. Decide: graph card? snippet? long answer? multiple options? Compare candidates: score by relevance, trust, freshness, and presentation quality. Compare candidates: score by relevance, trust, freshness, and presentation quality. Decide: graph card? snippet? long answer? multiple options? Decide: graph card? snippet? long answer? multiple options? That fusion layer is effectively a meta‑ranker over answers, not just documents. meta‑ranker over answers 4. KBQA: How Knowledge‑Graph Q&A Actually Works Let’s zoom in on the structured side. 4.1 Data Update Pipelines Real‑world knowledge graphs are never static. Updates usually run in three modes: three modes Automatic updates Web crawlers, APIs, database feeds. Good for high‑volume, low‑risk attributes (e.g., stock prices, product availability). Semi‑automatic updates Models extract candidate facts, humans review/correct/approve. Used for sensitive or ambiguous facts (health, legal, financial). Manual curation Domain experts edit entities and relations by hand. Critical for niche domains (e.g., TCM herbs, specific legal regulations). Automatic updates Web crawlers, APIs, database feeds. Good for high‑volume, low‑risk attributes (e.g., stock prices, product availability). Automatic updates Web crawlers, APIs, database feeds. Good for high‑volume, low‑risk attributes (e.g., stock prices, product availability). Web crawlers, APIs, database feeds. Good for high‑volume, low‑risk attributes (e.g., stock prices, product availability). Semi‑automatic updates Models extract candidate facts, humans review/correct/approve. Used for sensitive or ambiguous facts (health, legal, financial). Semi‑automatic updates Models extract candidate facts, humans review/correct/approve. Used for sensitive or ambiguous facts (health, legal, financial). Models extract candidate facts, humans review/correct/approve. review/correct/approve Used for sensitive or ambiguous facts (health, legal, financial). Manual curation Domain experts edit entities and relations by hand. Critical for niche domains (e.g., TCM herbs, specific legal regulations). Manual curation Domain experts edit entities and relations by hand. Critical for niche domains (e.g., TCM herbs, specific legal regulations). Domain experts edit entities and relations by hand. Domain experts edit entities and relations by hand. Critical for niche domains (e.g., TCM herbs, specific legal regulations). Critical for niche domains (e.g., TCM herbs, specific legal regulations). A production KG typically combines all three. 4.2 Two Retrieval Styles: Triples vs Graph DB You’ll see two dominant patterns. Direct triple index Store triples in inverted indices keyed by entity, relation, sometimes value. Great for simple, local queries: single hop (“capital of X”) attribute lookup (“height of Mount Everest”). Store triples in inverted indices keyed by entity, relation, sometimes value. entity relation value Great for simple, local queries: single hop (“capital of X”) attribute lookup (“height of Mount Everest”). simple, local queries single hop (“capital of X”) attribute lookup (“height of Mount Everest”). single hop (“capital of X”) single hop (“capital of X”) attribute lookup (“height of Mount Everest”). attribute lookup (“height of Mount Everest”). Fast, cacheable, simple. Graph database Load the graph into a proper graph DB (Neo4j, JanusGraph, or in‑house). Query with something like Cypher / Gremlin / SPARQL‑ish languages. Needed for multi‑hop reasoning and graph analytics: “Which movies were directed by someone who also acted in them?” “Find companies within 2 hops of this investor via board memberships.” Load the graph into a proper graph DB (Neo4j, JanusGraph, or in‑house). graph DB Query with something like Cypher / Gremlin / SPARQL‑ish languages. Needed for multi‑hop reasoning and graph analytics: “Which movies were directed by someone who also acted in them?” “Find companies within 2 hops of this investor via board memberships.” multi‑hop reasoning “Which movies were directed by someone who also acted in them?” “Find companies within 2 hops of this investor via board memberships.” “Which movies were directed by someone who also acted in them?” “Find companies within 2 hops of this investor via board memberships.” The system often does a cheap triple lookup first, then escalates to deeper graph queries only when necessary. cheap triple lookup first 4.3 Semantic Parsing Pipeline Semantic parsing is the KBQA piece that feels most like compiler construction. The pipeline roughly looks like this: Domain classification Route “Write a seven‑character quatrain” to a Chinese poetry handler. Route “Who is the mayor of Paris?” to a single‑entity handler. Route “Which movies did Nolan direct after 2010?” to a multi‑entity/constraint handler. Syntactic/dependency parsing Build a parse tree to figure out subjects, predicates, objects, modifiers, and constraints. Logical form construction Convert into something like a lambda‑calculus / SQL / SPARQL‑like intermediate form. E.g. Q: Which cities in Germany have population > 1 million? → Entity type: City → Filter: located_in == Germany AND population > 1_000_000 Graph querying & composition Execute logical form against the graph. Recursively stitch partial results (multi‑step joins). Rank, dedupe, and verbalize. Domain classification Route “Write a seven‑character quatrain” to a Chinese poetry handler. Route “Who is the mayor of Paris?” to a single‑entity handler. Route “Which movies did Nolan direct after 2010?” to a multi‑entity/constraint handler. Domain classification Route “Write a seven‑character quatrain” to a Chinese poetry handler. Route “Who is the mayor of Paris?” to a single‑entity handler. Route “Which movies did Nolan direct after 2010?” to a multi‑entity/constraint handler. Route “Write a seven‑character quatrain” to a Chinese poetry handler. Chinese poetry Route “Who is the mayor of Paris?” to a single‑entity handler. single‑entity Route “Which movies did Nolan direct after 2010?” to a multi‑entity/constraint handler. multi‑entity/constraint Syntactic/dependency parsing Build a parse tree to figure out subjects, predicates, objects, modifiers, and constraints. Syntactic/dependency parsing Build a parse tree to figure out subjects, predicates, objects, modifiers, and constraints. Build a parse tree to figure out subjects, predicates, objects, modifiers, and constraints. Logical form construction Convert into something like a lambda‑calculus / SQL / SPARQL‑like intermediate form. E.g. Q: Which cities in Germany have population > 1 million? → Entity type: City → Filter: located_in == Germany AND population > 1_000_000 Logical form construction Convert into something like a lambda‑calculus / SQL / SPARQL‑like intermediate form. E.g. Q: Which cities in Germany have population > 1 million? → Entity type: City → Filter: located_in == Germany AND population > 1_000_000 Convert into something like a lambda‑calculus / SQL / SPARQL‑like intermediate form. Convert into something like a lambda‑calculus / SQL / SPARQL‑like intermediate form. E.g. Q: Which cities in Germany have population > 1 million? → Entity type: City → Filter: located_in == Germany AND population > 1_000_000 E.g. Q: Which cities in Germany have population > 1 million? → Entity type: City → Filter: located_in == Germany AND population > 1_000_000 Q: Which cities in Germany have population > 1 million? → Entity type: City → Filter: located_in == Germany AND population > 1_000_000 Graph querying & composition Execute logical form against the graph. Recursively stitch partial results (multi‑step joins). Rank, dedupe, and verbalize. Graph querying & composition Execute logical form against the graph. Recursively stitch partial results (multi‑step joins). Rank, dedupe, and verbalize. Execute logical form against the graph. Execute logical form against the graph. Recursively stitch partial results (multi‑step joins). Recursively stitch partial results (multi‑step joins). Rank, dedupe, and verbalize. Rank, dedupe, and verbalize. This rule‑heavy approach has a huge upside: when it applies, it’s insanely accurate and interpretable. The downside is obvious: writing and maintaining rules for messy real‑world language is painful. when it applies, it’s insanely accurate and interpretable. 4.4 Neural KBQA: Deep Learning in the Loop Modern systems don’t rely only on hand‑crafted semantic rules. They add deep models to: deep models Detect entities even with typos/aliases/code‑mixed text. Map natural‑language relation phrases (“who founded”, “created by”, “designed”) to schema relations. Score candidate logical forms or graph paths by semantic similarity instead of exact string match. Detect entities even with typos/aliases/code‑mixed text. Detect entities even with typos/aliases/code‑mixed text. Map natural‑language relation phrases (“who founded”, “created by”, “designed”) to schema relations. Map natural‑language relation phrases (“who founded”, “created by”, “designed”) to schema relations. Score candidate logical forms or graph paths by semantic similarity instead of exact string match. Score candidate logical forms or graph paths by semantic similarity instead of exact string match. semantic similarity The result is a hybrid: deterministic logical execution + neural models for fuzzier pattern matching. 5. DeepQA: Search + Machine Reading Comprehension in the Wild On the unstructured side, things get noisy fast. 5.1 From IBM Watson to DrQA and Beyond Early DeepQA stacks (hello, Watson) had: separate modules for question analysis, passage retrieval, candidate generation, feature extraction, scoring, reranking… tons of feature engineering and fragile dependencies. separate modules for question analysis, passage retrieval, candidate generation, feature extraction, scoring, reranking… separate modules for question analysis, passage retrieval, candidate generation, feature extraction, scoring, reranking… tons of feature engineering and fragile dependencies. tons of feature engineering and fragile dependencies. The modern “open‑domain QA over the web” recipe is leaner: Use a search index to fetch top‑N passages. Encode question + passage with a deep model (BERT‑like or better). Predict answer spans or generate text (MRC). Aggregate over documents. Use a search index to fetch top‑N passages. Use a search index to fetch top‑N passages. Encode question + passage with a deep model (BERT‑like or better). Encode question + passage with a deep model (BERT‑like or better). Predict answer spans or generate text (MRC). Predict answer spans or generate text (MRC). Aggregate over documents. Aggregate over documents. DrQA was a landmark design: retriever + reader, trained on datasets like SQuAD. That template still underlies many production stacks today. retriever + reader 5.2 Short‑Answer MRC: Extractive Readers Short‑answer MRC means: Given a question + multiple documents, extract a single contiguous span that answers the question, and provide the supporting context. Given a question + multiple documents, extract a single contiguous span that answers the question, and provide the supporting context. single contiguous span Think “What is the capital of France?” or “How many bits are in an IPv4 address?” A typical architecture: Encode each of the top‑N passages plus the question. For each passage, predict: Is there an answer here? (answerability) Start/end token positions for the span. Then pick the best span across documents (and maybe show top‑k). Encode each of the top‑N passages plus the question. Encode each of the top‑N passages plus the question. For each passage, predict: Is there an answer here? (answerability) Start/end token positions for the span. For each passage, predict: Is there an answer here? (answerability) Start/end token positions for the span. Is there an answer here? (answerability) Is there an answer here? Start/end token positions for the span. Start/end token positions Then pick the best span across documents (and maybe show top‑k). Then pick the best span across documents (and maybe show top‑k). Challenge 1: Noisy search results Top‑N search hits include: irrelevant content, duplicate or conflicting answers, clickbait. irrelevant content, duplicate or conflicting answers, clickbait. A clean trick is joint training of: joint training an answer existence classifier per passage, and a span extraction head, an answer existence classifier per passage, and answer existence a span extraction head, span extraction So, the model learns to say “there is no answer here” and suppresses bad passages rather than being forced to hallucinate a span from every document. Multi‑document interaction layers then allow the model to compare evidence across pages, rather than treating each in isolation. compare evidence across pages Challenge 2: Commonsense‑dumb spans Purely neural extractors sometimes output “valid text that’s obviously wrong”: boundary off by a few tokens, wrong type (“yes” instead of a date), or nonsense phrases. boundary off by a few tokens, wrong type (“yes” instead of a date), or nonsense phrases. A proven fix is to inject external knowledge: inject external knowledge Mark entities in passages using entity linking to Wikipedia/KG. Give the model special embeddings for “this looks like a date/person/location/numeric measure”. During training, nudge the model to pay extra attention to spans with the correct type. Mark entities in passages using entity linking to Wikipedia/KG. entity linking Give the model special embeddings for “this looks like a date/person/location/numeric measure”. During training, nudge the model to pay extra attention to spans with the correct type. This improves both precision and “commonsense sanity.” Challenge 3: Robustness & R‑Drop Dropout is great for regularization, terrible for consistent outputs: tiny changes can flip the predicted span. One neat trick from production stacks: R‑Drop. R‑Drop Apply dropout twice to the same input through the model. Force the two predicted distributions to be similar via symmetric KL‑divergence. Add that term as a regularizer during training. Apply dropout twice to the same input through the model. Apply dropout twice to the same input through the model. twice Force the two predicted distributions to be similar via symmetric KL‑divergence. Force the two predicted distributions to be similar via symmetric KL‑divergence. symmetric KL‑divergence Add that term as a regularizer during training. Add that term as a regularizer during training. This pushes the model toward stable predictions under stochastic noise, which is crucial when users reload the same query and expect the same answer. Combined with data augmentation on semantically equivalent queries (different phrasings pointing to the same passage), this significantly boosts robustness. stable predictions data augmentation Challenge 4: Answer normalization & multi‑span answers Reality is messier than SQuAD: Different docs phrase the same fact differently: “3–5 years”, “three to five years”, “around five years depending on…”. Different docs phrase the same fact differently: “3–5 years”, “three to five years”, “around five years depending on…”. Different docs phrase the same fact differently: “3–5 years”, “three to five years”, “around five years depending on…”. Extractive models struggle with this. A common upgrade is to move to generative readers, e.g., Fusion‑in‑Decoder (FiD): generative readers Encode each retrieved document separately. Concatenate encodings into the decoder, which generates a normalized answer (“3–5 years” or “Xi Shi and Wang Zhaojun”). Optionally highlight supporting evidence. Encode each retrieved document separately. Encode each retrieved document separately. Concatenate encodings into the decoder, which generates a normalized answer (“3–5 years” or “Xi Shi and Wang Zhaojun”). Concatenate encodings into the decoder, which generates a normalized answer (“3–5 years” or “Xi Shi and Wang Zhaojun”). generates a normalized answer Optionally highlight supporting evidence. Optionally highlight supporting evidence. Two extra details from real systems: Use click logs to synthesize massive weak‑supervision data (query → clicked doc → pseudo answers). Train a dedicated confidence model on top of generated answers, because raw language‑model scores (perplexity) are terrible as calibrated confidence. Use click logs to synthesize massive weak‑supervision data (query → clicked doc → pseudo answers). click logs Train a dedicated confidence model on top of generated answers, because raw language‑model scores (perplexity) are terrible as calibrated confidence. confidence model 5.3 Long‑Answer MRC: Summaries, Not Just Spans Short answers are great, until the question is: “How does R‑Drop actually improve model robustness?” “Compare KBQA and DeepQA in terms of scalability and coverage.” “How does R‑Drop actually improve model robustness?” “Compare KBQA and DeepQA in terms of scalability and coverage.” You don’t want “Because it reduces KL‑divergence.” You want a paragraph‑level explanation. a paragraph‑level explanation So long‑answer MRC is defined as: Given question + docs, select or generate one or more longer passages that collectively answer the question, including necessary background. Given question + docs, select or generate one or more longer passages that collectively answer the question, including necessary background. one or more longer passages Two flavors show up in practice. 5.3.1 Compositional (Extractive) Long Answers Here, the system: Splits a document into sentences/segments. Uses a BERT‑like model to score each segment as “part of the answer” or not. Picks a set of segments to form a composite summary. Splits a document into sentences/segments. Uses a BERT‑like model to score each segment as “part of the answer” or not. Picks a set of segments to form a composite summary. composite summary Two clever tricks: HTML‑aware inputs Certain tags ( , , , etc.) correlate with important content. Encode those as special tokens in the input sequence so the model can exploit page structure. Structure‑aware pretraining Task 1: Question Selection (QS) – randomly replace the question with an irrelevant one and predict if it’s coherent. Task 2: Node Selection (NS) – randomly drop/shuffle sentences or structural tokens and train the model to detect that. Both push the model to understand long‑range document structure rather than just local token patterns. HTML‑aware inputs Certain tags ( , , , etc.) correlate with important content. Encode those as special tokens in the input sequence so the model can exploit page structure. HTML‑aware inputs Certain tags ( , , , etc.) correlate with important content. Encode those as special tokens in the input sequence so the model can exploit page structure. Certain tags ( , , , etc.) correlate with important content. Encode those as special tokens in the input sequence so the model can exploit page structure. Structure‑aware pretraining Task 1: Question Selection (QS) – randomly replace the question with an irrelevant one and predict if it’s coherent. Task 2: Node Selection (NS) – randomly drop/shuffle sentences or structural tokens and train the model to detect that. Both push the model to understand long‑range document structure rather than just local token patterns. Structure‑aware pretraining Task 1: Question Selection (QS) – randomly replace the question with an irrelevant one and predict if it’s coherent. Task 2: Node Selection (NS) – randomly drop/shuffle sentences or structural tokens and train the model to detect that. Both push the model to understand long‑range document structure rather than just local token patterns. Task 1: Question Selection (QS) – randomly replace the question with an irrelevant one and predict if it’s coherent. Task 1: Question Selection (QS) – randomly replace the question with an irrelevant one and predict if it’s coherent. Question Selection (QS) Task 2: Node Selection (NS) – randomly drop/shuffle sentences or structural tokens and train the model to detect that. Task 2: Node Selection (NS) – randomly drop/shuffle sentences or structural tokens and train the model to detect that. Node Selection (NS) Both push the model to understand long‑range document structure rather than just local token patterns. Both push the model to understand long‑range document structure rather than just local token patterns. understand long‑range document structure This delivers “best of both worlds”: extractive (so you can highlight exact sources) but capable of stitching together multiple non‑contiguous bits. 5.3.2 Opinion & Judgement QA: Answer + Rationale Sometimes the user asks a judgment question: judgment question “Is it okay to keep a rabbit in a cage?” “Should I disable automatic updates on my phone?” “Is it okay to keep a rabbit in a cage?” “Should I disable automatic updates on my phone?” A pure span extractor can’t safely output just “yes” or “no” from arbitrary web text. Instead, some production systems do: Evidence extraction (long answer): Same as compositional QA: select sentences that collectively respond to the question. Stance/classification (short answer): Feed question + title + top evidence sentence into a classifier. Predict label: support / oppose / mixed / irrelevant or yes / no / depends. Evidence extraction (long answer): Same as compositional QA: select sentences that collectively respond to the question. Evidence extraction Same as compositional QA: select sentences that collectively respond to the question. Same as compositional QA: select sentences that collectively respond to the question. Stance/classification (short answer): Feed question + title + top evidence sentence into a classifier. Predict label: support / oppose / mixed / irrelevant or yes / no / depends. Stance/classification Feed question + title + top evidence sentence into a classifier. Predict label: support / oppose / mixed / irrelevant or yes / no / depends. Feed question + title + top evidence sentence into a classifier. Feed question + title + top evidence sentence into a classifier. question + title + top evidence sentence Predict label: support / oppose / mixed / irrelevant or yes / no / depends. Predict label: support / oppose / mixed / irrelevant or yes / no / depends. support / oppose / mixed / irrelevant yes / no / depends The final UX: A concise verdict (“In most cases, keeping a rabbit in a small cage is not recommended.”). Plus, the evidence passages users can read and judge for themselves. A concise verdict (“In most cases, keeping a rabbit in a small cage is not recommended.”). not Plus, the evidence passages users can read and judge for themselves. Plus, That “show your work” property is crucial when answers may influence health, safety, or money. 6. A Minimal QA Stack in Code (Toy Example) To make this less abstract, here’s a deliberately simplified Python‑style sketch of a search + MRC pipeline. This is not production‑ready, but it shows how the pieces line up: not from typing import List from my_search_engine import search_passages # your BM25 / dense retriever from my_models import ShortAnswerReader, LongAnswerReader, KgClient short_reader = ShortAnswerReader.load("short-answer-mrc") long_reader = LongAnswerReader.load("long-answer-mrc") kg = KgClient("bolt://kg-server:7687") def answer_question(query: str) -> dict: # 1. Try KBQA first for clean factoid questions kg_candidates = kg.query(query) # internally uses semantic parsing + graph queries if kg_candidates and kg_candidates[0].confidence > 0.8: return { "channel": "kbqa", "short_answer": kg_candidates[0].text, "evidence": kg_candidates[0].path, } # 2. Fallback to DeepQA over the web index passages = search_passages(query, top_k=12) # 3. Short answer try short = short_reader.predict(query=query, passages=passages) if short.confidence > 0.75 and len(short.text) < 64: return { "channel": "deepqa_short", "short_answer": short.text, "evidence": short.supporting_passages, } # 4. Otherwise go for a long, explanatory answer long = long_reader.predict(query=query, passages=passages) return { "channel": "deepqa_long", "short_answer": long.summary[:120] + "...", "long_answer": long.summary, "evidence": long.selected_passages, } from typing import List from my_search_engine import search_passages # your BM25 / dense retriever from my_models import ShortAnswerReader, LongAnswerReader, KgClient short_reader = ShortAnswerReader.load("short-answer-mrc") long_reader = LongAnswerReader.load("long-answer-mrc") kg = KgClient("bolt://kg-server:7687") def answer_question(query: str) -> dict: # 1. Try KBQA first for clean factoid questions kg_candidates = kg.query(query) # internally uses semantic parsing + graph queries if kg_candidates and kg_candidates[0].confidence > 0.8: return { "channel": "kbqa", "short_answer": kg_candidates[0].text, "evidence": kg_candidates[0].path, } # 2. Fallback to DeepQA over the web index passages = search_passages(query, top_k=12) # 3. Short answer try short = short_reader.predict(query=query, passages=passages) if short.confidence > 0.75 and len(short.text) < 64: return { "channel": "deepqa_short", "short_answer": short.text, "evidence": short.supporting_passages, } # 4. Otherwise go for a long, explanatory answer long = long_reader.predict(query=query, passages=passages) return { "channel": "deepqa_long", "short_answer": long.summary[:120] + "...", "long_answer": long.summary, "evidence": long.selected_passages, } Real systems add dozens of extra components (logging, safety filters, multilingual handling, feedback loops), but the control flow is surprisingly similar. 7. Design Notes If You’re Building This for Real If you’re designing a search QA system in 2025+, a few pragmatic lessons from production stacks are worth keeping in mind: Invest in offline data quality first. A mediocre model plus clean data beats a fancy model on garbage. Treat QA as multi‑channel from day one. Don’t hard‑wire yourself into “only KG” or “only MRC.” Assume you’ll need both. Calibrate confidence explicitly. Don’t trust raw model logits or LM perplexity. Train separate calibration/rejection heads. Log everything and mine it. Query logs, click logs, and dissatisfaction signals (“people also ask”, reformulations) are your best supervision source. Plan for long answers and opinions. Short answers are the demo; long, nuanced replies are the reality in most domains. Expose evidence in the UI. Let users see why you answered something, especially in health, finance, and legal searches. Keep an eye on LLMs, but don’t throw away retrieval. LLMs with RAG are amazing, but in many settings, you still want: KG for hard constraints, business rules, and compliance. MRC and logs to ground generative answers in actual content. Invest in offline data quality first. A mediocre model plus clean data beats a fancy model on garbage. Invest in offline data quality first. Treat QA as multi‑channel from day one. Don’t hard‑wire yourself into “only KG” or “only MRC.” Assume you’ll need both. Treat QA as multi‑channel from day one. Calibrate confidence explicitly. Don’t trust raw model logits or LM perplexity. Train separate calibration/rejection heads. Calibrate confidence explicitly. Log everything and mine it. Query logs, click logs, and dissatisfaction signals (“people also ask”, reformulations) are your best supervision source. Log everything and mine it. Plan for long answers and opinions. Short answers are the demo; long, nuanced replies are the reality in most domains. Plan for long answers and opinions. Expose evidence in the UI. Let users see why you answered something, especially in health, finance, and legal searches. Expose evidence in the UI. why Keep an eye on LLMs, but don’t throw away retrieval. LLMs with RAG are amazing, but in many settings, you still want: KG for hard constraints, business rules, and compliance. MRC and logs to ground generative answers in actual content. Keep an eye on LLMs, but don’t throw away retrieval. KG for hard constraints, business rules, and compliance. MRC and logs to ground generative answers in actual content. KG for hard constraints, business rules, and compliance. MRC and logs to ground generative answers in actual content. 8. Closing Thoughts Modern search Q&A is what happens when we stop treating “search results” as the product and start treating the answer as the product. the answer Knowledge graphs give us crisp, structured facts and graph‑level reasoning. DeepQA + MRC gives us coverage and nuance over the messy, ever‑changing web. The interesting engineering work is in the seams: retrieval, ranking, fusion, robustness, and UX. crisp, structured facts coverage and nuance If you’re building anything that looks like a smart search box, virtual assistant, or domain Q&A tool, understanding these building blocks is the difference between “looks impressive in a demo” and “actually survives in production.” And the next time your browser nails a weirdly specific question in one line, you’ll know there’s a whole KBQA + DeepQA orchestra playing behind that tiny answer box.