Search and Extract: Why This AI Pattern Matters, Tutorial, iyo Isticmaalka

Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Markaas, data waa mid ka mid ah dhismaha ah, ka mid ah kontekst, in la mid ah, ama mid ka mid ka mid ka mid ka mid ka mid ah dhismaha. Sida loo yaabaa, waxaan ka mid ahay Waa in la yaabaa, sida loo isticmaali karaa in ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah! Search iyo Extract Qalabka dhismaha iyo dhismaha iyo dhismaha iyo dhismaha iyo dhismaha Sida loo yaqaan Statista, 402.74 million terabytes of data waa la soo saarka dayday. Dhismaha waaweyn oo ka mid ah dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha iyo dhismaha. Sida loo yaabaa, waxaa laga yaabaa in ay ka mid ah wax soo saarka iyo wax soo saarka iyo wax soo saarka. Markaas, waxaa laga yaabaa in ay ku yaalaa in ay ku yaalaa in ay ku yaalaa in ay ku yaalaa in ay ku yaalaa in ay ku yaalaa in ay ku yaalaa in ay ku yaalaa ... Shirkadda waxaa ka mid ah wax soo saarka ganacsiga, automation, iyo call-driven AI. Haku Haku Sida loo yaabaa in ay ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah. Sidaas waxaa loo yaqaan "LLMs hallucinate, AI agents give convincing but incorrect answers, and data pipelines quietly fail in production". In mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah. Ma rabtaa wax soo saarka ah ee wax soo saarka ah ee wax soo saarka ah ee wax soo saarka ah ee wax soo saarka ah ee wax soo saarka ah ee wax soo saarka. Haku Qalabka Qalabka Sida loo yaabaa, waxaan ka mid ahay Data waxaa loo isticmaali karaa oo ka mid ah Waa in la soo xiriir, sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale. Kuuma Marka Sida loo yaqaan Qalabka ah. Sida loo isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa? Sida loo isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa in la isticmaali karaa? Sidaa waxa uu ku saabsan pattern waa mid ka mid ah! Search and Extract Search and Extract: Waayo waxaa laga yaabaa oo loo yaqaan 'Solution' “Shock and Extract” waa a process for retrieving up-to-date, verifiable, and contextual-rich information from the web . Waxaa laga yaabaa in dhismaha ah, laakiin sidoo kale waxaa laga yaabaa in dhismaha. Nadiifinta Standard Markaas ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah. Search: nidaamka soo saarka webka live on search engines sida Google si aad u aragto mid ka mid ah mid ka mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah. Tusaale: From the search results, the system identifies the most relevant URLs, visits those pages, and retrieves the most useful content or data from them. This may include facts, statistics, definitions, or structured information analyzed from the page. The data retrieved then becomes the grounded, trusted basis for the downstream steps in the pipeline. Shuruudaha waxaa laga yaabaa in ka mid ah shuruudaha ah ee shuruudaha iyo shuruudaha. Sidaas, waxaa loo isticmaali karaa mashiinka mashiinka ah, wax soo saarka wax soo saarka iyo wax soo saarka data. Qalabka RAG Qalabka RAG Qalabka dhismaha "Search and Extract" waa si ay u dhismaha dhismaha dhismaha modern sida loo isticmaali karaa: Fresh, sida wax soo saarka web ka mid ah wax soo saarka web ka mid ah wax soo saarka web ka mid ah wax soo saarka iyo wax soo saarka. Shuruudaha dhismaha iyo dhismaha dhismaha iyo dhismaha dhismaha dhismaha iyo dhismaha dhismaha dhismaha iyo dhismaha dhismaha dhismaha iyo dhismaha dhismaha dhismaha dhismaha iyo dhismaha dhismaha dhismaha dhismaha dhismaha iyo dhismaha dhismaha dhismaha dhismaha. Qalabka dhismaha iyo dhismaha iyo dhismaha dhismaha iyo dhismaha dhismaha iyo dhismaha dhismaha, dhismaha dhismaha iyo dhismaha dhismaha, dhismaha dhismaha iyo dhismaha dhismaha, dhismaha dhismaha dhismaha, dhismaha dhismaha dhismaha, dhismaha dhismaha dhismaha, dhismaha dhismaha dhismaha, dhismaha dhismaha dhismaha, dhismaha dhismaha dhismaha dhismaha, dhismaha dhismaha dhismaha dhismaha. Sida loo isticmaali karaa Search and Extract? Markaas oo dhan, "Search and Extract" waxaa laga yaqaan "Shock and Extract" waxaa laga yaqaan "Workflow": Inta badan waxaa ka mid ah dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha, dhismaha. Markaas ka mid ah wax soo saarka, macluumaad oo aad u qaadi karaa "Search" component, oo loo yaqaan 'SERP API solution' waxay ka soo saarka web automated ee macluumaadka search target. Ka dib markii aad qaadi kartaa wax soo saarka, sidoo kale aad ka mid ah loo yaabaa webka oo dhan ama sidoo kale ka soo saarka mid ka mid ah macluumaad ka mid ah AI / ML ama raaxaysta ah (tusaale, si ay u soo saarka webka ka mid ah macluumaadka, xafiisyada, ama mid ka mid ah). Waayo, waxaa loo yaabaa in ka mid ah wax soo saarka ah oo ka mid ah wax soo saarka ah oo ka mid ah wax soo saarka iyo wax soo saarka. URLs ka dib markii ay ku xiran "Extract" node, oo aad u isticmaali karaa web scraping adeeg ka mid ah sida aad u soo saarka wax soo saarka. this extracted data is fresh, contextual, and verifiable, as desired. Sidaas, waxaan ka soo xiriir in ay ku saabsan adeegga ah ee "Search" iyo "Extract" fasaxa ah, soo xiriir! Haku Search engines sida Google, Bing, DuckDuckGo, Baidu, iyo Yandex waxay ku qorrago SERPs (Search Engine Results Pages) iyo CAPTCHAs si ay u socdaan scraping automated. Sida loo isticmaali karaa "Search" waa loo isticmaali karaa adeegga API SERP. Dhismaha waxaa loo isticmaali karaa si ay u isticmaali karaa macluumaadka macluumaadka macluumaadka iyo macluumaadka anti-bot. Waxaa ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah Shirkadda Serp API ee Data Shirkadda Serp API ee Data Markaas, API-ka SERP ee Bright Data waxay ka caawineysaa in ka mid ah wax soo saarka ah oo ka mid ah wax soo saarka. Waxaa kale oo ka mid ah in ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid. Qalabka ️ Shuruudaha webka waxaa laga yaabaa in ay ka mid ah wax soo saarka webka ah, iyo sidoo kale ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ka mid ah. Marka aad u baahan tahay adeegga baabuurta ah oo loo yaabaa in ay u isticmaali karaa qiyaastii web-pages, oo ka mid ah in ay ku salaysan, oo ka mid ah in ay u baahan tahay scalability. In this regard, Bright Data waa mid ka mid ah adeegga best-web-scraping solvents on the market, thanks to tools such as: Web Unlocker API: API waa in ay u qiyaastii wax soo saarka webka ah, si loo isticmaali karaa blocks, CAPTCHAs, fingerprinting, iyo rotating proxy automatically. Browsing API: A browser cloud full-hosted compatible with Puppeteer, Selenium, Playwright, iyo mid ka mid ah macluumaadka macluumaadka, oo ka mid ah macluumaadka macluumaadka CAPTCHA ah iyo macluumaadka macluumaadka. Marka: Web Unlocker API Marka: Web Unlocker API Haku API Haku API Web Unlocker waa mid ka mid ah wax soo saarka content page, ama in raw HTML ama AI-ready Markdown (a Waayo, waxaa laga yaabaa in ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah. Shuruudaha ugu fiican ee data for LLM ingestion Shuruudaha ugu fiican ee data for LLM ingestion Sidaas, waxaa laga yaabaa in ka mid ah wax soo saarka "Extract" ee "Search and Extract" workflow. Sida loo isticmaali karaa Search and Extract Workflow in Python Sida loo helo "Search and Extract" workflow ee Python: Waayo, waxaa loo isticmaali karaa API SERP ee Bright Data si loo isticmaali karaa search web live. Waayo, sidoo kale waxaa laga yaabaa in la isticmaali karaa Web Unlocker API si ay u isticmaali karaa iyo loo isticmaali karaa qiyaasadda. Nala soo xiriir in la aasaasay! ↓ Qalabka Sida loo isticmaali karaa, waxaa loo isticmaali karaa in la isticmaali karaa in la isticmaali karaa. Python 3.9+ waxaa la aasaasay. A account Bright Data waxaa laga yaabaa in ka mid ah a SERP API zone iyo a Web Unlocker API zone, sida ay ka mid ah API key. If you’re unfamiliar with how to get started with Bright Data’s SERP API or Web Unlocker API products, sidoo kale ka mid ah wax soo saarka ah: Qalabka ah ee API Sida loo yaqaan Bright Data Unlocker API. Qalabka ah ee API Qalabka ah ee API Sida loo yaqaan Bright Data Unlocker API. Sida loo yaqaan Bright Data Unlocker API. Sida loo yaqaan 'SERP API Zone' waxaa loo isticmaali karaa Sida loo yaqaan Web Unlocker API : serp_api web_unlocker Sidaa, waxaa laga yaabaa in la dhismaha waxaa laga yaabaa in "Full JSON" mode ee "Data Format" aasaasay tababarka dhismaha: serp_api Sida loo yaabaa, API Bright Data SERP waxay ku yaalaa resultada SERP ah ee JSON, oo loo isticmaali karaa in ay u isticmaali karaa si ay u isticmaali karaa si ay u isticmaali karaa. Waayo, . Waxaa loo isticmaali karaa si ay u adeegsanayo qaab-dhismaha ka mid ah adeegga ugu horeysay! generate your Bright Data API key Nala soo xiriir Bright Data API Key Waayo, waxaan ku dhigi karaa in aad u baahan tahay in ay ku habboonay Python oo ku habboonay macluumaadka virtual. Step #1: Raadi karaa Access Data Services via API Sida loo isticmaali karaa in ay ka mid ah wax soo saarka, waxaa loo isticmaali karaa in ay ka mid ah wax soo saarka iyo wax soo saarka. pip install aiohttp Sidaas, loo yaqaan 'SERP API' iyo 'Web Unlocker API' waa mid ka mid ah: # Replace these with your Bright Data API token and zone names BRIGHT_DATA_API_TOKEN = " " SERP_ZONE = "serp_api" # Your SERP API zone WEB_UNLOCKER_ZONE = "web_unlocker" # Your Web Unlocker API zone HEADERS = { "Authorization": f"Bearer {BRIGHT_DATA_API_TOKEN}", # To authenticate the API requests to Bright Data "Content-Type": "application/json" } async def fetch_serp(session, query, location="us", language="en"): url = "https://api.brightdata.com/request" data = { "zone": SERP_ZONE, "url": f"https://www.google.com/search?q={urllib.parse.quote_plus(query)}&hl={language}&gl={location}", "format": "json" } async with session.post(url, json=data, headers=HEADERS) as resp: response_data = await resp.json() return json.loads(response_data["body"]) async def fetch_page(session, page_url, data_format="markdown"): url = "https://api.brightdata.com/request" data = { "zone": WEB_UNLOCKER_ZONE, "url": page_url, "format": "raw", "data_format": data_format } async with session.post(url, json=data, headers=HEADERS) as resp: return await resp.text() EE.UU Function waxay ku saabsan API-ka Bright Data SERP si ay u sameeyaan searches Google oo ka soo saarka resultada JSON SERP (si kastaba ha ahaado in aad account). fetch_serp() Sidaas, Shuruudaha waxaa laga yaabaa in ay ka mid ah shuruudaha ugu horeysay ee shuruudaha ugu horeysay ee shuruudaha ugu horeysay ee shuruudaha ugu horeysay ee shuruudaha ugu horeysay ee shuruudaha ugu horeysay ee shuruudaha ugu horeysay ee shuruudaha ugu horeysay ee shuruudaha ugu horeysay ee shuruudaha ugu horeysay ee shuruudaha ugu horeysay ee shuruudaha ugu horeysay ee shuruudaha ugu horeysay ee shuruudaha ugu horeysay ee shuruudaha ugu horeysay ee shuruudaha ugu horeysay ee shuruudaha. fetch_page() Ma rabtaa in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan yahay. : In wax soo saarka, si ay u soo saarka hardcoding API-ka Bright Data. Note Markaas oo ka mid ah wax soo saarka ah oo ka mid ah wax soo saarka iyo wax soo saarka iyo wax soo saarka. Haku: async import asyncio async def main(): # "Search and Extract" logic... # Run the async main function asyncio.run(main()) Step #2: Dhammaan wax soo saarka Extract Waxaad ku yaalaa in ay ku yaalaa MCP integrations oo aad u baahan yahay in ay u soo saarka in ay ka mid ah wax soo saarka oo ka mid ah oo ka mid ah oo ka mid ah oo ka mid ah oo ka mid ah oo ka mid ah oo ka mid ah oo ka mid ah. LLMs caadiga ah waxay ka caawin karaa in ay ka badan, sida MCP waa caadiga ah oo ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah. "new version of MCP" Waayo, waxaa loo isticmaali karaa in aad u isticmaali karaa Google SERP si aad u isticmaali karaa: # Topics to search input_topic = "new version of MCP" async with aiohttp.ClientSession() as session: # Search phase: Get SERP results serp_results = await fetch_serp(session, input_topic) Waayo, oo ka mid ah ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ah. Functiisa waxaa loo isticmaali karaa API SERP oo loo isticmaali karaa search query. If you need to handle multiple queries, you can call Qalabka competitive . fetch_serp() fetch_serp() asyncio.gather() : By default, API SERP waxay ku yaalaa 10 result per query, laakiin waxaa laga yaabaa in ay ku yaalaa in ka badan 100 result. Note Waxaa la heli karaa Google SERP si loo isticmaali karaa dictionary Python. Haku: fetch_serp() organic serp_ranking_results = serp_results.get("organic", []) Waayo, waxaa loo isticmaali karaa in aad u isticmaali karaa in aad u isticmaali karaa in aad u isticmaali karaa. Markaad ka mid ah cadaadis ka mid ah wax soo saarka oo ku yaalaa in aad u heli karaa by searching for "new version of MCP" on Google: Step #3: Filter URLs of Interest Waayo, ka mid ah wax soo saarka SERP oo loo yaqaan "SERP API call" waa in ay u baahan tahay in ka badan 3 ka badan oo ka mid ah "SERP API call". “Domain” (domain ah ee ku yaala MCP). Markaas, si kastaba ha ahaatee, si aad u aragti Google si aad u aragti. modelcontextprotocol.io Nala soo xiriirProtocol.io Filter ka mid ah: # Where to store the page URLs to scrape in the "Extract" phase pages_of_interest = [] # Iterate over results and filter for URLs containing "modelcontextprotocol.io" for result in serp_data_results: page_url = result.get("url") if "modelcontextprotocol.io" in page_url: pages_of_interest.append(page_url) # Stop once reaching 3 results if len(page_url) == 3: break Markaas ka mid ah wax soo saarka, waxaa laga yaabaa in ka mid ah wax soo saarka ah oo ka mid ah wax soo saarka iyo wax soo saarka. [ "http://blog.modelcontextprotocol.io/posts/2025-11-25-first-mcp-anniversary/", "https://modelcontextprotocol.io/development/roadmap", "https://modelcontextprotocol.io/specification/2025-11-25" ] Waayo, waxaa laga yaabaa in ay ka soo xiriir in la soo xiriir oo ka mid ah wax soo saarka ah. Step #4: Dhammaan wax soo saarka Haku function to call the Web Unlocker API and scrape the three selected pages in parallel: fetch_page() tasks = [fetch_page(session, url) for url in pages_of_interest] web_page_results = await asyncio.gather(*tasks) If you’re dealing with a dynamic site that requires user interaction, qaadi karaa data ka dib u isticmaalo technology automatization browser sida Playwright, integrated with the Browser API. Note Sida loo yaqaan 'Markdown' waxaa loo yaqaan 'Markdown' oo loo yaqaan 'Markdown' oo loo yaqaan 'Markdown' oo loo yaqaan 'Markdown' oo loo yaqaan 'Markdown' oo loo yaqaan 'Markdown' oo loo yaqaan 'Markdown' oo loo yaqaan 'Markdown' oo ku saabsan: markdown_pages = [page for page in web_page_results] Ma rabtaa in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan yahay. for markdown_page in markdown_pages: print(markdown_page[:300] + "...\n---\n") Sidaas, wax soo saarka waa: Waxay ku yaalaa version Markdown ee cadaadiga ah, oo ku yaalaa Web Unlocker API ayaa ku yaalaa wax soo saarka data web si ay u sameeyaan, oo ka mid ah macluumaadka anti-bot ama anti-scraping ah. Sida loo yaabaa, dhismaha, iyo wax soo saarka, iibsaday in la soo saarka badan, si ay u isticmaali karaa LLM si ay u isticmaali karaa, loo isticmaali karaa si ay u isticmaali karaa macluumaadka dheeraad ah, si ay u isticmaali karaa si ay u isticmaali karaa, ama si ay isticmaali karaa in ka badan. Step #5: Dhammaan codka Haddii aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan yahay. # pip install aiohttp import asyncio import aiohttp import urllib.parse import json # Replace these with your Bright Data API token and zone names BRIGHT_DATA_API_TOKEN = " " SERP_ZONE = "serp_api" # Your SERP API zone WEB_UNLOCKER_ZONE = "web_unlocker" # Your Web Unlocker API zone HEADERS = { "Authorization": f"Bearer {BRIGHT_DATA_API_TOKEN}", # To authenticate the API requests to Bright Data "Content-Type": "application/json" } async def fetch_serp(session, query, location="us", language="en"): url = "https://api.brightdata.com/request" data = { "zone": SERP_ZONE, "url": f"https://www.google.com/search?q={urllib.parse.quote_plus(query)}&hl={language}&gl={location}", "format": "json" } async with session.post(url, json=data, headers=HEADERS) as resp: response_data = await resp.json() return json.loads(response_data["body"]) async def fetch_page(session, page_url, data_format="markdown"): url = "https://api.brightdata.com/request" data = { "zone": WEB_UNLOCKER_ZONE, "url": page_url, "format": "raw", "data_format": data_format } async with session.post(url, json=data, headers=HEADERS) as resp: return await resp.text() async def main(): # Topics to search input_topic = "new version of MCP" async with aiohttp.ClientSession() as session: # Search phase: Get SERP results serp_results = await fetch_serp(session, input_topic) # Where to store the page URLs to scrape in the "Extract" phase pages_of_interest = [] # Iterate over results and filter for URLs containing "modelcontextprotocol.io" for result in serp_results.get("organic", []): page_url = result.get("link") if page_url and "modelcontextprotocol.io" in page_url: pages_of_interest.append(page_url) if len(pages_of_interest) == 3: break print("Pages to scrape:", pages_of_interest) # Extract phase: Scrape pages concurrently tasks = [fetch_page(session, url) for url in pages_of_interest] web_page_results = await asyncio.gather(*tasks) # Access the content from each page markdown_pages = [page for page in web_page_results] print("Scraped page contents:") for markdown_page in markdown_pages: print(markdown_page[:300] + "...\n---\n") # AI processing, RAG pipelines, or further workflows... # Run the async main function asyncio.run(main()) Markaad ka mid ah wax soo saarka API-ka Bright Data, sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale sidoo kale. Qalabka Marka aad u aragto in aad u aragto wax soo saarka iyo wax soo saarka iyo wax soo saarka iyo wax soo saarka iyo wax soo saarka iyo wax soo saarka. Sida loo isticmaali karaa data, waxaa loo isticmaali karaa pattern AI, waxa uu ku yaalaa in ay ku yaalaa mid ka mid ah si ay u qiyaastii wax soo saarka ah, wax soo saarka web. wax soo saarka ah oo ka mid ah macluumaadka search, laakiin Qalabka waxaa loo isticmaali karaa in ay ka mid ah wax soo saarka ah ee loo isticmaali karaa. Search iyo Extract Haku Qalabka ️ Sida loo yaabaa oo dhan this artikel, loo isticmaali karaa platform web scraping AI-ready sida Bright Data waa loo isticmaali karaa this workflow oo ka mid ah isticmaali karaa. Waayo, sidoo kale waxaa laga yaabaa in la soo saarka web data ah! Waayo, waxaa laga yaqaan 'Bright Data Free Trial' ah. Waayo, waxaa laga yaqaan 'Bright Data Free Trial' ah. Sida loo yaqaan HackerNoon Business Blogging. Sida loo yaqaan HackerNoon Business Blogging.