Yaqhip sitios web ukanakax Selenium, Puppeteer ukat ukham uñtasitanak apnaqañakiw chiqak raspachañatakix utji, yaqha sitios web ukanakax nayrar sartañ seguridad ukarjam phuqhapxi, CAPTCHAs ukat IP jark’awinakas ch’amakïspawa. Uka jan walt’awinak atipjañataki ukhamarak 99% sitios web ukar inakiw Raspador apnaqañamp raspachañax wakisi, ukax aka qillqatanx lurasiniwa, ukatx mä... código ukanx ukax yanapt’aniw uka seguridad ukar jan yäqañataki. proxy ukax mä herramienta ukhamawa Ukampirus, yatiyawinak apthapiñax mä thakhikiwa; kuntix uka datos ukamp lurapkta ukax pachpakiwa, jan ukax juk’ampi, wali wakiskiriwa. Jila partejja, ukatakejj walja yatiyäwinak amparampiw wal chʼamachasiñasa. Ukampis ¿uka lurañ automáticamente lurasma ukhajja, ¿kamachasmasa? Mä modelo de lengua (LLM) ukar aprovechasax janiw datos ukanak apthapiñakikiti jan ukasti jiskt’asiraksnawa, ukhamat aski amuyunaka apsuñataki —tiempo ukat ch’ama qhispiyañataki. Aka guia ukanx kunjams web scraping ukax AI ukampiw mayacht’asispa uk yatxatapxäta, mä ch’aman herramienta lurañataki, ukax datos ukanak apthapiñatakiwa ukat escala ukan inakiw uñakipañataki. ¡Uñtʼayasiñäni! Nayraqata wakisirinaka Janïr qalltkasaxa, aka tuqinakatwa yatxatañama: Python yatiñanakax básicos, kunatix aka proyectox Python código qillqañamp amuyañampiw sarantaski. Python (3.7 jan ukax qhipa) ukax sistema ukan utt’ayatawa. Uka tuqitwa apkatasma . python.org ukan uñt’ayata Instalación ukat Setup ukanak luraña Aka yatichawimpi sarantañatakixa, aka lurawi phuqhañawa: Aka lurawinakampi phuqhañamawa pachamamaru utt’ayañataki ukhamaraki wakicht’asiñamawa AI-ampi ch’amanchata raspador lurañataki. 1. Mä Virtual Pachamama luraña Nayraqatxa, mä virtual ukan utt’ayañamawa, proyecto ukan dependencias ukanakap apnaqañataki. Ukhamatwa taqi wakiskir paquetes ukanakatakix mä sapa chiqawj utjañapa. Mä machaq proyecto directorio luraña: Terminal ukar jist’arañamawa (jan ukax Command Prompt/PowerShell Windows ukan) ukat machaq directorio proyecto ukar lurañamawa: mkdir ai-website-scraper cd ai-website-scraper Uka virtual ukan pacha luraña: Aka kamachixa virtual pacha lurañatakixa phuqhañawa: Windows ukanx: python -m venv venv macOS/Linux ukanx: python3 -m venv venv Ukax mä carpeta ukaw lurasi, ukax virtual ukan pachaparuw imatäni. venv 2. Pachamaman Virtual ukar ch’amanchaña Uka taypin irnaqañ qalltañatakix virtual medio ambiente ukar ch’amanchañamawa: Windows ukanx: .\venv\Scripts\activate macOS/Linux ukanx: source venv/bin/activate Terminal ukan jiskt’awipax uñacht’ayañatakiw mayjt’ayasini ( ), jichhax virtual ukan manqhankatamat chiqanchañataki. venv 3. Wakisir Dependencias ukanaka utt’ayaña Jichhaxa, proyectoman munaski uka bibliotecas ukanakaw utji. Mä qillqat proyecto directorio ukan luraña ukat aka dependencias ukanakamp yapxataña: requirements.txt streamlit selenium Beautifulsoup4 langchain langchain-ollama lxml html5lib Aka paquetes ukax wali wakiskiriwa raspado, datos proceso ukat UI lurañataki: : Akax mä interfaz de usuario interactivo lurañatakiw apnaqasi. streamlit : Sitio web ukan contenido ukar raspañataki. Selenium : HTML ukar uñakipt’añataki ukhamarak q’umachañataki. beautifulsoup4 ukat : Akax Ollama LLM ukamp chikt’atäñatakiwa ukat qillqat lurañatakiw. langchain langchain-ollama ukat : HTML nayrar sartañ uñakipañataki. lxml html5lib Uka dependencias ukanakaxa aka kamachimpiwa utt’ayata: (Janiräkipanxa, kawkhantï qillqatax utjki uka carpetaruw mantañama.) pip install -r requirements.txt Streamlit ukampi UI luraña ukax mä interfaz interactiva de usuario (UI) Python ukan apnaqawipatak lurañax jasakiwa. Aka t’aqapanx mä sanu, apnaqir interfaz ukaw lurasini, kawkhantix apnaqirinakax mä URL ukar mantapxaspa ukat raspado datos uñacht’ayapxaspaw. Ukax mä jach’a uñacht’äwiwa 1. Streamlit Script uka wakicht’aña Mä qillqat ui.py sutimp lurañaw proyecto directorio ukan. Aka script ukax UI ukax raspador ukatakiw qhanañchasi. Aka amparamp qillqt’at chimpumpiw aplicación ukar uñt’ayañax wakisi: import streamlit as st import pathlib from main import scrape_website # function to load css from the assets folder def load_css(file_path): with open(file_path) as f: st.html(f"<style>{f.read()}</style>") # Load the external CSS css_path = pathlib.Path("assets/style.css") if css_path.exists(): load_css(css_path) st.title("AI Scraper") st.markdown( "Enter a website URL to scrape, clean the text content, and display the result in smaller chunks." ) url = st.text_input(label= "", placeholder="Enter the URL of the website you want to scrape") if st.button("Scrape", key="scrape_button"): st.write("scraping the website...") result = scrape_website(url) st.write("Scraping complete.") st.write(result) ukat uka lurawinakax aplicación ukan sutip uñt’ayi ukatx apnaqirinakarux yatichäwinak churaraki. st.title st.markdown Componente ukax apnaqirinakaruw URL uka sitio web ukar mantañapatak jayti, jupanakax raspañ munapxi. st.text_input "Scrape" ukxaru ch’iqt’asaxa, lógica de raspado ukarux ch’amancharaki, apnaqasa nayrar sartañ yatiyawinak uñacht’ayañataki. st.info Jupanakatxa componentes streamlit ukanakat juk’amp yatxatapxasmawa . documentación ukanaka 2. Estilos Personalizados ukanaka yapxataña Aplicación ukar estilo lurañatakix, mä carpeta de activos ukar proyecto directorio ukan lurañamawa ukat mä style.css archivo ukar yapxatañamawa. Streamlit ukax CSS ukampiw uñt’ayasi: .stAppViewContainer { background-image: url("https://images.unsplash.com/photo-1732979887702-40baea1c1ff6?q=80&w=2832&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D"); background-size: cover; color: black; } .stAppHeader { background-color: rgba(0, 0, 0, 0); } .st-ae { background-color: rgba(233, 235, 234, 0.895); } .st-emotion-cache-ysk9xe { color: black; } .st.info, .stAlert { background-color: black; } .st-key-scrape_button button { display: inline-block; padding: 10px 20px; font-size: 16px; color: #fff; background-color: #007bff; border: none; border-radius: 5px; cursor: pointer; animation: pulse 2s infinite; } .st-key-scrape_button button:hover { background-color: #0056b3; color: #fff; } 3. Streamlit uka app apnaqaña Proyecto directorio ukanxa, aka kamachi phuqhañawa: streamlit run ui.py Ukax mä local servidor ukar uñstayañapawa, ukatx mä URL terminal ukan uñjañaw wakisi, jilapart . Aka URL ukax navegador ukan jist’aratawa, web ukan apnaqawimp chikt’atäñataki. http://localhost:8501 Raspado sitio web ukax Selenium ukampiw lurasi Ukxarusti, Selenium apnaqasa kuna web ukan HTML ukan utjki ukanak apsuñatakix código qillqt’aña. Ukampirus, código ukax irnaqañapatakix mä Chrome WebDriver ukawa. ChromeDriver ukax Selenium ukatakiw utt’ayata Selenium ukax mä WebDriver ukaruw munaraki, ukax web ukan uñt’ayatawa. Akax kunjams wakicht’asispa: ChromeDriver ukax akhamawa: Aka tuqiru visitt’apxañani ukat Google Chrome ukan navegador ukar uñtasit versión ukar apkatañamawa. ChromeDriver ukax mä sitio web ukankiwa ChromeDriver ukax PATH ukar yapxatatawa ChromeDriver apsuñ tukuyasaxa, archivo apsuña ukatxa aplicación archivo suti “ ” ukxa copiaña ukatxa proyecto carpeta ukar uchaña. chromedriver Kunawsatix ukham luratäki ukhax mä machaq qillqat ukham lurañaw ukat aka amparamp qillqt’at chimpunak phuqhaña: main.py from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.chrome.service import Service from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC # Function to scrape HTML from a website def scrape_website(website_url): # Path to WebDriver webdriver_path = "./chromedriver" # Replace with your WebDriver path service = Service(webdriver_path) driver = webdriver.Chrome(service=service) try: # Open the website driver.get(website_url) # Wait for the page to fully load WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.TAG_NAME, "body"))) # Extract the HTML source html_content = driver.page_source return html_content finally: # Ensure the browser is closed after scraping driver.quit() Uka codigox imaña ukat apnaqaña; taqi HTML ukax panka raspado uñacht’ayat streamlit aplicación ukan akham uñt’ayañapawa: Mä Proxy Provider apnaqañax sitio web ukar Bypass ukar Captcha ukat IP Bans ukanakampi Jichhax mä sitio web ukan HTML ukar apsuñjamakchisa, aka pata tuqinkir código ukax inas jan irnaqkchiti sitios ukanakatakix nayrar sartañ mecanismos anti-raspado ukanakampi kunjamakitix CAPTCHA ukan jan walt’awinakapa jan ukax IP jark’awinaka. Amuyt’añataki, Indeed jan ukax Amazon ukham Selenium apnaqañ sitio ukar raspañax mä CAPTCHA ukan pankapar mantañ jark’aqaspawa. Ukax lurasiwa kunatix sitio web ukax mä bot ukan contenido ukar mantañ yant’atap uñt’ayi. Uka sarnaqawix utjaskakispa ukhaxa, sitio ukax qhipharux IP ukar jark’aqaspawa, juk’amp mantañ jark’añataki. Uka askichañatakixa, mayachañawa guion ukar mantañamawa. Navegador de raspado ukax mä ch’aman herramienta ukawa, ukax walja redes proxy ukanakaruw aprovecha, IPs residenciales ukanakamp chika, anti-raspado ukan arxatäwinakap jan yäqañataki. Ukax p’iqinchäwinak jan jark’aqañ apnaqi, p’iqinchäwinak apnaqasa, navegador ukan amparanakap apnaqasa, CAPTCHA askichañampi, ukat juk’ampinakampiw apnaqi. Ukax raspañ ch’amanchawinakamax jan uñt’ataw qhiparañapataki, kunawsatix contenido ukar jan kuna jan walt’awimp mantañkama. Bright Data ukax Navegador de Raspado ukawa Bright Data ukax Scraping Browser ukax inakiw wakicht’ata Qillqt’asiña — saraña ukat “ ” ukar ch’iqt’añama. Ukax mä cuenta Bright Data ukampiw utji, ukax mäkiw mantasma. Bright Data ukax mä uñt’ayawiwa Inakiw yant’äw qalltaña Ukar mantañ tukuyasaxa, “ ” ukar ch’iqt’añamawa. Get Proxy Products “ ” siski ukaruw chʼetjjtäta ukat “ ” siski ukaruw chʼetjjtäta. Agregar Navegador de raspado Ukxarusti, “ ” uksaruw apataraki, ukanx machaq zona proxy de navegador de raspado ukatakix mä suti ajlliñaw wakisi. Uka qhepatjja, “ ” siski ukaruw chʼetjjtäta. Zona yapxataña Agregar Uka qhipatxa, zona proxy ukan credenciales ukanakax luratäniwa. Uka detalles ukanakax script ukanx munasiniw kuna mecanismos anti-raspado ukanakax kuna sitio web ukan apnaqatäki ukanakar jan yäqañataki. Ukhamarakiw Bright Data ukan desarrollador ukan qillqatanakap uñakipt’asma, juk’amp yatxatañatakix navegador de raspado uka tuqita. qillqatanx, uka chimpux mayjt’ayatawa. Uka codigox nayra codigot sipanx juk’amp q’uma ukat jisk’akiwa sasaw amuyapxäta. main.py from selenium.webdriver import Remote, ChromeOptions from selenium.webdriver.chromium.remote_connection import ChromiumRemoteConnection from selenium.webdriver.common.by import By from bs4 import BeautifulSoup AUTH = '<username>:<passord>' SBR_WEBDRIVER = f'https://{AUTH}@brd.superproxy.io:9515' # Function to scrape HTML from a website def scrape_website(website_url): print("Connecting to Scraping Browser...") sbr_connection = ChromiumRemoteConnection(SBR_WEBDRIVER, "goog", "chrome") with Remote(sbr_connection, options=ChromeOptions()) as driver: driver.get(website_url) print("Waiting captcha to solve...") solve_res = driver.execute( "executeCdpCommand", { "cmd": "Captcha.waitForSolve", "params": {"detectTimeout": 10000}, }, ) print("Captcha solve status:", solve_res["value"]["status"]) print("Navigated! Scraping page content...") html = driver.page_source return html ukat ukax raspado navegador ukan apnaqirin sutipampi ukat contraseña ukamp turkañawa. <usuario> <contraseña> Dom ukax q’umachañawa Mä sitio web ukan HTML ukan contenido ukar raspado tukuyatatxa, walja kutiw jan wakiskir elementos ukanakamp phuqhantat uñjasi, JavaScript, estilos CSS, jan ukax jan munat etiquetas ukanakax janiw yanapt’kiti kuna yatiyawinaktix apsusipkta ukaru. Datos ukanakax juk’amp wakicht’atäñapataki ukat juk’amp lurañatakix askïñapatakix, DOM ukan contenido ukar q’umachañax wakisiwa, jan wakiskir elementonak apsusaw ukat qillqat wakicht’añaw wakisi. Aka t’aqax kunjams HTML ukan q’umachañapa, aski qillqatanaka apsuña, ukatx jisk’a t’aqanakaruw jaljañapa, ukax alaya tuqin lurañatakiw qhanañchi. Q’umachañ lurawix lurawinakatakix datos wakicht’añatakix wali wakiskiriwa, kunjamatix natural lengua ukar apnaqaña jan ukax contenido uñakipaña. Código Walkthrough ukax DOM Contenido Q’umachañataki Akax mä código ukawa, ukax ukar yapxatatawa, DOM ukan q’umachañ apnaqañataki: main.py from bs4 import BeautifulSoup # Extract the body content from the HTML def extract_body_content(html_content): soup = BeautifulSoup(html_content, "html.parser") body_content = soup.body if body_content: return str(body_content) return "" # Clean the body content by removing scripts, styles, and other unwanted elements def clean_body_content(body_content): soup = BeautifulSoup(body_content, "html.parser") # Remove <script> and <style> tags for script_or_style in soup(["script", "style"]): script_or_style.extract() # Extract cleaned text with each line separated by a newline cleaned_content = soup.get_text(separator="\n") cleaned_content = "\n".join( line.strip() for line in cleaned_content.splitlines() if line.strip() ) return cleaned_content # Split the cleaned content into smaller chunks for processing def split_dom_content(dom_content, max_length=5000): return [ dom_content[i : i + max_length] for i in range(0, len(dom_content), max_length) ] Kuns Código ukax Luri Uka cuerpon utjki ukanak apsuña: lurawix BeautifulSoup ukampiw HTML ukar uñakipañataki ukat ukan chimpunakapar apsuñataki. extract_body_content <body> Mä chimpux utjki ukhax lurawix mä cadena ukham kutt’ayaraki. Jan ukhamäkanixa, mä ch’usa cadena kutt’ayaraki. <body> Uka Contenido qʼomachaña: 1.1. lurawix apsutanakax jan wakiskir elementonak apsuñatakiw lurasi: clean_body_content ukat uka chimpunakax JavaScript ukat CSS ukanak chhaqtayañatakiw apsutäna. <script> <style> Función ukax q’uma qillqatanakat q’uma qillqat apsu. Ch’usa chimpunaka ukat extraños chiqanakat apsusaw qillqat formato uñt’ayi. Contenido ukar jaljaña: lurawix q’umachata yänak apsu ukatx jisk’a chiqanakaruw jaljasi, ukax 5.000 chimpunak jach’a jayaruw uñt’ayasi. split_dom_content Ukax wali askiwa jach’a qillqatanakax apnaqañjam chiqanakan lurañataki, juk’ampirus kunawsatix datos ukanakax modelos ukar token jan ukax entrada tama límites ukanakamp pasañapataki. Mayjt’awinakam imañamawa ukat aplicación ukar yant’añamawa. Mä sitio web ukar raspañ tukuyasax mä salida ukham jikxatañamawa. Ollama ukar Dom ukan contenido ukar uñakipt’aña Mä kutix DOM ukax q’umachatawa ukatx wakicht’atarakiwa, jutir amtawix yatiyawinak uñakipt’añawa, ukhamat apnaqasa específicos detalles ukanaka apsuñataki , mä jach’a arut modelo (LLM) LangChain ukamp chikt’ata. Ollama ukax mä herramienta CLI ukawa, ukax LLMs locales ukar apkatañataki ukat apnaqañatakiw apnaqasi. Ukampirus janïr Ollama apnaqañkamax aka instalaciones ukanakaw lurasiñapa: Ollama Janitï ukhamäkchi ukhaxa, Ollama ukar apkatañamawa ukat instalañamawa . Mac ukanx Homebrew kamachimpiw uskt’asispa. sitio web oficial ukax uñt’ayatawa brew install ollama Ukxarusti, kuna modelos ukanakat instalaña ; kunayman modelos ukanakaw utji, Phi3, Mistral, Gemma 2, ukat juk’ampinaka; sapa mayniw sistema ukan mayiwinakapamp utji. Aka codigox phi3 uka apnaqi jilpachax kunatix k’achachata. aka lista ukaxa ollama pull phi3 Instalación tukuyatatxa, uka modelo ukarux script ukanx jawst’asmawa LangChain uka apnaqasa, ukax datos ukanakat aski amuyunaka churañataki, ukax ukaruw apayani. Akax kunjams DOM ukan contenido ukar modelo ukar uñakipañatakix funcionalidad ukar wakicht’añax phi3 Código walkthrough ukax llm.py ukatakiw Aka codigox Ollama ukamp DOM chunks ukar parse ukat wakiskir detalles ukar apsuñatakix lógica ukar phuqhi: from langchain_ollama import OllamaLLM from langchain_core.prompts import ChatPromptTemplate # Template to instruct Ollama for parsing template = ( "You are tasked with extracting specific information from the following text content: {dom_content}. " "Please follow these instructions carefully: \n\n" "1. **Extract Information:** Only extract the information that directly matches the provided description: {parse_description}. " "2. **No Extra Content:** Do not include any additional text, comments, or explanations in your response. " "3. **Empty Response:** If no information matches the description, return an empty string ('')." "4. **Direct Data Only:** Your output should contain only the data that is explicitly requested, with no other text." ) # Initialize the Ollama model model = OllamaLLM(model="phi3") # Function to parse DOM chunks with Ollama def parse_with_ollama(dom_chunks, parse_description): prompt = ChatPromptTemplate.from_template(template) chain = prompt | model parsed_results = [] for i, chunk in enumerate(dom_chunks, start=1): if not chunk.strip(): # Skip empty chunks print(f"Skipping empty chunk at batch {i}") continue try: print(f"Processing chunk {i}: {chunk[:100]}...") # Print a preview print(f"Parse description: {parse_description}") response = chain.invoke( { "dom_content": chunk, "parse_description": parse_description, } ) print(f"Response for batch {i}: {response}") parsed_results.append(response) except Exception as e: print(f"Error parsing chunk {i}: {repr(e)}") parsed_results.append(f"Error: {repr(e)}") return "\n".join(parsed_results) Kuntix codigox luraski. Yatichañ Plantilla: Ollamatakix kuna yatiyawinaksa apsuñapa ukxat chiqaparu irnaqawi churaraki. Uñakipt’añatakix mistuwix q’uma, jisk’a, ukatx parsing qhanañchäwimp chikaw wakisi. Chunk Procesamiento ukax akhamawa: parse_with_ollama lurawix DOM chimpunak tuqiw itera, sapa maynix LLM ukampiw lurasi. Ch’usa chunks ukanakax juk’amp suma lurañatakiw jaytawayi. Pantjasiwinak apnaqaña: 1.1. Pantjasiwinak suma uñji, qillqt’i, ukatx qhipharkir chimpunak luraskaki. Ui.py qillqatan machaqar tukuyaña Aka chimpu ui.py qillqataru yapxataña, ukhamata apnaqirinakaxa LLM ukarux parsing yatichäwinak mantapxañapataki ukat resultados uñakipapxañapataki: from main import scrape_website, extract_body_content, clean_body_content, split_dom_content from llm import parse_with_ollama if "dom_content" in st.session_state: parse_description = st.text_area("Enter a description to extract specific insights from your scraped data:") if st.button("Parse Content", key="parse_button"): if parse_description.strip() and st.session_state.get("dom_content"): st.info("Parsing the content...") dom_chunks = split_dom_content(st.session_state.dom_content) parsed_result = parse_with_ollama(dom_chunks, parse_description) st.text_area("Parsed Results", parsed_result, height=300) else: st.error("Please provide valid DOM content and a description to parse.") Kunjams UI ukan irnaqaski Usuario Input ukax akhamawa: Usuario ukax mä qillqat chiqan apsuñatakix datos ukanakat natural arut qhanañcht’i. Parsing Trigger ukax akhamawa: Kunawsatix Parse Content ukax ch’iqt’atäki ukhax q’uma DOM ukax apnaqañjam chimpunakaruw jaljasi ukatx parse_with_ollama ukaruw pasaraki. Ukax akham sañ muniwa: Uñakipt’atanakax mä qillqat chiqan uñacht’ayatawa, ukax apnaqirinakaruw apsut yatiyäwinak uñakipañapatak yanapt’i. Ukham lurasax, jichhax raspador ukax jiskt’awinakamar jaysäwinak churaspa, ukax datos raspado ukarjam luratawa. ¿Kunas jutïrin pasani? Web scraping ukat AI ukanakan mayacht’asitapax datos-driven insights ukanakatakix kusiskañ posibilidades ukanakaw jist’arasi. Datos apthapiñat sipansa ukat imañat sipansa, jichhax AI ukax aprovechasispawa, ukhamat datos raspado ukanakat amuyt’awinak jikxatañ thakhix juk’amp sumaptañapataki. Akax qhathunakataki ukhamarak aljirinakatakix wali askiwa, datos uñakipañataki, negocios ukankirinakataki, ukat juk’ampinakataki. Aka chiqanx AI raspador ukatakix taqpach código ukaw jikxatasi. Jan axsarasaw yantʼañama ukat kunatï jumatak wakiski ukarjamaw lurañama. Yanapt’awinakax wali askiwa —jumatix juk’amp askinak lurañ amtanakax utjktam ukhax mä jaqukipañ mayiw lurañ amtañamawa! Uka tuqitxa juk’ampiruw apasxaraksna. Akax mä qawqha amuyunakawa: Yant’awinakampi yant’aña: Jiskt’awinakamaxa wakicht’atawa mä qhawqha amuyunaka apsuñataki jan ukaxa proyecto ukana sapa mayni mayiwinakaparu uñjañataki. Usuario Interfaz ukampi Yaqha LLM Modelos ukanakamp mayachthapiña: Yaqha arut modelos ukanakamp yatxataña kunjamatix , . , ukat juk’ampinaka, juk’amp suma uñakipañataki datos uñakipañataki. OpenAI ukax mä juk’a pachanakanwa Géminis ukanaka
