Xpath

xpath

Groundbreaking Tips On How To Use XPath In Selenium 

Sometimes, you need to deal with sliders in your tests but Playwright does not have any built-in tools to interact with sliders. However, it is not a problem at all. In this guide, you will learn how to interact with sliders in Playwright. Let’s start by adding a simple slider element to our page using the setContent() method, which internally calls document.write() and adds anything you want in the DOM. import {test, expect, Page} from '@playwright/test';

let page: Page;


const sliderHTML = `<div>
    <p><input type="range" id="slider" min="0" max="100"/></p>
</div>`

test('Slider test', async ({page}) => {
  await page.setContent(sliderHTML);
}); In this case, all you get is the simple range slider on the page. I suggest making the task more complicated and setting the slider value random. This makes us consider how to interact with the slider depending on its value. import {test, expect, Page} from '@playwright/test';

let page: Page;

function getRandomInt(min, max) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

const sliderHTML = `<div>
    <p><input type="range" id="slider" min="0" max="100" value="${getRandomInt(0, 100)}"/></p>
</div>`

test('Slider test', async ({page}) => {
  await page.setContent(sliderHTML);
  await setSliderValue(page, "//*[@id = 'slider']",75);
}); So, the final task is to set the slider to the given value, let’s say 95%. The solution will include multiple steps: Find the slider element on a page. Detect its current value to be able to click on it. Move the slider into the final position (95%). It looks pretty simple. Starting with finding the slider, I will use the XPath locator to find it. Feel free to check my XPath mastering guide. For this case, I suggest using the following: "//*[@id = 'slider']" The next step is getting the bounding box of the slider element to determine its position and size. const sliderBound = await page.locator("//*[@id = 'slider']").boundingBox(); An element's bounding box is the smallest possible rectangle (aligned with the axes of that element’s user coordinate system) that entirely encloses it and its descendants. The bounding box will help us to click and move the mouse cursor inside that box. Since the slider pin can have a random position, we must find its actual coordinates. We need to obtain the current slider value from the to do this. It is possible by using the page.evaluate() method. HTML const currentSliderValue = await page.evaluate(`document.evaluate(""//*[@id = 'slider']"", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.value`); This method returns the current slider value as a number. This number will help you to calculate the slider pin coordinate. You need to calculate two points for X and Y coordinates. Because the slider pin moves horizontally, the only X coordinate is crucial for us. To calculate it, use the formula: const targetX = sliderBound.x + (sliderBound.width * currentSliderValue / 100); To get the Y, calculate the vertical middle point: const targetY = sliderBound.y + sliderBound.height / 2; Next, move the mouse cursor to the calculated point and hold its button: await page.mouse.move(targetX, targetY);
await page.mouse.down(); Now, you can move the mouse cursor horizontally to any position you want. await page.mouse.move(
    sliderBound.x + (sliderBound.width * 95) / 100,
    sliderBound.y + sliderBound.height / 2,
);
await page.mouse.up(); Let’s bring all parts together and create a method for setting a wanted value to any slider element by its XPath: async function setSliderValue(page: Page, sliderXPath: string, valueAsPercent: number) {
  // Find the slider element using the provided XPath and obtain its bounding box
  const sliderBound = await page.locator(sliderXPath).boundingBox();

  // Use page.evaluate to obtain the current slider value from the HTML using the same XPath
  const currentSliderValue = await page.evaluate(`document.evaluate("${sliderXPath}", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.value`);

  // Calculate the target X and Y coordinates for the mouse cursor based on the current slider value
  const targetX = sliderBound.x + (sliderBound.width * currentSliderValue / 100);
  const targetY = sliderBound.y + sliderBound.height / 2;

  // Move the mouse cursor to the calculated position
  await page.mouse.move(targetX, targetY);

  // Simulate a mouse click by pressing the mouse button
  await page.mouse.down();

  // Move the mouse cursor to the desired position by the provided valueAsPercent
  await page.mouse.move(
      sliderBound.x + (sliderBound.width * valueAsPercent) / 100,
      sliderBound.y + sliderBound.height / 2,
  );

  // Release the mouse button to complete the interaction
  await page.mouse.up();
} Now you can use it in your test just by passing the slider locator and a wanted value: import {test, expect, Page} from '@playwright/test';

let page: Page;

function getRandomInt(min, max) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

async function setSliderValue(page: Page, sliderXPath: string, valueAsPercent: number) {
  // Find the slider element using the provided XPath and obtain its bounding box
  const sliderBound = await page.locator(sliderXPath).boundingBox();

  // Use page.evaluate to obtain the current slider value from the HTML using the same XPath
  const currentSliderValue = await page.evaluate(`document.evaluate("${sliderXPath}", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.value`);

  // Calculate the target X and Y coordinates for the mouse cursor based on the current slider value
  const targetX = sliderBound.x + (sliderBound.width * currentSliderValue / 100);
  const targetY = sliderBound.y + sliderBound.height / 2;

  // Move the mouse cursor to the calculated position
  await page.mouse.move(targetX, targetY);

  // Simulate a mouse click by pressing the mouse button
  await page.mouse.down();

  // Move the mouse cursor to the desired position by the provided valueAsPercent
  await page.mouse.move(
      sliderBound.x + (sliderBound.width * valueAsPercent) / 100,
      sliderBound.y + sliderBound.height / 2,
  );

  // Release the mouse button to complete the interaction
  await page.mouse.up();
}

const sliderHTML = `<div>
    <p><input type="range" id="slider" min="0" max="100" value="${getRandomInt(0, 100)}"/></p>
</div>`

const sliderElementLocator = "//*[@id = 'slider']";
test('Slider test', async ({page}) => {
  await page.setContent(sliderHTML);
  await setSliderValue(page, sliderElementLocator, 75);
}); I hope this guide helped you to understand how to handle the sliders in Playwright. Happy testing! Also published here.

Interacting With Sliders Using Playwright

This tutorial will clarify various methods for selecting elements and data within XML or HTML documents. I’ve included for you the explanations and examples of standard XPath methods. Node Selection Node selection in XPath refers to choosing specific elements, attributes, or nodes within an XML or HTML document based on their type or location in the document’s hierarchy. //img Attribute Selection Attribute selection in XPath involves choosing elements within an XML or HTML document based on their attributes’ values. //*[@id = 'gridItemRoot'] Predicate Filtering Predicate filtering in XPath applies conditions or filters to select specific elements or nodes based on certain criteria. Use conditions inside square brackets to filter elements. //span[contains(@class, 'sc-price') and number(translate(., '$', '')) < 10.00] Positional Selection Positional selection in XPath involves choosing elements within an XML or HTML document based on their position or index in its structure. //*[@id = 'gridItemRoot'][4] Text Content Selection Text content selection in XPath refers to choosing elements within an XML or HTML document based on the textual content contained within those elements. //*[text()='The 48 Laws of Power'] Logical Operators Logical operators in XPath are used to combine or modify conditions within an XPath expression to make more complex selections. //div[@id='gridItemRoot' and //*[contains(@class, 'a-icon-star-small')] and .//span[contains(@class, 'sc-price') and number(translate(., '$', '')) < 10.00]] Axis Selection Axis selection in XPath involves navigating the document’s hierarchy based on the relationships between elements and nodes, allowing you to select elements related to a specific context node. Parent Selection The “parent” XPath is used to select the parent element of a given element. It allows you to navigate the document’s hierarchy to access a specific node's immediate or nearest enclosing parent element. //li[contains(.//span, 'Comics & Graphic Novels')]/parent::*
or
//li[contains(.//span, 'Comics & Graphic Novels')]/.. Preceding Sibling Selection Preceding sibling selection in XPath allows you to select elements that are siblings of a given context node and appear before it in the document’s hierarchy. //li[contains(.//span, 'Comics & Graphic Novels')]/preceding-sibling::* Following Sibling Selection Following sibling selection in XPath allows you to select elements that are siblings of a given context node and appear after it in the document’s hierarchy. //li[contains(.//span, 'Comics & Graphic Novels')]/following-sibling::* Child Selection Child selection in XPath involves selecting elements that are direct children of a given parent element or context node within the XML or HTML document. //li[contains(.//span, 'Comics & Graphic Novels')]/../child::* Wildcards Wildcard selection in XPath involves using wildcard symbols to match elements or attributes regardless of their specific names or values. //* Functions Functions in XPath are predefined operations or calculations that you can use within an XPath expression to manipulate or evaluate nodes, attributes, or values in XML or HTML documents. //*[@class = 'a-size-small']//child::*[starts-with(text(), 'It')] These methods offer versatile ways to locate specific elements, attributes, or data within XML and HTML documents, making XPath a powerful tool for tasks such as web scraping, data extraction, and test automation. Also published . here

Getting Your Way Around With Common XPath Methods

In the context of web scraping, is a nice tool to have in your belt, as it allows you to write specifications of document locations more flexibly than CSS selectors. In case you’re looking for a tutorial, . XPath here is a XPath tutorial with nice examples In this post, we’ll show you some tips we found valuable when using XPath in the trenches, using for our examples. Scrapy Selector API Avoid using contains(.//text(), ‘search text’) in your XPath conditions. Use contains(., ‘search text’) instead. Here is why: the expression yields a collection of text elements — a . And when a node-set is converted to a string, which happens when it is passed as argument to a string function like or , results in the text for the element only. .//text() node-set contains() starts-with() first >>> from scrapy import Selector>>> sel = Selector(text='<a href="#">Click here to go to the <strong>Next Page</strong></a>')>>> xp = lambda x: sel.xpath(x).extract() # let's type this only once>>> xp('//a//text()') # take a peek at the node-set[u'Click here to go to the ', u'Next Page']>>> xp('string(//a//text())') # convert it to a string[u'Click here to go to the '] A converted to a string, however, puts together the text of itself plus of all its descendants: node >>> xp('//a[1]') # selects the first a node[u'<a href="#">Click here to go to the <strong>Next Page</strong></a>']>>> xp('string(//a[1])') # converts it to string[u'Click here to go to the Next Page'] So, in general: GOOD: >>> xp("//a[contains(., 'Next Page')]")[u'<a href="#">Click here to go to the <strong>Next Page</strong></a>'] BAD: >>> xp("//a[contains(.//text(), 'Next Page')]")[] GOOD: >>> xp("substring-after(//a, 'Next ')")[u'Page'] BAD: >>> xp("substring-after(//a//text(), 'Next ')")[u''] You can read . more detailed explanations about string values of nodes and node-sets in the XPath spec Beware of the difference between //node[1] and (//node)[1] selects all the nodes occurring first under their respective parents. //node[1] selects all the nodes in the document, and then gets only the first of them. (//node)[1] >>> from scrapy import Selector>>> sel=Selector(text="""....: <ul class="list">....: <li>1</li>....: <li>2</li>....: <li>3</li>....: </ul>....: <ul class="list">....: <li>4</li>....: <li>5</li>....: <li>6</li>....: </ul>""")>>> xp = lambda x: sel.xpath(x).extract()>>> xp("//li[1]") # get all first LI elements under whatever it is its parent[u'<li>1</li>', u'<li>4</li>']>>> xp("(//li)[1]") # get the first LI element in the whole document[u'<li>1</li>']>>> xp("//ul/li[1]") # get all first LI elements under an UL parent[u'<li>1</li>', u'<li>4</li>']>>> xp("(//ul/li)[1]") # get the first LI element under an UL parent in the document[u'<li>1</li>'] Also, gets a collection of the local anchors that occur first under their respective parents. //a[starts-with(@href, '#')][1] gets the first local anchor in the document. (//a[starts-with(@href, '#')])[1] When selecting by class, be as specific as necessary If you want to select elements by a CSS class, the XPath way to do that is the rather verbose: *[contains(concat(' ', normalize-space( ), ' '), ' someclass ')] @class Let’s cook up some examples: >>> sel = Selector(text='<p class="content-author">Someone</p><p class="content text-wrap">Some content</p>')>>> xp = lambda x: sel.xpath(x).extract() doesn’t work because there are multiple classes in the attribute BAD: >>> xp("//*[ ='content']")[] @class gets more than we want BAD: >>> xp("//*[contains( ,'content')]")[u’<p class=”content-author”>Someone</p>’] @class GOOD: >>> xp("//*[contains(concat(' ', normalize-space( ), ' '), ' content ')]")[u'<p class="content text-wrap">Some content</p>'] @class And many times, you can just use a CSS selector instead, and even combine the two of them if needed: ALSO GOOD: >>> sel.css(".content”).extract()[u'<p class="content text-wrap">Some content</p>']>>> sel.css('.content').xpath(' ').extract()[u'content text-wrap'] @class Read . more about what you can do with Scrapy’s Selectors here Learn to use all the different axes It is handy to know how to use the axes, you can to quickly review this. follow through the examples given in the tutorial In particular, you should note that and are not the same thing, this is a common source of confusion. The same goes for and , and also and . following following-sibling preceding preceding-sibling ancestor parent Useful trick to get text content Here is another XPath trick that you may use to get the interesting text contents: 1//*[not(self::script or self::style)]/text()[normalize-space(.)] This excludes the content from and tags and also skip whitespace-only text nodes. Source: script style http://stackoverflow.com/a/19350897/2572383 Do you have another XPath tip? Please, leave us a comment with your tips or questions. Originally published on the . Scrapinghub Blog This post was written by Elias Dorneles ( ), a Software Developer at Scrapinghub. @eliasdorneles Please heart “Recommend” and share these tips far and wide. Find out what web scraping and web data can do for you .

XPath Tips from the Web Scraping Trenches

Web Scraping Optimization: Tips for Faster, Smarter Scrapers

Web Scraping Optimization: Daha Sürətli, Daha Ağıllı Scrapers üçün Məsləhətlər

Optimización del web scraping: consejos para que los scrapers sean más rápidos e inteligentes

Web スクレイピングの最適化: より高速でスマートなスクレイパーのためのヒント

Žiniatinklio grandymo optimizavimas: greitesnių ir išmanesnių grandiklių patarimai

Fanamafisana ny fakosoka amin'ny Internet: Toro-hevitra ho an'ny mpikirakira haingana kokoa

वेब स्क्र्यापिंग अप्टिमाइजेसन: छिटो, स्मार्ट स्क्र्यापरहरूको लागि सुझाव

Optymalizacja Web Scraping: Wskazówki dotyczące szybszych i inteligentniejszych Scraperów

Gukuramo Urubuga Optimisiyoneri: Inama zo Kwihuta, Byoroheje

Оптимизација веб-стругања: савети за брже и паметније стругаче

Оптимизатсияи веб скрепинг: Маслиҳатҳо барои скреперҳои тезтар ва оқилтар

Web Scraping Optimization: Mga Tip para sa Mas Mabilis, Mas Matalino na mga Scraper

I-Web Scraping Optimization: Amathiphu Okwenza Ama-Scrapers Asheshayo, Ahlakaniphile

In Part 1 of this series, we went through the process of setting up and installing Appium, along with the necessary dependencies and drivers for mobile automation testing. Now that we have everything installed, let's take a closer look at what exactly we’ve set up and understand the role each component plays in our mobile test automation workflow. Understanding these components will give us a clearer picture of how Appium enables us to automate tests for both Android and iOS applications. 1. Appium Appium is the heart of our mobile automation setup. It acts as a bridge between the test scripts you write and the mobile device or simulator/emulator that the tests are being executed on. Think of Appium as a tool that translates your test scripts (which are written in programming languages like Java, Python, or JavaScript) into commands that the mobile operating system can understand. When we write a test case, Appium takes each action—such as tapping a button, swiping a screen, or entering text—and converts it into a sequence of commands that the mobile OS interprets. Appium is designed to support both Android and iOS, making it a cross-platform solution for mobile test automation. This means that we can write one test and run it across different devices and platforms with minimal changes to our code. 2. UiAutomator Driver The UI Automator driver is a powerful tool for interacting with the UI components on an Android device or emulator. It provides access to all the features needed to automate the interaction with Android app elements. This driver enables actions such as tapping buttons, scrolling through lists, entering text in forms, and verifying the presence of elements on the screen. One key benefit of the UI Automator Driver is its ability to interact with both the app's UI and system-level features, such as accessing notifications and interacting with native Android components. If you’re focusing on testing Android apps, the UI Automator Driver is a critical component of your setup. 3. XCUITest Driver XCUITest is a testing framework developed by Apple for iOS applications. When we need to interact with elements in an iOS app, we use the XCUITest Driver. Just like the UI Automator Driver does for Android, XCUITest translates the actions from your test script into commands that the iOS operating system can execute. XCUITest supports a wide range of iOS-specific features, allowing you to automate gestures, verify UI elements, and even test complex interactions such as push notifications or in-app purchases. Since iOS and Android are very different platforms, the XCUITest Driver provides a tailored approach to testing iOS apps, ensuring that your automation scripts can work seamlessly on Apple's devices. 4. Wait Plugin The Wait Plugin is an essential part of test automation, especially for mobile applications where delays or changes in the UI might cause timing issues. This plugin allows you to pause your test execution until a specific condition is met, such as waiting for an element (like a button or text field) to appear or become clickable. Without such a feature, your tests could fail prematurely, particularly in scenarios where elements load dynamically or after a delay (such as network latency or complex animations). By incorporating the Wait Plugin, you can avoid flaky tests—tests that fail intermittently because the necessary elements haven't loaded in time. The plugin ensures that the test doesn't proceed until it is sure the element is available for interaction, improving the reliability of your automated tests. 5. Appium Inspector Think of Appium Inspector as a visual tool to help you inspect the app's user interface during the test design phase. It provides a graphical interface that displays the app's components, such as buttons, text fields, and images, making it easy to identify elements you want to interact with. You can use the Appium Inspector to locate elements by inspecting their properties like their ID, class, or XPath. This tool is invaluable when designing your test scripts, as it allows you to visually explore the app’s UI and interact with specific elements without writing code immediately. With Appium Inspector, you can quickly determine the exact properties and attributes of UI components, and once you have identified the elements, you can use them in your test scripts to automate interactions. It significantly simplifies the process of writing and debugging automation tests. Conclusion In summary, the components we've installed: Appium, the UI Automator Driver, the XCUITest Driver, the Wait Plugin, and the Appium Inspector, each serve a unique purpose, but they work together to enable powerful mobile automation testing. Appium provides the foundation by connecting the test script with the device, while the drivers for Android and iOS (UI Automator and XCUITest) provide platform-specific functionality. The Wait Plugin ensures reliable test execution by managing timing issues, and the Appium Inspector makes it easier to identify and interact with UI elements. In Part 1 of this series, we went through the process of setting up and installing Appium, along with the necessary dependencies and drivers for mobile automation testing. Now that we have everything installed, let's take a closer look at what exactly we’ve set up and understand the role each component plays in our mobile test automation workflow. Understanding these components will give us a clearer picture of how Appium enables us to automate tests for both Android and iOS applications. Part 1 1. Appium Appium is the heart of our mobile automation setup. It acts as a bridge between the test scripts you write and the mobile device or simulator/emulator that the tests are being executed on. Think of Appium as a tool that translates your test scripts (which are written in programming languages like Java, Python, or JavaScript) into commands that the mobile operating system can understand. When we write a test case, Appium takes each action—such as tapping a button, swiping a screen, or entering text—and converts it into a sequence of commands that the mobile OS interprets. Appium is designed to support both Android and iOS, making it a cross-platform solution for mobile test automation. This means that we can write one test and run it across different devices and platforms with minimal changes to our code. 2. UiAutomator Driver The UI Automator driver is a powerful tool for interacting with the UI components on an Android device or emulator. It provides access to all the features needed to automate the interaction with Android app elements. This driver enables actions such as tapping buttons, scrolling through lists, entering text in forms, and verifying the presence of elements on the screen. One key benefit of the UI Automator Driver is its ability to interact with both the app's UI and system-level features, such as accessing notifications and interacting with native Android components. If you’re focusing on testing Android apps, the UI Automator Driver is a critical component of your setup. 3. XCUITest Driver XCUITest is a testing framework developed by Apple for iOS applications. When we need to interact with elements in an iOS app, we use the XCUITest Driver. Just like the UI Automator Driver does for Android, XCUITest translates the actions from your test script into commands that the iOS operating system can execute. XCUITest supports a wide range of iOS-specific features, allowing you to automate gestures, verify UI elements, and even test complex interactions such as push notifications or in-app purchases. Since iOS and Android are very different platforms, the XCUITest Driver provides a tailored approach to testing iOS apps, ensuring that your automation scripts can work seamlessly on Apple's devices. 4. Wait Plugin The Wait Plugin is an essential part of test automation, especially for mobile applications where delays or changes in the UI might cause timing issues. This plugin allows you to pause your test execution until a specific condition is met, such as waiting for an element (like a button or text field) to appear or become clickable. Without such a feature, your tests could fail prematurely, particularly in scenarios where elements load dynamically or after a delay (such as network latency or complex animations). By incorporating the Wait Plugin, you can avoid flaky tests—tests that fail intermittently because the necessary elements haven't loaded in time. The plugin ensures that the test doesn't proceed until it is sure the element is available for interaction, improving the reliability of your automated tests. 5. Appium Inspector Think of Appium Inspector as a visual tool to help you inspect the app's user interface during the test design phase. It provides a graphical interface that displays the app's components, such as buttons, text fields, and images, making it easy to identify elements you want to interact with. You can use the Appium Inspector to locate elements by inspecting their properties like their ID, class, or XPath. This tool is invaluable when designing your test scripts, as it allows you to visually explore the app’s UI and interact with specific elements without writing code immediately. With Appium Inspector, you can quickly determine the exact properties and attributes of UI components, and once you have identified the elements, you can use them in your test scripts to automate interactions. It significantly simplifies the process of writing and debugging automation tests. Conclusion In summary, the components we've installed: Appium, the UI Automator Driver, the XCUITest Driver, the Wait Plugin, and the Appium Inspector, each serve a unique purpose, but they work together to enable powerful mobile automation testing. Appium provides the foundation by connecting the test script with the device, while the drivers for Android and iOS (UI Automator and XCUITest) provide platform-specific functionality. The Wait Plugin ensures reliable test execution by managing timing issues, and the Appium Inspector makes it easier to identify and interact with UI elements.

Mobile App Test Automation - Getting Started With Appium Part 1B

Selenium Versus Karate: a Solid Comparative Approach

A brief understanding of the difference between C# and JavaScript web scraping As a compiled language, C# provides a wealth of libraries and frameworks, such as HtmlAgilityPack, HttpClient, etc., which facilitate the implementation of complex web crawling logic, and the code is concise and efficient, with strong debugging and error handling capabilities. At the same time, C# has good cross-platform support and is suitable for a variety of operating systems. However, the learning curve of C# may be relatively steep and requires a certain programming foundation. In contrast, JavaScript, as a scripting language, is more flexible in web crawling and can be run directly in the browser without the need for additional installation environment. JavaScript has a rich DOM operation API, which is convenient for direct operation of web page elements. In addition, JavaScript is also supported by a large number of third-party libraries and frameworks, such as Puppeteer, Cheerio, etc., which further simplifies the implementation of web crawling. However, JavaScript’s asynchronous programming model may be relatively complex and requires a certain learning cost. Summary of C# vs JavaScript for web scraping Differences in language and environment‌ C#:‌ Requires .NET environment,‌ suitable for desktop or server-side applications. ‌
JavaScript:‌ Built-in in the browser,‌ suitable for front-end and Node.js environment. ‌ Crawl tools and libraries‌:‌ C#: Commonly used HttpClient,‌ combined with HtmlAgilityPack parsing. ‌
JavaScript:‌ Libraries such as Axios can be used,‌ with Cheerio parsing. ‌ Execution environment and restrictions‌ C#:‌ Executed on the server or desktop,‌ less restricted by browsers. ‌
JavaScript:‌ Executed in the browser,‌ restricted by the same-origin policy, etc. ‌ Processing dynamic content‌ Both require additional processing,‌ such as Selenium assistance. ‌ JavaScript has a natural advantage in the browser environment. ‌ Summary‌ Choose based on project requirements,‌ development environment and resources. ‌ Which one is better for crawling complex dynamic web pages, C# or JavaScript? For crawling complex dynamic web pages, C# and JavaScript each have their own advantages, but C# combined with tools such as Selenium is usually more suitable. ‌ JavaScript‌:‌ As a front-end scripting language, JavaScript is executed in a browser environment and naturally supports processing dynamic content. ‌However, when JavaScript is executed on the server side or in desktop applications, it requires the help of tools such as Node.js, and may be limited by the browser’s homology policy, etc. ‌ ‌


C#‌:‌ By combining libraries such as Selenium WebDriver, C# can simulate browser behavior and process JavaScript-rendered content, including login, click, scroll, and other operations. ‌This method can more comprehensively crawl dynamic web page data, and C#’s strong typing characteristics and rich library support also improve development efficiency and stability. ‌ Therefore,‌ in scenarios where complex dynamic web pages need to be crawled,‌ it is recommended to use C# combined with tools such as Selenium for development‌ What technologies and tools are needed for web scraping with C#? Web scraping with C# requires the following technologies and tools: ‌ ‌ HttpClient or WebClient class‌:‌ used to send HTTP requests and obtain web page content. ‌HttpClient provides more flexible functions and is suitable for handling complex HTTP requests. ‌ ‌
HTML parsing library‌:‌ such as HtmlAgilityPack,‌ used to parse the obtained HTML document and extract the required data from it. ‌HtmlAgilityPack supports XPath and CSS selectors, which is convenient for locating HTML elements. ‌ ‌
Regular expression‌:‌ used to match and extract specific text content in HTML documents, but attention should be paid to the accuracy and efficiency of regular expressions. ‌ ‌
Selenium WebDriver‌:‌ For scenarios that need to simulate browser behavior (such as logging in,‌ processing JavaScript rendered content),‌ Selenium WebDriver can be used to simulate user operations. ‌
JSON parsing library‌:‌ such as Json.NET,‌ used to parse JSON formatted data,‌ which is very useful when processing data returned by API. ‌
Exception handling and multithreading: In order to improve the stability and efficiency of the program, you need to write exception handling code and consider using multithreading technology to process multiple requests concurrently. ‌
Proxy and User-Agent settings: In order to bypass the anti-crawling mechanism of the website, you may need to set the proxy and custom User-Agent to simulate different access environments. ‌ The combination of these technologies and tools can efficiently implement the C# web crawling function. ‌ How to crawl dynamic web pages with C# combined with Selenium? ‌ How to use C# combined with Selenium to crawl dynamic web pages? ‌C# combined with Selenium to crawl dynamic web pages‌ Environment preparation‌:‌ Make sure that the C# development environment is installed. ‌
Install Selenium WebDriver, which is used to simulate browser behavior. ‌
Download and set up the browser driver, such as ChromeDriver, to ensure that it is consistent with the browser version. ‌ Usage steps‌:‌ Import Selenium-related external libraries, such as WebDriver, WebDriverWait, etc. ‌
Initialize WebDriver, set up the browser driver, and open the target web page. ‌
Use the methods provided by Selenium to simulate user behaviors, such as clicking, input, scrolling, etc., to handle operations such as dynamically loading content or logging in. ‌
Parse the web page source code and extract the required data. ‌
Close the browser and WebDriver instance. ‌ By combining C# with Selenium, you can effectively crawl dynamic web page content, handle complex interactions, and avoid being blocked by website detection. ‌‌ Conclusion In summary, C# and JavaScript each have their own advantages and disadvantages in web crawling. The choice of language depends on specific needs and development environment. A brief understanding of the difference between C# and JavaScript web scraping As a compiled language, C# provides a wealth of libraries and frameworks, such as HtmlAgilityPack, HttpClient, etc., which facilitate the implementation of complex web crawling logic, and the code is concise and efficient, with strong debugging and error handling capabilities. At the same time, C# has good cross-platform support and is suitable for a variety of operating systems. However, the learning curve of C# may be relatively steep and requires a certain programming foundation. In contrast, JavaScript, as a scripting language, is more flexible in web crawling and can be run directly in the browser without the need for additional installation environment. JavaScript has a rich DOM operation API, which is convenient for direct operation of web page elements. In addition, JavaScript is also supported by a large number of third-party libraries and frameworks, such as Puppeteer, Cheerio, etc., which further simplifies the implementation of web crawling. However, JavaScript’s asynchronous programming model may be relatively complex and requires a certain learning cost. Summary of C# vs JavaScript for web scraping Differences in language and environment‌ C#:‌ Requires .NET environment,‌ suitable for desktop or server-side applications. ‌ JavaScript:‌ Built-in in the browser,‌ suitable for front-end and Node.js environment. ‌ C#:‌ Requires .NET environment,‌ suitable for desktop or server-side applications. ‌ JavaScript:‌ Built-in in the browser,‌ suitable for front-end and Node.js environment. ‌ Crawl tools and libraries‌:‌ C#: Commonly used HttpClient,‌ combined with HtmlAgilityPack parsing. ‌ JavaScript:‌ Libraries such as Axios can be used,‌ with Cheerio parsing. ‌ C#: Commonly used HttpClient,‌ combined with HtmlAgilityPack parsing. ‌ JavaScript:‌ Libraries such as Axios can be used,‌ with Cheerio parsing. ‌ Execution environment and restrictions‌ C#:‌ Executed on the server or desktop,‌ less restricted by browsers. ‌ JavaScript:‌ Executed in the browser,‌ restricted by the same-origin policy, etc. ‌ C#:‌ Executed on the server or desktop,‌ less restricted by browsers. ‌ JavaScript:‌ Executed in the browser,‌ restricted by the same-origin policy, etc. ‌ Processing dynamic content‌ Both require additional processing,‌ such as Selenium assistance. ‌ JavaScript has a natural advantage in the browser environment. ‌ Summary‌ Choose based on project requirements,‌ development environment and resources. ‌ Which one is better for crawling complex dynamic web pages, C# or JavaScript? For crawling complex dynamic web pages, C# and JavaScript each have their own advantages, but C# combined with tools such as Selenium is usually more suitable. ‌ JavaScript‌:‌ As a front-end scripting language, JavaScript is executed in a browser environment and naturally supports processing dynamic content. ‌However, when JavaScript is executed on the server side or in desktop applications, it requires the help of tools such as Node.js, and may be limited by the browser’s homology policy, etc. ‌ ‌ C#‌:‌ By combining libraries such as Selenium WebDriver, C# can simulate browser behavior and process JavaScript-rendered content, including login, click, scroll, and other operations. ‌This method can more comprehensively crawl dynamic web page data, and C#’s strong typing characteristics and rich library support also improve development efficiency and stability. ‌ JavaScript‌:‌ As a front-end scripting language, JavaScript is executed in a browser environment and naturally supports processing dynamic content. ‌However, when JavaScript is executed on the server side or in desktop applications, it requires the help of tools such as Node.js, and may be limited by the browser’s homology policy, etc. ‌ ‌ JavaScript‌:‌ As a front-end scripting language, JavaScript is executed in a browser environment and naturally supports processing dynamic content. ‌However, when JavaScript is executed on the server side or in desktop applications, it requires the help of tools such as Node.js, and may be limited by the browser’s homology policy, etc. ‌ ‌ C#‌:‌ By combining libraries such as Selenium WebDriver, C# can simulate browser behavior and process JavaScript-rendered content, including login, click, scroll, and other operations. ‌This method can more comprehensively crawl dynamic web page data, and C#’s strong typing characteristics and rich library support also improve development efficiency and stability. ‌ C#‌:‌ By combining libraries such as Selenium WebDriver, C# can simulate browser behavior and process JavaScript-rendered content, including login, click, scroll, and other operations. ‌This method can more comprehensively crawl dynamic web page data, and C#’s strong typing characteristics and rich library support also improve development efficiency and stability. ‌ Therefore,‌ in scenarios where complex dynamic web pages need to be crawled,‌ it is recommended to use C# combined with tools such as Selenium for development‌ What technologies and tools are needed for web scraping with C#? Web scraping with C# requires the following technologies and tools: ‌ ‌ HttpClient or WebClient class‌:‌ used to send HTTP requests and obtain web page content. ‌HttpClient provides more flexible functions and is suitable for handling complex HTTP requests. ‌ ‌ HTML parsing library‌:‌ such as HtmlAgilityPack,‌ used to parse the obtained HTML document and extract the required data from it. ‌HtmlAgilityPack supports XPath and CSS selectors, which is convenient for locating HTML elements. ‌ ‌ Regular expression‌:‌ used to match and extract specific text content in HTML documents, but attention should be paid to the accuracy and efficiency of regular expressions. ‌ ‌ Selenium WebDriver‌:‌ For scenarios that need to simulate browser behavior (such as logging in,‌ processing JavaScript rendered content),‌ Selenium WebDriver can be used to simulate user operations. ‌ JSON parsing library‌:‌ such as Json.NET,‌ used to parse JSON formatted data,‌ which is very useful when processing data returned by API. ‌ Exception handling and multithreading: In order to improve the stability and efficiency of the program, you need to write exception handling code and consider using multithreading technology to process multiple requests concurrently. ‌ Proxy and User-Agent settings: In order to bypass the anti-crawling mechanism of the website, you may need to set the proxy and custom User-Agent to simulate different access environments. ‌ HttpClient or WebClient class‌:‌ used to send HTTP requests and obtain web page content. ‌HttpClient provides more flexible functions and is suitable for handling complex HTTP requests. ‌ ‌ HTML parsing library‌:‌ such as HtmlAgilityPack,‌ used to parse the obtained HTML document and extract the required data from it. ‌HtmlAgilityPack supports XPath and CSS selectors, which is convenient for locating HTML elements. ‌ ‌ Regular expression‌:‌ used to match and extract specific text content in HTML documents, but attention should be paid to the accuracy and efficiency of regular expressions. ‌ ‌ Selenium WebDriver‌:‌ For scenarios that need to simulate browser behavior (such as logging in,‌ processing JavaScript rendered content),‌ Selenium WebDriver can be used to simulate user operations. ‌ JSON parsing library‌:‌ such as Json.NET ,‌ used to parse JSON formatted data,‌ which is very useful when processing data returned by API. ‌ Json.NET Exception handling and multithreading: In order to improve the stability and efficiency of the program, you need to write exception handling code and consider using multithreading technology to process multiple requests concurrently. ‌ Proxy and User-Agent settings: In order to bypass the anti-crawling mechanism of the website, you may need to set the proxy and custom User-Agent to simulate different access environments. ‌ The combination of these technologies and tools can efficiently implement the C# web crawling function. ‌ How to crawl dynamic web pages with C# combined with Selenium? ‌ How to use C# combined with Selenium to crawl dynamic web pages? ‌C# combined with Selenium to crawl dynamic web pages‌ Environment preparation‌:‌ Make sure that the C# development environment is installed. ‌ Install Selenium WebDriver, which is used to simulate browser behavior. ‌ Download and set up the browser driver, such as ChromeDriver, to ensure that it is consistent with the browser version. ‌ Make sure that the C# development environment is installed. ‌ Install Selenium WebDriver, which is used to simulate browser behavior. ‌ Download and set up the browser driver, such as ChromeDriver, to ensure that it is consistent with the browser version. ‌ Usage steps‌:‌ Import Selenium-related external libraries, such as WebDriver, WebDriverWait, etc. ‌ Initialize WebDriver, set up the browser driver, and open the target web page. ‌ Use the methods provided by Selenium to simulate user behaviors, such as clicking, input, scrolling, etc., to handle operations such as dynamically loading content or logging in. ‌ Parse the web page source code and extract the required data. ‌ Close the browser and WebDriver instance. ‌ Import Selenium-related external libraries, such as WebDriver, WebDriverWait, etc. ‌ Initialize WebDriver, set up the browser driver, and open the target web page. ‌ Use the methods provided by Selenium to simulate user behaviors, such as clicking, input, scrolling, etc., to handle operations such as dynamically loading content or logging in. ‌ Parse the web page source code and extract the required data. ‌ Close the browser and WebDriver instance. ‌ By combining C# with Selenium, you can effectively crawl dynamic web page content, handle complex interactions, and avoid being blocked by website detection. ‌‌ Conclusion In summary, C# and JavaScript each have their own advantages and disadvantages in web crawling. The choice of language depends on specific needs and development environment.

Web Scraping: Is C# or JavaScript the Superior Choice?

(This post originally appeared on AdatoSystems.com) Nobody cares about security. There. I said it. I said the thing everyone feels, some people think, but very few have the temerity to say out loud. But before you call me a blasphemous heathen, I will ask for just a few moments of your time to offer context. I even have some ideas and solutions. When I say “Nobody cares about security,” I mean it in the same way I remind folks that nobody cares about backups. What they care about is RESTORES. To quote my friend and colleague Tom LaRock, backups are worthless. Restores are priceless. Ultimately, the thing we need is the ability to get our data back if something catastrophic happens. This means we need A) the data itself and B) the ability to restore it. In order to have those things, we must first perform backups, and we’re forced to perform them regularly. (And, I can’t emphasize enough, that those backups must be tested. Otherwise, you have Schrodinger’s Backup ). Over the decades that the IT industry has existed, we’ve consistently improved both the technology and techniques of backups, from the “grandfather, father, son” rotation scheme of the 80’s and 90’s to the “3-2-1” rule first used in 2009 and onward. We’ve done so NOT because we care about the backups themselves, but the sheer frequency of unexpected and catastrophic data loss events serve as both object lessons and cautionary tales that drive our willingness to invest effort and money in backups nevertheless. So much so that, when some part of the organization questions the necessity (and the costs – from software to hardware to staff time) of backups, there’s usually a recent event that IT professionals can easily point to and say, “Remember how horrible that was? Remember the cost? We don’t want THAT to happen again, do we?” The ability to back up and efficiently restore data is an insurance policy that guards the company against even greater lost revenue. The Opposite of Love Isn’t Hate; It’s Apathy This brings me back to my original point: Nobody (i.e., business leaders) cares about security. What they care about is avoiding lost revenue due to application downtime, extortion, and lawsuits. Now, that’s a pretty hefty list, and one might think it’s more than enough to justify (almost) any cost of beefing up a company’s information security posture and capabilities. But look at that list of consequences again and consider it from a purely business point of view. Each of those concerns can be addressed in other ways, from built-in redundancy to cyber liability insurance. These mitigations are (or at least appear to be, from a business standpoint) less expensive than the investment needed to improve infosec. What’s worse, many of you reading this will notice that I left two commonly cited consequences off my initial list: fines and damage to reputation. I did that because the prevailing attitude among business leaders is: Damage to the company’s reputation SOUNDS bad, but (so the thinking goes) it’s really too amorphous to quantify. Plus, many companies in recent memory were the victims of massive cyber attacks, took a hit to their reputation or stock price, but saw it rebound a week later with no other ill effects. (again, that’s the belief. More on this later) The fines currently in place appear to be lower than the expected cost to improve the company’s security posture. However, the financial bottom line is only part of the reason businesses ignore their information needs. The other reasons are equally challenging to address: Infosec issues are, almost by their very nature, complex. They are certainly more complex than having up-to-date backups. They might be more complex than any other category of risk the company will face. Cyber threats are constantly changing and expensive to preemptively address. There’s usually no single “right” answer to “How much infosec do we need?” To put it another way, the broad answer to addressing both the issue of backups and infosec is that one needs to establish healthy habits. But the habits for backups are akin to remembering to brush your teeth in the morning. Important, but straightforward. Whereas the habits needed for proper information hygiene are akin to making healthy heart choices: the path to optimizing your cardiovascular health might involve not just one but a range of habits – from eating healthy foods (and avoiding unhealthy ones) to getting the right amount of exercise and sleep. But even within those broad instructions lie a range of nuances. What, exactly, are “healthy foods”? (Organic? Low-starch? Unprocessed? Keto friendly?) What type of exercise will be most effective? (Cardio? Weight training? Crossfit?) Underlying all of those variations is the very real chance that none of it may help – not because the person isn’t committed, but because there are so many other factors – from age to genetics to individual metabolism – that it still might not work. And even after adjusting for every single other variable, there’s the genuine possibility that a person can do everything right and still succumb to some other disease or physical failure. That last point – that despite our health goals, we may still have health problems – is far closer to the reality of infosec than many readers might realize. A few years ago, Cisco CEO John Chambers stated, “There are only two types of companies: those that have been hacked and those who don’t know they’ve been hacked yet.” Abe Silber, CEO at CyberCure.com, succinctly identified the core issue: “The problem with security is that it’s impossible to measure your ROI. Even if we can measure the cost of a security incident (not an easy task) it’s almost impossible to measure the likelihood of preventing them (hence ROI) based on different security solutions. So it’s not that business people don’t care. It’s that we (IT practitioners) have no way to show them what they get for their money. If I spend x amount on honeypots, how less likely am I to get hacked?
(We can try to make the case that) Security helps you not lose money, like insurance. But unlike insurance, it’s not guaranteed protection. I can tell you firsthand that it’s easier to sell a CEO a $25,000 cyber insurance policy than a $10,000 security solution.” This Is Only the Beginning. In my next post, I’ll start to propose a way forward – a set of actionable decisions you (and your company) can make to get people to not only care about security but to support and participate in healthy security behaviors. In the meantime, if you want to share your own ideas, I’d love to hear them in the comments below. (This post originally appeared on AdatoSystems.com) (This post originally appeared on AdatoSystems.com ) AdatoSystems.com Nobody cares about security. There. I said it. I said the thing everyone feels, some people think, but very few have the temerity to say out loud. But before you call me a blasphemous heathen, I will ask for just a few moments of your time to offer context. I even have some ideas and solutions. When I say “Nobody cares about security,” I mean it in the same way I remind folks that nobody cares about backups. What they care about is RESTORES. To quote my friend and colleague Tom LaRock, backups are worthless. Restores are priceless. are worthless. Restores are priceless. Ultimately, the thing we need is the ability to get our data back if something catastrophic happens. This means we need A) the data itself and B) the ability to restore it. In order to have those things, we must first perform backups, and we’re forced to perform them regularly. (And, I can’t emphasize enough, that those backups must be tested. Otherwise, you have Schrodinger’s Backup ). get our data back get our data back Schrodinger’s Backup Over the decades that the IT industry has existed, we’ve consistently improved both the technology and techniques of backups, from the “grandfather, father, son” rotation scheme of the 80’s and 90’s to the “3-2-1” rule first used in 2009 and onward. We’ve done so NOT because we care about the backups themselves, but the sheer frequency of unexpected and catastrophic data loss events serve as both object lessons and cautionary tales that drive our willingness to invest effort and money in backups nevertheless. “grandfather, father, son” “3-2-1” So much so that, when some part of the organization questions the necessity (and the costs – from software to hardware to staff time) of backups, there’s usually a recent event that IT professionals can easily point to and say, “Remember how horrible that was? Remember the cost? We don’t want THAT to happen again, do we?” The ability to back up and efficiently restore data is an insurance policy that guards the company against even greater lost revenue. and efficiently restore data The Opposite of Love Isn’t Hate; It’s Apathy This brings me back to my original point: Nobody (i.e., business leaders) cares about security. What they care about is avoiding lost revenue due to application downtime, extortion, and lawsuits. Now, that’s a pretty hefty list, and one might think it’s more than enough to justify (almost) any cost of beefing up a company’s information security posture and capabilities. But look at that list of consequences again and consider it from a purely business point of view. Each of those concerns can be addressed in other ways, from built-in redundancy to cyber liability insurance . These mitigations are (or at least appear to be, from a business standpoint) less expensive than the investment needed to improve infosec. cyber liability insurance What’s worse, many of you reading this will notice that I left two commonly cited consequences off my initial list: fines and damage to reputation. I did that because the prevailing attitude among business leaders is: Damage to the company’s reputation SOUNDS bad, but (so the thinking goes) it’s really too amorphous to quantify. Plus, many companies in recent memory were the victims of massive cyber attacks, took a hit to their reputation or stock price, but saw it rebound a week later with no other ill effects. (again, that’s the belief. More on this later) Damage to the company’s reputation SOUNDS bad, but (so the thinking goes) it’s really too amorphous to quantify. Plus, many companies in recent memory were the victims of massive cyber attacks, took a hit to their reputation or stock price, but saw it rebound a week later with no other ill effects. (again, that’s the belief. More on this later) The fines currently in place appear to be lower than the expected cost to improve the company’s security posture. The fines currently in place appear to be lower than the expected cost to improve the company’s security posture. However, the financial bottom line is only part of the reason businesses ignore their information needs. The other reasons are equally challenging to address: Infosec issues are, almost by their very nature, complex. They are certainly more complex than having up-to-date backups. They might be more complex than any other category of risk the company will face. Cyber threats are constantly changing and expensive to preemptively address. There’s usually no single “right” answer to “How much infosec do we need?” To put it another way, the broad answer to addressing both the issue of backups and infosec is that one needs to establish healthy habits. But the habits for backups are akin to remembering to brush your teeth in the morning. Important, but straightforward. Whereas the habits needed for proper information hygiene are akin to making healthy heart choices: the path to optimizing your cardiovascular health might involve not just one but a range of habits – from eating healthy foods (and avoiding unhealthy ones) to getting the right amount of exercise and sleep. But even within those broad instructions lie a range of nuances. What, exactly, are “healthy foods”? (Organic? Low-starch? Unprocessed? Keto friendly?) What type of exercise will be most effective? (Cardio? Weight training? Crossfit?) Underlying all of those variations is the very real chance that none of it may help – not because the person isn’t committed, but because there are so many other factors – from age to genetics to individual metabolism – that it still might not work. And even after adjusting for every single other variable, there’s the genuine possibility that a person can do everything right and still succumb to some other disease or physical failure. That last point – that despite our health goals, we may still have health problems – is far closer to the reality of infosec than many readers might realize. A few years ago, Cisco CEO John Chambers stated, “There are only two types of companies: those that have been hacked and those who don’t know they’ve been hacked yet.” “There are only two types of companies: those that have been hacked and those who don’t know they’ve been hacked yet.” Abe Silber, CEO at CyberCure.com , succinctly identified the core issue: Abe Silber, CEO at CyberCure.com “The problem with security is that it’s impossible to measure your ROI. Even if we can measure the cost of a security incident (not an easy task) it’s almost impossible to measure the likelihood of preventing them (hence ROI) based on different security solutions. So it’s not that business people don’t care. It’s that we (IT practitioners) have no way to show them what they get for their money. If I spend x amount on honeypots, how less likely am I to get hacked? (We can try to make the case that) Security helps you not lose money, like insurance. But unlike insurance, it’s not guaranteed protection. I can tell you firsthand that it’s easier to sell a CEO a $25,000 cyber insurance policy than a $10,000 security solution.” “The problem with security is that it’s impossible to measure your ROI. Even if we can measure the cost of a security incident (not an easy task) it’s almost impossible to measure the likelihood of preventing them (hence ROI) based on different security solutions. So it’s not that business people don’t care. It’s that we (IT practitioners) have no way to show them what they get for their money. If I spend x amount on honeypots, how less likely am I to get hacked? “The problem with security is that it’s impossible to measure your ROI. Even if we can measure the cost of a security incident (not an easy task) it’s almost impossible to measure the likelihood of preventing them (hence ROI) based on different security solutions. So it’s not that business people don’t care. It’s that we (IT practitioners) have no way to show them what they get for their money. If I spend x amount on honeypots, how less likely am I to get hacked? (We can try to make the case that) Security helps you not lose money, like insurance. But unlike insurance, it’s not guaranteed protection. I can tell you firsthand that it’s easier to sell a CEO a $25,000 cyber insurance policy than a $10,000 security solution.” (We can try to make the case that) Security helps you not lose money, like insurance. But unlike insurance, it’s not guaranteed protection. I can tell you firsthand that it’s easier to sell a CEO a $25,000 cyber insurance policy than a $10,000 security solution.” This Is Only the Beginning. In my next post, I’ll start to propose a way forward – a set of actionable decisions you (and your company) can make to get people to not only care about security but to support and participate in healthy security behaviors. In the meantime, if you want to share your own ideas, I’d love to hear them in the comments below.

Here's the Truth: Nobody Cares About Security

Mastering Dynamic Web Scraping

ডায়নামিক ওয়েব স্ক্র্যাপিং আয়ত্ত করা

Dynamisches Web Scraping beherrschen

Dominar el raspado web dinámico

Maîtriser le Web Scraping dynamique

गतिशील वेब स्क्रैपिंग में निपुणता

動的ウェブスクレイピングをマスターする

동적 웹 스크래핑 마스터하기

Dominando o Web Scraping Dinâmico

Освоение динамического парсинга веб-страниц

Dinamik Web Scraping'de Uzmanlaşmak

Làm chủ việc quét web động

掌握动态网页抓取

Stories, News & Mentions about XPATH

Xpath’s Stories on HackerNoon