Web Automation with Python and Selenium

Web automation is one of the best ways companies can test a product in development, especially the app's functionalities, such as clicking, scrolling, and other actions. Essentially, web automation is about mimicking human action as it is crucial to ensure the software works across all device types of users. In this article, we will learn how to use Selenium as an automation tool to test a website with Python and automate the entire process without using a mouse or keyboard on the browser. Selenium doesn’t work with Python alone but with many other programming languages. What is Selenium Used For? Selenium allows us to browse or use a browser without a human involved and automate processes through code, such as typing into a user input and interacting with the website. For example, automating form submissions with Selenium is possible. Selenium does everything all by itself without a single click from a human. Getting Started Before Selenium can carry out any action through code, we need the packages and tools to be able to run browser automation and make it seamless. Installation This section, we install the Selenium package and the WebDriver executable file. First, check the version of the Chrome browser so that it is equivalent to the same version of the WebDriver exec file. The following needs to be installed on our local machine: The Chrome browser. Download it here Download the latest stable release of the . On the ChromeDriver web page, check to confirm the version of Chrome matches the version of the ChromeDriver ChromeDriver Now, run this command to install Selenium in your terminal: pip install selenium Automating a Web Page To begin, create a new folder containing the file and the downloaded ChromeDriver exec files. Selenium requires the driver to interface with the chosen browser, which in this tutorial, we are using the Chrome browser. It is essential to place the ChromeDriver in the same directory as the Python file. The directory for the project should look like this: Another way to use the ChromeDriver in this project is to find the path location of the file and copy it, which should look like this: MacBook users: /Users/ /Desktop/chromedriver Windows users: C:\Users\<computer user name\Desktop\chromdriver The WebDriver is the bridge that bridges the Selenium code to work with the Chrome browser. The Chrome browser provides the bridge. Without it, automation will not be possible. Next, let’s copy and paste the following code into the file: automation.py service = Service(chrome_drive_path) driver = webdriver.Chrome(service=service) driver.get(url) drive.close() # automation.py from selenium import webdriver from selenium.webdriver.chrome.service import Service from selenium.webdriver.common.by import By from selenium.webdriver.common.keys import Keys chrome_driver_path = './chromedriver' url = "https://en.wikipedia.org/wiki/Main_Page" article_count = driver.find_element(By.CSS_SELECTOR, "#articlecount a" ) print (article_count.text) The following occurs in the code snippet above as we will use Selenium to navigate Wikipedia: Import the function from the selenium module webdriver The module provides all the WebDriver implementations, and the Service object is for handling the browser driver selenium.webdriver The class is used to locate elements within the document of a web page using its CSS classes, ids, and so on By The Keys class simulates the actions from the keypress of the keyboard like the RETURN / Enter, F1, Alt, and so on Assign the chromedriver exec file to a variable Call the executable path of the driver to the service object Next is to instantiate the Chrome browser with the with the service argument webdriver.Chrome() Define the url of the web page to check Using the method will navigate to the page given by the url .get() On the Wikipedia page, open the inspect element to access the elements we will use to get the details from the page for automation. Now to the task of finding the article count. WebDriver offers us a way to do so by targeting the element using the find_element method and using the as the first parameter with the selector name, By.CSS_SELECTOR #articlecount a. Before we run the Python program, the script has the method, , that closes the browser window after running the test. driver.close() Run the program with the command: python automation.py The result from the terminal should display the article count, which shows the exact figure on the web page, . 6,545,457 Simulating a Click Action From what we have done so far, it is evident that Selenium navigates a website without the use or click of a mouse. Now, let’s simulate an action where the browser reacts to a click of a link on the page. Update the with these lines of code: automation.py contact_us.click() # automation.py # import modules contact_us = driver.find_element(By.LINK_TEXT, "Contact us" ) With this code snippet, we are finding the element to the link text, “Contact us” with the . After locating the elements, attach the to an action with method. By.LINK_TEXT contact_us .click() Again, we run the command: python automation.py The command above gives the result of the contact us page. Search a Web Page There are so many use cases with Selenium. Searching a web page can be achieved with automation, which gives you the result for a search word. Still, in the same file, automation.py, update the file with this code: # automation.py search = driver.find_element(By.NAME, "search" ) search.send_keys( "React" , Keys.ENTER) The code above does a similar thing to the previous by using the method with the input's name attribute, search. find_element Also, the simulate pressing the ENTER key, searching the keyword . send_keys React The result of the page should look like this: Conclusion This article is an overview of how to use Selenium in Python to automate the web, especially for professional developers who want to know if there are errors in the code and how to fix them. The possibilities of using Selenium with Python for web automation are endless. Some use cases include filling out an online form, applying for LinkedIn jobs, playing a clicker game, and so on. Do you know other use cases for using Selenium? Kindly share your favorite use case in the comments below. Learn More Selenium documentation XPath Tutorial