A Guide to Scraping HTML Tables with Pandas and BeautifulSoup

It’s very common to run into HTML tables while scraping a webpage, and without the right approach, it can be a little tricky to extract useful, consistent data from them. In this article, you’ll see how to perform a quick, efficient scraping of these elements with two main different approaches: using only the Pandas library and using the traditional scraping library BeautifulSoup. As an example, I scraped the Premier League classification table. This is good because it’s a common table that can be found on basically any sports website. Although it makes sense to inform you this, the table being is scraped won’t make much difference while you read as I tried to make this article as generalistic as possible. pandas.read_html(): The Shortcut If all you want is to get some tables from a page and nothing else, you don’t even need to set up a whole scraper to do it as can get this job done by itself. The pandas.read_html() function uses some scraping libraries such as and to return a list containing all the tables on a page as DataFrames. You just need to pass the URL of the page. Pandas BeautifulSoup Urllib dfs = pd.read_html(url) All you need to do now is to select the DataFrame you want from this list: df = dfs[ 4 ] If you’re not sure about the order of the frames in the list or if you don’t want your code to rely on this order (websites can change), you can always search the DataFrames to find the one you’re looking for by its length… the_one = df for df in dfs: if len (df) == 20 : break … or by the name of its columns, for example. the_one = df for df in dfs: if df.columns == [ '#' , 'Team' , 'MP' , 'W' , 'D' , 'L' , 'Points' ]: break But Pandas isn’t done making our lives easier. This function accepts some helpful arguments to help you get the right table. You can use to specify a string o regex that the table should match; header to get the table with the specific headers you pass; the parameter allows you to identify the table by its class or id, for example. match attrs However, if you’re not scraping only the tables and are using, let’s say, Requests to get the page, you’re encouraged to pass to the function instead of the URL: page.text dfs = pd.read_html(page.text) page = requests.get(url) soup = BeautifulSoup(page.text, 'html.parser' ) The same goes if you’re using Selenium’s web driver to get the page: dfs = pd.read_html(driver.page_source) That’s because by doing this you’ll significantly reduce the time your code takes to run since the read_html() function does not need to get the page anymore. Check the average time elapsed for one hundred repetitions in each scenario: Using page.text: Using the URL: Average time elapsed: 0.2345 seconds Average time elapsed: 0.0774 seconds Using the URL made the code about three times slower. So it only makes sense to use it if you’re not going to get the page first using other libraries. Getting the Table’s Elements with BeautifulSoup Although Pandas is really great, it does not solve all of our problems. There will be times when you’ll need to scrape a table element-wise, maybe because you don’t want the entire table or because the table’s structure is not consistent or for whatever other reason. To cover that, we first need to understand the standard structure of an HTML table: . . . Where stands for “table row”, stands for “table header” and stands for “table data”, which is where the data is stored as text. tr th td The pattern is usually helpful, so all we have left to do is select the correct elements using BeautifulSoup. The first thing to do is to find the table. The method returns a list of all elements that satisfied the requirements we pass to it. We then must select the table we need in that list: find_all() table = soup.find_all( 'table' )[ 4 ] Depending on the website, it will be necessary to specify the table class or id, for instance. The rest of the process is now almost intuitive, right? We just need to select all the tags and the text in the and tags inside them. We could just use again to find all the tr tags, yes, but we can also iterate over these tags in a more straightforward manner. tr th td find_all() The attribute returns an iterable object with all the tags right beneath the parent tag, which is table, therefore it returns all the tr tags. As it’s an iterable object, we need to use it as such. children After that, each is a tag. We just need to extract the text of each tag inside it. Here’s the code for all this: child tr td for child in soup.find_all( 'table' )[ 4 ].children: for td in child: print (td.text) And the process is done! You then have the data you were looking for and you can manipulate it the way it best suits you. Other Possibilities Let’s say you’re not interested in the table’s header, for instance. Instead of using , you could select the first tag, which contains the header data, and use the attribute. This, just like the attribute, will return an iterable, but with all the other tags, which are the of the first one we selected. You’d be then skipping the header of the table. children tr next_siblings children tr siblings for sibling in soup.find_all( 'table' )[ 4 ].tr.next_siblings: for td in sibling: print (td.text) Just like children and the next siblings, you can also look for the previous siblings, parents, descendants, and way more. The possibilities are endless, so make sure to check the to find the best option for your scraper. BeautifulSoup documentation A Real-Life Example We’ve so far written some very straightforward code to extract HTML tables using Python. However, when doing this for real you’ll, of course, have some other issues to consider. For instance, you need to know how you’re going to store your data. Will you directly write it in a text file? Or will you store it in a list or in a dictionary and then create the file? Or will you create an empty DataFrame and fill it with the data? There certainly are lots of possibilities. My choice was to store everything in a big list of lists that will be later transformed into a DataFrame and exported as a file. .csv .csv In another subject, you might want to use some and clauses in your code to make it prepared to handle some exceptions it may find along the way. Of course, you’ll also want to insert some random pauses in order not to overload the server and avoid getting blocked. try except In this example, I scraped the Premier League table after every round in the entire 2019/20 season using most of what I’ve covered in this article. This is the entire code for it: page = requests.get(round_url) rows = [] row = [] rows.append(row) import pandas as pd import numpy as np import requests from bs4 import BeautifulSoup from time import sleep def get_table ( round , url=url ): round_url = f' {url} / { round } ' soup = BeautifulSoup(page.text, 'html.parser' ) for child in soup.find_all( 'table' )[ 4 ].children: for td in child: try : row.append(td.text.replace( '\n' , '' )) except : continue if len (row) > 0 : df = pd.DataFrame(rows[ 1 :], columns=rows[ 0 ]) return df for round in range ( 1 , 39 ): table = get_table( round ) table.to_csv( f'PL_table_matchweek_ { round } .csv' , index= False ) sleep(np.random.randint( 1 , 10 )) Everything is there: gathering all the elements in the table using the children attribute, handling exceptions, transforming the data into a DataFrame, exporting a .csv file, and pausing the code for a random number of seconds. After all this, all the data gathered by this code produced this interesting chart: You’re not going to find the data needed to plot a chart like that waiting for you on the internet. But that’s the beauty of scraping: you can go get the data yourself! As a wrap this up I hope was somehow useful and that you never have problems when scraping an HTML table again. If you have a question, a suggestion, or just want to be in touch, feel free to contact me through , , or . Twitter GitHub Linkedin Thanks for reading! Also published here

A Guide to Scraping HTML Tables with Pandas and BeautifulSoup

Too Long; Didn't Read

Companies Mentioned

@otavioss

RELATED STORIES