For my first ever wrote article, I've decided to write about web scraping and how I built a Stock Market Watcher using Ruby and Nokogiri Gem. First of all some requirements to follow this guide: Ruby HTML/CSS basics The purpose of this guide is to show what you can do using Ruby, probably out there you'll find better/professional ways to do this. The repository containing the complete code used on this guide will be linked at the end of the article. Let's begin First of all, set up your environment, create a ruby file that will contain our scrapper code. In this case, I'll create a file called stock_watcher.rb and I'll add the following code to it. #stock_watcher.rb require 'nokogiri' require 'open-uri' This code requires a Ruby Gem called (Help us to handle extracted data from the website) and the open-uri module to extract data from the web. Nokogiri After this, we need to define from where we're going to get the data for our scrapper, here you can use any website you wish, for this guide I'm going to use data. TradingView With the source defined now we need to get the data from the page, Nokogiri got some useful parsing methods to help us, we're going to use the parser since we're scraping an HTML page. Nokogiri::HTML Add the following code to stock_watcher.rb page = Nokogiri::HTML(URI.open( )) #stock_watcher.rb 'https://www.tradingview.com/markets/stocks-usa/market-movers-large-cap' This simple get the data from the web using the URI module and parse it using Nokogiri, it returns a . Nokogiri::HTML::Document With our page object, we have in our hands a bunch of useful methods. Now we have to use our HTML/CSS acknowledgment to define from where on the page we want to get the data. Chrome dev tools is a good friend here. The page in question is , this page shows us the US STOCK MARKET companies ordered by large capitalization. This page stores data on the main table as you can see, what we have to find is a way to identify this table. https://www.tradingview.com/markets/stocks-usa/market-movers-large-cap/ Nokogiri offers us a method called "css" that lets us search for specific CSS rules on our , we can search for specific CSS classes, ids, etc... Nokogiri::HTML::Document Let's try to find out what's the class or ID of the table using the Chrome inspect tool, just right click on any table element then inspect, you'll find a window similar to that one: Scroll a bit till you find the <table> tag like this: As you can see our table uses the "tv-data-table" class, that one is going to be our target class. Now we got our table class, add the following code to : stock_watcher.rb table = page&.css( ); #stock_watcher.rb '.tv-data-table' This code get our page object and find an HTML element with "tv-data-table" class assigned to it and returns a . Nokogiri::XML::NodeSet The only thing we need from the table is the table data, the table data (td) is stored inside the table rows (tr). offers more useful methods that allow us to search for specific keys inside it. Nokogiri::XML::NodeSet Add the following code to : stock_watcher.rb rows = table.search( ); #stock_watcher.rb 'tr' This returns another with a subset of keys and values from the table that contains 'tr'. Nokogiri::XML:NodeSet We're getting close to our goal. Now add the following code to : stock_wacher.rb rows.each stock_content = row.search( ).map { td.text.strip } stock_content[ ]&.gsub!( , ) do |row| 'td' |td| 0 "\n\t\t\t\t\t\t\t\t\t" " " end Now we iterate over our XML subset, in this case, it's our rows containing the data from stocks. Then we search in every row for the table data (td) using the same method search, it returns an array with all the data of the iterated row. We need to strip our text because Nokogiri returns a lot of useless pieces of text inside our strings. Now you can play a bit with that code, try to print the stock content you got using "puts stock_content" before the end keyword of the last code and see what you got, it should return something similar to this: MSFT Microsoft Corp. % Buy M B Technology Services "[" ", " 182.51 ", " -2.27 ", " -4.23 ", " ", " 31.952 ", " 1384.054 ", " 31.09 ", " 6.06 ", " 144000.00 ", " "]" With this data in hand who define what should be done next is you, there are thousands of possibilities of what could be done with this, I'll give a few examples: Store data on a file for later checking, store data on a database to handle it later even showing the data on another website (Check for permissions first), create triggers if a stock value is on a price that you want. As I said, there are a lot of possibilities. Now to finish this guide I've created a class to handle the data we got from our last piece of code. First, create another file on the same folder called stock.rb and add the following code: , , , , , , @name = stock_data[ ] @last = stock_data[ ] @change_percentage = stock_data[ ] @change_value = stock_data[ ] @rating = stock_data[ ] @vol = stock_data[ ] @market_cap = stock_data[ ] #stock.rb class Stock attr_reader :name :last :change_percentage :change_value :rating :vol :market_cap def initialize (stock_data) 0 1 2 3 4 5 6 end end I'm not going to explain how classes work on Ruby since that is not the purpose of this guide. But basically, that class receive the data we got on the "stack_content" variable and store it elegantly. Now replace your entire code on file to look like this: stock_watcher.rb require_relative page = Nokogiri::HTML(URI.open( )) table = page&.css( ); rows = table.search( ); stock_arr = [] rows.each stock_content = row.search( ).map { td.text.strip } stock_content[ ]&.gsub!( , ) stock_arr << Stock.new(stock_content) stock_content.length.positive? #stock_watcher.rb require 'nokogiri' require 'open-uri' 'stock' 'https://www.tradingview.com/markets/stocks-usa/market-movers-large-cap' '.tv-data-table' 'tr' do |row| 'td' |td| 0 "\n\t\t\t\t\t\t\t\t\t" " " if end It requires our new class file, creates a new array called stock_arr, and store every new stock object on it. If you print the first object on stock_arr you'll get something like this: MSFT Microsoft Corp. - % - Buy M B "#<Stock:0x00007fffd443e2f8 @name=" ", @last=" 182.51 ", @change_percentage=" 2.27 ", @change_value=" 4.23 ", @rating=" ", @vol=" 31.977 ", @market_cap=" 1384.054 ">" As you can see, it's a better way to store our stocks, and you can access its properties using dot notation, like stock.name, stock.last, stock.change_value, etc... Thank you for reading this guide, I hope it helps you in some way. Feel free to contact me on any social media listed on HackerNoon. Useful links: Complete code repository: https://github.com/kazumaki/stock-market-watcher TradingView website: https://www.tradingview.com/ Nokogiri documentation: https://nokogiri.org/rdoc/index.html
