Hi everyone. In this article we are going to talk about how can you write a simple web scraper and a little search application using well known existing technologies which you perhaps didn’t know they can do that. From practical point of view the “product” we will have in the end will be barely capable of being used in production for mass web crawling, but if you just need to be able to crawl your own site or a site of your competitor or someone else’s and want to use an advanced search syntax and not just grep — this article should be useful for you. The article should be also useful for those who are just starting with docker-compose or Manticore Search. TL;DR Our solution will be based on: wget crawling a site recursively tiny php script to pass the crawled content from wget to Manticore Search tiny php script to make a minimalistic search UI all wrapped in docker and docker-compose Once we are done you should be able to run it via docker-compose like this: domain=who.int docker-compose up which will start crawling and indexing https://who.int and will immediately run another container with a web server, so you can search in the crawled pages: Technologies So what technologies will we use in our solution? Wget Everyone probably knows wget. When you need to download something in terminal in Linux, FreeBSD or MacOS most likely you will use wget. But did you know that wget can not just download a single file, but can be easily used as a simple web crawler which respects robots.txt, can follow links and doesn’t overload your system? Well if not, you know now. Yes, it doesn’t come with a load distribution among a network of your crawling servers or even ability to do searches in parallel. It’s actually not scalable at all, but it’s simple and it’s tried and trusted tool which suits our idea very well since the whole job can be done in just one call of the wget: wget -nv -r -H -nd --connect-timeout= --read-timeout= --tries= --follow-tags=a -R --domains=${domain} | php load.php 2 10 1 "*.css*,*.js*,*.png,*.jpg,*.gif" "http://${domain}/" Let’s go through the most important parameters: disables verbosity since we don’t need it in wget’s output which will be parsing -nv turns on recursive retrieving. Obviously the most important part for us -r enables spanning across hosts when doing recursive retrieving -H disables creating directories when retrieving recursively -nd limits the HTML tags to follow by just the hyperlink tag <a> --follow-tags=a lists patterns to ignore. Obviously we don’t need any images or css/js files for full-text search, so we are ignoring them -R "*.css*,*.js*,*.png,*.jpg,*.gif" is our starting point. It will be the first page wget will download "http://${domain}/" lets us define the domains to be followed. In our case we are limiting by the same domain we are crawling --domains=${domain} and after all we want to pipe wget’s output to load.php | php load.php load.php This is a simple and straightforward 15 lines of code script which: makes a connection to Manticore Search using a MySQL library creates a new table if it doesn’t exist yet with the morphology settings we need reads info about downloaded pages from wget at STDIN reads each page and puts it to Manticore Here is the full script with each line commented: $f = fopen( , ); $manticore = mysqli( , , , , ); $manticore->query( ); (!feof($f)) { $s = fgets($f); (!preg_match( , $s, $match)) ; { $content = @file_get_contents( .$match[ ]); usleep( ); } (!$content); (preg_match( , $content, $content_match)) $title = trim(html_entity_decode($content_match[ ])); ; .strlen($content). ; $manticore->query( .crc32($title). .$manticore->escape_string($title). .$manticore->escape_string($match[ ]). .$manticore->escape_string($content). ); } <?php 'php://stdin' 'r' # we'll be waiting for data at STDIN new 'manticore' '' '' '' 9306 # let's connect to Manticore Search via MySQL protocol "CREATE TABLE IF NOT EXISTS rt(title text, body text, url text stored) html_strip='1' html_remove_elements='style,script,a' morphology='stem_en' index_sp='1'" /* creating a table "rt" if it doesn't exist with the following settings:
- html_strip='1': stripping HTML is on
- html_remove_elements='style,script,a': for HTML tags <style>/<script>/<a> we don't need their contents, so we are stripping them completely
- morphology='stem_en': we'll use English stemmer as a morphology processor
- index_sp='1': we'll also index sentences and paragraphs for more advanced full-text search capabilities and better relevance
*/ while # reading from STDIN while there's something /* getting one line. Here is an example of wget returns:
    2020-04-08 07:39:33 URL:https://www.who.int/westernpacific/ [98667/98667] -> "index.html.3" [1]
    which means that:
    - the original URL was https://www.who.int/westernpacific/
    - that it saved the contents to index.html.3
    */ if '/URL:(?<url>http.*?) \[.*?\] -> "(?<path>.*?)"/' continue # if wget returns smth else we are just skipping it, otherwise we use regexp to put the url and the path to $match do # it may be that wget returns the info about a download earlier than the file appears, so we are looping until can read from the file: './' 'path' # reading from the file 10000 # sleeping 10 milliseconds while # end the loop when we have the content if '/<title>(?<title>.*?)<\/title>/is' 'title' # here we are doing a simple HTML page parsing to get <title> from that else continue # we are not interested in pages without a title echo "{$match['path']}: $title {$match['url']} " " bytes\n" # let's say something about our progress "REPLACE INTO rt (id,title,url,body) VALUES(" ",'" "','" 'url' "','" "')" # and we are finally putting the contents to Manticore. We use crc32(title) as a document ID to avoid duplicates. # and we are going back to the next page wget reports as downloaded So as soon as wget downloads at least something it will appear in Manticore immediately and will be searchable. Your data collection will grow until wget can’t download anything else or until you stop the container. Manticore Search Another important component is . Manticore Search Manticore is a lightweight database written in C++ created specifically for search purposes with a powerful full-text search capabilities It can speak SQL over MySQL protocol as well as JSON over HTTP. What’s important for our purpose is that: it can strip HTML it has built-in NLP capabilities so we can split our texts into words, sentences and paragraphs efficiently and use stemmed forms of words (so e.g. “running” will find “run” etc.) it’s doesn’t require any configuration at all by default so we can just use 2 SQL queries: one to create a new table and another to add a new document to it official docker image it starts in milliseconds and is very cost-efficient in terms of RAM. No java heap which takes all your memory or garbage collection which ruins your search performance adding a document requires just one line of code to make an SQL query So all we need to do to hook up Manticore in our case is these 3 lines in docker-compose.yml: : : : manticoresearch/ : . services manticore image manticore 3.4 0 Docker compose file is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration Docker Compose Our docker-compose YAML looks like this: : '2.2' version services: # Manticore Search is a small yet powerful database for search with awesome full-text search capabilities manticore: # we'll just use their official image image: manticoresearch/manticore:3.4.0 # and create a volume for data persistency volumes: - ./data:/var/lib/manticore # we also need php php: # which we'll build ourselves from Dockerfile build: php # no point to run the php container before manticore, hence the dependency depends_on: - manticore # the command below just runs wget to start crawling the domain passed in the env. variable # and lets the wget output flow to "php load.php" which insert into into Manticore Search command: /bin/bash -c 'wget -nv -r -H -nd --connect-timeout=2 --read-timeout=10 --tries=1 --follow-tags=a -R "*.css*,*.js*,*.png,*.jpg,*.gif" "http://${domain}/" --domains=${domain} 2>&1 | php load.php' # let's also add a tiny php script to visualize what we have in Manticore web: # we'll use php 7.2. + Apache for that image: php:7.2-apache # it also depends on Manticore depends_on: - manticore # let's bind it to 8082 port locally ports: - 8082 :80 # we'll mirror folder "www" to /var/www/html/ inside the web server container so ./www/index.php will be the front page volumes: - ./www/:/var/www/html/ and there is also a Dockerfile for php+wget+mysql extension: php: -cli # Let's take php 7.4 as a base image FROM 7.4 # We'll also install wget and PHP mysqli extension RUN apt-get update \
&& apt-get -y install wget \
&& docker-php-source extract \
&& docker-php-ext-install mysqli \
&& docker-php-source delete # We'll use load.php, so we need to copy it to the image COPY load.php /usr/src/myapp/ # And let's change the working dir WORKDIR /usr/src/myapp Please go through the comments in them. In a nutshell it includes 3 services: manticore: just using the official image php: we build it ourselves from php/Dockerfile php+wget+mysqli extension and we copy the load.php script to it — from Dockerfile. Depends on manticore web: from php+apache official image. Depends on manticore Feel free to override the port from 8082 to whatever you want. We also use the environment variable to specify the domain to crawl. So when you run it like this: $domain domain=who.int docker-compose up it runs the above 3 services and starts crawling: snikolaev@dev:~/crawler$ domain=who.int docker-compose up
Starting crawler_manticore_1 … Recreating crawler_web_1 … Starting crawler_php_1 … ...
php_1        | data.5: GHO https://www.who.int/data/gho 125537 bytes
php_1        | fact-sheets.4: Fact sheets https://www.who.int/news-room/fact-sheets 83345 bytes
php_1        | facts-in-pictures.3: Facts pictures https://www.who.int/news-room/facts-in-pictures 70227 bytes
php_1        | publications.7: WHO | Publications https://www.who.int/publications/en/ 92069 bytes
php_1        | questions-answers.3: WHO | Online Q&A https://www.who.int/features/qa/en/ 78145 bytes
php_1        | popular.3: Health topics https://www.who.int/health-topics/ 123263 bytes
php_1        | ebola-virus-disease.8: Ebola virus disease https://www.who.int/health-topics/ebola/ 112116 bytes done done done in Search bar The last component we haven’t covered yet is index.php which runs when you open (or another port if you changed it in the compose file). The full script is just 13 lines of code: http://hostname:8082 <form><h1>Manticore</h1><input name= type= style= value= ></form>
<hr> ( ($_GET[ ])) { $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, ); curl_setopt($ch, CURLOPT_POST, ); curl_setopt($ch, CURLOPT_RETURNTRANSFER, ); curl_setopt($ch, CURLOPT_POSTFIELDS, ); ($json = json_decode(curl_exec($ch))) { ($json->data $result) ; }
} "search" "text" "width: 50%; border: 1px solid" "<?=$_GET['search']?>" <?php if isset 'search' # we have a search request, let's process it # initializing curl "http://manticore:9308/sql" # we'll connect to Manticore's /sql endpoint via HTTP. There's also /json/search/ which gives much more granular control, but for the sake of simplicity we'll use the /sql endpoint 1 # we'll send via POST true # we need the response back, don't output it "mode=raw&query=SELECT url, highlight({}, 'title') title, highlight({}, 'body') body FROM rt WHERE MATCH('{$_GET['search']}') LIMIT 10" /* here we are SELECTing :
 - url
 - highlighted title
 - highlighted body
 - from the index called "rt"
 - we want all documents that MATCH() our search query
 - and we need only the first 10, hence LIMIT 10
*/ if # running the query and decoding the JSON foreach as echo "<small>{$result->url}</small><br><a href=\"{$result->url}\">{$result->title}</a><br>{$result->body}<br><br>" # and here we just output the results: url, title and body Here unlike load.php we connect to Manticore over HTTP and use it’s JSON api endpoint /sql which allows to transmit any SQL command over HTTP. In a production environment it might make more sense to use Manticore’s endpoint which allows to break down the request into pieces much more granularly which is often important if your search form is not just one text area, but multi-field or in other cases. But we don’t need that all now. The logic of the script is simple: /json/search render a simple search form with text area named “search” if you press enter the form sends the typed value as an http parameter “search” then the script just takes the value and passes it to Manticore in a very compact and clear SQL query: SELECT url, highlight({}, 'title') title, highlight({}, 'body') body FROM rt WHERE MATCH(' ') LIMIT 10"); {$_GET['search']} gets the results and renders them as HTML That’s it. Nothing complicated. What can it do? Let’s now see what we can do with what we’ve built. Why didn’t we just dump wget output to files and use to search in them? Here is why: grep Not just our search engine finds what matches your query, but it using improved ranking formula similar to . For example as you can see on this picture the results containing “IPC precaution recommendations” go first since they have the whole phrase: highlights the results and sorts them properly BM25 Second, you can use to do many interesting things. For example you might want to find only those documents that have “covid” and “caught” in the same or : Manticore’s extended query syntax sentence paragraph Or you can match by a whole phrase, use OR and NOT and many more. Third, do you remember when we were doing CREATE TABLE we turned on English stemming? Here is how we can now use it — if I enter “coronaviruses” it finds just “coronavirus” too: So even though the crawling part is very basic the search part of our solution is quite powerful. You definitely can’t do anything like this with wget. How do I run it myself? git https://github.com/manticoresoftware/demos.git manticore_demos manticore_demos/crawler/
domain=who.int docker-compose up clone cd If you run it first time you’ll have to wait few minutes for docker to download the images to build the php service. Afterwards it will start crawling (or another domain you specify), and the search UI will be available at unless you run it on a remote server. http://who.int http://localhost:8082 Thanks for reading! You can access the code . here

Flow

Apache

2022 - HackerNoon Contributor of the Year - Database

Read My Stories

Nominated for 2022 - HackerNoon Contributor of the Year - Database

Too Long; Didn't Read

Cassandra Forward a free Cassandra-focused community event

Build A Web Crawler with Search bar Using Wget and Manticore [A Step By Step Guide]

Too Long; Didn't Read

Companies Mentioned

Coin Mentioned

@snikolaev

RELATED STORIES