Whether you're interested in penetration testing, OSINT, or bug bounty, this quick and easy Golang-based crawler is an essential addition to your recon toolbox. On a newly configured Ubuntu ARM64 virtual machine, we will install Hakrawler, run practical examples, and learn how to use it efficiently. Setup "Let's start by installing Hakrawler on a fresh Ubuntu installation." Terminal Commands: sudo apt update && sudo apt upgrade -y
sudo apt install golang git -y sudo apt update && sudo apt upgrade -y
sudo apt install golang git -y "First, we must install Go and Git. We'll use go install to install Hakrawler after that is finished. go install github.com/hakluke/hakrawler@latest go install github.com/hakluke/hakrawler@latest The binary can be found at ~/go/bin/hakrawler after installation. Let us permanently include it in our PATH. echo 'export PATH=$PATH:~/go/bin' >> ~/.bashrc
source ~/.bashrc echo 'export PATH=$PATH:~/go/bin' >> ~/.bashrc
source ~/.bashrc "Now, let's see if it works." hakrawler --help hakrawler --help "Awesome! The hakrawler is now set up and operational. Usage "Let's crawl our initial website. For testing, I'll use https://www.openexploit.in as a secure target. https://www.openexploit.in echo https://www.openexploit.in | hakrawler echo https://www.openexploit.in | hakrawler "Hakrawler begins crawling after reading URLs from standard input." Multiple URLs can also be entered from a file in the manner described here: cat urls.txt | hakrawler cat urls.txt | hakrawler "To illustrate that, let's make a brief urls.txt file." echo -e "https://httpbin.org\nhttps://openexploit.in" > urls.txt
cat urls.txt | hakrawler echo -e "https://httpbin.org\nhttps://openexploit.in" > urls.txt
cat urls.txt | hakrawler Bug Bounty-Style Recon An actual bug bounty recon chain might look like this. Assume that you are testing target.com. You use Hakrawler to crawl them after first gathering subdomains with Haktrails and filtering live hosts with httpx. echo openexploit.in | haktrails subdomains | httpx | hakrawler echo openexploit.in | haktrails subdomains | httpx | hakrawler "You receive a list of all subdomains, determine which are active, and then send them all to Hakrawler in a single line for crawling!" Options & Flgs "Hakrawler provides a number of helpful flags." Let's review a few that I use frequently. Subdomains: Subdomains echo https://www.openexploit.in | hakrawler -subs echo https://www.openexploit.in | hakrawler -subs Depth: Depth: echo https://example.com | hakrawler -subs echo https://example.com | hakrawler -subs HTTP Headers: HTTP Headers: echo https://www.openexploit.in | hakrawler -h "User-Agent: OpenExploitBot;;Referer: https://google.com" echo https://www.openexploit.in | hakrawler -h "User-Agent: OpenExploitBot;;Referer: https://google.com" JSON output: JSON output: echo https://www.openexploit.in | hakrawler -json echo https://www.openexploit.in | hakrawler -json Proxy(such as Burp Suite): Proxy(such as Burp Suite): echo https://www.openexploit.in | hakrawler -proxy http://127.0.0.1:8080 echo https://www.openexploit.in | hakrawler -proxy http://127.0.0.1:8080 "You have more control over how and what you crawl with each of these flags." Typical Problem and Solutions You may not always receive URLs back. This is typically the result of the domain redirecting to a subdomain, so don't panic. As an example: echo http://www.openexploit.in | hakrawler echo http://www.openexploit.in | hakrawler "You will not receive any results if this reroutes to https://www.openexploit.in unless you either add -subs to include subdomains or crawl the redirected URL directly." https://www.openexploit.in Bonus: Using Docker Would you rather use Docker or not install Go? Here's how to use Docker with Hakrawler. DockerHub: echo https://httpbin.org | docker run --rm -i hakluke/hakrawler:v2 -subs echo https://httpbin.org | docker run --rm -i hakluke/hakrawler:v2 -subs Local: git clone https://github.com/hakluke/hakrawler
cd hakrawler
sudo docker build -t hakluke/hakrawler .
echo https://httpbin.org | sudo docker run --rm -i hakluke/hakrawler -subs git clone https://github.com/hakluke/hakrawler
cd hakrawler
sudo docker build -t hakluke/hakrawler .
echo https://httpbin.org | sudo docker run --rm -i hakluke/hakrawler -subs Final Words It is quick, easy, and very useful for reconfiguring and finding endpoints quickly during bug bounty hunts or pentests. Until the next time, be safe and curious! Prefer watching instead of reading? Here’s a quick video guide Prefer watching instead of reading? Here’s a quick video guide Prefer watching instead of reading? Here’s a quick video guide Prefer watching instead of reading? Here’s a quick video guide https://youtu.be/eSR-1e3vMto?embedable=true https://youtu.be/eSR-1e3vMto?embedable=true

Mastering MCP Server Management with ToolHive

Read My Stories

Too Long; Didn't Read

Boost your HackerNoon story @ $159.99! 🚀

Bug Bounty Recon Made Easy with Hakrawler

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A Beginner’s Guide to Reconnaissance in PenTesting

Windows Sticky Keys Exploit: The War Veteran That Never Dies

06/02/2018: Biggest Stories in the Cryptosphere

0-Days are on the Rise and that Means a Lot More Work for SOC Teams

Time Bombs Inside Software: 0-Day Log4Shell is Just the Tip of The Iceberg

A Beginner’s Guide to Reconnaissance in PenTesting

Windows Sticky Keys Exploit: The War Veteran That Never Dies

06/02/2018: Biggest Stories in the Cryptosphere

0-Days are on the Rise and that Means a Lot More Work for SOC Teams

Time Bombs Inside Software: 0-Day Log4Shell is Just the Tip of The Iceberg

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps