Use Kali Linux Docker Containers to Support Covert Web Scraping

Unlock the power of Kali Linux on Docker networks by adding a GUI back to the container and level up information gathering with anonymized web scraping powered by Python, Selenium, and Tor Browser. As a DevOps Engineer, I have both shipped, and enabled other Engineers to ship, many hundreds of thousands of containers. I use Continuous Delivery and a variety of orchestration methods: Kubernetes, Lambda Containers on AWS, Elastic Container Service (ECS) backed by both Elastic Compute Cloud(EC2) and Fargate. Containers are useful for custom build systems, automated deployments, vulnerability scanning, research purposes, data analytics, machine learning, legacy software archival, local development, and web scraping. In this guide I will go over the steps I went through to add a GUI to the official using Xvfb and a VNC server. The code I discuss here is published in the open source repository, and is continuously delivered to for general use. Finally, in Part 2 of this series, everything in the containers will be put to the test. I will use a custom Python script, Pytest, and Selenium to automate the Tor Browser instance and gather data from the web. Kali Linux container from DockerHub kalilinux-xvfb Docker Hub Container Networks Part of the motivation behind having Kali Linux in a container is to gain visibility into the private network space created by the host machine for containers. The default setup from Docker Desktop is to use a NAT network. Unless the port(s) are bound to the host machine, with , that service will not be accessible from the host. docker run -p The Software Here are added summary sections with links for each of the following software packages: Kali Linux Docker Desktop Tor Browser Tor Browser Launcher Xvfb Selenium Additional links for software I used: Xfce 4 Tight VNC Server OpenSSH Server and Python3 Pip3 Geckodriver Kali Linux Kali Linux is a Debian-based Linux distribution aimed at advanced Penetration Testing and Security Auditing. 1 I chose Kali Linux for the sheer amount of security focused tools that are available. The Kali Linux team provides a Docker image that is updated regularly. Much of the required software is available to the Kali Linux Docker image and I will be using the package manager to install them. Docker Desktop is a framework for running containers locally. Containers are a way of packaging operating system and application state in a reusable operating system image. Additional resources are available on the . Docker Desktop Docker website Containers and virtual machines have similar resource isolation and allocation benefits, but function differently because containers virtualize the operating system instead of hardware. 2 Since Docker containers are more lightweight than virtual machines, I favor them heavily in my development efforts. Long term, this state can be stored as a small text file and is a lot easier to track changes over time with git and similar tools. The ability to derive an image again from a text file is powerful. However, it is entirely possible for a Dockerfile to fall into disrepair over time and could fail to build at a later date. Much like any other Operating System, and Infrastructure as Code, it is important to keep everything up to date. An alternative to Docker would be to use a Virtual Machine, or a bare metal installation. However, I believe containers are the perfect testbed for running ephemeral web crawlers because they are easy to provision and cleanup. I can run many containers on a single machine, in parallel, that can be started and stopped almost instantly. Then whenever I require additional isolation this whole setup can be run from inside a virtual machine as well. Tor Browser The Onion Router (Tor) is a global network of computers that route traffic through a series of encrypted hops. This allows for a user to browse the internet somewhat anonymously, and also allows for a user to access websites that are only available on the Tor Network. is a heavily configured version of Firefox with a focus on privacy and security. It can use the Tor Network out of the box. Tor Browser is included in many Linux operating systems and makes starting a web browser on the Tor Network extremely easily. torbrowser-launcher Selenium is a tool for automating web browsers. I went with Python in this example, but there are many languages and tools available. Selenium I will be using Selenium to automate this Tor Browser on the Tor Network. I am attempting to take advantage of the security conscious browser's existing config to limit sharing of information to targets. I will be able to clean up the containers runtime states by stopping them. The resulting WebDriver session will have a random IP address vended through Tor, making it a very powerful tool for web scraping. Xvfb The XOrg component can be used to simulate a screen to the X window system. I will then use this screen to start an X server in the container. This emulation technique is not particularly fast, and graphics heavy applications will benefit from a GPU. Xvfb Overview of the Files The Dockerfiles should build on any Linux, macOS, or Windows distribution that can run Docker. I have tested this on Docker Desktop using an Intel processor. There is an ARM variant for the official Kalilinux base container. I would appreciate feedback from anyone that tries running this on ARM. Dockerfile TorBrowser.Dockerfile TorBrowser-root.Dockerfile startx-once.sh start-vnc-server-once.sh ssh-keys.sh torrc .github/workflows/main.yml Dockerfile This is the Dockerfile that I used to build the base image. I have added a few comments to explain what each line does. # From the official Kali Linux Docker image rolling release FROM kalilinux/kali-rolling # Leaving this here for now, but it's not needed ARG DISPLAY_NUMBER=1 ENV DISPLAY=:$DISPLAY_NUMBER # Install the packages we need # Got lucky with this one, there was a ticket about this: # https://github.com/dnschneid/crouton/issues/4461 `xfce4 dbus-launch not found` in this version of the image so installing dbus-x11 RUN apt-get update -y -q \ && apt-get install -y -q xvfb xfce4 xfce4-goodies tightvncserver net-tools curl openssh-server dbus-x11 # This line copies the shell scripts from the local directory to the container, and makes them executable. COPY *.sh /opt/ RUN chmod 755 /opt/*.sh # This line runs the shell scripts that start the X server, and generate the SSH keys. RUN /opt/startx-once.sh RUN /opt/ssh-keys.sh # This line sets the entrypoint to bash, so that we can run commands in the container. ENTRYPOINT /bin/bash startx-once.sh Helps start the X server. I am using the Xvfb command to start the X server. If a PID is present, and when something is wrong with the X server, I can kill the process and start it again. I experimented with blocking X sessions from network, while it may be unnecessary, i am still adding it anyway: -nolisten tcp discussion on super user #!/bin/bash FILE=/opt/.x.pid if [[ -f "$FILE" ]]; then echo "$FILE exist" echo "PID: " `cat $FILE` else Xvfb -nolisten tcp $DISPLAY & echo $? > $FILE fi ssh-keys.sh VNC Clients are less than ideal for copying and pasting text. This can be remedied by adding SSH keys to the container to help other hosts connect to it. This opens up the possibility of using SSH tunnels to connect to the container. The following command was referenced from this ssh-keygen Stack Exchange Article #!/bin/bash # Generate SSH keys ssh-keygen -b 2048 -t rsa -f ~/.ssh/id_rsa -q -N "" cat ~/.ssh/id_rsa >> ~/.ssh/authorized_keys If I am going to use these keys, I need the private key on my workstation. I am using the following commands to copy the private key to my host machine from docker, but there are a multitude of ways to do this. There was a good discussion around this on . Stack Overflow Finding the container ID with the : ps command docker ps Using the container ID to copy the with the : cp command # For root docker cp :/root/.ssh/id_rsa /target/path/on/host # Otherwise for the user configured in the Dockerfile docker cp :/home/username/.ssh/id_rsa /target/path/on/host torrc This needs to be copied into the container, but is just uncommented sections from the boilerplate configuration the Tor Project provides . here # /etc/tor/torrc ## Tor opens a SOCKS proxy on port 9050 by default -- even if you don't ## configure one below. Set "SOCKSPort 0" if you plan to run Tor only ## as a relay, and not make any local application connections yourself. SOCKSPort 9050 # Default: Bind to localhost:9050 for local connections. #SOCKSPort 192.168.0.1:9100 # Bind to this address:port too. ## Uncomment this to start the process in the background... or use ## --runasdaemon 1 on the command line. This is ignored on Windows; ## see the FAQ entry if you want Tor to run as an NT service. RunAsDaemon 1 ## The port on which Tor will listen for local connections from Tor ## controller applications, as documented in control-spec.txt. ControlPort 9051 ## If you enable the controlport, be sure to enable one of these ## authentication methods, to prevent attackers from accessing it. #HashedControlPassword 16:872860B76453A77D60CA2BB8C1A7042072093276A3D701AD684053EC4C CookieAuthentication 1 TorBrowser.Dockerfile This Dockerfile builds on the previous Dockerfile, and adds Tor Browser and Python to the mix. I am using the package to install Tor Browser. torbrowser-launcher # From the base Dockerfile: FROM excitingtheory/kalilinux-xvfb ARG DISPLAY_NUMBER=1 ENV DISPLAY=:$DISPLAY_NUMBER # More info about torbrowser-launcher: # https://github.com/micahflee/torbrowser-launcher RUN apt-get update -y -q \ && apt-get install -y -q torbrowser-launcher python3 python3-pip # Create a user and add them to the sudo group, sudo still TBD but leaving for now, will not be able to switch users in this container. RUN useradd -s /bin/bash -m username \ && usermod -aG sudo username # Install python packages RUN pip3 install selenium pytest COPY torrc /etc/tor/torrc # Set the user and home directory for the container USER username ENV USER=username ENV HOME=/home/username # Run the shell scripts that start the X server, and generate the SSH keys. RUN /opt/startx-once.sh RUN /opt/ssh-keys.sh # This line sets the entrypoint to bash, so that we can run commands in the container. ENTRYPOINT /bin/bash TorBrowser-root.Dockerfile Just like before but with less configuration. Will use the root user. # From the base Dockerfile: FROM excitingtheory/kalilinux-xvfb ARG DISPLAY_NUMBER=1 ENV DISPLAY=:$DISPLAY_NUMBER # More info about torbrowser-launcher: # https://github.com/micahflee/torbrowser-launcher RUN apt-get update -y -q \ && apt-get install -y -q torbrowser-launcher python3 python3-pip # Install python packages RUN pip3 install selenium pytest # Copy the tor config file into the containera COPY torrc /etc/tor/torrc # Run the shell scripts that start the X server, and generate the SSH keys. RUN /opt/startx-once.sh RUN /opt/ssh-keys.sh # This line sets the entrypoint to bash, so that we can run commands in the container. ENTRYPOINT /bin/bash .github/workflows/main.yml The Github Actions workflow file builds the Docker image and pushes it to Docker Hub. This is the stock workflow from , with additional tags being pushed. I added a to trigger this Action once a day. Docker's documentation schedule name: docker build and push on: push: branches: - "main" schedule: - cron: '20 16 * * *' jobs: build: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v3 - name: Login to Docker Hub uses: docker/login-action@v2 with: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} - name: Set up Docker Buildx uses: docker/setup-buildx-action@v2 - name: Build and push base image uses: docker/build-push-action@v4 with: context: . file: ./Dockerfile push: true tags: excitingtheory/kalilinux-xvfb:latest - name: Build and push torbrowser root image uses: docker/build-push-action@v4 with: context: . file: ./TorBrowser-root.Dockerfile push: true tags: excitingtheory/kalilinux-xvfb:torbrowser-root - name: Build and push torbrowser image uses: docker/build-push-action@v4 with: context: . file: ./TorBrowser.Dockerfile push: true tags: excitingtheory/kalilinux-xvfb:torbrowser Using the Containers Now I can test out the containers and see if they work. I am going to use the tag for this example. torbrowser Building the Containers (Optional) See the for more information about the command. To build containers locally I use the following commands, otherwise they can be pulled from Docker Hub on demand during the next steps. Docker CLI Build Documentation docker build docker build -t excitingtheory/kalilinux-xvfb . docker build -t excitingtheory/kalilinux:torbrowser -f TorBrowser.Dockerfile . docker build -t excitingtheory/kalilinux:torbrowser-root -f TorBrowser-root.Dockerfile . Running the Containers with a Volume I need to mount a volume to the container so I can access the files that I am working on. I am using the directory as a volume. ${HOME}/src Some brief notes on the flags that we are using are included below, otherwise see the for more information about the command. Docker CLI Run Documentation docker run is used to run the container in interactive mode. -it is used to remove the container layers when it exits. --rm is used to map the port from the container to the host machine. -p is used to mount a volume from the host machine to the container. -v # Run the base container docker run -it --rm -p 5901:5901 -v "${HOME}/src":/src excitingtheory/kalilinux-xvfb # Run the container with the default user docker run -it --rm -p 5901:5901 -v "${HOME}/src":/src excitingtheory/kalilinux-xvfb:torbrowser # Run the container with the root user docker run -it --rm -p 5901:5901 -v "${HOME}/src":/src excitingtheory/kalilinux-xvfb:torbrowser-root Using Tor Browser from the Container Next I want to open the user container, login through VNC, and run the Tor Browser Launcher to verify that everything is working so far. Tor browser launcher won't work with the root user. Start the container with the default user: docker run -it --rm -p 5901:5901 -v "${HOME}/src":/src excitingtheory/kalilinux-xvfb:torbrowser Starting the VNC server in the container, the server will prompt for the main password, I am setting an additional view-only password. /opt/start-vnc-server-once.sh Below is an example of the output. Connect on macOS The connect to server dialog can be found from the finder menu: -> . Go Connect to Server Once the dialog is open, enter the address and click . I have saved this address as a favorite for easy access. vnc://localhost:5901 Connect Connect on Windows A vnc client like TightVNC can connect to this on . localhost:5901 Connected to Kali Linux Xfce4 Desktop Once connected, I can see the xfce4 desktop. Now I can start the Tor Browser Launcher from the terminal in the VNC session. torbrowser-launcher Once the browser is open, I can check to see if it is working by going to the Tor Project Connection Checker An example of a successful test: I opened the site on my host machine, that wasn’t connected, to compare: Progress so Far This is the Operating System setup I wanted to finish before I started working with Selenium and Python. To recap, I have done the following: Installed dependencies for Selenium, Python, and a simple xfce4 desktop environment. Built and pushed three interdependent Docker images with GitHub Actions. Successfully started a VNC Server in a Docker container. Connected to a running container's desktop environment with a VNC client. Tested out running Tor Browser Launcher from the container, and verified it is connecting to the Tor network. Now that there is a working container environment. Check out the next post in this series on covert Python web scraping.