Developer’s Toolkit: 4 linux command-line tools for debugging everyday issues that all developers…

We spend a large proportion of our lives debugging issues. Sometimes we just need an answer to a very simple question but getting that answer is a real struggle and can be time-consuming. On occassions I’ve seen a complicated 3rd party piece of technology thrown at the problem but often in these sorts of scenarios, linux already has you covered. Many of these command-line utilties either already come with a standard linux distribution or are one line away from an install from your specific distro’s preferred package manager. Here’s a list of command-line tools I think all developers should know how to use to answer simple questions.

ps

For answering the question:

What’s running on my machine and what process ID does it have?

There are many useful flags you can append to the command such as:

ps -ef

To list all processes in unix format OR:

ps -aux

To list all processes in BSD format.

You can then grep this list to find the PID of the desired application, for example trying to find the PID of a running tomcat:

ps -ef | grep tomcat

Here’s some example output:

andy@desktop:~$ ps -ef | grep tomcatandy 19822 1590 99 14:08 pts/1 00:00:06 /usr/lib/jvm/java-8-oracle/bin/java -Djava.util.logging.config.file=/home/andy/tomcat/apache-tomcat-7.0.56/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -XX:MaxPermSize=512m -Xmx2048m -Dcom.sun.management.jmxremote.port=9012 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.endorsed.dirs=/home/andy/tomcat/apache-tomcat-7.0.56/endorsed -classpath /home/andy/tomcat/apache-tomcat-7.0.56/bin/bootstrap.jar:/home/andy/tomcat/apache-tomcat-7.0.56/bin/tomcat-juli.jar -Dcatalina.base=/home/andy/tomcat/apache-tomcat-7.0.56 -Dcatalina.home=/home/andy/tomcat/apache-tomcat-7.0.56 -Djava.io.tmpdir=/home/andy/tomcat/apache-tomcat-7.0.56/temp org.apache.catalina.startup.Bootstrap startandy 19848 7377 0 14:08 pts/1 00:00:00 grep --color=auto tomcat

Here I can see a process running with a PID of 19822

Note: You will also see your grep in this output (the PID of _19848_).

netstat

For answering the question:

What’s running on my machine and on what port?

If I want to determine what’s listening on port 8000:

sudo netstat tulpn | grep 8000

Here’s an explanation of what the different flags mean — more can be found with netstat --help:

-t # tcp ports-u # udp ports-l # only listening ports-p # output PIDs-n # resolve hardware names

And if I have something running on this port, I get something along the lines of the following output:

andy@desktop:~$ netstat -tulpn | grep 8000tcp 0 0 127.0.0.1:8000 0.0.0.0:* LISTEN 16740/node

Allowing me to kill <pid> or inspect the process that’s running as necessary.

tcpdump

For answering the question:

What requests / packets are coming from and to my applications?

If I’m interested in the requests and packets coming from and to port 8080, I can get a full dump of the traffic, requests and message body with this command:

sudo tcpdump -A -s 0 -i lo port 8080

-i is the interface name. lo is the loopback interface and all traffic originating from your machine will go through this network interface.

-A will debug and omit some strange characters.

-s 0 will print all packets since tcpdump doesn’t do this by default.

Here’s some example output showing a simple POST request to an application which performs a simple authenticated redirect with a username and password combo:

13:49:04.701180 IP6 (flowlabel 0x8a77f, hlim 64, next-header TCP (6) payload length: 817) ip6-localhost.52870 > ip6-localhost.http-alt: Flags [P.], cksum 0x0339 (incorrect -> 0x3fa7), seq 1861:2646, ack 2002, win 1373, options [nop,nop,TS val 4948070 ecr 4947571], length 785: HTTP, length: 785POST /login/?sessionId=941b5ac4-5524-4b43-b4b1-b9e371a7f212 HTTP/1.1Host: localhost:8080Connection: keep-aliveContent-Length: 49Accept: */*Origin: http://localhost:8080X-Requested-With: XMLHttpRequestUser-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36Content-Type: application/x-www-form-urlencoded; charset=UTF-8Referer: http://localhost:8080/login/?redirect=/myAppAccept-Encoding: gzip, deflate, brAccept-Language: en,en-GB;q=0.9Cookie: sessionId=941b5ac4-5524-4b43-b4b1-b9e371a7f212;

    username=username&password=password\[!http\]

There are many other additional flags you can add to the command such as monitoring more than one port:

sudo tcpdump -A -s 0 -i lo port 8080 or port 8000

Or capturing packets only from a single host:

sudo tcpdump -A -s 0 -i lo port 8080 and src host 1.2.3.4

telnet

For answering the question:

Is there a network issue preventing me from connecting to my resource?

telnet <host> <port>

Here’s some example output — I have no active processes running and listening on port 8000 on my local machine so I can’t connect to them:

andy@desktop:~$ telnet localhost 8000Trying 127.0.0.1...telnet: Unable to connect to remote host: Connection refused

Say for example I’m trying to determine whether I can connect to a remote instance of postgres in my network, I already know the instance is up and there are no networking issues — I already know the hostname and know that postgres listens on TCP 5432:

andy@desktop:~$ telnet myRDSinstance.eu-west-1.rds.amazonaws.com 5432Trying 172.20.15.200...Connected to myRDSinstance.eu-west-1.rds.amazonaws.com.Escape character is '^]'.

I will receive a message which shows I have made a TCP connection to my resource. Or if I fail to connect I will see something similar to the following message:

andy@desktop:~$ telnet myRDSinstance.eu-west-1.rds.amazonaws.com 5432Trying 172.20.15.200...telnet: Unable to connect to remote host: Connection refused

Giving me evidence to suggest network connectivity issues as the culprit for my particular issue.