Calling Shell Commands from Python: OS.system vs Subprocess

If you are a Python programmer, it is quite likely that you have experience in shell scripting. It is not uncommon to face a task that seems trivial to solve with a shell command. Therefore, it is useful to be familiar with how to call these commands efficiently from your Python code and know their limitations.

In this short article, I discuss how to use the older (although still relatively common)

os.system

subprocess

External commands might be an attractive option for small tasks if you are familiar with them and the pythonic alternative would imply more learning/implementation time. However, Python functions will save you time and trouble in the long run.

Option 1: OS.system (deprecated)

I show below a simple example of

os.system

head -n 1

#!/usr/bin/env python3
import os
# Define command and options wanted
command = "head -n 1 "
# Ask user for file name(s) - SECURITY RISK: susceptible to shell injection
filename = input("Please introduce name of file of interest:\n")
# Run os.system and save return_value
return_value = os.system(command+filename)
print('###############')
print('Return Value:', return_value)

The

os.system

subprocess

Note that this function will simply execute the shell command and the result will be printed to the standard output, but the output that the function returns is the return value (0 if it ran OK, and different than 0 otherwise).

1.1: Susceptibility to shell injection

Among other drawbacks,

os.system

Shell injection is an issue any time that

os.system

dummy; touch harmful_file

This will result in the shell doing:

head -n 1 dummy; touch harmful_file

As a result, the program will first execute

head -n 1 dummy

touch harmful_file

Granted that this empty file is not much of a threat, but you can imagine a user adding extra commands to create a file with actual nefarious purposes. This shell injection can also be used to simply transfer or delete information. For example, one could use

;rm -rf ~

;rm -rf /

Option 2: subprocess.call (recommended for Python<3.5)

2.1: Not-secure way to run external shell commands

A preferable alternative is

subprocess.call

shell=True

os.system

os.system

#!/usr/bin/env python3
import subprocess

# Define command and options wanted
command = "head -n 1 "
# Ask user for file name(s) - SECURITY RISK: susceptible to shell injection
filename = input("Please introduce name of file of interest:\n")
# Run subprocess.call and save return_value
return_value = subprocess.call(command+filename, shell=True)
print('###############')
print('Return value:', return_value)

2.2: Preferred way to run external commands

A preferable way to run

subprocess.call

shell=False

subprocess.call(args)

agrs[0]

args[1:]

#!/usr/bin/env python3
import subprocess

# Define command and options wanted
command = "head"
options = "-n 1"
# Ask user for file name(s) - now it's safe from shell injection
filename = input("Please introduce name(s) of file(s) of interest:\n")
# Create list with arguments for subprocess.call
args=[]
args.append(command)
args.append(options)
for i in filename.split():
    args.append(i)
# Run subprocess.call and save return_value
return_value = subprocess.call(args)
print('###############')
print('Return value:', return_value)

2.3: Getting standard output:

subprocess.check_output

As mentioned before,

subprocess.call

subprocess.check_output

To make our script more robust, we can add a try/exclude statement to deal with situations when

check_output

As shown below, we can use

subprocess.CalledProcessError

#!/usr/bin/env python3
import subprocess
# Define command and options wanted
command = "head"
options = "-n 1"
# Ask user for file name(s) - now it's safe from shell injection
filename = input("Please introduce name(s) of file(s) of interest:\n")
# Create list with arguments for subprocess.check_output
args=[]
args.append(command)
args.append(options)
for i in filename.split():
    args.append(i)
# Run subprocess.check_output and save command output
try:
    output = subprocess.check_output(args)
    # use decode function to convert to string
    print('###############')
    print('Output:', output.decode("utf-8"))                                                      
# If check_output returns an error: 
except subprocess.CalledProcessError as error:
    print('Error code:', error.returncode, '. Output:', error.output.decode("utf-8"))

Option 3: subprocess.run (recommended since Python 3.5)

The recommended way to execute external shell commands, since Python 3.5, is with the

subprocess.run

By default, this function returns an object with the input command and the return code. One can very easily also get the standard output by using the option

capture_output=True

output.returncode

output.stdout

#!/usr/bin/env python3                                                                            
import subprocess
# Define command and options wanted
command = "head"
options = "-n 1"
# Ask user for file name(s) - now it's safe from shell injection
filename = input("Please introduce name(s) of file(s) of interest:\n")
# Create list with arguments for subprocess.run
args=[]
args.append(command)
args.append(options)
for i in filename.split():
    args.append(i)
# Run subprocess.run and save output object
output = subprocess.run(args,capture_output=True)
print('###############')
print('Return code:', output.returncode)
# use decode function to convert to string
print('Output:',output.stdout.decode("utf-8"))

Final Note on Shell Commands in Python

If you are thinking about using any of the methods discussed here to call an external command, it might be worth considering if there is a standard Python function that allows you to do the same task and will avoid creating a new process. Note that using external commands also makes your program less cross-platform friendly.

External commands might be an attractive option for small tasks if you are familiar with them and the pythonic alternative would imply more learning/implementation time.

However, sticking to Python functions will save you computation time and trouble in the long run, so external commands should be saved for tasks that cannot be achieved with standard Python libraries.

I hope this article helped you to call external commands from your python code!

• Featured image by JPuckett Design on Instagram, used with permission.