Before you go, check out these stories!

0
Hackernoon logoCalling Shell Commands from Python: OS.system vs Subprocess by@marcosdelcueto

Calling Shell Commands from Python: OS.system vs Subprocess

Author profile picture

@marcosdelcuetoMarcos del Cueto

PhD in Theoretical Chemistry. Interested in Machine Learning applied to materials discovery

If you are a Python programmer, it is quite likely that you have experience in shell scripting. It is not uncommon to face a task that seems trivial to solve with a shell command. Therefore, it is useful to be familiar with how to call these commands efficiently from your Python code and know their limitations.

In this short article, I discuss how to use the older (although still relatively common)

os.system
command and the newer
subprocess
command. I will show some of their potential risks, limitations and provide complete examples of their use.

External commands might be an attractive option for small tasks if you are familiar with them and the pythonic alternative would imply more learning/implementation time. However, Python functions will save you time and trouble in the long run.

Option 1: OS.system (deprecated) Shell Commands

I show below a simple example of

os.system
to print the first line of a file provided by the user, using
head -n 1
.

#!/usr/bin/env python3
import os
# Define command and options wanted
command = "head -n 1 "
# Ask user for file name(s) - SECURITY RISK: susceptible to shell injection
filename = input("Please introduce name of file of interest:\n")
# Run os.system and save return_value
return_value = os.system(command+filename)
print('###############')
print('Return Value:', return_value)

The

os.system
function is easy to use and interpret: simply use as input of the function the same command you would use in a shell. However, it is deprecated and it is recommended to use
subprocess
now.

Note that this function will simply execute the shell command and the result will be printed to the standard output, but the output that the function returns is the return value (0 if it ran OK, and different than 0 otherwise).

1.1: Susceptibility to shell injection

Among other drawbacks,

os.system
directly executes the command in a shell, which means that it is susceptible to shell injection (aka command injection). You can read more about it in 10 common security gotchas in Python and how to avoid them by Anthony Shaw).

Shell injection is an issue any time that

os.system
is receiving unformatted input, like for example when a user can introduce a filename, as in the example above. You can try to execute the script above and give the following input:

dummy; touch harmful_file

This will result in the shell doing:

head -n 1 dummy; touch harmful_file

As a result, the program will first execute

head -n 1 dummy
, as expected, but then it will execute the command
touch harmful_file
to create a file named 'harmful_file'.

Granted that this empty file is not much of a threat, but you can imagine a user adding extra commands to create a file with actual nefarious purposes. This shell injection can also be used to simply transfer or delete information. For example, one could use

;rm -rf ~
,
;rm -rf /
or any other potentially dangerous command (please do not try these!).

Option 2: subprocess.call (recommended for Python<3.5)

2.1: Not-secure way to run external shell commands

A preferable alternative is

subprocess.call
. As os.sys, the function subprocess.call returns the return value as its output. A naive first approach to subprocess is using the
shell=True
option. This way, the desired command will also be run in a subshell. Note that this is not recommended, as it has the same potential security risks as
os.system
. For example, the code below would be equivalent to the previous
os.system
example.

#!/usr/bin/env python3
import subprocess

# Define command and options wanted
command = "head -n 1 "
# Ask user for file name(s) - SECURITY RISK: susceptible to shell injection
filename = input("Please introduce name of file of interest:\n")
# Run subprocess.call and save return_value
return_value = subprocess.call(command+filename, shell=True)
print('###############')
print('Return value:', return_value)

2.2: Preferred way to run external commands

A preferable way to run

subprocess.call
is by using
shell=False
(it is the default option, so there is no need to specify it). Then, we can simply call
subprocess.call(args)
, where
agrs[0]
contains the command, and
args[1:]
contains all the extra options to the command.

#!/usr/bin/env python3
import subprocess

# Define command and options wanted
command = "head"
options = "-n 1"
# Ask user for file name(s) - now it's safe from shell injection
filename = input("Please introduce name(s) of file(s) of interest:\n")
# Create list with arguments for subprocess.call
args=[]
args.append(command)
args.append(options)
for i in filename.split():
    args.append(i)
# Run subprocess.call and save return_value
return_value = subprocess.call(args)
print('###############')
print('Return value:', return_value)

2.3: Getting standard output:
subprocess.check_output

As mentioned before,

subprocess.call
returns the return value. However, sometimes we might be interested in the standard output returned by the shell command. In this case, we can make use of
subprocess.check_output
.

To make our script more robust, we can add a try/exclude statement to deal with situations when

check_output
raises an error (in this case, for example, when no file is found).

As shown below, we can use

subprocess.CalledProcessError
in the except clause to deal with the error in a controlled manner and get information about it.

#!/usr/bin/env python3
import subprocess
# Define command and options wanted
command = "head"
options = "-n 1"
# Ask user for file name(s) - now it's safe from shell injection
filename = input("Please introduce name(s) of file(s) of interest:\n")
# Create list with arguments for subprocess.check_output
args=[]
args.append(command)
args.append(options)
for i in filename.split():
    args.append(i)
# Run subprocess.check_output and save command output
try:
    output = subprocess.check_output(args)
    # use decode function to convert to string
    print('###############')
    print('Output:', output.decode("utf-8"))                                                      
# If check_output returns an error: 
except subprocess.CalledProcessError as error:
    print('Error code:', error.returncode, '. Output:', error.output.decode("utf-8"))

Option 3: subprocess.run (recommended since Python 3.5)

The recommended way to execute external shell commands, since Python 3.5, is with the

subprocess.run
function. It is more secure and user-friendly than the previous options discussed.

By default, this function returns an object with the input command and the return code. One can very easily also get the standard output by using the option

capture_output=True
, and finally retrieve the return code and command output using:
output.returncode
and
output.stdout
, respectively.

#!/usr/bin/env python3                                                                            
import subprocess
# Define command and options wanted
command = "head"
options = "-n 1"
# Ask user for file name(s) - now it's safe from shell injection
filename = input("Please introduce name(s) of file(s) of interest:\n")
# Create list with arguments for subprocess.run
args=[]
args.append(command)
args.append(options)
for i in filename.split():
    args.append(i)
# Run subprocess.run and save output object
output = subprocess.run(args,capture_output=True)
print('###############')
print('Return code:', output.returncode)
# use decode function to convert to string
print('Output:',output.stdout.decode("utf-8"))

Final Note on Shell Commands in Python

If you are thinking about using any of the methods discussed here to call an external command, it might be worth considering if there is a standard Python function that allows you to do the same task and will avoid creating a new process. Note that using external commands also makes your program less cross-platform friendly.

External commands might be an attractive option for small tasks if you are familiar with them and the pythonic alternative would imply more learning/implementation time.

However, sticking to Python functions will save you computation time and trouble in the long run, so external commands should be saved for tasks that cannot be achieved with standard Python libraries.

I hope this article helped you to call external commands from your python code!

Tags

Join Hacker Noon

Create your free account to unlock your custom reading experience.