Writing BASH scripts in Python with Subprocess and Fabric
Supbrocess
The most fundamental way to execute BASH from Python is the subprocess module.
suck_it_world.py
import subprocess result = subprocess.call(['echo', 'suck it world']) print result
result
suck it world 0
not so bad; but what if we care about the output of a bash command?
probe.py
import subprocess
child = subprocess.Popen('ls -la', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
child_out, child_err = child.communicate()
print 'out', child_out.split("\n")
print 'err', child_err
result
out ['total 16', 'drwxr-xr-x 4 dave staff 136 Sep 16 11:00 .', 'drwxr-xr-x 22 dave staff 748 Sep 16 10:57 ..', '-rw-r--r-- 1 dave staff 215 Sep 16 11:00 probe.py', '-rw-r--r-- 1 dave staff 113 Sep 16 10:56 suck_it_world.py', ''] err
Two things to note about bash and subprocess:
- with Popen, you pass the comand as a single string; with call, the arguments are passed as an array.
- if you are using call, clump quoted strings as a single element. i.e., ['echo', 'this is a space seperated string'], not ['echo', 'this', 'is'...]
Fabric: spamming SSH servers
Fabric lets you run bash processs on other servers. It is best to have SSH key files for your target servers. The power of fabric is in a fabfile, a super library that has all the methods you want to run remotely. Say you wanted to explore the home contents of a bunch of remote users. This is the kind of fabfile you would write:
fabfile.py
from fabric.api import *
def init():
key_user = "root"
env.user = key_user
env.key_filename = ['/fabkeys/server_one.dev', '/fabkeys/server_two.dev']
def look_in():
run('ls -la')
Note that Fabric is not just a library - it is a command line program. In fact its a little tricky to use Fabric from within Python scripts.
To call the above fabfile, assuming you're in the same dirctory as the fabfile.py file, you execute the following from the command line:
fab -H root@server_one,root@server_two init look_in
Result
[root@server_one] Executing task 'init' [root@server_two] Executing task 'init' [root@server_one] Executing task 'look_in' [root@server_one] run: ls -la [root@server_one] out: total 8748 [root@server_one] out: drwxr-x--- 7 root root 4096 Aug 3 17:31 . [root@server_one] out: drwxr-xr-x 24 root root 4096 Jul 26 12:08 .. [root@server_one] out: -rw------- 1 root root 14046 Sep 15 18:06 .bash_history [root@server_one] out: -rw-r--r-- 1 root root 24 Jan 6 2007 .bash_logout [root@server_one] out: -rw-r--r-- 1 root root 201 Jul 22 00:20 .bash_profile ... [root@server_one] out: -rw-r--r-- 1 root root 129 Jan 6 2007 .tcshrc [root@server_one] out: drwxrwxrwx 17 20 games 4096 Nov 11 2010 ImageMagick-6.5.0-10 [root@server_one] out: -rw-r--r-- 1 root root 8796079 Nov 11 2010 ImageMagick-6.5.0-10.tar.bz2 [root@server_one] out: -rw-r--r-- 1 root root 19954 Jul 9 2010 install.log [root@server_one] out: -rw-r--r-- 1 root root 0 Jul 9 2010 install.log.syslog [root@server_one] out: -rw-r--r-- 1 root root 251 Jul 9 2010 upgrade.log [root@server_one] out: -rw-r--r-- 1 root root 0 Jul 9 2010 upgrade.log.syslog [root@server_one] out: [root@server_two] Executing task 'look_in' [root@server_two] run: ls -la [root@server_two] out: total 8752 [root@server_two] out: drwxr-x--- 7 root root 4096 Aug 17 18:48 . [root@server_two] out: drwxr-xr-x 24 root root 4096 Aug 3 15:49 .. [root@server_two] out: -rw------- 1 root root 13805 Sep 14 20:14 .bash_history [root@server_two] out: -rw-r--r-- 1 root root 24 Jan 6 2007 .bash_logout [root@server_two] out: -rw-r--r-- 1 root root 201 Jul 22 00:20 .bash_profile [root@server_two] out: -rw-r--r-- 1 root root 176 Jan 6 2007 .bashrc [root@server_two] out: -rw-r--r-- 1 root root 100 Jan 6 2007 .cshrc ... [root@server_two] out: -rw-r--r-- 1 root root 19954 Jul 9 2010 install.log [root@server_two] out: -rw-r--r-- 1 root root 0 Jul 9 2010 install.log.syslog [root@server_two] out: -rw-r--r-- 1 root root 70 Aug 17 18:48 stall.rb [root@server_two] out: -rw-r--r-- 1 root root 251 Jul 9 2010 upgrade.log [root@server_two] out: -rw-r--r-- 1 root root 0 Jul 9 2010 upgrade.log.syslog [root@server_two] out: Done. Disconnecting from root@server_one... done. Disconnecting from root@server_two... done.
Things to note:
- The methods to be called are called in the order they are passed in, and they are applied to each server in the -H list, one by one; i.e., if you call "fab -H FooServer,BarServer,ZedServer AlphaMethod BetaMethod GammaMethod" then AlphaMethod is applied ot all three servers before the first BetaMethod is called.
- Server names are comma seperated, with no spaces allowed. Methods are space seperated.
Getting a little trickier with Fabric
Fabric allows for parametrized method arguments. Say, for instance, that you want to look inside the '/etc' directory across all the target servers. A little surgery on the fabfile:
from fabric.api import *
def init():
key_user = "root"
env.user = key_user
env.key_filename = [ "/opt/deploy/keys/tyger.dev"]
def look_in(path_to_look_in):
run("cd '%s'; ls -la" % (path_to_look_in))
[root@gecko examples]#
Something else to note - I originally tried putting the cd and the ls on two different runs as in
run("cd '%s'" % (path_to_look_in))
run(ls -la")
However, and I guess I should have seen this coming but, these runs executed in different SSH sessions and therefore, I got a listing of the home directory again.
Moving along: executing
fab -H root@server_one,root@server_two init look_in:/etc
returns a listing of the /etc directory.
Given the simplicity of Fabric, you can do a lot of administration without a lot of deep Python experience.
Further tricks with Fabric
There are two straightforward reasons why you would want to tweak with the settings in Fabric:
- You want to run a series of functions, most of them over the same servers, but a few of them on just one server.
- You want to ignore any irrelevant errors thrown by BASH.
The latter came up for me when I wanted to run "killall -9 ruby". If there are no processes to kill it throws an error (really?). But I don't want that error to be a show stopper.
There are a bunch of ways to change your "hit list" of servers. the Fabric Documentation covers them as well as I could - have a look. But here I will highlight the use of the settings construct which has options for both of the above scenarios.
Say you wanted to inspect a series of servers, then kill ruby processes on a single server. Also you got tired of typing server names, so you wanted to embed that list in the script. Here's the modifed fabfile.py:
from fabric.api import *
env.hosts=['server_one', 'server_two']
env.user='root'
def init():
key_user = "root"
env.user = key_user
env.key_filename = [ "/opt/deploy/keys/tyger.dev"]
def look_in(path_to_look_in):
run("cd '%s'; ls -la" % (path_to_look_in))
runs = 0
def kill_all_ruby():
global runs
if runs == 0:
runs += 1
with settings(host_string='mongodb-dev',warn_only=True):
run('killall -9 ruby')
Note that there is a bit of hackery in this script that I do to further limit the scope of executions. Because fab by definition loops each of the tasks you pass it, the kill_all_ruby() task will be executes several times - once for each host. To ensure it only executes once, I use a "runs" variable to track how many times it executes - to once. Also, with the host_string parameter, ony a single host can be targeted.
Another, and more "fabric'y" way of doing this is with a task that switches the env.hosts value. Its important to note the following:
- You cannot modify the env.hosts AND execute a run FROM THE SAME DEF. they must be seperated.
- Any changes to env.hosts in this manner are persistent. Changing the host list in one def changes the host target list for all subsequent defs.
from fabric.api import *
env.hosts=['server_one', 'server_two', 'server_three']
env.user='root'
def init():
key_user = "root"
env.user = key_user
env.key_filename = [ "/opt/deploy/keys/tyger.dev"]
def look_in(path_to_look_in):
run("cd '%s'; ls -la" % (path_to_look_in))
def kill_all_ruby():
with settings(warn_only=True):
run('killall -9 ruby')
def point_to_mongo():
env.hosts = ['server_mongo']
shows what is needed for def-based env.host switching. If called like this:
fab init look_in:/root point_to_mongo kill_all_ruby look_in:/root
(note the absence of -H) you will see the following flow:
- look in root of server_one
- look in root of server_two
- look in root of server three
- kill ruby on server_mongo
- look in root of server_mongo
In Closing
I hope this flyover of the Fabric system is useful to your script writing efforts.
Post new comment