Tired of shell scripting?

Python-based alternatives

Lucas Inojosa / lucas.inojosa@gmail.com

Who am I?

System commands


ls
ls -l
df
df -h
ps aux
touch testfile
ls
rm testfile
					

Examples

Shell script


df -h | grep /dev/disk1
df -h | grep /dev/disk1 | awk '{ print $4 }'
echo Hello World! > hello.txt
cat hello.txt
rm hello.txt
					

Combining commands and redirecting output

Shell script


PROCESSES=`ps aux`
for process in $PROCESSES
do
  echo $process
done
					

Showing current system processes


USER
PID
%CPU
%MEM
VSZ
RSS
TT
STAT
STARTED
TIME
COMMAND
linojosa
663
7.8
1.1
2947004
93820
??
S
8:55AM
0:08.89
/Applications/Google
Chrome.app/Contents/Versions/51.0.2704.79/Google
Chrome
Helper.app/Contents/MacOS/Google
Chrome
Helper
--type=ppapi
--channel=238.34.645195462
--ppapi-flash-args
--lang=en-US
linojosa
245
6.4
3.3
4065052
274080
??
S
8:48AM
0:52.02
/Applications/Slack.app/Contents/MacOS/Slack
-psn_0_32776
_windowserver
176
6.1
1.4
3942984
119428
??

.
.
.
					

Shell script


IFS=$'\n'
PROCESSES=`ps aux`
for process in $PROCESSES
do
  echo $process
done
					

Fixing the 'each line' problem


USER              PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
linojosa          663   8.5  1.3  2948056 109076   ??  S     8:55AM   1:02.57 /Applications/Google Chrome.app/Contents/Versions/51.0.2704.79/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper --type=ppapi --channel=238.34.645195462 --ppapi-flash-args --lang=en-US
_windowserver     176   6.8  1.4  3968076 118956   ??  Us    8:47AM   1:36.33 /System/Library/Frameworks/ApplicationServices.framework/Frameworks/CoreGraphics.framework/Resources/WindowServer -daemon
linojosa          245   6.5  3.2  4056184 265440   ??  S     8:48AM   1:32.96 /Applications/Slack.app/Contents/MacOS/Slack -psn_0_32776
linojosa          486   3.3  1.5  2902504 129964   ??  S     8:49AM   0:17.09 /opt/homebrew-cask/Caskroom/iterm2/2.1.1/iTerm.app/Contents/MacOS/iTerm2
root              619   1.2  0.0  2470552   1992   ??  Ss    8:50AM   0:00.04 /usr/libexec/systemstatsd
linojosa          372   0.2  1.8  4021184 152468   ??  S     8:48AM   0:17.46 /Applications/Dropbox.app/Contents/MacOS/Dropbox
linojosa          241   0.2  1.1  3747124  88884   ??  S     8:48AM   0:08.78 /Applications/Screenhero.app/Contents/MacOS/Screenhero -psn_0_24582
root               39   0.1  0.2  2500660  15976   ??  Ss    8:47AM   0:00.56 /usr/libexec/UserEventAgent (System)

.
.
					

Why Shell script?

  • Call and connect powerful system commands and executables
  • Used by SysAdmins to automate tasks
  • Native support in UNIX systems
  • Let's assume bash as default
    • Coming soon to Windows 10!

Why seek an alternative?

  • It's powerful, the problem is the language itself...
  • No modern language features available
    • Weak data structure support
  • Weird syntax

VARIABLE = "content"
if [-n $VARIABLE]
then
  echo "VARIABLE is not empty"
fi
          

Wrong...


bash: VARIABLE: command not found
bash: [-n: command not found
          

VARIABLE="content"
if [ -n $VARIABLE ]
then
  echo "VARIABLE is not empty"
fi
          

Spaces matter


VARIABLE is not empty
          

Python-based alternatives

Pure python


import os
os.system("ls -la")
					

Calling the ls command

Pure python


import subprocess
p = subprocess.Popen(['ls', '-la'], stdout=subprocess.PIPE)
(out, err) = p.communicate()
print(out.decode())
					

Retrieving the output to a variable

Pure Python advantages

  • No external libraries required
  • Unix-based OSes usually come with pre-installed Python

Pure Python drawbacks

  • Too much code compared to shell scripting

Plumbum


from plumbum import local
ls = local['ls']
ls
ls()
					

Listing files

Plumbum


from plumbum import local
ls = local['ls']

files = ls().split('\n')
print(files)
print(len(files))

files = ls['-a']().split('\n')
print(files)
					

Listing files


['Applications', 'Desktop', 'Documents', 'Downloads', 'Dropbox', 'Google Drive', 'Library', 'Movies', 'Music', 'Pictures', 'Projects', 'Public', 'Snapshots', 'VirtualBox VMs', '']

15

['.', '..', '.CFUserTextEncoding', '.DS_Store', '.IntelliJIdea15', '.NERDTreeBookmarks', '.Spotlight-V100', '.Trash', '.Trashes', '.Xauthority', '.atom', '.bash_history', '.bash_profile', '.bashrc', '.berkshelf', '.bundler', '.cache', '.chef', '.chefdk', '.com.apple.timemachine.donotpresent', '.config', '.cpan', '.cpanm', '.cups', '.dbshell', '.dropbox', '.emacs.d', '.fseventsd', '.gbas', '.gem', '.gitattributes', '.gitconfig', '.gitignore', '.gnome2', '.gnupg', '.gradle', '.groovy', '.histfile', '.ipython', '.jenv', '.lesshst', '.local', '.mongorc.js', '.node-gyp', '.npm', '.oh-my-zsh', '.oracle_jre_usage', '.pry_history', '.python_history', '.rnd', '.sh_history', '.sonar', '.ssh', '.subversion', '.swo', '.swp', '.vagrant', '.vagrant.d', '.vim', '.viminfo', '.vimrc', '.virtualenvs', '.xonsh_history', '.xonsh_history.json', '.xonsh_man_completions', '.yjp', '.zcompdump', '.zcompdump-LAlinojosa-5.0.5', '.zcompdump-LAlinojosa-5.2', '.zsh-update', '.zsh_history', '.zshrc', 'Applications', 'Desktop', 'Documents', 'Downloads', 'Dropbox', 'Google Drive', 'Library', 'Movies', 'Music', 'Pictures', 'Projects', 'Public', 'Snapshots', 'VirtualBox VMs', '']
					

Plumbum


from plumbum import local
grep = local['grep']
wc = local['wc']
ls_grep = ls['-a'] | grep['\\.']

ls_grep
ls_grep().split('\n')
					

Nesting commands


Pipeline(BoundCommand(LocalCommand(/bin/ls), ['-a']), BoundCommand(LocalCommand(/usr/bin/grep), ['\\.']))

['.', '..', '.CFUserTextEncoding', '.DS_Store', '.IntelliJIdea15', '.NERDTreeBookmarks', '.Spotlight-V100', '.Trash', '.Trashes', '.Xauthority', '.atom', '.bash_history', '.bash_profile', '.bashrc', '.berkshelf', '.bundler', '.cache', '.chef', '.chefdk', '.com.apple.timemachine.donotpresent', '.config', '.cpan', '.cpanm', '.cups', '.dbshell', '.dropbox', '.emacs.d', '.fseventsd', '.gbas', '.gem', '.gitattributes', '.gitconfig', '.gitignore', '.gnome2', '.gnupg', '.gradle', '.groovy', '.histfile', '.ipython', '.jenv', '.lesshst', '.local', '.mongorc.js', '.node-gyp', '.npm', '.oh-my-zsh', '.oracle_jre_usage', '.pry_history', '.python_history', '.rnd', '.sh_history', '.sonar', '.ssh', '.subversion', '.swo', '.swp', '.vagrant', '.vagrant.d', '.vim', '.viminfo', '.vimrc', '.virtualenvs', '.xonsh_history', '.xonsh_history.json', '.xonsh_man_completions', '.yjp', '.zcompdump', '.zcompdump-LAlinojosa-5.0.5', '.zcompdump-LAlinojosa-5.2', '.zsh-update', '.zsh_history', '.zshrc', '']
					

Plumbum advantages

  • Pure Python, no syntax changes
  • Better readability

Plumbum drawbacks

  • Requires an external library
  • Less similar to classic shell scripting

IPython


%env HOME
!pwd
!cd
!ls -l
!ls -l | grep Downloads
					

Accessing env variables and listing

IPython


files = !ls -l # it automatically parses the output into a list

type(files)
files.grep('Downloads')

for fileline in files.fields():
    print(fileline)
					

The SList object


IPython.utils.text.SList

['drwx------+ 18 linojosa  staff   612 Jun  7 09:39 Downloads']

['total', '0']
['drwx------', '10', 'linojosa', 'staff', '340', 'Feb', '24', '13:08', 'Applications']
['drwx------+', '4', 'linojosa', 'staff', '136', 'Jun', '6', '10:56', 'Desktop']
['drwx------+', '14', 'linojosa', 'staff', '476', 'Jun', '6', '16:14', 'Documents']
['drwx------+', '18', 'linojosa', 'staff', '612', 'Jun', '7', '09:39', 'Downloads']
['drwx------@', '14', 'linojosa', 'staff', '476', 'Jun', '7', '08:48', 'Dropbox']
['drwx------@', '13', 'linojosa', 'staff', '442', 'Jun', '7', '08:48', 'Google', 'Drive']
['drwx------@', '54', 'linojosa', 'staff', '1836', 'May', '27', '18:42', 'Library']
['drwx------+', '3', 'linojosa', 'staff', '102', 'Aug', '11', '2015', 'Movies']
['drwx------+', '4', 'linojosa', 'staff', '136', 'Apr', '25', '17:22', 'Music']
['drwx------+', '10', 'linojosa', 'staff', '340', 'Jun', '1', '15:21', 'Pictures']
['drwxr-xr-x', '3', 'linojosa', 'staff', '102', 'Jun', '7', '09:50', 'Projects']
['drwxr-xr-x+', '5', 'linojosa', 'staff', '170', 'Aug', '11', '2015', 'Public']
['drwxr-xr-x', '2', 'linojosa', 'staff', '68', 'Mar', '2', '17:58', 'Snapshots']
['drwx------', '3', 'linojosa', 'staff', '102', 'Apr', '7', '11:38', 'VirtualBox', 'VMs']
					

IPython advantages

  • Clear and simple syntax to call system commands
  • More alike to a shell
  • All power of Python

IPython drawbacks

  • Changes the language's syntax
  • Still not a native shell

xonsh


# https://github.com/scopatz/xonsh
# http://xon.sh/
pip3 install xonsh
					

Installing xonsh

xonsh


ls
ps aux
df -h
					

Basic shell commands work!

xonsh


arquivos = $(ls).split('\n')
print(arquivos)
echo @(arquivos)
					

Printing variables in both Shell and Python statements


['Applications/', 'Desktop/', 'Documents/', 'Downloads/', 'Dropbox/', 'Google Drive/', 'Library/', 'Movies/', 'Music/', 'Pictures/', 'Projects/', 'Public/', 'Snapshots/', 'VirtualBox VMs/', '']

Applications/ Desktop/ Documents/ Downloads/ Dropbox/ Google Drive/ Library/ Movies/ Music/ Pictures/ Projects/ Public/ Snapshots/ VirtualBox VMs/
					

xonsh


def ls_files(dir):
    return [file for file in $(ls @(dir)).split('\n') if file]

ls_files('/')
ls_files($HOME)
					

Functions: plain old Python


['Applications/', 'Library/', 'Network/', 'System/', 'Users/', 'Volumes/', 'bin/', 'dev/', 'etc', 'home/', 'hs_err_pid5502.log', 'installer.failurerequests', 'local.cfg', 'mnt/', 'net/', 'opt/', 'private/', 'sbin/', 'sockets.log', 'tmp', 'usr/', 'var']

['Applications/', 'Desktop/', 'Documents/', 'Downloads/', 'Dropbox/', 'Google Drive/', 'Library/', 'Movies/', 'Music/', 'Pictures/', 'Projects/', 'Public/', 'Snapshots/', 'VirtualBox VMs/']
					

xonsh


NAME    AGE    EMAIL
Alice   23     alice@email.com
Bob     27     bob@email.com
Eve     31     eve@email.com
					

Example: building a nice data structure from output


[
  {'Name': 'Alice',
   'Age': '23',
   'Email': 'bob@email.com'},
  {'Name': 'Bob',
   'Age': '27',
   'Email': 'bob@email.com'},
  {'Name': 'Eve',
   'Age': '31',
   'Email': 'eve@email.com'}
]
					

xonsh


procs_info = [proc.split() for proc in $(ps aux).split('\n') if proc][:11]
headers = procs_info[0]
procs = []
for proc_info in procs_info[1:]:
    proc_dict = {}
    for (index, header) in enumerate(headers):
        proc_dict[header] = proc_info[index]
    procs.append(proc_dict)

import json
echo @(json.dumps(procs, indent=4)) > procs.json

cat procs.json
					

Building a dict from the first 10 processes

xonsh


print([
    proc['%CPU'] for proc in procs
      if 'Screenhero' in proc['COMMAND']
][0])
					

Screenhero's CPU usage?


0.2
					

xonsh advantages

  • Is a native shell
  • Bash-like commands
  • All power of Python 3

xonsh drawbacks

  • Not mature enough (v. 0.3.3)

Questions?

Thank You

ThoughtWorks is hiring

https://www.thoughtworks.com/careers