What To Expect When You’re Expect Scripting
I’ve been playing with Expect lately. Expect is an extension of the TCL scripting language developed in the 1990s. Its main purpose in life is to automate terminal interactions and it does that job very well.
I spend most of my day in a shell and automate as much as humanly possible so that I can be as lazy as humanly possible. Using tools like ssh and scp it’s very easy to automate simple commands and simple file transfers. But when these tasks become complex enough that they need to respond to terminal prompts, or provide arbitrary changing input, those tools fall apart.
My particular use case was a need to grep through logs on multiple Linux servers. This would be a trivial task to achieve using plain old ssh except for the fact that I use a Yubi key to log on to the servers. I need to interactively provide the PIN for my Yubi at each login. The same problem exists for encrypted public keys. For a while I just copied the PIN and pasted it at every prompt, but that became a pain pretty quickly so I started casting around for other options.
There’s a few solutions that specifically address the ssh password problem such as sshpass and ssh-agent, but I wanted something more generic that I could continue to use once logged in to do complex things. Enter expect.
Expect can handle some very complex interactions, but at its core is a very simple game of catch. Tell your script to “expect” some output and then “send” the appropriate input. Here’s a very simple example of SSH-into a server with a password.
#!/usr/bin/expect -f
set host [lindex $argv 1]
set password "password"
spawn ssh "$host"
expect "password: "
send "$password\r"
interact
You can chmod this script to make it executable chmod +x script and run it using the format ./script host.com
The interact command is important. It tells expect to release the terminal so you can type things and do things. Without the interact command, you won’t be able to…well…interact. If you’re just planning on executing a lot of commands without any interaction that is fine.
This little snippet is meant as an example and as such it stores the password in the script. Obviously, that’s crap security and you should not do that, but what options are there? One option is to store the credential in a separate file that does not comprise the codebase. Expect can read external files using this format (assuming a file named PASS with just one single line it, your password):
set fp [open "PASS" r]
set PASS [read $fp]
This will store the contents of the file PASS in the variable $PASS. You can then replace the send "$password\r" with send "$PASS\r".
Another useful feature is Expect’s logging function. In my case, I am running a bunch of greps on log files and I want to save the output locally to my machine so I can review it later. The expect command log_file works perfectly for this.
Using the command log_file by itself will write a log file into the current directory on your local machine. You can also specify a log file using the format log_file output.log
To start logging, you simply put the log_file command in the script where you'd like the logging to start. If you want to end logging at some point, put log_file by itself again (regardless of whether you specified at log file name or not when you began logging). Expect is smart enough to know it already has a log file open and knows to close it when it encounters a single log_file all by itself.
Let’s put this all together to log into a machine, grep the Apache log file for something, and then save the log for review.
#!/usr/bin/expect -f
set host [lindex $argv 1]
set fp [open "PASS" r]
set PASS [read $fp]
spawn ssh "$host"
expect "password: "
send "$PASS\r"
expect "$ "
log_file website_logins.log
send "grep login /var/log/httpd/access.log\r"
log_file
Now run ./script host.com and sit back. In a few seconds you should have a file named website_logins.log in the script directory that contains the output of your grep.
There’s a lot more to learn, but these basic building blocks will get you started.
my shorter content on the fediverse: https://the.mayhem.academy/@jdw