Creating Powerful Command Line Tools in Ruby

When it comes to software development, a majority of the tools available to us are command-line applications. It is also well worth noting that many of the tools used on the command line are quite powerful in what they can accomplish, from the trivial to the tedious. Taking this further, you can combine command-line applications to chain together a sequence of work to attain a desired output. Learning existing command-line commands enables you to increase your productivity and better understand what capabilities are at hand and what tasks you may need to implement on your own.

When you design a command-line tool, you really need to pay attention to who, or what, your target audience is. For example, if all the people who will be using this tool use the same operating system and environment, then you have the most flexibility in taking advantage of the full ecosystem. However if you need to implement your command-line app to work across multiple operating systems, you have now restricted the tools that you can use to what is available on all of those systems -- or you have to implement exceptions for each operating system into your app. (That can be very tedious to research and test for.)

The solution most often used to handle diverse OS environments is to avoid any OS-specific external tools when possible or delegate the responsibility out to libraries that have already done the hard work of implementing features to work for multiple architectures. The more you can keep implementation down to one core language without depending on external dependencies, the easier it will be to maintain your project.

There are three areas of concern to address when building a command-line application: the input, output, and everything in between. If you're developing a one-off command-line tool that simply takes an input or source and processes/formats it and spits out the information, then your work is rather simple. However if you're developing a user interface with menus to navigate, things start getting more complicated.

To begin processing input, we have parameters that may be given to your application. These arguments are the most typical way of starting a command, with details on how you want it to execute and avoid the need for menu systems. In this example:

ruby -e "puts RUBY_VERSION"

Ruby is the command-line program being run, -e is called a flag, and "puts RUBY_VERSION" is the value given for the flag. In this case, Ruby's flag -e means to execute the following as Ruby code. When I run the above on the command line, I get back 2.4.0 printed out to standard output (which simply shows on the next line).

Argument input, or parameters, are all the extra bits of text entered after a command and separated by a space. Almost all commands allow for a help flag argument. A flag argument has a dash or two in front of it.

The standard help flags are -h, and --help. Back in the MSDOS days, it was (and may still be) /?. I don't know if the trend for flags in Windows has kept on using forward-slash as a flag marker, but cross-platform command-line scripts these days use dashes.

In Ruby, the input from a command line is placed into two different variables: ARGV and ARGF. ARGV handles the input parameters as an array of strings; ARGF is for handling streams of data. You can use ARGV for parameters directly, but this may be more work than you need to do. There are a couple of Ruby libraries built for working with command-line arguments.

OptionParser has the advantage of being included with Ruby, so it's not an external dependency. OptionParser gives you an easy way to both display what command-line options are available and process the input into any Ruby object you'd like. Here's an excerpt from one of my command-line tools dfm:

require 'optionparser'
options = {}
printers = Array.new
OptionParser.new do |opts|
  opts.banner = "Usage: dfm [options] [path]\nDefaults: dfm -xd ." + File::SEPARATOR
  opts.on("-f", "--filters FILTERS", Array, "File extension filters") do |filters|
    options[:filters] = filters
  end
  opts.on("-x", "--duplicates-hex", "Prints duplicate files by MD5 hexdigest") do |dh|
    printers << "dh"
  end
  opts.on("-d", "--duplicates-name", "Prints duplicate files by file name") do |dh|
    printers << "dn"
  end
  opts.on("-s", "--singles-hex", "Prints non-duplicate files by MD5 hexdigest") do |dh|
    printers << "sh"
  end
  opts.on("-n", "--singles-name", "Prints non-duplicate files by file name") do |dh|
    printers << "sn"
  end
end.parse!

In this example, each opts.on block has code to execute if the flag was passed in on the command line. The bottom four options are (within their block) simply appending the flag info into an array for me to use later.

The first one has Array given as one of the parameters to on, so the input for this flag will be converted to a Ruby array and stored in my hash named options under the key filters.

The rest of the parameters given to the on method are the flag details and the description. Both the short and long flags will work to execute the code given in the following block.

OptionParser also has the -h and --help flags built in by default, so you don't need to reinvent the wheel. Simply type dfm -h for the above command-line tool, and it nicely outputs a helpful description:

Usage: dfm [options] [path]
Defaults: dfm -xd ./
    -f, --filters FILTERS            File extension filters
    -x, --duplicates-hex             Prints duplicate files by MD5 hexdigest
    -d, --duplicates-name            Prints duplicate files by file name
    -s, --singles-hex                Prints non-duplicate files by MD5 hexdigest
    -n, --singles-name               Prints non-duplicate files by file name

OptionParser is a bit verbose when it comes to writing out the command-line definitions. The gem Slop is designed to enable you to write your own command-line parsing with much less effort. Instead of having you provide the blocks of code in the flag definition, Slop simply creates an object you can query in your application to see if a flag was given and what value(s) may have been provided for it.

# Excerpt from https://github.com/leejarvis/slop
opts = Slop.parse do |o|
  o.string '-h', '--host', 'a hostname'
  o.integer '--port', 'custom port', default: 80
  o.bool '-v', '--verbose', 'enable verbose mode'
  o.bool '-q', '--quiet', 'suppress output (quiet mode)'
  o.bool '-c', '--check-ssl-certificate', 'check SSL certificate for host'
  o.on '--version', 'print the version' do
    puts Slop::VERSION
    exit
  end
end
ARGV #=> -v --host 192.168.0.1 --check-ssl-certificate
opts[:host]                 #=> 192.168.0.1
opts.verbose?               #=> true
opts.quiet?                 #=> false
opts.check_ssl_certificate? #=> true
opts.to_hash  #=> { host: "192.168.0.1", port: 80, verbose: true, quiet: false, check_ssl_certificate: true }

This simplicity can help simplify your code base and your test suite, as well as speed up your development time.

When writing a simple command-line tool, you often want it to output the results of the work performed. Keeping in mind what the target for this tool is will largely determine how you want the output to look.

You could format the output data as a simple string, list, hash, nested array, JSON, XML, or other form of data to be consumed. If your data is going to be streamed over a network connection, then you'll want to pack the data into a tightly packed string of data. If it's meant for a user's eyes to see, then you'll want to expand it out in a presentable way.

Many existing Linux/Mac command-line tools can print out details in pairs or sets of values. Information can be separated by a colon, spaces, tabs, and indentation blocks. When you're not sure of how exactly it may be used, go for the simplest and most presentable way of presenting the data.

One example of a target you may need to consider is an API testing tool. Many APIs provide a JSON response and can be accessed with a command-line tool like curl. If bandwidth is a concern, then use JSON's to_json method, but if it's meant for local machine work, use pretty_generate.

x = {"hello" => "world", this: {"apple" => 4, tastes: "delicious"}}
require 'json'
puts x.to_json
# {"hello":"world","this":{"apple":4,"tastes":"delicious"}}
puts JSON.pretty_generate( x )
# {
#   "hello": "world",
#   "this": {
#     "apple": 4,
#     "tastes": "delicious"
#   }
# }

You may likewise use YAML for data.

require 'yaml'
puts x.to_yaml
# ---
# hello: world
# :this:
#   apple: 4
#   :tastes: delicious

If you want to have more complex output nicely formatted, then I highly recommend using the awesome_print gem. This will give you specific control of your presentation on output.

This is another kind of output that may occur from command-line applications. When something is wrong or has gone wrong, it's customary to have the command-line tool write to the output known as STDERR. It will appear as regular output, but other tools can verify that the command wasn't successful.

STDERR.puts "Oops! You broke it!"

The more common practice is to use a Logger to write specific details on errors. You may route that to STDERR output on the command line or possibly a log file.

Writing a user interface can be one of the most rewarding things to accomplish. It allows you to apply some artistic design and provide different ways for a user to interact.

The minimal user interface is a display of some text with a prompt waiting for input. It can look as simple as:

Who is your favorite super hero [Superman]/Batman/Wonder-Woman ?

The above indicates the question, the options, and a default option should the user choose to hit the enter key without entering anything in. In plain Ruby, this would look like:

favorite = "Superman"
printf "Who is your favorite hero [Superman]/Batman/Wonder Woman?"
input = gets.chomp
favorite = input unless input.empty?

This is very crude code to be using for input and output, and if you're going to be writing a UI, I highly recommend trying out a gem like highline or tty.

With the highline gem, you could write the above as:

require 'highline/import'
favorite = ask("Who is your favorite hero Superman/Batman/Wonder Woman?") {|question|
  question.in = ["Superman", "Batman", "Wonder Woman"]
  question.default = "Superman"
}

Here, highline will show the question, show the default, and catch any incorrect options entered, notifying the user that they haven't selected one of the options provided.

Both highline and tty have a vast amount of additional things you can do for a menu and user experience, with niceties such as adding colors to your display. But again you need to take into consideration who your target audience is.

The more of a visual experience you provide, the more you need to pay attention to the cross-platform capabilities of these tools. Not all command lines handle the same presentation data the same way, which creates a bad user experience.

The great news is Ruby has a lot of the solutions needed for tooling across different operating systems. When Ruby is compiled for any specific operating system, the source file for RbConfig is generated with absolute values native to the system it was built on, saved in hash format. It is therefore the key to detecting and using OS features.

To see this file with your favorite text editor, you can run the following Ruby code:

editor = "sublime" # your preferred editor here
exec "#{editor} #{RbConfig.method(:ruby).source_location[0]}"

This will show you everything in a presentable manner, which I feel is better than seeing the hash via RbConfig::CONFIG. This hash includes useful things like the commands used for handling the filesystem, Ruby version, system architecture, where everything important to Ruby resides at, and other things like this.

To check the operating system, you may do:

case RbConfig::CONFIG['target_os']
when /mingw32|mswin/
  # Probably a Windows operating system
else
  # Most likely a Unix compatible system like BSD/Mac/Linux
end

For other operating system-specific values, you may look at the source code for Gem::Platform.

Now, the filesystem-specific commands stored here aren't meant to be used from this resource. Ruby has the classes Dir, File, and Pathname written for that.

When you need to know whether an executable exists in the system's PATH, then you'll want to use MakeMakefile.find_executable. Ruby supports building C extensions and one of the nice features they added for that is this ability to find out if the executable exists to call.

But this will generate a log file in your system's current directory every time you run it. So to avoid writing this log file, you will need to do the following:

require 'mkmf'
MakeMakefile::Logging.instance_variable_set(:@log, File.open(File::NULL, 'w'))
executable = MakeMakefile.find_executable('clear') # or whatever executable you're looking for

When you draw a visual windowed menu on a command line, it's preferred that the menu should stay in a fixed position when the display updates. The simplest way to do this is to clear the command line between every write of the full display.

In the example above, I searched for the clear command. On Windows, the shell command for clearing the command line is cls, but you won't be able to find an executable for it because it's part of command.com's internal code.

The good news is that grabbing the string output from Linux's clear command produces this escape code sequence: \e[3J\e[H\e[2J. I and others have tested this across Windows, Mac, and Linux, and this does exactly what we want: clearing the screen for redrawing on it.

This escape string has three different actions it's performing:

CLEAR = (ERASE_SCOLLBACK = "\e[3J") + (CURSOR_HOME = "\e[H") + (ERASE_DISPLAY = "\e[2J")

However, it is preferable to use a command-line library rather than write escape codes yourself. But this one is well worth mentioning.

With anything that writes to something like STDOUT or STDERR, it would be wisest to create an internal variable in your UI Ruby object that can be changed with an optional parameter to new. So when your program normally runs, the value will be STDOUT. But when you write tests, you'll pass in StringIO.new, which you can easily test against.

When trying to read IO from a command executed outside of Ruby, the testing is a bit more involved. You'll likely need to look into Open3.popen3 to handle each IO stream STDIN, STDOUT, and STDERR.

Bash has come to Windows (WSL)! This means you have access to bash commands across the vast majority of operating systems used worldwide. This opens up more possibilities for your own command-line tool.

It's typical to "pipe" commands through each other, sending the output of one command as a stream to the next. Knowing this, you may want to consider adding streaming input support, or think of how to better format your output for the consumption of other command-line tools.

Here's an example of a regex substitution with Bash's sed command receiving its input piped from echo:

echo hello | sed "s/ll/ck n/g"

This outputs heck no. The more you learn about Bash, the better enabled you'll be to get things accomplished on a command line and the better equipped you'll be to write better command-line tools of your own.

There is much to learn when it comes to writing your own command-line tools. The more situations you have to account for, the more complex things become.

Take the time to learn the tools you have at hand, and you'll reap many rewards you didn't know you were missing. I wish you all the best!