Unifying Shell and Programming

Introduction

One of the great things about UNIX1 is the immensely powerful command line. The power of the command line comes from the sophisticated shells that are used on UNIX systems, among which are the classic Bourne shell, the C shell and the Bourne again shell. All these shells support environment variables, pipes2 and functions. This, in conjunction with the many commands that ship with UNIX systems, allows the creation of sophisticated shell scripts that approach compiled programs written in a programming language in terms of power, and are often easier to write.

As with anything, shell scripts suffer from a number of problems. Some of these may not apply to specific shells, but the focus of this article is not on particular shells, but on the general picture. First of, shell scripts are a lot slower than compiled programs. Many software projects have been prototyped as shell scripts first, and later rewritten in C for speed. Secondly, shell syntax tends to get ugly when more sophisticated functionality is used (i.e. functionality that hasn't traditionality been part of the shell, such as trapping signals3). This is a problem, because projects may start out simple enough to be implemented in shell, and then grow beyond what can be elegantly expressed in shell, resulting in either a move to a Real programming language, or a maintenance nightmare due to illegible code. A third problem is that shell syntax is different from that of commonly used programming language, so that developers who use both must learn two syntaxes (and more if we consider makefiles and possible other supplements).

This article is an exploration of the possibility of solving these problems by using a programming language that can be used as a shell, as has been done on, for example, LISP machines and old home computers running BASIC. This could unify the syntaxes of shell and programs, allow a smooth transition from simple to more sophisticated programs, and eliminate speed problems of shell scripts.

Differences Between Shell and Programming Languages

The differences between shell and programming languages mainly arise from their different purposes. Whereas programming languages are designed to write software, the main purpose of the shell is to launch the software that has been written in programming languages. The ability to define functions and gather multiple commands in a shell script, which can then be executed like a program, is more a nice feature than the core function of the shell. This has implications for syntax, speed and the functionality offered.

Shell syntax is primarily designed to be easy for a user to type in. The user will type a command name, often followed by arguments, and then press <return> to execute the command. Usually, there is no need for nesting commands inside other commands, a practice which is very common in programming languages. Thus, programming language typically make use of parentheses to group arguments together and indicate which arguments belong to which command. In the shell, requiring parentheses would induce extra typing, often for no gain, which is clearly undesirable. Programming languages trade simplicity for flexibility, shells sacrifice flexibility in return for simplicity.

Programs are often used to perform operations on large sets of data, making speed an important issue. Typically, programming languages allow a program to be compiled to machine code, the native language of the computer, for maximum speed. In the shell, commands are typically executed once after they have been entered, meaning that first compiling to machine code and then executing may induce extra overhead and actually make the shell slower. Simply interpreting the command entered and invoking a precompiled program to carry out the requested action is usually fast enough that there is no noticible delay for users. Compilation could be useful for many shell scripts, which are executed repeatedly or iterate over larger sets of data.

Whereas many programming languages can be used to access functionality that is provided on a fairly low level, shells typically do not offer these facilities. For example, a program written in C could be used to control a video card, something a shell script would not be able to do. Programming languages can also often be extended by loading libraries or modules, which makes additional functions available to the program. The shell can very easily be extended by installing additional programs (so that, for example, a shell script could control the video card), but these are external programs, that have to be loaded on each invocation and could not be written in shell script. Functions in a library are typically written in the language the library is used with, and are loaded once, after which they can be used indefinitely.

Programming languages also provide types for purposes of optimization or functionality. The C programming language, for instance, provides a number of built-in types such as integers, floating point numbers and characters, which can be used to represent data in a format that the machine can efficiently work with. Lisp provides linked lists to flexibly store data, and provides functions as a data type, so that they can be passed as arguments to function calls, after which they could, for example, be applied to each element in a list. Shells typically provide only strings as a data type, which is good enough for most uses of the shell, but certainly not enough for general purpose programming.

Closing the Gap

A language that is to be used both interactively and for programming needs to have a simple yet powerful syntax, and also have access to the system API. There are numerous examples that come close. The following subsections discuss a few of them.

Perl

Perl is a versatile language that has been successfully employed for system administration tasks, dynamic websites, and programs of various kinds. It has some powerful built-in functionality, such as hashes4 and regular expressions. It provides an interface to many POSIX system calls and library functions, and extension modules have been developed for almost any imaginable task.

There are a number of ways to make Perl into a usable shell. The simplest one is to write a Perl program that reads a line of input, and evaluates that as a Perl program. This provides a reasonably convenient and very powerful shell. Convenience can be further improved by implementing additional functionality, such as word completion, history buffers, and various tweaks to the syntax. This is exactly what is being done by various Perl shell projects, such as psh.

On the other end of the spectrum, we see that Perl is being ported to the Parrot virtual machine. This will allow Perl programs to be compiled ahead of time, which may improve performance at run time. Also, the Parrot code generated from Perl will probably be interoperable with Parrot code generated from other languages, which further increases Perl's utility as a programming language.

scsh

scsh is a very strong contender in the competition for best programming shell. Scheme is a LISP dialect with a very clean syntax and API. It is a full-blown programming language, and LISP dialects have been used as shells before, and LISP is indeed very suitable for interactive use.

scsh is an attempt to bring the power of scheme to the UNIX command line. It offers some features typically found in shells, almost complete access to POSIX functions, and a number of extensions such as support for symbolic links, regular expressions, and networking. In addition, because it runs on top of a full Scheme implementation, it can be extended with Scheme modules. Thus, it scales up all the way from shell to programming language, and provides access to common UNIX APIs to better blend in with the rest of UNIX.

There are a number of issues that stand in the way of scshbeing the perfect unifying language. One is the syntax, which is pretty clean, but requires a little more typing than I find comfortable for a shell. I have a problem with the outer parentheses, and I agree this is nitpicking, but if you have to type an opening and closing paren around every command you enter, you start wondering why not just leave them out. scsh is tightly bound to Scheme 48, which is a decent Scheme implementation, but performance and portability could certainly be better.

Instead of scsh, any Scheme implementation (or other LISP dialect, for that matter) could be used as a shell, without the need for an extra program on top of it. With the advent of the Scheme library, Scheme now offers a full system interface (not just a POSIX interface in Scheme, but a system interface that fits in with Scheme's programming style). Very efficient Scheme implementations exist. If an existing language is to be used to unify shell and programming, Scheme would make a very good choice.

1 UNIX here should be read as any UNIX-like system, including GNU/Linux and any other system with some POSIX compliance and the ability to run UNIX shells

2 Pipes are a very essential part of a UNIX system. A pipe allows the output of a command to be redirected to a file or to another command. Thus, commands can cooperate, allowing individual programs to be small and simple.

3 On UNIX, signals are used to communicate certain events to processes. Some frequently used signals are SIGHUP for reloading configuration files, SIGTERM for gracefully terminating a process, and SIGKILL for killing a process that is running wild. Trapping refers to the process catching the signal and performing certain actions in response to it.

4 Hashes are often called associative arrays, association lists, or dictionaries. They relate keys to values.