Regex and grep: Data flow and building blocks

In Introducing regular expressions, I covered what they are and why they’re useful. Now let’s take a deeper look at how they’re created. Because GNU grep is one of the tools I use the most (that provides a more or less standardized implementation of regular expressions), I will use that set of expressions as the basis for this article. We will then look at sed (another tool that uses regular expressions) in a later article.

All implementations of regular expressions are line-based. A pattern created by a combination of one or more expressions is compared against each line of a data stream. When a match is made, an action is taken on that line as prescribed by the tool being used.

For example, when a pattern match occurs with grep, the usual action is to pass that line to STDOUT and discard lines that do not match the pattern. As we saw in Getting started with regular expressions: An example, the -v option reverses those actions, so that the lines with matches are discarded.

Each line of the data stream is evaluated on its own. Think of each data stream line as a record, where the tools that use regexes process one record at a time. When a match is made, an action defined by the tool in use is taken on the line that contains the matching string.

Regex building blocks

The following table contains a list of the basic building block expressions and metacharacters implemented by the GNU grep command (and most other regex implementations), and their descriptions. When used in a pattern, each of these expressions or metacharacters matches a single character in the data stream being parsed:

Let’s explore these building blocks before continuing on with some of the modifiers. The text file we will use for Experiment 3 is from a lab project I created for an old Linux class I used to teach. It was originally in a LibreOffice Writer odt file but I saved it to an ASCII text file. Most of the formatting of things like tables was removed, but the result is a long ASCII text file that we can use for this series of experiments.

Example: TOC entries

Let’s take a look at an example to explore what we’ve just learned. First, make the ~/testing directory your PWD (create it if you didn't already in the previous article in this series), and then download the sample file from the GitHub.

[student@studentvm1 testing]$  wget https://raw.githubusercontent.com/opensourceway/reg-ex-examples/master/Experiment_6-3.txt

To begin, use the less command to look at and explore the Experiment_6-3.txt file for a few minutes to get an idea of its content.

Now, let’s use some simple grep expressions to extract lines from the input data stream. The Table of Contents (TOC) contains a list of projects and their respective page numbers in the PDF document. Let’s extract the TOC starting with lines ending in two digits:

[student@studentvm1 testing]$  grep [0-9][0-9]$ Experiment_6-3.txt

This command is not really what we want. It displays all lines that end in two digits and misses TOC entries with only one digit. We'll look at how to deal with an expression for one or more digits in a later experiment. Looking at the whole file in less, we could do something like this.

[student@studentvm1 testing]$ grep "^Lab Project" Experiment_6-3.txt | grep "[0-9]$"

This command is much closer to what we want, but it is not quite there. We get some lines from later in the document that also match these expressions. If you study the extra lines and look at those in the complete document, you can see why they match while not being part of the TOC.

This command also misses TOC entries that do not start with "Lab Project." Sometimes this result is the best you can do, and it does give a better look at the TOC than we had before. We will look at how to combine these two grep instances into a single one in a later experiment.

Now, let’s modify this command a bit and use the POSIX expression. Note the double square braces ([[]]) around it:

[student@studentvm1 testing]$ grep "^Lab Project" Experiment_6-3.txt | grep "[[:digit:]]$"

Single braces generate an error message.

This command gives the same results as the previous attempt.

Example: systemd

Let’s look for something different in the same file:

[student@studentvm1 testing]$ grep systemd Experiment_6-3.txt

This command lists all occurrences of "systemd" in the file. Try using the -i option to ensure that you get all instances, including those that start with uppercase letters (the official form of "systemd" is all lowercase). Or, you could change the literal expression to Systemd.

Count the number of lines containing the string systemd. I always use -i to ensure that all instances of the search expression are found regardless of case:

[student@studentvm1 testing]$ grep -i systemd Experiment_6-3.txt | wc
20      478     3098

As you can see, I have 20 lines, and you should have the same number.

Example: Metacharacters

Here is an example of matching a metacharacter: the left bracket ([). First, let’s try without doing anything special:

[student@studentvm1 testing]$  **grep -i "[" Experiment_6-3.txt**
grep: Invalid regular expression

This error occurs because [ is interpreted as a metacharacter. We need to escape this character with a backslash (\) so that it is interpreted as a literal character and not as a metacharacter:

[student@studentvm1 testing]$ grep -i "\[" Experiment_6-3.txt

Most metacharacters lose their special meaning when used inside bracket expressions:

• To include a literal ], place it first in the list.
• To include a literal ^, place it anywhere but first.
• To include a literal [, place it last.

Repetition

Regular expressions can be modified using operators that let you specify zero, one, or more repetitions of a character or expression. These repetition operators are placed immediately following the literal character or metacharacter used in the pattern:

As an example, run each of the following commands and examine the results carefully, so that you understand what is happening:

[student@studentvm1 testing]$  **grep -E files? Experiment_6-3.txt**
[student@studentvm1 testing]$  **grep -Ei "drives*" Experiment_6-3.txt**
[student@studentvm1 testing]$  **grep -Ei "drives+" Experiment_6-3.txt**
[student@studentvm1 testing]$  **grep -Ei "drives{2}" Experiment_6-3.txt**
[student@studentvm1 testing]$  **grep -Ei "drives{2,}" Experiment_6-3.txt**
[student@studentvm1 testing]$  **grep -Ei "drives{,2}" Experiment_6-3.txt**
[student@studentvm1 testing]$  **grep -Ei "drives{2,3}" Experiment_6-3.txt**

Be sure to experiment with these modifiers on other text in the sample file.

Metacharacter modifiers

There are still some interesting and important modifiers that we need to explore:

We now have a way to specify word boundaries with the \< and \> metacharacters. This means that we can now be even more explicit with our patterns. We can also use logic in more complex patterns.

As an example, start with a couple of simple patterns. This first one selects all instances of drives but not drive, drivess, or additional trailing "s" characters:

 [student@studentvm1 testing]$  **grep -Ei "\<drives\>" Experiment_6-3.txt**

Now let’s build up a search pattern to locate references to tar (the tape archive command) and related references. The first two iterations display more than just tar-related lines:

[student@studentvm1 testing]$ grep -Ei "tar" Experiment_6-3.txt
[student@studentvm1 testing]$ grep -Ei "\<tar" Experiment_6-3.txt
[student@studentvm1 testing]$  grep -Ein "\<tar\>" Experiment_6-3.txt

The -n option in the last command above displays the line numbers for each line in which a match occurred. This option can assist in locating specific instances of the search pattern.

Tip: Matching lines of data can extend beyond a single screen, especially when searching a large file. You can pipe the resulting data stream through the less utility and then use the less search facility which implements regexes, too, to highlight the occurrences of matches to the search pattern. The search argument in less is: \<tar\>.

This next pattern searches for "shell script," "shell program," "shell variable," "shell environment," or "shell prompt" in our test document. The parentheses alter the logical order in which the pattern comparisons are resolved:

[student@studentvm1 testing]$ grep -Eni "\<shell (script|program|variable|environment|prompt)" Experiment_6-3.txt

Note: This article is a slightly modified version of Chapter 6 from Volume 2 of my Linux book, "Using and Administering Linux: Zero to SysAdmin," due out from Apress in late 2019.

Remove the parentheses from the preceding command and run it again to see the difference.

Wrapping up

Although we have now explored the basic building blocks of regular expressions in grep, there are an infinite variety of ways in which they can be combined to create complex yet elegant search patterns. However, grep is a search tool, and does not provide any direct capability to edit or modify a line of text in the data stream when a match is made. For that purpose, we need a tool like sed, which I cover in my next article.