Video summary
The video introduces the concept of pipelines in Linux as a powerful mechanism for chaining commands together to process data efficiently. It begins by demonstrating how standard input redirection works with tools like `less` to view file contents page by page, but highlights a limitation when trying to apply this to command output that is not stored in a file. To solve this, the tutorial explains that Linux allows users to connect the output of one command directly to the input of another using a vertical bar symbol known as a pipe. This capability transforms individual utilities into a cohesive workflow, enabling tasks that no single tool could perform alone.
Several practical examples illustrate how pipelines streamline complex operations. The presenter shows how to list directory contents and immediately sort them alphabetically while removing duplicate entries by chaining `ls`, `sort`, and `unique` commands. Another scenario involves searching for specific patterns; by piping the output of an `ls` command into `grep`, users can filter files containing a particular string, and further piping that result into `wc` allows for counting matches without manually reviewing every line. These examples emphasize how small utilities can be combined to achieve sophisticated data processing goals with minimal effort.
The tutorial concludes with an advanced demonstration of a long pipeline chain designed to extract unique words from a text file about Gertrude Stein. The process involves using `cat` to read the file, followed by multiple instances of the `tr` utility to remove punctuation, convert text to lowercase, and replace spaces with newlines so that each word appears on its own line. After cleaning the data, the output is sorted and deduplicated before being displayed through `less`. While the presenter notes that such complex chains might not be necessary for everyday tasks, they serve to showcase the immense flexibility and power inherent in Linux pipelines for manipulating text streams.
Read the full video transcript
Let's start with some things we've done
before.
I'm going to see what's in my current
directory with ls -l.
If I want to find out what's in this
file,
I can use the less command.
Less is a program that accepts standard
input.
So, I'm going to explicitly redirect
here and say that less should get its
input from the file euro_us.csv.
And there are the file contents exactly
as expected.
And we can page through that.
And again, nothing new here.
But let's take a look behind the scenes.
We can think of less as a sort of black
box
that has an input, in this case a file.
It processes that input and gives us
some output, namely the file in pages.
Keep that in mind.
And now let's return to the console.
Let's do another command that produces a
lot of output. We're going to do a long
listing of /user/bin.
There are so many files in this
directory
that even if I were to scroll back to
the very beginning of my window,
I wouldn't be at the start of the
listing.
It would be nice if we could use less to
see all the output from this listing one
page at a time.
Except for one problem.
We can't use redirection here because
the output of ls -l isn't a file.
What we would like is some way to take
the output of ls -l
and feed that into the input for less.
Linux provides exactly this solution
with something called a pipeline.
We do the ls -l command
and use a vertical bar
to pipe the output of ls
to the input of the less command.
Here it is in action.
We list the directory ls -l /user/bin
and we pipe it
to less.
There, that's much better.
Now I can page through the results at my
leisure.
And I can see the beginning of the
output as well.
Pipelines are one of the best features
of Linux.
We can take several small utilities and
chain them together to do a task that no
single utility could do.
For example, here are two directories.
What I would like is a sorted list of
those names with duplicates eliminated.
Notice that this name exists both in
/user and also in /var.
I can use a pipeline to do this.
I can do the ls command,
pipe that result to sort to get
everything in alphabetical order,
and then pipe that to
to get rid of duplicates.
I'll bring back the command
and add a pipe
to sort
and then pipe that to unique.
And there I have it and you'll notice
that this name appears only once because
I got rid of duplicates.
Here's another example.
If I want to find how many files in
/usr/bin
have HTML as part of their name,
I can list the file
and then pipe it to grep.
The grep utility prints all lines that
match a specific pattern and the pattern
I'm looking for is HTML.
And then I'll pipe that
to the word count utility
to see how many lines there are.
And it turns out there are 11 such
files.
If I wanted to see what those files were
and omit the final pipe
and that would show me all the lines
that match the pattern HTML.
For my final demonstration here, I'm
going to do a very long pipeline chain.
Here are the contents of a text file
about Gertrude Stein
who is an American poet, author, and
playwright.
And what I want as a result is a sorted
list of all the unique words in this
file.
Here's my plan.
I'm going to show the file using cat.
Then I'm going to get rid of all the
punctuation marks.
I'll convert everything to lowercase
and convert all spaces to new lines,
which will give me one word per line.
Now, I can sort that into alphabetical
order
and eliminate duplicates
and use less to show the result one page
at a time.
I'm going to cat stein.text
and then pipe that to tr, the translate
utility,
and tell it to delete
anything that is punctuation.
To keep this manageable, I'm going to
continue to the next line. To continue
to the next line, I'm going to end this
line with a backslash.
And now, I have gotten rid of the
punctuation
and I need to pipe that to another
invocation of tr that will change
all my uppercase
to lowercase.
Continue to next line.
And pipe that result to yet another
invocation of tr
that will take all of my spaces
and convert them into new line
characters, which in Linux is denoted by
backslash n.
Now that I have my file cleaned up with
one word per line,
I can pipe that to sort
and pipe that result to unique to get
rid of duplicates
and finally
pipe everything to less.
And there are all the unique words in
that file
in alphabetical order.
I don't expect you to do things this
complicated with pipelines right away,
but I wanted to show you this to show
you some of the power of pipelines.
In summary, when you want to take the
output of one program and feed it to the
input of another program,
use a pipe.