|[<--] [Cover] [Table of Contents] [Concept Index] [Program Index] [-->]|
There are many ways to use command-line tools to analyze text in various ways, such as counting the number of words in a text, creating a concordance, and comparing texts to see if (and where) they differ. There are also other tricks you can do with text that count as analysis, such as finding anagrams and palindromes, or cutting up text to generate unexpected combinations of words. This chapter covers all these topics.
Use the "word count" tool,
wc, to count characters, words, and
lines in text.
Give the name of a file as an argument; if none is given,
works on standard input. By default,
wc outputs three
columns, displaying the counts for lines, words, and characters in the
$ wc outline [RET]
The following subsections describe how to specify just one kind of count
wc, and how to count text in Emacs.
NOTE: You can get a count of how many different words
are in a text, too -- see Making a Concordance of a Text. To count the average length of words, sentences, and
style (see Checking Text for Readability).
wc with the `-c' option to specify that just the number
of characters be counted and output.
$ wc -c classified.ad [RET]
wc with the `-w' option to specify that just the number
of words be counted and output.
$ wc -w story [RET]
To output counts for several files, first concatenate the files with
cat, and then pipe the output to
$ cat *.txt | wc -w [RET]
NOTE: To read more about concatenation with
wc with the `-l' option to specify that just the number
of lines be counted and output.
$ wc -l outline [RET]
To find the number of occurrences of some text string or pattern in a
file or files, use
grep to search the file(s) for the text
string, and pipe the output to
wc with the `-l' option.
$ grep chapter outline | wc -l [RET]
NOTE: For more recipes for searching text, and more about
grep, see Searching Text.
count-lines-page function in Emacs outputs in the minibuffer
the number of lines on the current page (as delimited by pagebreak
characters, if any---see Paginating Text),
followed by the number of lines in the buffer before the line that point
is on, and the number of lines in the buffer after point.
Emacs outputs the number of lines per page of the current buffer in the echo area.
For example, if the output in the minibuffer is
Page has 351 lines (69 + 283)
this means that the current page contains 351 lines, and point is on line number 70 -- there are 69 lines before this line, and 283 lines after this line.
A concordance is an index of all the words in a text, along with
their contexts. A concordance-like functionality -- an alphabetical
listing of all words in a text and their frequency -- can be made fairly
easily with some basic shell tools:
$ tr ' ' ' [RET] > ' < naked_lunch | sort | uniq -c [RET]
These commands translate all space characters to newline
characters, outputting the text with each word on its own line; this is
then sorted alphabetically, and that output is passed to
which outputs only the unique lines -- that is, all non-duplicate
lines -- while the `-c' option precedes each line with its
count (the number of times it occurs in the text).
To get a word frequency count -- that is, the total number of
different words in a text -- just pipe the output of the frequency
wc with the `-l' option. This counts all the lines
of its input, which in this case will be the list of unique words, one
$ tr ' ' ' [RET] > ' < naked_lunch | sort | uniq -c | wc -l [RET]
The following recipes show how to analyze a given text for its relevancy to other text, either to keywords or to whole files of text.
You can also use the
diff family of tools to analyze differences
in text; those tools are especially good for comparing different
revisions of the same file (see Comparing Files).
Debian: `rel' WWW: http://www.johncon.com/
rel to analyze text files for relevance to a given set of
keywords. It outputs the names of those files that are relevant to the
given keywords, ranked in order of relevance; if a file does not meet
the criteria, it is not output in the relevance listing.
rel takes as an option the keyword to search for in quotes; you
can build a boolean expression by grouping multiple keywords in
parentheses and using any of the following operators between them:
$ rel "(saving & profit)" report.a report.b report.c [RET]
$ rel "(invitation | request)" ~/mail [RET]
$ rel "(invitation ! wedding)" ~/mail [RET]
$ rel "(invitation & party)" ~/mail [RET]
Debian: `remembrance-agent' WWW: http://www.media.mit.edu/~rhodes/RA/
The purpose of the Remembrance Agent is to analyze the text you type in an Emacs session and, in the background, find similar or relevant passages of text within your other files. It then outputs in a smaller window a list of suggestions -- those files that it has found -- which you can open in a new buffer.
When installing the Remembrance Agent, you create three databases of the
files to use when making relevance suggestions; when
remembrance-agent is running, it searches these three databases
in parallel, looking for relevant text. You could create, for example,
one database of saved email, one of your own writings, and one of saved
C-c r t
remembrance-agent is running, suggested buffers will be
displayed in the small `*Remembrance*' buffer at the bottom of the
screen. To open a suggestion in a new buffer, type C-c r
number, where number is the number of the suggestion.
C-c r 2
An anagram is a word or phrase whose characters consist entirely of all the characters of a given word or phrase -- for example, `stop' and `tops' are both anagrams of `pots'.
an to find and output anagrams. Give as an argument the word
or quoted phrase to use;
an writes its results to the standard
$ an lake [RET]
$ an 'lakes and oceans' [RET]
To limit the anagrams output to those containing a given string, specify that string with the `-c' option.
$ an -c seas 'lakes and oceans' [RET]
To print all of the words that some or all letters in a given word or phrase can make, use the `-w' option. This outputs words that are not anagrams, since anagrams must contain all of the letters of the other word or phrase.
$ an -w seas [RET]
This command outputs all of the words that can be formed from all or some of the characters in `seas', including `see' and `as'.
A palindrome is a word that reads the same both forwards and backwards; for example, "Mom," "madam," and "nun" are all palindromes.
To find palindromes in a file, use this simple Perl "one-liner," and substitute file for the name of the file to check:
perl -lne 'print if $_ eq reverse' file
To check for palindromes in the standard input, specify `-' as the file name to check. This is useful for putting at the end of a pipeline.
$ perl -lne 'print if $_ eq reverse' /usr/dict/words [RET]
A cut-up is a random rearrangement of a physical layout of text, made with the intention of finding unique or interesting phrases in the rearrangement. Software for rearranging text in random ways has existed since the earliest text-processing tools; the popularity of these tools will never die.
The cut-up technique in literature was discovered by painter Brion Gysin and American writer William S. Burroughs in 1959; they believed it brought the montage technique of painting to the written word.
"All writing is in fact cut-ups," Burroughs wrote.(21) "A collage of words read heard overheard ... [u]se of scissors renders the process explicit and subject to extension and variation."
These recipes describe a few of the common ways to make text cut-ups; more free software tools for making cut-ups are listed at http://dsl.org/comp/cutups.shtml.
To perform a simple cut-up of a text, use
cutup. It takes the
name of a file as input and cuts it both horizontally and vertically
along the middle, rearranges the four sections to their diagonally
opposite corners, and then writes that cut-up to the standard
output. The original file is not modified.
$ cutup nova [RET]
Debian: `dadadodo' WWW: http://www.jwz.org/dadadodo/
No simple cut-up filter, Jamie Zawinski's
dadadodo uses the
computer to go one step beyond -- it generates passages of random text
whose structure and characters are similar to the text input you give
it. The program works better on larger texts, where more subtleties can
be analyzed and hence more realistic-looking text is output.
Give as an argument the name of the text file to be used; by default,
dadadodo outputs text to standard output until you interrupt it
by typing C-c.
$ dadadodo nova [RET]
This command will output passages of random text based on the text in the file `nova' until it is interrupted by the user.
You can analyze a text and save the analysis to a file of compiled data;
this analysis can then be used to generate random text when the original
input text is not present. The following table describes this and other
||Generate integer sentences (default is 0, meaning "generate an infinite amount until interrupted").|
||Load compiled data in file and use it to generate text.|
||Output compiled data to file file for later use.|
||Pause for integer seconds between paragraphs.|
dissociated-press function in Emacs makes random cut-ups of
the current buffer in a new buffer called `*Dissociation*'; the
original buffer is not modified. The text in the new buffer is generated
by combining random portions of the buffer by overlapping characters or
words, thus (usually) creating plausible-sounding sentences. It pauses
occasionally and asks whether or not you want to continue the
M-x dissociated-press [RET]
Give a positive argument to the
dissociated-press function to
specify the number of characters to use for overlap; give a negative
argument to specify the number of words for overlap.
C-u 3 M-x dissociated-press [RET]
C-u -1 M-x dissociated-press [RET]
[<--] [Cover] [Table of Contents] [Concept Index] [Program Index] [-->]