Searching COCA from R: a Shiny app for tagged corpus extraction

If you work with large annotated corpora, you will know the feeling. You have a research question. You know roughly what you are looking for: a construction, a collocation, a specific part-of-speech sequence. You open the corpus interface, click around for a while, export something that is almost but not quite what you need, and then spend the next hour massaging a spreadsheet into shape, filtering out the noise. The question was linguistic. Somehow, you have ended up doing data engineering. This post is about a Shiny app I built to short-circuit that process, at least for one specific corpus: the quite famous Corpus of Contemporary American English (COCA), as distributed by English Corpora and purchasable from corpusdata.org. I purchased the word/lemma/POS version (the so-called WLP files) about two years ago. If you have done the same, this app may save you a considerable amount of time.

What the app does

COCA’s WLP files are plain text files in which every token is annotated with its lemma and part-of-speech tag, following the UCREL CLAWS7 tagset. A line of corpus text looks roughly like this (source: wlp_fic_2010.inline.txt):

Mortars_nn2 roared_vvd_vvn@ and_cc the_at ground_nn1 shook_vvd beneath_ii Travis_np1 ._y Rock_nn1_vv0@ chips_nn2 flew_vvd from_ii the_at boulders_nn2 behind_ii which_ddq he_pphs1 crouched_vvd ,_y as_csa bullets_nn2 screamed_vvn_vvd past_rl ._y

Each word is glued to its tag by an underscore. Reading left to right, we have:

Token	Word	Tag(s)	Tag meaning
Mortars_nn2	mortars	nn2	Plural common noun
roared_vvd_vvn@	roared	vvd / vvn	Past tense / past participle of lexical verb
and_cc	and	cc	Coordinating conjunction
the_at	the	at	Article
ground_nn1	ground	nn1	Singular common noun
shook_vvd	shook	vvd	Past tense of lexical verb
beneath_ii	beneath	ii	General preposition
Travis_np1	Travis	np1	Singular proper noun
._y	.	y	Punctuation
Rock_nn1_vv0@	Rock	nn1 / vv0	Singular common noun / base-form lexical verb
chips_nn2	chips	nn2	Plural common noun
flew_vvd	flew	vvd	Past tense of lexical verb
from_ii	from	ii	General preposition
the_at	the	at	Article
boulders_nn2	boulders	nn2	Plural common noun
behind_ii	behind	ii	General preposition
which_ddq	which	ddq	Wh-determiner
he_pphs1	he	pphs1	3rd person singular subjective pronoun
crouched_vvd	crouched	vvd	Past tense of lexical verb
,_y	,	y	Punctuation
as_csa	as	csa	Subordinating conjunction, comparative sense
bullets_nn2	bullets	nn2	Plural common noun
screamed_vvn_vvd	screamed	vvn / vvd	Past participle / past tense of lexical verb
past_rl	past	rl	Locative adverb
._y	.	y	Punctuation

Some tokens carry ambiguous or double tags, separated by @, a charming quirk of the tagger that you quickly learn to either love or tolerate:

• roared_vvd_vvn@ — roared carries two tags separated by an underscore: past tense of a lexical verb (vvd) and past participle (vvn). The @ at the end signals that the tagger was uncertain and left both candidates in place rather than committing to one. This is what the documentation calls an ambiguous or double tag.
• Rock_nn1_vv0@ — Rock gets two tags: singular common noun (nn1) and base form of a lexical verb (vv0). Again the @ marks the tagger’s indecision — rock can be either, and without more context the tagger declined to choose.
• screamed_vvn_vvd — screamed with two tags again: past participle (vvn) and past tense (vvd), no @ this time: both tags are written consecutively with an underscore, which is a slightly different presentation of the same ambiguity. The extractor handles both forms.

Two things worth pointing out to beginners. First, ambiguous tags are not errors. They are the automatic tagger’s record of genuine lexical ambiguity. Out of context, rock really is both a noun and a verb. Roared in certain constructions really could be either a past tense or a past participle. Second, the sentence boundary marker is just another token with the tag y: the corpus has no special sentence-level structure beyond what the punctuation itself provides. When the extractor strips tags, all of this disappears and you are left with plain readable text. The annotation is only there for the search.

The files are named systematically: wlp_acad_1990.inline.txt, wlp_blog_01.inline.txt, and so on, encoding genre and, where applicable, year.

The app lets you search across all of this (or a carefully selected subset of it) using a simple token-by-token pattern builder. You specify what you are looking for (a word, a tag, or a combination of both), choose your genres and year range, set a context window, click Extract, and get a CSV file with left context, match, right context, and metadata. No R knowledge required to run it. No command line. No waiting for a corpus interface to time out.

Before you start

You need two things. First, the raw corpus files. The app works specifically with the WLP inline format from COCA. The filenames follow two naming conventions:

• Dated genres (academic, fiction, magazine, news, spoken, TV & movies): wlp_genre_YYYY.inline.txt — e.g. wlp_acad_1990.inline.txt, wlp_fic_2015.inline.txt
• Undated genres (blog and web): wlp_genre_NN.inline.txt — e.g. wlp_blog_01.inline.txt, wlp_web_29.inline.txt

In both cases, the genre code and the identifier (year or sequence number) are separated by underscores. Blog and web files receive NA in the year column of the output.

As I wrote in the introduction, these are not freely available. You need to purchase them from corpusdata.org. Mark Davies offers several formats and several corpora; the one you want is COCA with word, lemma, and part-of-speech annotation. Once purchased and downloaded, unzip everything into a single folder. That folder path is the first thing the app will ask you for.

Second, you need R of course, with a handful of packages. Here is the install command, which you only need to run once:

install.packages(c("shiny", "shinydashboard", "shinyWidgets", "DT", "stringi", "data.table", "future", "promises", "shinyjs"))

That is nine packages, all from CRAN, none of them exotic. The heaviest dependency is stringi, which does the actual regex work, and future plus promises, which handle the asynchronous extraction so the interface does not freeze while you wait.

Loading the app

Download the app file coca_extractor_app.R from my GitHub repository (https://github.com/GuillaumeDesa/COCA-extractor). Then, from R or RStudio, run:

shiny::runApp("path/to/coca_extractor_app.R")

That is it. The app opens in your browser. If you are on RStudio, it may open in the built-in viewer pane instead; I recommend clicking “Open in Browser” for a more comfortable experience, particularly if you are working with the Tag Guide or the results table.

There is no installation in the traditional sense, no package to build, no configuration file to edit. The .R file is the app.

A tour of the interface

The app is organised into six tabs, accessible from the left-hand sidebar.

Setup is where you tell the app where your corpus lives and where you want the output to go.

You provide the path to your folder of .inline.txt files, click “Validate”, and the app counts how many files it finds and reports back.

You also set your context window here (how many words to show left and right of each match) and decide whether to include the combined full-context column in the output.

Pattern Builder is where the interesting decisions happen. Rather than asking you to type a raw regular expression (though the generated regex is shown, if you are curious), the app uses a token-by-token grammar. Each line in the text box represents one position in your pattern. The syntax is:

• how/rgq — the word how with the tag rgq (wh- degree adverb)
• */jj — any word with the tag jj (general adjective)
• */[vbz|vbr] — any word tagged either vbz (is) or vbr (are)
• */rg? — an optional degree adverb (the ? marks the token as optional)
• ... — a flexible gap of zero to three words

So to search for how followed immediately by an adjective, you type how/rgq on the first line and */jj on the second. The app assembles the regex, shows you a preview, and lets you run a quick sanity check on the first matching file before committing to the full extraction (“Test on first matched file”). This step may take a while if your pattern is rare or if your pattern is messy or wrong because the script will move on to explore the next 20 corpus files.

The Tag Guide tab, available at all times, lists all CLAWS7 tags with their descriptions, so you do not have to keep a separate reference open.

Corpus is where you filter by genre and year. COCA covers eight genres: academic, blog, fiction, magazine, news, spoken, TV and movies, and web. The year slider runs from 1990 to 2019 and applies to the six dated genres; blog and web files use numeric identifiers rather than years and are included in full when selected. The tab reports how many files match your current selection, which is useful for gauging how long the extraction is likely to take.

Extract shows a pre-flight checklist (corpus files found, pattern valid, output folder exists, filename ends in .csv) before you click the Start button. The extraction runs in a background process, so the interface stays live throughout. A progress log updates every 800 milliseconds: it shows which file is being processed and how many hits it yielded. When it finishes, the app switches automatically to the Results tab.

Results displays the extracted data in a searchable, filterable table, with summary boxes showing the total number of matches, genres covered, year span, and number of files that returned at least one hit. You can filter by genre, by year, or by a substring of the match itself. The green “Download CSV” button writes the current filtered view to a CSV with UTF-8 encoding and a byte-order mark, which means it opens cleanly in Excel without any of the usual encoding theatrics.

What happens under the hood

Each corpus file is read with latin1 encoding and immediately re-encoded to UTF-8. This is not optional; the raw files may contain characters that will silently corrupt if you try to read them as UTF-8 directly. The entire file is collapsed into a single lowercase string, which stringi then searches with the assembled regular expression.

Matches are located with stri_locate_all_regex(), which returns character offsets. For each match, the app slices a buffer of 600 characters to the left and right, strips the POS tags from both buffers with a second regex pass, and takes the outermost N words as context. The tag-stripping handles simple tags (_jj), ambiguous tags (_jj@_nn1), ditto tags (_ii21), and the stray @, <, > symbols that appear as separators in the COCA files. What you get in the output should be clean, readable text.

Metadata (genre, year, file identifier) is parsed from the filename. The app recognises both naming conventions: wlp_acad_1990.inline.txt (genre + four-digit year) and wlp_blog_01.inline.txt (genre + two-digit sequence number). The latter receive NA in the year column, which is expected.

The extraction runs sequentially through your selected files in a future() call. This is a proper background R process, not a forked thread with all the attendant instability that implies on Windows. Progress is written to a temporary log file line by line; a reactiveTimer in the main session polls that file every 800 milliseconds and pushes the updates to the interface. It took me years of R practice to make a progress report like that work nicely. I can now say that I am very proud to have finally implemented it nicely!

Limitations I am not going to pretend do not exist

Speed. The app processes one file at a time in a single background worker. COCA has 248 .inline.txt files. On my machine (a powerful 2022 Macbook pro with an Apple M1 Max chip and 64 Go of RAM), a simple two-token pattern across all genres and all years takes somewhere between 3 and 5 minutes. That is great for large extractions that you run locally. But if you are running dozens of queries in a session, you may find yourself spending a lot of time watching the progress log. A file-level parallelisation layer is on the list.

Memory. Very large result sets (tens of thousands of matches) are held in memory until you download them. This has not caused problems in practice because I have lots of RAM, but it is something to be aware of if you are working on a machine with limited RAM and a very common pattern.

The pattern builder has limits. The token grammar handles the vast majority of POS-sequence queries I have needed, but it is not a full query language. You cannot express constraints across non-adjacent positions (e.g., agreement in number between a subject and a verb separated by an intervening clause), and there is no lookahead or lookbehind. For those cases, the generated regex is exposed in the Pattern Builder tab. You can in principle edit it by hand, though the app does not currently support freehand regex input. That is a deliberate choice: the token grammar is there to lower the barrier, not to replace the regex entirely.

Will this work with other corpora?

Almost certainly yes, with some adaptation. The CLAWS7 tagset and the word_TAG annotation format are not unique to COCA. The British National Corpus XML edition, the BNC Baby, and several other corpora distributed by UCREL use the same or a closely related tagset. The filename parsing would need to be updated (it is currently hardcoded to the wlp_genre_year.inline.txt convention) but the core extraction logic is indifferent to the specific tags in your tagset. If your corpus uses word_TAG tokens separated by whitespace and collapsed into flat text files, it will work.

The more interesting question is whether this approach generalises to syntactically annotated corpora such as treebanks, dependency-parsed corpora, and the like. Here I am more cautious. A corpus annotated with Universal Dependencies, for instance, stores its annotation in CoNLL-U format, where each token occupies a separate line with multiple tab-separated fields. Searching that with a flat regex over a collapsed string is neither natural nor reliable. You would want to search over dependency arcs, not character sequences. That is a fundamentally different problem, and one that would require a fundamentally different app, one that works with a parsed representation rather than raw text. I have thought about this. I have not yet built it.

What I can say is that the architecture of this app (the token grammar, the background extraction, the metadata handling, the context stripping) would translate reasonably well to any flat, whitespace-tokenised annotation format. Penn Treebank–style POS-tagged files, for instance, would need only a new filename parser and a tag-stripping regex adjusted to the word/TAG convention rather than word_TAG. The rest would carry over.

A note on the corpus itself

Again, COCA is not free. A licence for the WLP files costs money, and that money goes to Mark Davies and his company English-corpora.org. I have no affiliation with either. I just think the resource was worth paying for, from my position as a corpus linguist and I happened to have a research budget available. If your institution has a subscription that covers corpus data access, check before purchasing individually.

Closing remarks

On GitHub, the code is commented throughout, including the token grammar builder and the async extraction logic, so if you want to understand what is happening, or adapt it for your own corpus, you have somewhere to start. Feedback, bug reports, and suggestions are welcome in the usual places.

If you end up using this in your research or your teaching, I would like to hear about it. Not for the usual reasons of academic vanity: I am genuinely curious about what people actually search for, and whether the token grammar covers it, or whether there are query types I have not thought of. That kind of feedback is how apps get better.

---

The text only may be used under licence Creative Commons Attribution Non Commercial 4.0 International. All other elements (illustrations, imported files) are “All rights reserved”, unless otherwise stated.

---

---