textpress

CRAN version CRAN downloads

textpress is an R toolkit for building text corpora and searching them – no custom object classes, just plain data frames from start to finish. It collects URLs from durable sources – RSS and Atom feeds, Wikipedia and its citations, or URLs you already have – then reads, processes, and searches the resulting text through a consistent four-step API: Fetch, Read, Process, Search.

Installation

From CRAN:

install.packages("textpress")

Development version:

remotes::install_github("jaytimm/textpress")

A source-to-corpus example

Select a small set of known publishers, retrieve their recent feed entries, then read the linked pages. read_urls() retains the RSS discovery metadata.

feeds <- textpress::rss_local_rags |>
  subset(state_abbr == "NM") |>
  head(2)

articles <- textpress::fetch_rss(feeds$url)
corpus <- textpress::read_urls(articles)

corpus$text
corpus$meta

There are three ways into the same pipeline:

Starting point Entry point Best for
Known publishers rss_politics or rss_local_rags -> fetch_rss() Recent, source-led collections
A topic fetch_wiki_urls() -> fetch_wiki_refs() Topic- and citation-led discovery
Existing URLs read_urls() directly Curated or externally collected corpora

Each route converges on read_urls() -> nlp_*() -> search_*().

The textpress API

Conventions: corpus is a data frame with a text column plus identifier column(s) passed to by (default doc_id). All outputs are plain data frames or data.tables; pipe-friendly.

1. Fetch (fetch_*)

Collect URLs and provenance from stable sources – not full article text. textpress does not provide general-purpose live web search. Pass fetch results to read_urls() to retrieve their content.

  • fetch_rss(feed_url) – Retrieve recent entries from RSS and Atom feeds; bundled catalogs are available as rss_politics and rss_local_rags.
  • fetch_wiki_urls(query, limit) – Wikipedia article URLs matching a search phrase.
  • fetch_wiki_refs(url, n) – External citation URLs from a Wikipedia article’s References section.

2. Read (read_*)

Scrape and parse URLs into a structured corpus.

  • read_urls(x, ...) – URL vector or fetch-result table → list(text, meta). Passing a table preserves its discovery metadata and doc_id. text is one row per node; meta is one row per URL.

3. Process (nlp_*)

Prepare text for search or indexing.

4. Search (search_*)

Four retrieval modes over the same corpus. Data-first, pipe-friendly.

Function Query type Use case
search_regex(corpus, query) Regex pattern Specific strings, KWIC with inline highlighting.
search_dict(corpus, terms) Term vector Exact phrases and MWEs; built-in dict_generations, dict_political.
search_index(index, query) Keywords BM25 ranked retrieval over a token index.
search_vector(embeddings, query) Numeric vector Semantic nearest-neighbor search; use util_fetch_embeddings() to embed.

RAG & LLM pipelines

textpress is designed to compose cleanly into retrieval-augmented generation pipelines.

Hybrid retrieval – run search_index() and search_vector() over the same chunks, then merge with reciprocal rank fusion (RRF). Chunks that rank well under both term frequency and meaning rise to the top.

Context assembly – nlp_roll_chunks() with context_size > 0 gives each chunk a focal sentence plus surrounding context, so retrieved passages are self-contained when passed to an LLM.

Because the stages exchange plain data frames, their outputs can also be inspected, filtered, saved, or passed to an LLM without conversion to a custom corpus class.

Vignettes

License

MIT © Jason Timm

Citation 

citation("textpress")

Issues

Report bugs or request features at https://github.com/jaytimm/textpress/issues