Read content from URLs — read

Input: character vector of URLs. Output: structured data frame (one row per node: headings, paragraphs, lists). Like read_csv or read_html: bring an external resource into R. Follows fetch_rss or fetch_wiki_urls in the pipeline—fetch gets locations, read gets text. Wikipedia uses high-fidelity selectors; use parent_heading to see which section each node belongs to. External links and empty text rows are omitted; optionally exclude References/See also/Bibliography/Sources sections for wiki URLs.

read_urls(
  x,
  cores = 1,
  detect_boilerplate = TRUE,
  remove_boilerplate = TRUE,
  exclude_wiki_refs = TRUE
)

Arguments

x: Character vector of URLs, or a data frame containing a url column. Data-frame metadata is preserved in the returned meta table; an existing doc_id is honored.
cores: Number of cores for parallel requests (default 1).
detect_boilerplate: Logical. Detect boilerplate (e.g. sign-up, related links).
remove_boilerplate: Logical. If detect_boilerplate is TRUE, remove boilerplate rows; if FALSE, keep them and add is_boilerplate.
exclude_wiki_refs: Logical. For Wikipedia URLs only, drop nodes whose parent_heading is References, See also, Bibliography, or Sources. Default TRUE.

Value

A list with text (node-level data: doc_id, url, node_id, parent_heading, text, and optionally type, is_boilerplate) and meta (one row per URL with doc_id, url, scraped fields, and any metadata supplied in x). For character input, doc_id is an integer key in first-appearance order.

Examples

if (FALSE) { # \dontrun{
feeds <- subset(rss_politics, category == "polling")
articles <- fetch_rss(feeds$url)
out <- read_urls(articles, cores = 1)
nodes <- out$text
meta <- out$meta
} # }