Build a BM25 index for ranked keyword search

Source: R/nlp_index_tokens.R

Build a weighted BM25 index for ranked keyword search. Creates a searchable index from a named list of token vectors. The unit-id column name is taken from attr(tokens, "id_col") when present (e.g. from nlp_tokenize_text), else "uid".

nlp_index_tokens(tokens, k1 = 1.2, b = 0.75, stem = FALSE)

Arguments

tokens

Named list of character vectors (e.g. from nlp_tokenize_text).

k1

BM25 saturation parameter (default 1.2).

b

BM25 length normalization (default 0.75).

stem

Logical. If TRUE, stem tokens (default FALSE).

Value

Data.table with unit-id column, token, score; attr(., "id_col") set for search_index.