Skip to main content

Edit this page

Synonyms

Shoppers and your catalog rarely use the same word. Someone types "telly", the product says "television" and nothing matches. A synonym dictionary closes that gap: you declare the words that mean the same thing and SereneDB expands them at analysis time, on both the indexed text and the query, so either side finds the other.

Synonyms are a step in a text search dictionary. You wrap a tokenizer and a synonym filter in a pipeline: the tokenizer lowercases and splits the text, then the synonym step rewrites the tokens. There are two filters. solr_synonyms takes a map you write by hand and keeps the surface words, and wordnet_synonyms normalizes words to a shared sense from the WordNet database. Most of this recipe uses solr; the last section shows WordNet.

Four products sit in a catalog indexed on name, behind a dictionary that treats tv, television and telly as one word and files every laptop under notebook.

Schema and sample data
Query
CREATE TEXT SEARCH DICTIONARY syn (    template = 'pipeline',    step1_template = 'text',    step1_locale = 'en_US.UTF-8',    step1_case = 'lower',    step1_stemming = false,    step2_template = 'solr_synonyms',    step2_synonyms = 'tv, television, tellylaptop => notebook',    frequency = true,    position = true);
CREATE TABLE catalog (id INTEGER PRIMARY KEY, name VARCHAR);
CREATE INDEX catalog_idx ON catalog USING inverted (id, name syn);
INSERT INTO catalog VALUES(1, 'wall mounted tv'),(2, 'smart television screen'),(3, 'gaming laptop'),(4, 'wireless headphones');
VACUUM (REFRESH_TABLE) catalog;

Find the word the shopper did not type​

The catalog never says "telly", but the query does. Expansion runs on the indexed text too, so the tv and television rows both carry "telly" and the search lands.

Query
SELECT id, name FROM catalog_idx WHERE name @@ 'telly' ORDER BY id;
Result
 id | name----+-------------------------  1 | wall mounted tv  2 | smart television screen

It works both ways​

A comma group is symmetric, so it does not matter which member is in the document and which is in the query. Searching "tv" finds the row that says "television" just as well.

Query
SELECT id, name FROM catalog_idx WHERE name @@ 'tv' ORDER BY id;
Result
 id | name----+-------------------------  1 | wall mounted tv  2 | smart television screen

Rewrite one word onto another​

Use laptop => notebook when you want a one-way rewrite rather than a group. Every "laptop" is filed as "notebook" so a search for "notebook" finds it, without dragging unrelated notebook senses back onto laptops.

Query
SELECT id, name FROM catalog_idx WHERE name @@ 'notebook' ORDER BY id;
Result
 id | name----+---------------  3 | gaming laptop

See what a word expands to​

ts_lexize runs a single value through the dictionary so you can see the expansion the index and the query both get. A two-way group returns every member, a one-way rule returns the target.

Query
SELECT ts_lexize('syn', 'telly') AS telly, ts_lexize('syn', 'laptop') AS laptop;
Result
 telly                 | laptop-----------------------+------------ {television,telly,tv} | {notebook}

Normalize to a shared sense with WordNet​

The wordnet_synonyms filter takes a different tack. Instead of adding synonyms to the surface word it replaces each word with the numeric id of its WordNet sense, so words that mean the same thing collapse to one term and match each other. Here "couch", "sofa" and "divan" share a sense, so any of them finds all three.

Query
SELECT id, name FROM furniture_idx WHERE name @@ 'sofa' ORDER BY id;
Result
 id | name----+---------------  1 | leather couch  2 | fabric sofa  3 | velvet divan

The catch is that WordNet only knows the words in its map. A word with no sense is dropped, not passed through, so ts_lexize returns an empty list for it and that word becomes unsearchable. Load the full WordNet database when you want broad coverage, or reach for solr when you would rather hand-write a few groups and keep every other word intact.

Query
SELECT ts_lexize('wn', 'sofa')  AS sofa,       ts_lexize('wn', 'couch') AS couch,       ts_lexize('wn', 'table') AS unknown;
Result
 sofa        | couch       | unknown-------------+-------------+--------- {100000001} | {100000001} | {}

See also​