Improving the ‘Relevance’ of Website Search Queries using ElasticSearch

Sep 12, 2020
|
Posted by: Rohit Paliwal
|
Category: Tips and Tricks

Relevance is essential when you want to serve relatable data to your users which meets their requirements. It’s the key functionality of any business. A good marketing strategy to engage your audience is to satisfy the customers needs by giving them what they are searching for.

Having bad search can result in a noticeable decrease in the traffic on your platform. This is because users have been trained to expect results similar to Google searches which is not easy to acquire but with Elasticsearch, you can achieve it by tuning your search results with multiple factors.

Scoring in Elasticsearch

Elastic search uses a scoring system to filter and rank the query results that get displayed to the users. Scoring is done on the basis of field matches with the search query and other applied configuration which we will talk about later in this article. Elasticsearch calculates scores of the documents and uses this as a relevance factor to sort the documents. Higher the score implies higher the relevance of the document. Each clause in the query contributes to the score of the document.

The Practical Scoring Function

Since Elasticsearch is built over the Lucene library, so for calculating the relevance score elasticsearch uses Lucene’s Practical Scoring Function. This takes the boolean model, term frequency (TF), inverse document frequency (IDF) and vector space model for multi-term queries and combines them to assemble the matching documents and scores them as it goes.

For example, a multi-term query be like

GET /my_index/doc/_search
{
"query": {
"match": {
"text": "quick fox"
          }
      }
}

is rewritten internally to look like this:

GET /my_index/doc/_search
{
  "query": {
    "bool": {
      "should": [
        {"term": { "text": "quick" }},
        {"term": { "text": "fox"   }}
      ]
    }
  }
}

When a document matches the query, Lucene calculates the score by combining the score of each matching term. This scoring calculation is done by the practical scoring function.

score(q,d)  =  
           queryNorm(q)  
          · coord(q,d)    
          · ∑ (           
                tf(t in d)   
              · idf(t)²      
              · t.getBoost() 
              · norm(t,d)    
            ) (t in q) 

 

Now, let’s get more familiar with each of the scoring mechanisms that make up the Practical Scoring Function:

tf = sqrt(termFreq)

The idea behind calculating term frequency is that the more time a term appears in a document, the more relevant the document is.

idf = 1 + ln(maxDocs/(docFreq + 1))

A common term, for example: ‘the’, which appears in almost all the documents should be considered less important than other terms which are included in fewer documents. This is the reason for calculating inverse document frequency.

norm = 1/sqrt(numFieldTerms)

For field length normalization, a document containing less number of terms and having the query term considered to be more relevant than a document containing more terms along with the search term.

 

Explain API:

Now we know how elasticsearch’s practical scoring function works. Before going any further, let’s talk about an important tool that we will be using for debugging the score for any query in Elasticsearch – “explain”.

Explain API returns information about why a specific document matches (or doesn’t match) a query. For example :

GET /twitter/_explain/0
{
      "query" : {
        "match" : { "message" : "elasticsearch" }
      }
}

The API returns the following response:

{
   "_index":"twitter",
   "_type":"_doc",
   "_id":"0",
   "matched":true,
   "explanation":{
      "value":1.6943598,
      "description":"weight(message:elasticsearch in 0) [PerFieldSimilarity], result of:",
      "details":[
         {
            "value":1.6943598,
            "description":"score(freq=1.0), computed as boost * idf * tf from:",
            "details":[
               {
                  "value":2.2,
                  "description":"boost",
                  "details":[]
               },
               {
                  "value":1.3862944,
                  "description":"idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                  "details":[
                     {
                        "value":1,
                        "description":"n, number of documents containing term",
                        "details":[]
                     },
                     {
                        "value":5,
                        "description":"N, total number of documents with field",
                        "details":[]
                     }
                  ]
               },
               {
                  "value":0.5555556,
                  "description":"tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                  "details":[
                     {
                        "value":1.0,
                        "description":"freq, occurrences of term within document",
                        "details":[]
                     },
                     {
                        "value":1.2,
                        "description":"k1, term saturation parameter",
                        "details":[]
                     },
                     {
                        "value":0.75,
                        "description":"b, length normalization parameter",
                        "details":[]
                     },
                     {
                        "value":3.0,
                        "description":"dl, length of field",
                        "details":[]
                     },
                     {
                        "value":5.4,
                        "description":"avgdl, average length of field",
                        "details":[]
                     }
                  ]
               }
            ]
         }
      ]
   }

As you can see, this is the detailed view of how the scoring is done for all the factors for a query. Hence, ‘explain’ API can help knowing whether a document matches the search query or not. 

Factors Affecting Relevance :

Now let’s take a look at other factors that help us in tuning the relevancy.

Indexing & Mapping : Indexing your data is quite important and has a major impact on your search. By default elasticsearch indexes all the fields and dynamically maps them with matching data type. This could be done explicitly considering the case if you want to use the field for full-text search or keyword search.

For example :

PUT /my-index
{
  "mappings": {
    "properties": {
      "age":    { "type": "integer" },  
      "email":  { "type": "keyword"  }, 
      "name":   { "type": "text"  }     
    }
  }
}

Here, age is an integer field, email is a keyword field and name is text field.

Boosting : Fields can be boosted to elevate the relevance score to get the most relatable document in a specific scenario. Consider an example of a blog website where you want the title matching the search query up rather than the body matching the query. For such scenarios, boosting fields can play an important role.

Boosting can be done at both index time and query time. See the example below:

Index time boosting :

PUT blog-index
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "boost": 2 
      },
      "body": {
        "type": "text"
      }
    }
  }
}

Query time boosting :

POST _search
{
    "query": {
        "match" : {
            "title": {
                "query": "quick brown fox",
                "boost": 2
            }
        }
    }
}

In both the cases, when the query matched the title field, the score gets double than that of the body field.

Often index time boosting is not advisable because of the following reasons:

Synonyms : Setting up synonyms can be useful in those cases where you want multiple words to refer to the same context and should retrieve the same documents. For example, when users search for the terms [ ‘water’, ‘rain’, ‘ocean’ ], you want them to have the same set of documents as a query result because they all share the same meaning in your case.

Similar to boosting, synonyms can be done at indexing time or at query time. Let’s find out the differences:

 

Summary:

Elasticsearch is a very powerful tool and provides an entire ecosystem around the search. This blog only talks about the basic capabilities of Elasticsearch to improve the relevance of the search as this is often the foundation of creating a great user experience in website search.

Following are the important things covered in the blog

How elasticsearch scores the documents – The Practical Scoring Function

Read more about the above discussed topics on Elastic Blogs & Documentation.

References:

Elasticsearch Reference

Elastic Blogs