Optional Filters

About optional filters

Unlike filters, optional filters don’t remove records from your search results when your query doesn’t match them. Instead, they divide your records into two sets: the results that match the optional filter, and the ones that don’t.

You can leverage this to boost results that you want to show before others, or bury results that you want to show last.

Filter scoring

You can add extra nuance to your ranking by specifying scores for different optional filters. For example, let’s say you want to display your matching products in the following order:

• Apple products first
• Samsung products second
• Huawei products last

In this scenario, products from any other brand than these three must appear after the Apple and Samsung products, and before the Huawei products. You can achieve this by using scored optional filters to boost Apple and Samsung products, and negative filters to bury the Huawei products.

1
2
3
4
5
6
7
$index->search('phone', array(
    'optionalFilters' => array(
        'brand:Apple<score=3>',
        'brand:Samsung<score=2>',
        'brand:-Huawei',
    )
));

1
2
3
4
5
6
7
index.search('phone', {
    optionalFilters: [
      'brand:Apple<score=3>',
      'brand:Samsung<score=2>',
      'brand:-Huawei',
    ]
  })

1
2
3
4
5
6
7
index.search('phone', {
  optionalFilters: [
    'brand:Apple<score=3>',
    'brand:Samsung<score=2>',
    'brand:-Huawei',
  ],
});

1
2
3
4
5
6
7
index.search("phone", {
    "optionalFilters": [
        "brand:Apple<score=3>",
        "brand:Samsung<score=2>",
        "brand:-Huawei",
    ],
})

1
2
3
4
5
6
7
8
9
index.search(query: Query("phone")
  .set(\.optionalFilters, to: ["brand:Apple<score=3>",
                               "brand:Samsung<score=2>",
                               "brand:-Huawei"])
) { result in
  if case .success(let response) = result {
     print("Response: \(response)")
  }
}

1
2
3
4
5
6
7
8
9
10
val query = query("phone") {
    optionalFilters {
        and {
            facet("brand", "Apple", score = 3)
            facet("brand", "Samsung", score = 2)
            facet("brand", "-Huawei")
        }
    }
}
index.search(query = query)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
index.Search<Contact>(new Query {
    SearchQuery = "phone",
    OptionalFilters = new List<List<string>> {
        new List<string> {
            "brand:Apple<score=3>",
            "brand:Samsung<score=2>",
            "brand:-Huawei"
        }
    }
});

// Asynchronous
await index.SearchAsync<Contact>(new Query {
    SearchQuery = "phone",
    OptionalFilters = new List<List<string>> {
        new List<string> {
            "brand:Apple<score=3>",
            "brand:Samsung<score=2>",
            "brand:-Huawei"
        }
    }
});

1
2
3
4
5
6
7
8
9
List<List<String>> optionalFilters = Collections.singletonList(
    Arrays.asList("brand:Apple<score=3>", "brand:Samsung<score=2>", "brand:-Huawei"));
Query query = new Query("phone")
    .setOptionalFilters(optionalFilters);

index.search(query);

// Asynchronous
index.searchAsync(query);

1
2
3
4
5
6
7
8
res, err := index.Search(
  "phone",
  opt.OptionalFilterAnd(
    "brand:Apple<score=3>", 
    "brand:Samsung<score=2>", 
    "brand:-Huawei",
  ),
)

1
2
// Creating Rules isn't currently possible with this client.
// Please use the dashboard instead.

When you don’t specify a score for an optional filter, it defaults to 1.

For performance reasons, you shouldn’t use filter scoring on searches that may return more than 100,000 results.

Score calculation

The engine has three ways of calculating scores:

• calculate the sum of optional filters,
• calculate the maximum score for optional filters,
• or a combination of both.

By default, the engine sums the score of each optional filter in your optionalFilters. The engine determines this score by taking the maximum score of each optional filter.

Let’s see what this looks like in practice.

Simple optional filters

To understand how the engine calculates scores for optionalFilters, let’s score a few records step by step.

1
2
3
4
5
6
$index->search('', array(
    'optionalFilters' => array(
        'brand:Apple<score=2>',
        'type:tablet'
    )
));

1
2
3
4
5
6
index.search('', {
    optionalFilters: [
      'brand:Apple<score=2>',
      'type:tablet'
    ]
  })

1
2
3
index.search('', {
  optionalFilters: ['brand:Apple<score=2>', 'type:tablet'],
});

1
2
3
4
5
6
index.search("phone", {
    "optionalFilters": [
        "brand:Apple<score=2>",
        "type:table",
    ],
})

1
2
3
4
5
6
7
8
index.search(query: Query("phone")
  .set(\.optionalFilters, to: ["brand:Apple<score=2>",
                               "type:Tablet"])
) { result in
  if case .success(let response) = result {
     print("Response: \(response)")
  }
}

1
2
3
4
5
6
7
8
9
val query = query {
    optionalFilters {
        and {
            facet("brand", "Apple", score = 2)
            facet("type", "tablet")
        }
    }
}
index.search(query = query)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
index.Search<Contact>(new Query {
    SearchQuery = "phone",
    OptionalFilters = new List<List<string>> {
        new List<string> {
            "brand:Apple<score=2>",
            "type:table"
        }
    }
});

// Asynchronous
await index.SearchAsync<Contact>(new Query {
    SearchQuery = "phone",
    OptionalFilters = new List<List<string>> {
        new List<string> {
            "brand:Apple<score=2>",
            "type:table"
        }
    }
});

1
2
3
4
5
6
7
8
9
10
List<List<String>> optionalFilters = Collections.singletonList(
    Arrays.asList("brand:Apple<score=2>", "type:tablet"));

Query query = new Query()
    .setOptionalFilters(optionalFilters);

index.search(query);

// Asynchronous
index.searchAsync(query);

1
2
3
4
5
6
7
res, err := index.Search(
  "",
  opt.OptionalFilterAnd(
    "brand:Apple<score=2>",
    "type:tablet>",
  ),
)

1
2
// Creating Rules isn't currently possible with this client.
// Please use the dashboard instead.

The engine starts by calculating the maximum score for each optional filter you send. With the above example, the engine first checks if a record’s brand attribute has Apple as its value. If this is the case, the record gets a score of 2 on this filter, and a score of 0 otherwise. Finally, it checks the type attribute to see if it matches tablet and assigns a score of 1.

Let’s consider the following records:

1
2
3
4
5
6
7
8
9
10
11
12
[
  {
    "name": "iPad Pro",
    "brand": "Apple",
    "type": "tablet"
  },
  {
    "name": "iPhone 11",
    "brand": "Apple",
    "type": "phone",
  }
]

The first record matches both the brand:Apple<score=2> and type:tablet optional filters, which respectively get a score of 2 and 1. Its total score is 3. However, the second record only matches on the brand:Apple<score=2> optional filter, so the record gets a total score of 2. Therefore, the iPad Pro record shows up higher than the iPhone one.

With these optional filters, the engine ranks results as follows:

1. Apple tablets (score: 3)
2. Other Apple products (score: 2)
3. Tablets by other brands (score: 1)
4. Other products by other brands (score: 0)

Complex optional filters

The engine calculates the maximum score for each element of your optional filters, so we can combine filters to perform more complex score calculations.

For example, imagine we want to promote red and blue products, but we want to avoid over-promoting items that are both red and blue. To do so, we can use the following optional filters:

1
[["color:red<score=2>", "color:blue"], ["type:-jeans"]]

And the following record:

1
2
3
4
5
6
7
8
{
  "name": "Long-sleeved shirt",
  "type": "shirt",
  "color": [
    "red",
    "blue"
  ]
}

We start by calculating the maximum score for the first optional filter. In this case, it’s an array: ["color:red<score=2>", "color:blue"]. Because this optional filter is an array, the engine checks all nested optional filters and uses the highest matching score.

The above record contains both red and blue colors, so it gets a score of 2 (match on red) and 1 (match on blue). The engine uses the highest matching score instead of a sum, meaning this record gets a score of 2 on this filter instead of 3.

Then, the engine matches the second optional filter (type:-jeans). The above record also matches as well (it’s not jeans), so it gets a score of 1 on this filter. Adding this to the previous optional filter’s score, we end up with a total score of 3 for this record.

Our results are ranked as follows when we use these optional filters:

With these optional filters, the engine ranks results as follows:

1. Products that have “red” listed as color, and aren’t jeans (score: 3)
2. Jeans that have “red” listed as color (score: 2), and products that have “blue” listed as color but aren’t jeans (score: 2)
3. Jeans that have “blue” listed as color (score: 1), and products that aren’t jeans (score: 1)
4. Jeans in colors other than “red” and “blue” (score: 0)

When several records have a similar score, the engine ranks them according to the ranking formula, as with any other search.

sumOrFiltersScores

In the previous example, we use filter scoring to boost products that are either red or blue, and that aren’t jeans. We also keep items that are both red and blue from getting a higher score than other records.

We can change the behavior of the engine by adding the sumOrFiltersScores parameter to our searches. This makes the engine add all scores together, regardless of the structure of your filters.

With our example, this means that products that have both color “red” and “blue” would show up even higher:

1. Products that have “red” and “blue” listed as color, and aren’t jeans (score: 4)
2. Products that have only “red” listed as color, and aren’t jeans (score: 3)
3. Jeans that have only “red” listed as color (score: 2), and products that have only “blue” listed as color and aren’t jeans (score: 2)
4. Jeans that have only “blue” listed as color (score: 1), and products that aren’t jeans (score: 1)
5. Jeans in colors other than “red” and “blue” (score: 0)

The sumOrFiltersScores parameter only changes the scoring for OR optional filters, which you create with a nested array of optional filters (optionalFilters: [["attr:val"]]). The engine always sums AND filter scores (optionalFilters: ["attr:val1", "attr:val2"]).

Performance

Optional filters are powerful, but they come with some important limitations, notably in terms of performance. Here are a few recommendations:

• Define optionalFilters with the filterOnly modifier, which tells the engine to treat the attribute as a filter instead of a facet. For example:

1
2
3
4
5
$index->setSettings([
  'attributesForFaceting' => [
    "filterOnly(brand)"
  ]
]);

1
2
3
4
5
index.set_settings({
  attributesForFaceting: [
    "filterOnly(brand)"
  ]
})

1
2
3
4
5
6
7
index.setSettings({
  attributesForFaceting: [
    "filterOnly(brand)"
  ]
}).then(() => {
  // done
});

1
2
3
4
5
index.set_settings({
    'attributesForFaceting': [
        'filterOnly(brand)'
    ]
})

1
2
3
4
5
6
7
let settings = Settings()
  .set(\.attributesForFaceting, to: [.filterOnly("brand")])
index.setSettings(settings) { result in
  if case .success(let response) = result {
     print("Response: \(response)")
  }
}

1
2
3
4
5
6
7
val settings = settings {
    attributesForFaceting {
        +"filterOnly(brand)"
    }
}

index.setSettings(settings)

1
2
3
4
5
6
7
8
9
10
11
12
IndexSettings settings = new IndexSettings
{
    AttributesForFaceting = new List<string>
    {
      "filterOnly(brand)"
    }
};

index.SetSettings(settings);

// Asynchronous
await index.SetSettingsAsync(settings);

1
index.setSettings(new IndexSettings().setAttributesForFaceting(Collections.singletonList("allergens")));

1
2
3
res, err := index.SetSettings(search.Settings{
  AttributesForFaceting: opt.AttributesForFaceting("filterOnly(brand)"),
})

1
2
3
4
5
6
7
client.execute {
  changeSettings of "myIndex" `with` IndexSettings(
    attributesForFaceting = Some(Seq(
      "filterOnly(brand)"
    ))
  )
}

• Prefer attributes with a low cardinality for your optionalFilters. For example, it’s preferable to use it with attributes like “brand” rather than “product_name”, as there are usually more unique product names than brands.
• Minimize the number of optionalFilters as much as possible.
• Aim for queries to return a smaller result set.
• Finally, talk to us. Depending on your plan, we can apply sharding and other custom settings to optimize for your use case.