Custom Fields and ElasticSearch

Searching on websites is an important and downright integral feature to offer your users. Having search capabilities on eCommerce sites is even more important in that you want to drive your customers to find products to make a sale.

For Optimizely (formerly known as Episerver) sites, the primary search provider is Optimizely Search & Navigation (AKA Episerver Find). This product can be quite costly and, for some of Diagram’s clients, is not an option. Luckily, there is a great open-source alternative made by Epinova called Epinova.Elasticsearch.

Note: This post does not go into detail on how to setup Epinova.ElasticSearch. For configuring Epinova.Elasticsearch in your solution please read the setup README on their GitHub.

Epinova.Elasticsearch offers many of the same features as Optimizely Search & Navigation. Some of these features include:

• Typed Search
• Facets
• Filtering
• Best Bets
• Synonyms
• Commerce Support
• Boosting

A full set of features supported can be found on the GitHub repo under features.

Some notable features I’ve noticed missing are:

• Caching
• Multiple Queries in One Request
• Projections
• Searching Over Multiple Types

One of the best things, in my opinion, about Epinova.Elasticsearch is the ability to include custom fields for objects in your Elasticsearch index. These are custom fields you can set up against Epinova.Elasticsearch’s client conventions, and when an object of the type you’ve specified is indexed, the indexing engine will call that field for the data.

This can be especially useful when you have data for entities across multiple systems. In our case one of our eCommerce clients has product data in both Episerver Commerce and a 3^rd party system. We need the data from the 3^rd party system for searching, but don’t want to import it directly into the Episerver Commerce database. A great way to solve this problem is to configure custom fields for the product data to pull from the 3^rd party system during indexing of each individual product.

There are multiple ways to include custom fields during indexing. The method I find most useful is by using extension methods. To set up custom fields for indexing, the standard approach is to configure your custom fields against Epinova.Elasticsearch’s client conventions in an initialization module.

Here is the code for registering a custom field using an extension method against Epinova.Elasticsearch’s client conventions:

using Epinova.ElasticSearch.Core.Conventions;

using EPiServer.Framework;
using EPiServer.Framework.Initialization;

using Search.Extensions;
using Search.Models.Catalog;

namespace Search.Infrastructure
{
    [InitializableModule]
    [ModuleDependency(typeof(EPiServer.Commerce.Initialization.InitializationModule))]
    public class SingleCustomFieldSearchInitialization : IInitializableModule
    {
        public void Initialize(InitializationEngine context)
        {
            // Search Configurations
            Indexing.Instance
                .ForType().IncludeField(x.ThirdPartyData());
        }

        public void Uninitialize(InitializationEngine context) { }
    }
}

You can set up multiple custom fields with Epinova.Elasticsearch like so:

using Epinova.ElasticSearch.Core.Conventions;

using EPiServer.Framework;
using EPiServer.Framework.Initialization;

using Search.Extensions;
using Search.Models.Catalog;

namespace Search.Infrastructure
{
    [InitializableModule]
    [ModuleDependency(typeof(EPiServer.Commerce.Initialization.InitializationModule))]
    public class MultipleCustomFieldsSearchInitialization : IInitializableModule
    {
        public void Initialize(InitializationEngine context)
        {
            // Search Configurations
            Indexing.Instance
                .ForType().IncludeField(x => x.ThirdPartyData())
                .ForType().IncludeField(x => x.AnotherCustomField())
                .ForType().IncludeField(x => x.YetAnotherCustomField());
        }

        public void Uninitialize(InitializationEngine context) { }
    }
}

How the extension method retrieves data is not important, but for the sake of this post here is the custom field’s extension method code:

using System.Collections;
using System.Collections.Generic;

using Epinova.ElasticSearch.Core.Contracts;

using EPiServer.ServiceLocation;

using Search.Extensions;
using Search.Models.Catalog;

namespace Search.Services
{
    [ServiceConfiguration(typeof(IProductItemSearchService), Lifecycle = ServiceInstanceScope.Singleton)]
    public static class ProductItemDataSearchExtensions
    {
        private static IThirdPartyDataRepository _dataRepository;

        public static IEnumerable ThirdPartyData(this ProductItemData product)
        {
            if (_dataRepository == null)
            {
                _dataRepository = ServiceLocator.Current.GetInstance();
            }

            return _dataRepository.GetMoreData(product.Code) ?? Enumerable.Empty();
        }
    }
}

After you run Epinova.Elasticsearch’s scheduled jobs for indexing, the custom data should be sitting alongside your objects in the index. You can verify this by logging into the Optimizely CMS backend and navigating to Search Engine  Search Index. Using this built-in interface offered by Epinova.Elasticsearch, you can query for data against the type you added custom fields to.

For this post’s examples, the custom field ThirdPartyData should show up for all objects of type ProductItemData in the index. The custom fields show as properties on the JSON object for the type you retrieve from the index. Here is a screenshot of the JSON for ProductItemData showing the custom field ThirdPartyData (appears as the property directly following the Types property).

After you have your custom field included in your Elasticsearch index you can then search and/or filter against it as well as retrieve it from the index. Searching and/or filtering against custom fields is similar to doing so against concrete properties on your objects, but you use the extension methods for your custom fields instead.

Here is an example of how to specifically search against a custom field (Epinova.Elasticsearch should automatically search against this field by default, but this is an example of how to only search against a custom field):

using Epinova.ElasticSearch.Core.Contracts;

using EPiServer.ServiceLocation;

using Search.Extensions;
using Search.Models.Catalog;

namespace Search.Services
{
    [ServiceConfiguration(typeof(IProductItemSearchService), Lifecycle = ServiceInstanceScope.Singleton)]
    public class ProductItemSearchService : IProductItemSearchService
    {
        private readonly IElasticSearchService _searchService;
        
        public ProductItemSearchService(IElasticSearchService searchService)
        {
            _searchService = searchService;
        }
        
        /// <summary>
        ///  Create Elasticsearch query that will return a list of products
        /// </summary>
        /// <param name="term">The search term to search against</param>
        public IElasticSearchService<ProductItemData> CreateSearchQuery(string term)
        {
            return _searchService
                .Search(term)
                .InField(x => x.ThirdPartyData());
        }
    }
}

In this example, I am using IElasticSearchService’s extension method InField(), offered by Epinova.Elasticsearch.

And here is an example of how to filter against a custom field:

using Epinova.ElasticSearch.Core.Contracts;

using EPiServer.ServiceLocation;

using Search.Extensions;
using Search.Models.Catalog;

namespace Search.Services
{
    [ServiceConfiguration(typeof(IProductItemSearchService), Lifecycle = ServiceInstanceScope.Singleton)]
    public class ProductItemSearchService : IProductItemSearchService
    {
        private readonly IElasticSearchService _searchService;
        
        public ProductItemSearchService(IElasticSearchService searchService)
        {
            _searchService = searchService;
        }
        
        /// <summary>
        ///  Create Elasticsearch query using a simple filter that will return a list of products
        /// </summary>
        /// <param name="term">The search term to search against</param>
        /// <param name="thirdPartyDataFilter">Third party filter to use against the ThirdPartyData enumerable.</param>
        public IElasticSearchService<ProductItemData> CreateSearchQuery(string term, string thirdPartyDataFilter)
        {
            return _searchService
                .Search<ProductItemData>(term)
                .Filter(x => x.ThirdPartyData(), thirdPartyDataFilter);
        }
    }
}

In this example, I am using IElasticSearchService’s extension method Filter(), offered by Epinova.Elasticsearch.

Many times, you’ll find yourself wanting to also display your custom fields in addition to searching and filtering against them. Epinova.Elasticsearch has a way to pull this data from a dictionary of each result item returned from a query. This is a piece of the package I think could use some improvement, and Optimizely’s Search & Navigation does this really well with their projections.

Epinova.Elasticsearch returns the following when you tell it to retrieve your results from the query you have built:

using System.Collections.Generic;
using System.Linq;
using Epinova.ElasticSearch.Core.Models;
using EPiServer.Commerce.Catalog.ContentTypes;

namespace Epinova.ElasticSearch.Core.EPiServer.Commerce
{
    /// <summary>
    /// Contains the materialization of the search query
    /// </summary>
    public sealed class CatalogSearchResult<T> : SearchResultBase<CatalogSearchHit<T>>
        where T : EntryContentBase
    {
        public CatalogSearchResult(SearchResult searchResult, IEnumerable<CatalogSearchHit<T>> filteredHits)
        {
            Query = searchResult.Query;
            Took = searchResult.Took;
            Facets = searchResult.Facets;
            Hits = filteredHits ?? Enumerable.Empty<CatalogSearchHit<T>>();
            TotalHits = searchResult.TotalHits;
            DidYouMeanSuggestions = searchResult.DidYouMeanSuggestions;
        }
    }
}

On CatalogSearchResult there is a collection of CatalogSearchHit<T> on the property Hits. The class CatalogSearchHit<T> has a dictionary containing the custom fields indexed for that type and against that specific CatalogSearchHit<T> item. This dictionary is of type Dictionary<string, object>.

After you have your results by calling GetContentResults() or GetCatalogResults(), you can loop through the results one-by-one to retrieve your custom fields and map them to properties on a type of your choosing. Gist containing an example of how to do this:

using System.Collections;
using System.Collections.Generic;

using Epinova.ElasticSearch.Core.Contracts;
using Epinova.ElasticSearch.Core.EPiServer;

using Search.Models.Catalog;
using Search.Models.Search;

namespace Search.Services
{
    public static class PullingCustomFields
    {
        public static IEnumerable<SearchHit> ToSearchHits(this IEnumerable<CatalogSearchHit<ProductItemData>> productHits)
        {
            foreach (var productHit in productHits)
            {
                productHit.Custom.TryGetValue("ThirdPartyData", out object thirdPartyData);
                var thirdPartyDataCollection = thirdPartyData as IEnumerable;
                
                yield return new SearchHit
                {
                    ContentLink = productHit.Content.ContentLink,
                    ProductSku = productHit.Content.Code,
                    ThirdPartyData = thirdPartyDataCollection?.Cast<string>() ?? Enumerable.Empty<string>(),
                    Url = "/" + productHit.Content.SeoUri
                };
            }
        }
    }
}

One thing to note with custom fields is that sometimes the type coming back from Epinova.Elasticsearch isn’t quite right. The indexer is great at indexing simple types or collections of simple types and retrieving them. A problem I’ve found is when I index complex types or collections of complex types; Newtonsoft.JSON will choke when deserializing it from the index. Another problem I’ve found is sometimes the type returned from the indexer isn’t correct; for example, if I index decimals, they come back as doubles (this is more of a Newtonsoft issue).

I have submitted a pull request on Github for these issues and at the time of writing this post, it has yet to be reviewed: https://github.com/Epinova/Epinova.Elasticsearch/pull/134. For right now I would recommend only using and indexing custom fields of primitive types, strings, collections of primitive types, and collections of strings. I have come up with a way to get complex types out of the custom fields without issue, but that is another post I will write up at a later date if my pull request hasn’t been merged by then.

Overall, I have been very pleased with Epinova.Elasticsearch and plan to use it more in the future when Optimizely Search & Navigation is not an option. Another open-source option is Vulcan. We have a couple of blog posts on how to use Vulcan. You can find those blog posts here:

•  https://www.wearediagram.com/blog/improving-episerver-search-with-vulcan-2.0
•  https://www.wearediagram.com/blog/getting-started-using-vulcan-search-in-episerver