This sample shows how to add recommendations to an Azure Search index.

Give the following demo a try to see what we will be building by searching for a movie and then click on the movie to see recommendations.

Recommendations is a technique for surfacing more items from a catalog based on existing search User activity (such as web logs) used to recommend items and improve conversion. Recommendation engines are often trained using past customer activity or by collecting data directly from digital store Common recommendations include:

• Frequently Bought Together: a customer who bought this also bought that
• Customer to Item Recommendations: a customer like you also bought that

• Open the solution AzureSearchMovieRecommendations.sln and set ImportAzureSearchIndexData as the Default Project.
• Open App.config within ImportAzureSearchIndexData and update the SearchServiceName and SearchApiKey to point to your Azure Search service
• Download hetrec2011-movielens-2k.zip from http://grouplens.org/datasets/hetrec-2011/ and copy the files Movies.dat & user_ratedmovies.dat to the \ImportAzureSearchIndexData\data
• Buid &Run the project to create an index and load Movie data
• At the end, the application will execute a test search

Open the search.js file within \WebSite\starter-template-complete and update the apiKey and azureSearchService with your Azure Search service details.

You should be able to open this file in a browser such as Chrome to now view movies by typing into the search box.

Azure Search Traffic Analytics is a feature of Azure Search that tracks all activity against your search service. This includes statistic as well as details of individual search operations. You can learn more about this feature in my video as well as the following blog.

For this demo, we will be extracting all of the individual document lookups. This would happen when a user see's search results and then clicks on an individual result to get details. An example of a lookup of a document with ID 30 might look like the following.

https://azs-playground.search.windows.net/indexes/movies/docs('32')?api-version=2015-02-28&userid=75

Notice how this example also appends a parameter userid. This has no impact on the search results, but it is important for our demo because it allows us to uniquely identify which users looked at which items, and will be later used in our recommendations analysis.

In the Search Traffic Analytics, this operation will be logged and will look like this:

{
	"records": 
	[
		{
			 "time": "2016-01-25T23:00:00.0113871Z",
			 "resourceId": "/SUBSCRIPTIONS/[REDACTED]/RESOURCEGROUPS/[REDACTED]/PROVIDERS/MICROSOFT.SEARCH/SEARCHSERVICES/[REDACTED]",
			 "operationName": "Query.Lookup",
			 "operationVersion": "2015-02-28",
			 "category": "OperationLogs",
			 "resultType": "Success",
			 "resultSignature": 200,
			 "durationMS": 2,
			 "properties": { "Description" : "GET /indexes('movies')/docs('30')" , "Query" : "?userid=75&api-version=2015-02-28" , "Documents" : 0, "IndexName" : "movies" }
		}
	]
}

Ultimately, our goal will be to extract each UserID / ItemID values from all of this lookup data. But first, let's simulate a ton of user lookups.

At this point you will need to turn on Search Traffic Analytics for your Azure Search service. If you are not familiar with how to do this, please refer to my video or the following blog.

To simulate these user requests you can run the SimulateAzureSearchLookupRequests project from within the AzureSearchMovieRecommendations Solution. Before running, open App.config within this project and update the SearchServiceName and SearchApiKey to point to your Azure Search service.

This process will take some time since it is simulating over 600,000 users lookups.

There are numerous ways that item recommendations can be created using this usage data. The two options I prefer are Azure ML Recommendations API and Apache Mahout. Both options I feel have strengths and weaknesses which often direct you to use one over the other such as:

Azure ML Recommendations API

• Very simple to use either through the .NET API or through their UI interface
• Provides really good recommendations without the need for deep Machine Learning experience
• Currently limited to 100,000 items (meaning in our example, no more than 100,000 movies)
• Pricing can get expensive if you send large numbers of recommendation requests

Apache Mahout

• Able to support extremely large numbers of items
• Provides good recomendations out of the box, but can be complex to improve without Machine Learning experience
• Can be a good option for Java developers
• Can be difficult to operationalize

My personal preference is to use the Azure ML Recommendations API if my content size fits their limits because it is much simpler (IMO) to use and operationalize. I will focus the following sections on how to do this, but if you would prefer to do this using Machout, please take a look here.

Now that we have simulated a huge number of user search requests, we will want to export this data into a csv file (which we will call usage.csv) that is in format userId, productId indicating that the user of id userId looked at an item with id of productId. For example:

78,1391
78,1193
78,1653
127,2013
127,2719
127,5523

To do this, within the AzureSearchAnalyticsExtraction project, configure the Storage Account information in the app.config as well as the folder where the container.GetDirectoryReference can find your search operations logs.

In the previous step, we created a usage.csv file from Azure Search Traffic Analytics. To do this, we will use the AzureMLRecommendations project. Before running this project, you will need to do the following:

• Register for an Azure ML Recommendations API account here
• Get the Azure ML Recommendations Email & Account Key and update the AzureMLRecommendations project with these values
• Run this project and make note of the Model ID and Build ID that are created as you will use this in the Web Site to call recommendations

At this point you should be able to go back to the web application and update it with the Model ID and Build ID from the previous step in the Search.js file.

var azureMLModelId = ""
var azureMLBuildId = ""

If you want to see how the recommendations were returned when you clicked on this image, open Search.js and look at the openMovieDetails() function.

At this point you should be able to launch the web page, search for movies and return recommendations.

To this point we have been running console applications to extract the usage data and then update the Azure ML Recommendations engine. A good way to operationalize this is by publishing the code as an Azure WebJob to allow yourself to schedule this execution.

Data was provided by GrouLens (http://grouplens.org/datasets/hetrec-2011/)

Please refer to this page for details on the licensing of this data: http://files.grouplens.org/datasets/hetrec2011/hetrec2011-movielens-readme.txt