Now, let’s assume you have Quepid running in your browser – there’s a 30 day free trial which lets you create a single test case, which is a great way to try it out. A Case is used to illustrate a particular problem with search relevancy – say, how searching for ‘iPhone’ shows iPhone cases higher up the list than actual iPhones. Each Case contains a number of Queries. Note in this example we’re using Solr, but Quepid also works with Elasticsearch.

1. Hooking Quepid up to your search engine.

You’re going to need the help of your search developer for this one! He’ll need to tell you the URL of your Solr or Elasticsearch engine – and this will need to be accessible from the PC you’re running Quepid on. Since Quepid runs in the browser (although it stores its data in the Cloud) you shouldn’t have any trouble setting up secure access to your search engine – after all, your own PC is probably already within your corporate network. In Quepid, Click ‘Relevancy cases’ and ‘Create a case’. Give the case a name, like ‘iPhone_problem_English’ or ‘Two_word_queries’.

Enter the URL provided by your developer: for Solr, it will probably look a bit like:
http://your domain/solr/name of a Solr index/select
e.g.
http://www.mycompany.com/solr/myproducts/select

Quepid will then check it can see the Solr index – if it can’t, check that the URL is correct.

2. Setting up the right query fields

Now you need to tell Quepid an ID field (which must be unique) and a title field for each result. If you start typing, Quepid will show some suggestions – check with your developer for which ones to use as these will be defined in the schema configuration for your search engine. You can select any other fields to be displayed for each result: let Quepid suggest some by clicking in the Additional Display Fields box. All the above can be changed the Settings pane of the Tune Relevance panel later, so don’t worry if you don’t add everything now.

3. Adding some queries

You can now add some queries to test – ‘iPhone’, ‘iPhone case’, ‘iphone’ or whatever fits the test you’re creating. Add a few for now, you can add more later. Once you’re done click Continue, then Finish and Quepid will try these queries out. Don’t worry if you don’t get many results for now.

4. Using the right query parameters

By default, Quepid only sends a very simple query to Solr or Elasticsearch (click on Tune Relevance and check the Tune panel, you should see just ‘#$query##’ – a token that represents the various test queries you added above), and your search application almost certainly sends something a lot more complicated! So you can be sure you’re testing the same configuration as your search application uses, you need to tell Quepid what query pattern is being used.

One way to start is to use Solr’s log files to see what actual queries are being run by your search application. Your search developer should be able to find a section that looks like this:

INFO - 2016-06-03 09:12:37.964; [ mydomain.com] org.apache.solr.core.SolrCore; [mydomain.com] webapp=/solr path=/select params={hl.fragsize=70&sort=+score+desc,+date_text+desc&hl.mergeContiguous=true&qf=tm_body:summary^1.0&qf=tm_body:value^1.0&qf=tm_field_product^5.0&hl.simple.pre=[HIGHLIGHT]&json.nl=map&hl.fl=spell&wt=json&hl=true&rows=8&fl=*,score&hl.snippets=3&start=0&q="iphone"&hl.simple.post=[/HIGHLIGHT]&fq=bs_status:"true"&fq=index_id:"node_index"} hits=5147 status=0 QTime=46

Stripping out the query gives us:

hl.fragsize=70&sort=+score+desc,+date_text+desc&hl.mergeContiguous=true&qf=tm_body:summary^1.0&qf=tm_body:value^1.0&qf=tm_field_product^5.0&hl.simple.pre=[HIGHLIGHT]&json.nl=map&hl.fl=spell&wt=json&hl=true&rows=8&fl=*,score&hl.snippets=3&start=0&q="iphone"&hl.simple.post=[/HIGHLIGHT]&fq=bs_status:"true"&fq=index_id:"node_index"

We need to replace the query (highlighted above, we’re searching for ‘iphone’) with a special token so Quepid can use this string to send all its test queries:

hl.fragsize=70&sort=+score+desc,+date_text+desc&hl.mergeContiguous=true&qf=tm_body:summary^1.0&qf=tm_body:value^1.0&qf=tm_field_product^5.0&hl.simple.pre=[HIGHLIGHT]&json.nl=map&hl.fl=spell&wt=json&hl=true&rows=8&fl=*,score&hl.snippets=3&start=0&q=#$query##&hl.simple.post=[/HIGHLIGHT]&fq=bs_status:"true"&fq=index_id:"node_index"

If you paste this string into Quepid’s Tune panel (click Tune Relevance to toggle this) then you know Quepid is sending the same type of queries as your search application. Click ‘Rerun my Searches’ and the results you see should be in a similar, if not identical, order to your search application.

5. Starting the tuning process

You should now have Quepid connected to your actual Solr index and running queries the same way that your search application does – you can now start the process of ranking the results. Once you have some scores, you can ask your search developer to try changing the query in the Tune panel to see if he can improve the relevance scores. Your journey towards better relevance has begun!

Do get in touch if you’d like more information about Quepid or how Flax can help you develop a process of test-based relevancy tuning.

The post Setting up your first Quepid test case appeared first on Flax.

When starting on your journey to improve on site search relevancy investing time in understanding current performance is essential. Whilst the level of detail available to you will vary by industry and business, there are multiple sources of information you will have access to that provide insight. Combining these will ensure you have a holistic view of your customers experience.

The main sources of information to consider are:

• Web Analytics
• Current Relevancy Scores
• Customer Feedback
• Known Areas of Improvement
• Competitors

Web Analytics

The metrics you use will be dependent upon your business, the role that search plays on your site and what you currently measure. What is key is to develop a view of how core search queries are performing. Classifying and creating an aggregated view of performance for different search queries allows you to identify any differences by search type, which you might want to focus on as part of your testing.

This approach also helps to prevent reacting to the opinions of HIPPOS and LIPPOS (Highest Paid Persons and Loudest Paid Persons Opinions) when constructing test matrixes.

Another measure to consider is zero results – what percentage of your search queries lead customers to a see the dreaded ‘no results found’ message. Don’t react to the overall percentage as a figure per se (a low percentage could mean too many irrelevant results are being returned, a high percentage that you don’t have the product / information your customers are looking for). Again what you’re trying to get to is an understanding of the root cause so you can build changes into your overall test plan. It’s a manual process but even a review of the top 200 zero results will throw up meaningful insights.

Current Relevancy Scores

Very closely linked to Web Analytics is a view of current search relevancy scores. It’s good practice to develop a benchmark as to how search queries are performing through creating a search relevancy framework. Simply put, this is a score assigned to each search result based on how well that result answers the original query.

Use queries from your search logs so you know you are scoring the queries important to your customers (and not just those important to those HIPPO’s and LIPPO’s). And whilst scoring will always be subjective providing guidelines to your testers helps mitigate this.

Tools like Quepid, which can sit on top of open source search engines Apache Solr and Elasticsearch (and also incorporate scores from other engines) and automatically recalculate scores when configuration changes are made, can support ongoing search tuning processes.

Customer Feedback

Whether in the form of structured or unstructured feedback, with site search critical to a successful customer experience, your customers will undoubtedly be providing you with a wealth of feedback.

Take the time to read through as much of it as possible. Even better, walk through some of the journeys yourself to understand the experience from the eyes of your customers. Whilst the feedback might be vague you’ll quickly find you can classify and pull out key themes.

Internal customer service departments can also provide you with customer logs and real life scenarios. Involving them up front to identify problem areas can help in the long term as they can be an invaluable resource when testing different search set ups.

Known Areas of Improvement

You’ve probably already got a list of search configurations on your backlog you want to review and test. Your developers will too, as will multiple teams across the organisation. Pulling together all these different views can provide a useful perspective on how to tackle problem areas.

Whilst you need to develop your search strategy based on customers needs (not just what other people like) it’s always useful to have sight of what search functionality exists that has helped them to find the right product, so capture these as you go along.

Competitor Review

A very important question for e-commerce sites is how are your competitors answering common queries? As you have for your own site, scoring common search queries across multiple sites provides a view of how you fare compared to your competitors.

Getting Started!

Now you have all this insight you can start to build out your search test plans with your developers. In my next blog I’ll cover how to start developing search tuning processes.

Karen Renshaw is an independent On Site Search consultant and an associate of Flax. Karen was previously Head of On Site Search at RS Components, the world’s largest electronic component distributor.

Flax can offer a range of consulting, training and support, provide tools for test-driven relevancy tuning and we also run Search Workshops. If you need advice or help please get in touch.

The post Auditing your site search performance appeared first on Flax.