As I discussed, it’s necessary to record scores assigned to each search result based on how well that result answers the original query. Having this framework in place is necessary to ensure that you avoid the ‘see-saw’ effect of fixing one query but breaking many others further down the chain.

The challenge with this is the time taken to re-score queries once configuration changes have been made – especially given you could be testing thousands of queries.

That’s why it’s great to see a tool like Quepid now available. Quepid sits on top of open source search engines Apache Solr and Elasticsearch (it can also incorporate scores from other engines, which is useful for comparison purposes if you are migrating) and it automatically recalculates scores when configuration changes are made, thus reducing the time taken to understanding the impact of your changes.

Business and technical teams benefit

Quepid is easy to get going with. Once you have set up and scored an initial set of search queries (known as cases), developers can tweak configurations within the Quepid Sandbox (without pushing to live) and relevancy scores are automatically recalculated enabling business users to see changes in scores immediately.

This score, combined with the feedback from search testers, provides the insight into how effective the change has been – removing uncertainty about whether you should publish the changes to your live site.

Improved stakeholder communication

Having figures that shows how search relevancy is improving is also a powerful tool for communicating search performance to stakeholders (and helps to overcome those HIPPO and LIPPO challenges I’ve mentioned before too). Whilst a relevancy score itself doesn’t translate to a conversion figure, understanding how your queries are performing could support business cases and customer metric scores.

Test and Learn

As the need to manually re-score queries is removed, automated search testing is possible and combined with greater collaboration and understanding across the entire search team means that the test and learn process is improved.

Highly Customisable

Every organisation has a different objective when it comes to improving search, but Quepid is designed so that it can support your organisation and requirements:

• Choose from a range of available scorers or create your own
• Set up multiple cases so that you can quickly understand how different types of queries perform
• Share cases amongst users for review and auditing
• Download and export cases and scores
• Assist with a ‘deep dive’ into low scoring queries
• Identify if there are particular trends or patterns you need to focus on as part of your testing
• Create a dashboard to share with category managers and other stakeholders

Flax are the UK resellers for Quepid, built by our partners OpenSource Connections – contact us for a demo and free 30-day trial.

Karen Renshaw is an independent On Site Search consultant and an associate of Flax. Karen was previously Head of On Site Search at RS Components, the world’s largest electronic component distributor.

Flax can offer a range of consulting, training and support, provide tools for test-driven relevancy tuning and we also run Search Workshops. If you need advice or help please get in touch.

The post Measuring search relevance scores appeared first on Flax.

1. How to get started on improving Site Search Relevancy
2. A suggested approach to running a Site Search Tuning Workshop
3. Auditing your site search performance
4. Developing ongoing search tuning processes
5. Measuring search relevance scores

In my last blog I wrote about how to create an audit of your current site search performance. In this blog I cover how to develop search tuning processes.

Once started on your search tuning journey developing ongoing processes is a must. Search tuning is an iterative process and must be treated as such. In the same way that external search traffic – PPC and SEO – is continually reviewed and optimised, so must on site search be: otherwise you have invested a lot of time and money to get people to your site but then leave them wandering aimlessly in the aisles wondering if you have the product or information you so successfully advertised!

There are 2 key areas to focus on when developing search processes:

1. Ongoing review of search performance
2. Dedicated resource

1. Ongoing review of search performance

Develop a framework for measuring relevancy scores

It’s good practice to develop a benchmark as to how search queries are performing through creating a search relevancy framework. Simply put, this is a score assigned to each search result based on how well that result answers the original query.

You can customise the scoring system you use to score your search results. Whatever you choose the key is to ensure that your search analysts are consistent in their approach, the best way to achieve that is through providing documented guidelines.

Understanding how query scores change with different configurations is an integral part of search tuning process but you should also run regular reviews on how queries are performing. This way you’ll know the impact loading new documents and products into your site is having on overall relevancy and highlight changes you need to feed into your product backlog.

Process for manually optimising important or problematic queries

Even with a search tuning test and learn plan in place there will be some queries that don’t do as well as well as expected or for which a manual custom build response provides a better customer experience.

Whilst manually tuning a search can sometimes be viewed in a negative light – after all search should ‘just work’ – it shouldn’t be seen as such. Manually optimising important search queries means that you can provide a tailored response for your customer. The queries you optimise will be dependent on your metrics and what you deem as being a good or bad experience.

With manual optimisation you can should also build in continual reviews and take the opportunity to test different landing pages.

Competitive review

I’ve talked about this in a few of my other blogs but it is especially important for eCommerce sites to understand how your competitors are answering your customers’ queries. As you create a search relevancy framework for your site it’s easy to score the same queries on your competitors to draw out any comparisons and understand opportunities for improvements.

2. Dedicated Resource

Creating and maintaining the above reviews needs resource. Ideally you would have a staff member dedicated to reviewing search and responsible for updating product backlog configuration changes, working alongside developers to ensure changes are tested and deployed successfully.

If you don’t have a dedicated person responsible, the right skills will undoubtedly exist within your organisation. You will have teams who understand your product / information set, and within that team you will find a sub-set of individuals who have problem solving skills combined with a passion to improve the customer experience. Once you’ve found them, providing them with some light search knowledge will be enough to get you started.

Whether it’s a full-time role or part-time having someone focus on reviewing search queries should be part of your plan.

What’s next?

Now you have processes and a team in place it’s time to consider what to measure (and how). In my next blog I’ll cover how to measure search relevancy scores.

Karen Renshaw is an independent On Site Search consultant and an associate of Flax. Karen was previously Head of On Site Search at RS Components, the world’s largest electronic component distributor.

Flax can offer a range of consulting, training and support, provide tools for test-driven relevancy tuning and we also run Search Workshops. If you need advice or help please get in touch.

The post Developing ongoing search tuning processes appeared first on Flax.

When starting on your journey to improve on site search relevancy investing time in understanding current performance is essential. Whilst the level of detail available to you will vary by industry and business, there are multiple sources of information you will have access to that provide insight. Combining these will ensure you have a holistic view of your customers experience.

The main sources of information to consider are:

• Web Analytics
• Current Relevancy Scores
• Customer Feedback
• Known Areas of Improvement
• Competitors

Web Analytics

The metrics you use will be dependent upon your business, the role that search plays on your site and what you currently measure. What is key is to develop a view of how core search queries are performing. Classifying and creating an aggregated view of performance for different search queries allows you to identify any differences by search type, which you might want to focus on as part of your testing.

This approach also helps to prevent reacting to the opinions of HIPPOS and LIPPOS (Highest Paid Persons and Loudest Paid Persons Opinions) when constructing test matrixes.

Another measure to consider is zero results – what percentage of your search queries lead customers to a see the dreaded ‘no results found’ message. Don’t react to the overall percentage as a figure per se (a low percentage could mean too many irrelevant results are being returned, a high percentage that you don’t have the product / information your customers are looking for). Again what you’re trying to get to is an understanding of the root cause so you can build changes into your overall test plan. It’s a manual process but even a review of the top 200 zero results will throw up meaningful insights.

Current Relevancy Scores

Very closely linked to Web Analytics is a view of current search relevancy scores. It’s good practice to develop a benchmark as to how search queries are performing through creating a search relevancy framework. Simply put, this is a score assigned to each search result based on how well that result answers the original query.

Use queries from your search logs so you know you are scoring the queries important to your customers (and not just those important to those HIPPO’s and LIPPO’s). And whilst scoring will always be subjective providing guidelines to your testers helps mitigate this.

Tools like Quepid, which can sit on top of open source search engines Apache Solr and Elasticsearch (and also incorporate scores from other engines) and automatically recalculate scores when configuration changes are made, can support ongoing search tuning processes.

Customer Feedback

Whether in the form of structured or unstructured feedback, with site search critical to a successful customer experience, your customers will undoubtedly be providing you with a wealth of feedback.

Take the time to read through as much of it as possible. Even better, walk through some of the journeys yourself to understand the experience from the eyes of your customers. Whilst the feedback might be vague you’ll quickly find you can classify and pull out key themes.

Internal customer service departments can also provide you with customer logs and real life scenarios. Involving them up front to identify problem areas can help in the long term as they can be an invaluable resource when testing different search set ups.

Known Areas of Improvement

You’ve probably already got a list of search configurations on your backlog you want to review and test. Your developers will too, as will multiple teams across the organisation. Pulling together all these different views can provide a useful perspective on how to tackle problem areas.

Whilst you need to develop your search strategy based on customers needs (not just what other people like) it’s always useful to have sight of what search functionality exists that has helped them to find the right product, so capture these as you go along.

Competitor Review

A very important question for e-commerce sites is how are your competitors answering common queries? As you have for your own site, scoring common search queries across multiple sites provides a view of how you fare compared to your competitors.

Getting Started!

Now you have all this insight you can start to build out your search test plans with your developers. In my next blog I’ll cover how to start developing search tuning processes.

Karen Renshaw is an independent On Site Search consultant and an associate of Flax. Karen was previously Head of On Site Search at RS Components, the world’s largest electronic component distributor.

Flax can offer a range of consulting, training and support, provide tools for test-driven relevancy tuning and we also run Search Workshops. If you need advice or help please get in touch.

The post Auditing your site search performance appeared first on Flax.