A series of blogs by Karen Renshaw on improving site search:
- How to get started on improving Site Search Relevancy
- A suggested approach to running a Site Search Tuning Workshop
- Auditing your site search performance
- Developing ongoing search tuning processes
- Measuring search relevance scores
In my last blog I wrote in depth about how to run a search workshop. In this blog I cover how to create an audit of your current site search performance.
When starting on your journey to improve on site search relevancy investing time in understanding current performance is essential. Whilst the level of detail available to you will vary by industry and business, there are multiple sources of information you will have access to that provide insight. Combining these will ensure you have a holistic view of your customers experience.
The main sources of information to consider are:
- Web Analytics
- Current Relevancy Scores
- Customer Feedback
- Known Areas of Improvement
- Competitors
Web Analytics
The metrics you use will be dependent upon your business, the role that search plays on your site and what you currently measure. What is key is to develop a view of how core search queries are performing. Classifying and creating an aggregated view of performance for different search queries allows you to identify any differences by search type, which you might want to focus on as part of your testing.
This approach also helps to prevent reacting to the opinions of HIPPOS and LIPPOS (Highest Paid Persons and Loudest Paid Persons Opinions) when constructing test matrixes.
Another measure to consider is zero results – what percentage of your search queries lead customers to a see the dreaded ‘no results found’ message. Don’t react to the overall percentage as a figure per se (a low percentage could mean too many irrelevant results are being returned, a high percentage that you don’t have the product / information your customers are looking for). Again what you’re trying to get to is an understanding of the root cause so you can build changes into your overall test plan. It’s a manual process but even a review of the top 200 zero results will throw up meaningful insights.
Current Relevancy Scores
Very closely linked to Web Analytics is a view of current search relevancy scores. It’s good practice to develop a benchmark as to how search queries are performing through creating a search relevancy framework. Simply put, this is a score assigned to each search result based on how well that result answers the original query.
Use queries from your search logs so you know you are scoring the queries important to your customers (and not just those important to those HIPPO’s and LIPPO’s). And whilst scoring will always be subjective providing guidelines to your testers helps mitigate this.
Tools like Quepid, which can sit on top of open source search engines Apache Solr and Elasticsearch (and also incorporate scores from other engines) and automatically recalculate scores when configuration changes are made, can support ongoing search tuning processes.
Customer Feedback
Whether in the form of structured or unstructured feedback, with site search critical to a successful customer experience, your customers will undoubtedly be providing you with a wealth of feedback.
Take the time to read through as much of it as possible. Even better, walk through some of the journeys yourself to understand the experience from the eyes of your customers. Whilst the feedback might be vague you’ll quickly find you can classify and pull out key themes.
Internal customer service departments can also provide you with customer logs and real life scenarios. Involving them up front to identify problem areas can help in the long term as they can be an invaluable resource when testing different search set ups.
Known Areas of Improvement
You’ve probably already got a list of search configurations on your backlog you want to review and test. Your developers will too, as will multiple teams across the organisation. Pulling together all these different views can provide a useful perspective on how to tackle problem areas.
Whilst you need to develop your search strategy based on customers needs (not just what other people like) it’s always useful to have sight of what search functionality exists that has helped them to find the right product, so capture these as you go along.
Competitor Review
A very important question for e-commerce sites is how are your competitors answering common queries? As you have for your own site, scoring common search queries across multiple sites provides a view of how you fare compared to your competitors.
Getting Started!
Now you have all this insight you can start to build out your search test plans with your developers. In my next blog I’ll cover how to start developing search tuning processes.
Karen Renshaw is an independent On Site Search consultant and an associate of Flax. Karen was previously Head of On Site Search at RS Components, the world’s largest electronic component distributor.
Flax can offer a range of consulting, training and support, provide tools for test-driven relevancy tuning and we also run Search Workshops. If you need advice or help please get in touch.
