As I discussed, it’s necessary to record scores assigned to each search result based on how well that result answers the original query. Having this framework in place is necessary to ensure that you avoid the ‘see-saw’ effect of fixing one query but breaking many others further down the chain.

The challenge with this is the time taken to re-score queries once configuration changes have been made – especially given you could be testing thousands of queries.

That’s why it’s great to see a tool like Quepid now available. Quepid sits on top of open source search engines Apache Solr and Elasticsearch (it can also incorporate scores from other engines, which is useful for comparison purposes if you are migrating) and it automatically recalculates scores when configuration changes are made, thus reducing the time taken to understanding the impact of your changes.

Business and technical teams benefit

Quepid is easy to get going with. Once you have set up and scored an initial set of search queries (known as cases), developers can tweak configurations within the Quepid Sandbox (without pushing to live) and relevancy scores are automatically recalculated enabling business users to see changes in scores immediately.

This score, combined with the feedback from search testers, provides the insight into how effective the change has been – removing uncertainty about whether you should publish the changes to your live site.

Improved stakeholder communication

Having figures that shows how search relevancy is improving is also a powerful tool for communicating search performance to stakeholders (and helps to overcome those HIPPO and LIPPO challenges I’ve mentioned before too). Whilst a relevancy score itself doesn’t translate to a conversion figure, understanding how your queries are performing could support business cases and customer metric scores.

Test and Learn

As the need to manually re-score queries is removed, automated search testing is possible and combined with greater collaboration and understanding across the entire search team means that the test and learn process is improved.

Highly Customisable

Every organisation has a different objective when it comes to improving search, but Quepid is designed so that it can support your organisation and requirements:

• Choose from a range of available scorers or create your own
• Set up multiple cases so that you can quickly understand how different types of queries perform
• Share cases amongst users for review and auditing
• Download and export cases and scores
• Assist with a ‘deep dive’ into low scoring queries
• Identify if there are particular trends or patterns you need to focus on as part of your testing
• Create a dashboard to share with category managers and other stakeholders

Flax are the UK resellers for Quepid, built by our partners OpenSource Connections – contact us for a demo and free 30-day trial.

Karen Renshaw is an independent On Site Search consultant and an associate of Flax. Karen was previously Head of On Site Search at RS Components, the world’s largest electronic component distributor.

Flax can offer a range of consulting, training and support, provide tools for test-driven relevancy tuning and we also run Search Workshops. If you need advice or help please get in touch.

The post Measuring search relevance scores appeared first on Flax.

1. How to get started on improving Site Search Relevancy
2. A suggested approach to running a Site Search Tuning Workshop
3. Auditing your site search performance
4. Developing ongoing search tuning processes
5. Measuring search relevance scores

In my last blog I wrote about how to create an audit of your current site search performance. In this blog I cover how to develop search tuning processes.

Once started on your search tuning journey developing ongoing processes is a must. Search tuning is an iterative process and must be treated as such. In the same way that external search traffic – PPC and SEO – is continually reviewed and optimised, so must on site search be: otherwise you have invested a lot of time and money to get people to your site but then leave them wandering aimlessly in the aisles wondering if you have the product or information you so successfully advertised!

There are 2 key areas to focus on when developing search processes:

1. Ongoing review of search performance
2. Dedicated resource

1. Ongoing review of search performance

Develop a framework for measuring relevancy scores

It’s good practice to develop a benchmark as to how search queries are performing through creating a search relevancy framework. Simply put, this is a score assigned to each search result based on how well that result answers the original query.

You can customise the scoring system you use to score your search results. Whatever you choose the key is to ensure that your search analysts are consistent in their approach, the best way to achieve that is through providing documented guidelines.

Understanding how query scores change with different configurations is an integral part of search tuning process but you should also run regular reviews on how queries are performing. This way you’ll know the impact loading new documents and products into your site is having on overall relevancy and highlight changes you need to feed into your product backlog.

Process for manually optimising important or problematic queries

Even with a search tuning test and learn plan in place there will be some queries that don’t do as well as well as expected or for which a manual custom build response provides a better customer experience.

Whilst manually tuning a search can sometimes be viewed in a negative light – after all search should ‘just work’ – it shouldn’t be seen as such. Manually optimising important search queries means that you can provide a tailored response for your customer. The queries you optimise will be dependent on your metrics and what you deem as being a good or bad experience.

With manual optimisation you can should also build in continual reviews and take the opportunity to test different landing pages.

Competitive review

I’ve talked about this in a few of my other blogs but it is especially important for eCommerce sites to understand how your competitors are answering your customers’ queries. As you create a search relevancy framework for your site it’s easy to score the same queries on your competitors to draw out any comparisons and understand opportunities for improvements.

2. Dedicated Resource

Creating and maintaining the above reviews needs resource. Ideally you would have a staff member dedicated to reviewing search and responsible for updating product backlog configuration changes, working alongside developers to ensure changes are tested and deployed successfully.

If you don’t have a dedicated person responsible, the right skills will undoubtedly exist within your organisation. You will have teams who understand your product / information set, and within that team you will find a sub-set of individuals who have problem solving skills combined with a passion to improve the customer experience. Once you’ve found them, providing them with some light search knowledge will be enough to get you started.

Whether it’s a full-time role or part-time having someone focus on reviewing search queries should be part of your plan.

What’s next?

Now you have processes and a team in place it’s time to consider what to measure (and how). In my next blog I’ll cover how to measure search relevancy scores.

Karen Renshaw is an independent On Site Search consultant and an associate of Flax. Karen was previously Head of On Site Search at RS Components, the world’s largest electronic component distributor.

Flax can offer a range of consulting, training and support, provide tools for test-driven relevancy tuning and we also run Search Workshops. If you need advice or help please get in touch.

The post Developing ongoing search tuning processes appeared first on Flax.

1. How to get started on improving Site Search Relevancy
2. A suggested approach to running a Site Search Tuning Workshop
3. Auditing your site search performance
4. Developing ongoing search tuning processes
5. Measuring search relevance scores

You know your search experience isn’t working – your customers, your colleagues, your bosses are telling you – you know you need to fix it, fix something but where do you start?

Understanding and improving search relevancy can often feel like a never ending journey and it’s true – tuning search is not a one-off hit – it’s an iterative ongoing process that needs investment. But the resources, companies and tools needed to support you are available.

Here, I’ll take a quick look at how to get started on your search tuning journey. I’ll be following up in subsequent blog posts with more details of each step.

Getting Started

Like any project, to be successful you need to understand what you want to achieve. The best way is to kick off the process with a multi-functional Search Workshop.

Typically ran over 2 days, this workshop is designed to identify what to focus on and how. It becomes the key to developing ongoing search tuning processes and driving collaborative working across teams.

Workshop Agenda

Whilst the agenda can be adapted to be specific to your organisation, in the main there are 4 key stages to it:

1. Audit
2. Define
3. Testing Approach
4. Summary

1. Audit – Where are we are now?

Spend time understanding in depth what the issues are. There are many sources of information you can call on:

• Web Analytics – How are queries performing today?
• Customer Feedback – What are the key areas that your customers complain about?
• Known Areas of Improvement – What’s already on your product backlog?
• Competitive Review – Very important for eCommerce sites – how are your competitors responding to your customers queries?

2. Define – Where do we want to be?

As a team agree what the objectives for the project are:

• What are the issues you want to address?
• Are there specific types of search queries you want to focus on?
• Is a overhaul of all search queries something you want to achieve?
• What are the technical opportunities you haven’t yet exploited?

3. Testing Approach – What’s the plan of attack?

This is the time to plan out what changes you will make and what methodology for testing and deployment you are going to use.

• What order should you make your configuration changes in?
• Are there any constraints / limitations you need to plan around?
• What resources do you need to support search configuration testing?
• How are you going to measure and track your changes so you know they are successful?
• Do you need to build in a communication plan for stakeholders?

4. Summary

Ensure that all actions are captured in a project plan with clear owners and timescales.

Workshop Attendees

Within an organisation multiple teams have responsibility for making search better, so at a minimum a subject matter expert from each team should attend.

Key attendees:

• Business Owner
• Search Developer
• Content Owner
• Web Analyst

Benefits of the workshop

There are practical and cultural benefits to approaching search in this way:

• Collaborative working practices across the different disciplines are improved
• Shared objectives and issues leads to better engagement and understanding of the approach
• A test and learn approach can be developed with the time between testing iterations reduced
• The workshop itself is an indicator to the wider business that search is now a key strategic priority and that it is getting the love and attention it needs

In my next blog I’ll cover how to run the workshop in more detail.

Karen Renshaw is an independent On Site Search consultant and an associate of Flax. Karen was previously Head of On Site Search at RS Components, the world’s largest electronic component distributor.

Flax can offer a range of consulting, training and support, provide tools for test-driven relevancy tuning and we also run Search Workshops. If you need advice or help please get in touch.

The post How to get started on improving Site Search Relevancy appeared first on Flax.