We’ve spent a lot of time working with clients who recognise that their search engine isn’t delivering relevant results to users. Often this is seen as solely a technical problem, which can be resolved simply by changing query parameters or the search engine configuration – but technical teams need clear direction on why a result should or should not appear at a certain position, not just request for general relevance improvements.
It’s thus important to consider relevance as a business-wide issue, with multiple stakeholders providing input to the tuning process. We recommend the creation of a search relevance team – in a perfect world this should consist of dedicated staff, but even in the largest organisations this can be difficult to resource. It’s possible however to create a team to share the responsibility of improving relevance, contributing as they can.
The team should be drawn from the following business areas. Note that in some organisations some of these roles will be shared.
- Content – the content team create and manage the source data for the search engine, are responsible for keeping this data clean and consistent with reliable metadata. They may process external data into a database or other repository as well as creating it from scratch. The best search engine in the world can’t give good results if the underlying data is unreliable, inconsistent or badly formatted.
- Vendor – if the search engine is a commercial product, the vendor must provide sufficient documentation, training and support to the client to allow the engine to be tuned. If the engine is an open source project this information should be openly available and backed up by specialist consultancies who can provide training and technical support (such as Flax).
- Development – the development team are responsible for integrating the search engine into the client’s systems, indexing the source data, maintaining the configuration, writing the search queries and adding new features. They will make any changes that will improve relevance.
- Testing – the test team should create a process for test-driven relevance tuning, using tools such as Quepid to gather relevance judgements from the business. The test cases themselves can be built up from a combination of query logs, known important query terms (e.g. new products, common industry terms, SEO terms) and those queries deemed most valuable to the business.
- Operations – this team is responsible for keeping the search engine running at best performance with appropriate server provision and monitoring, plus providing a failover capacity as required.
- Sales & marketing, product owners – these teams should know why a particular result is more relevant than another to a customer or other user, by gathering online feedback, talking to users and knowing the current business goals. This team can thus help create the test cases discussed above.
- Management – management support of the relevance tuning process is essential, to commit whatever resources are required to the technical implementation and test process and to lead the search relevance team.
The search relevance team should meet on a regular basis to discuss how to build test cases for important search queries, examine the current position in terms of search relevance and set out objectives for improving relevance. The metrics chosen to measure progress should be available to all of the team.
Search relevance tuning should be seen as a shared responsibility, rather than simply a technical issue or something that can be easily resolved by building or buying a new search engine (a new, un-tuned search engine is unlikely to be as good as the current one). A well structured and resourced search relevance team can make huge strides towards improving search across the business – reducing the time users take to find information and improving responsiveness. For businesses that trade online, relevant search results are simply essential for retaining customers and a high level of conversion.
Flax regularly visit clients to discuss how to build an effective search team – do get in touch if we can help your business in this way.
