Relevance Engineering is a relatively new concept but companies such as Flax and our partners Open Source Connections have been carrying out relevance engineering for many years. So what is a relevance engineer and what do they do?
In this series of blog posts I’ll try to explain what I see as a new, emerging and important profession.
When Flax is working with clients on relevance tuning engagements we aim to gain an overview of the various technology the client uses and how it is obtained, deployed, managed and maintained. This will include not just the search engine but the various systems that supply data to it, host it, monitor it and interface to it to pass results to users. In addition we must understand who is responsible for the various areas, be it in-house staff, consultants, outsourcing or third party suppliers.
We try to answer the following questions in detail, including who supplies, modifies, maintains and supports the various systems concerned, what versions are used and where and how they are hosted and configured. We hope for full access to inspect the systems but this is not always possible – at the least, we need copies of configuration files and settings.
- What systems supply the source data for search?
- What is the current search technology?
- Is the search engine part of another system (such as a content management system or product information system)?
- What interface is there between the systems that supply source data and the search engine?
- What systems monitor and manage the search engine?
- What systems are used to submit queries to the search engine?
- What query logging is performed and at what level?
- How are development, test, staging and production systems arranged and what access is available to these?
- What are the processes used to deploy new software and configuration?
- What testing is performed?
It’s common to find flaws in the overall technical landscape – as an example, we’ll often find that there is no effective source control of search engine configuration files, with these having been originally derived from an example setup not intended for production use and since modified ad-hoc as issues arose. In this case it’s quite common that no-one knows why a particular setting has been used!
Without a good overall idea of the technology landscape it will be hard if not impossible to improve relevance. External processes (such as how hard it is to obtain a recent and complete log file from a production system) will also impact how effective these improvements will be.
Finally, as search is often owned by the IT department (and by the time we arrive, search is usually viewed as ‘broken’) we sometimes find a ‘bunker mentality’ – those responsible for the implementation are hunkered down and used to being harried and complained at by others who are unhappy with how search is (not) working. It’s important to communicate that only by being open and honest about the current situation can we all work together to improve things and build better search.
In the next post I’ll cover the tools a relevance engineer can use. In the meantime you can read the free Search Insights 2018 report by the Search Network. Of course, feel free to contact us if you need help with relevance engineering.
