He was kind enough to ask me to join a recent podcast. During the podcast, we had a great conversation about open source search, enterprise search, our recent book and whether Cognitive Search actually exists. Listen to the podcast here.

There is a small group of us working in the search business who believe that a major obstacle to the success of search projects is the lack of guidance, support and training for search managers.  It’s not just about technology – bear in mind that no matter what their marketing tells you, most search engines are basically the same – but how you apply that technology to business problems. The career path to becoming a search manager is unclear and rocky, with no formal training available, no professional association to join and little peer recognition. Since few at executive level understand much about search technology and many are swayed by the latest marketing buzzwords it can be a thankless task. I suspect many search managers have to explain the same things again and again: why you can’t just ‘make it work like Google’, why promises of Artificial Intelligence and Machine Learning by the big search vendors won’t help if your content and metadata is still a mess, and why all the problems with the current search implementation won’t be fixed overnight simply by buying or building a new search engine.

Jeff’s workshop was only scheduled for three hours and we quickly realised there was so much to discuss that the agenda had to be curtailed. He’s running a similar workshop this week in Washington DC at the Enterprise Search & Discovery and I’m looking forward to hearing how it goes. Hopefully this will feed into the ongoing discussions amongst the professional community (comprising independent search experts from across the world with decades of experience, working in varying areas such as Sharepoint, open source, legacy search technology and intranet consultancy – we all think things have to change) around how we can better support search managers with effective training, qualifications, reports and other resources.

Watch this space – and in the meantime, if you need help with either the technical or management aspects of search, do get in touch.

The post It’s not just about technology – training for search managers is vital appeared first on Flax.

Solr, being the older and more mature project, is often chosen by organisations who are comfortable with building and managing enterprise Java applications, already using other Apache open source projects and whose data is generally complex and of many different sizes and types. The queries used may also be quite complex. They like the fact that there are many places to obtain support and development services from and the wide range of documentation, books and articles on Solr. They also like its proven stability at large scale. Some examples from our own client list are Bloomberg, News UK, Reed Specialist Recruitment and Infomedia.

Elasticsearch is often chosen by organisations who are following the latest trends in terms of programming languages and frameworks, aren’t necessarily thinking they need a ‘search engine’ (they may for example be building analytics tools) and are likely to be ingesting a large number of reasonably small items of data. They want a tool that will scale easily and automatically without them having to think about how to do it. Although there isn’t so much documentation on Elasticsearch, and the support and training options are pretty much centred around one company (who also control the development roadmap), they like the fact that it’s quick to get started with. Some examples from our own client list are Arachnys, the Office for National Statistics, Westcoast and Eagle Genomics.

There’s exceptions of course and several of our clients use both Solr and Elasticsearch in different situations and for different purposes. You can build site search, enterprise search, Big Data and analytics systems with either – and we’re happy to offer consulting, training and support for both. If you need advice or help with your choice do get in touch or come along to a Meetup and chat to us in person.

The post Choosing between Elasticsearch and Solr appeared first on Flax.

• Building an open source search team is hard – let us help with training & mentoring on Solr and Elasticsearch
• RS Components: Flax & Quepid help us to make “crucial” data driven decisions for tuning search
• 40x faster indexing with Elasticsearch for Hadoop – over a gigabyte per second!

The post Flax Newsletter November 2015 appeared first on Flax.

You might think this is great news for Flax, as we’ll always be in demand, but it’s not quite that simple! As consultants we’re often brought asked to develop proofs-of-concept or to work as part of a team developing production systems, however our clients will also need to develop search engine skills internally so the project can be supported and continue to evolve. We’re also keen to drive development of the core technologies as part of our commitment to the open source community – which we already support by running Meetups, Hackdays and Workshops.

We can help in several ways:

• Helping to identify internal and external candidates with the right base skills and experience
• Delivering formal training on Solr and Elasticsearch, basic and advanced topics (either on-site or classroom based)
• Mentoring focused on a particular project – for example, we might lead a team or individual through some examples in the morning, then they try these techniques on their own data in the afternoon, repeating the process for a week or two – a great way to bootstrap a search project!
• On-site training for business analysts, managers and others on what is possible with search engines and how to think about querying and analysing your data
• Providing an introduction to the wider community of search – which books to read, which mailing lists to join and which events to attend

Our experts have years of experience in both software training and open source search and are highly active in the open source community. Do let us know if we can help build your search team.

The post Why building an open source search team is hard – and how we can help appeared first on Flax.

The search idea

So you’ve got this idea and you’re convinced that you need search as part of the puzzle, but you’re not sure where it fits, whether it will be performant or how to gather and transform your data so it’s ready for searching. Perhaps you’re from a startup, or maybe part of a skunkworks projects in a larger organisation. What you need is someone who really understands search software and what can be done with it to sit with you for a day or two, validate your technical choices, help you understand how to shape your data, even play with some basic indexing.

The proof of concept

You’re a little further along – you know what technology you’ll be using and you have some data all ready for indexing. However, before your funders or boss will release more budget you need to build something they can see (and search) – you’ll need an indexer and a basic search application. You could do it yourself but time is limited and you’ve not built a search application before. You’re expecting to spend a week or two developing something to show others, that lets them search real data and see real results. You might also want to experiment with scale – see what happens to performance when you add a few million items to the index, even if the schema isn’t quite right yet.

The big one

You’re building the big one – indexing complex data or many millions of items, and/or for a huge user base. You need to be very sure your indexing pipeline is fast, scales well, copes with updates and can transform data from many sources. You need to develop the very best search schema. Your search architecture must be resilient, cope with heavy load, failover cleanly and give the correct results. You’re assembling a team to build it but you need specialist help from people who have built this kind of system at scale before.

The migration

Finally you’ve secured budget to move away from the slow and innacurate search engine that everyone hates! Search really does suck, but you now have a chance to make it better. However, although you know how to keep the old engine running you don’t have much experience of open source search. Even though the old engine isn’t great, you’re doing a lot of business with it and you want to be confident that relevance is as good (and hopefully better) with the new engine – maybe you want to develop a testing framework?

We’re also increasingly delivering training (both for business users who want to know the capabilities of open source search and for technical users who want to improve their knowledge – we can tailor this to your requirements) and ongoing support – but everything starts with a search project of some kind!

The post The four types of open source search project appeared first on Flax.