My main motivation for talking at Essex (and at City University a couple of weeks later) will be to inspire students to consider working in the world of open source software and more specifically the commercial applications of what academics call information retrieval – search engines.  It’s an interesting field to work in – we have clients in a huge variety of sectors including e-commerce, law, publishing and government; we deal with both small startups and multinational businesses and help build systems indexing a few thousand to several billion items. It’s constantly changing as new requirements, ideas and innovations appear. It’s taken our staff around the world (Singapore, Malaysia, the USA, Denmark as a small sample from the last couple of years) and led to us gaining a global reputation and becoming part of a select group of independent search specialists. From being somewhat of a curiosity when we started, open source search engines have now gained huge acceptance and have changed the search market beyond recognition – no longer can vendors charge six or seven figures for mysterious black boxes (and more to make them actually do something useful).

However our sector needs more people – not just developers, but business-focused search managers who understand how to build search engines that truly deliver value to employees and customers. As I’ll say to the students next week there’s a skill shortage, plainly illustrated by the plaintive slide that ends nearly all search conference and Meetup presentations – “We’re Hiring!”. Time to learn to code, download Lucene/Solr or Elasticsearch, try out the examples, read our book and look forward to a great career in search!

The post Inspiring students to work in Open Source Search appeared first on Flax.

Our intention was to combine our skills and experience, drawn both from academia and industry, and create something that would be useful for both those interested in researching enterprise search (perhaps for a PhD) and those developing actual search systems. Once you’ve read the book itself there are over 20 pages of references (derived both from academia and industry) to follow up. Hopefully the book will complement Martin White’s seminal Enterprise Search (Martin has kindly reviewed it for us here).

Huge thanks must go of course to my co-author Udo, Mark Sanderson at Now Publishing, those who reviewed early drafts including Martin White and David Hawking and all those who have provided help and information. The book will be available in printed form at the SIGIR 2017 conference in Tokyo and the publishers have also made the entire book available for free download until July 29th. We would of course be very grateful for any feedback! We’re also planning some joint presentations this autumn to introduce the book.

We’d also like to hope that our book goes a little way towards explaining what is still a much misunderstood, over-hyped and over-sold field – as Martin writes today.

The post Announcing our new book, Searching the Enterprise appeared first on Flax.

Amir Dotan started us off with details of his work in designing user interfaces for the financial services sector, describing some of the challenges involved in designing for a high-pressure and highly regulated environment. Although he didn’t talk about search specifically we heard a lot about how to design useful interfaces. Two quotes stood out: “The right user interface can help make billions”, and as a way to get feedback “find someone nice in the business and never let them go”.

Gregory Grefenstette of Exalead was next, talking about his new book on Search Based Applications. He explained how SBAs have advantages over traditional databases in the three areas of agility, usability and performance and went on to show some examples, before an unfortunate combination of a broken slide deck and a failing laptop battery brought him to a halt: in retrospect a great advertisement for a physical book over a computer!

Upayavira of Sourcesense was next with details of a new search built for online news aggregator Moreover. This dealt with scaling Lucene/Solr to cope with indexing 2 million new documents a day, for a rolling 2 month index. He showed how some initial memory and performance problems had been solved with a combination of pre-warming caches, tweaks to the JVM and Java garbage collector and eventually profiling of their custom code. Particularly interesting was how they had developed a system for spinning up a complete copy of the searchable database (for load balancing purposes) on the Amazon EC2 cloud – from a standing start they can allocate servers, install software and copy across searchable indexes in around 40 minutes. This was a great demonstration of the power of the open source model – no more licenses to buy! Search performance over this large collection is pretty good as well, with faceted queries returning in a second or two and unfaceted in half a second.

We also heard from Martin White about an exciting new search related conference to be held in October this year in London in association with Information Today, Inc., and I managed a quick plug for our inaugural Cambridge Enterprise Search Meetup on Wednesday 16th February.

The post Enterprise Search London – Financial applications, SBA book and Solr searching 120m documents appeared first on Flax.