Last week Elastic, the company behind Elasticsearch, landed in London for one of their current series of one-day events. The £50 entrance fee has been put to good use, raising £16750 for AbilityNet who work on accessible IT – a very generous offer by Elastic.
Shay Banon, creator of Elasticsearch, kicked off with a brief history of the project which started when he built the Compass search engine, pretty much as a hobby project while his wife was training as a chef in London. Things have moved on somewhat: today there is a 35,000 strong community with over 35 million downloads of the Elasticsearch software and a number of high-profile users including NASA, Wikimedia and Verizon (who apparently have an impressive 500 billion items indexed).
Clinton Gormley led the next session, talking about new features in the recent 2.0 release. Resiliency, performance and analytics were major themes, with the latter leveraging Lucene’s DocValues as an off-heap column store to build various prediction and detection capabilities. Also mentioned was a new scriptable Ingest Node incorporating parts of the Logstash project. Steve Mayzak then told us about the new version 4 of the Kibana visualisation package, which has now grown in a general UI framework incorporating D3.js for charting and providing an extension API. Shay returned to tell us more about Logstash, which provides over 200 plugins for ingesting data into Elasticsearch. Next up was Uri Boness telling us about the various closed-source parts of the Elasticsearch ecosystem (including the Marvel performance monitor and Shield secuurity module) and we then heard from Morten Ingebrigtsen of Found (a hosted Elasticsearch solution, who Elastic acquired a while ago). For me the most interesting item here was news of an on-premise version of Found Premium – yes, like Lucidworks Fusion, you can now buy a packaged open source search engine from Elastic as a product. This isn’t something we generally recommend as it does remove one of the key advantages of open source, which is the lack of vendor lock-in, but it’s interesting to see Elastic plough such a familiar furrow.
The afternoon consisted of case studies including The Guardian (which I’ve written about previously), a good talk from Jay Chin on using Elasticsearch for Grid Computing for the financial services sector and a couple of use cases from Goldman Sachs. We also heard about the elasticsearch-hadoop connector – note that for high-performance indexing this may not be the best option. I missed a couple of the other talks due to a phone call but returned to hear Shay again, with a controversial statement that ‘the top 8 Lucene committers now work for Elastic’ – how exactly are you measuring that and have you told the other committers? He did however conclude reassuringly with ‘we’re not trying to force anyone to use commercial versions [of Elasticsearch]’ – good to hear!
By the way, if you want to hear how we helped a billion-pound UK IT supplier use Elasticsearch for their e-commerce website, we’ll be presenting with them at the Elasticsearch London Meetup later this month.
