The second day started with Eric Pugh’s second talk on The (Unofficial) State of Elasticsearch, bringing us all up to date on the meteoric rise of this technology and the opportunities it opens up especially in analytics and visualisation. Eric foresees Elastisearch continuing to specialise in this area, with Solr sticking closer to its roots in information retrieval. Giovanni Tumarello followed with a fast-paced demonstration of Kibi, a platform built on Elasticsearch and Kibana. Kibi allows one to very quickly join, visualise and explore different data sets and I was impressed with the range of potential applications including in the life sciences.

Evan Bolton of the US-based NCBI was next, talking about the massive PubChem dataset (80 million unique chemical structures, 200 million chemical substance descriptions, and 230 million biological activities, all heavily crosslinked). Although both Solr and CLucene had been considered, they eventually settled on the Sphinx engine with its great support for SQL queries and JOINs, although Evan admitted this was not a cloud-friendly solution. His team are now considering knowledge graphs and how to present up to 100 billion RDF triples. Andrea Pierleoni of the Centre for Therapeutic Target Validation then talked about an Elasticsearch cluster he has developed to index ‘evidence strings’ (which relate targets to diseases using evidence). This is a relatively small collection of 2.1 million association objects, pre-processed using Python and stored in Redis before indexing.

Next up was Nikos Marinos from the EBI Literature Services team talking about their recent migration from Lucene to Solr. As he explained most of this was a straightforward task, with one wrinkle being the use of DIH Transformers where array data was used. Rafael Jimenez then talked about projects he has worked on using both Elasticsearch and Solr, and stressed the importance of adhering to open standards and re-use of software where possible – key strengths of open source of course. Michal Nowotka then talked about a proposed system to replace the current ChEMBL search using Solr and django-haystack (the latter allows one to use a variety of underlying search engines from Django). Finally, Nicola Buso talked about EBISearch, based on Lucene.

We then concluded with another hands-on session, more aimed at Elasticsearch this time. As you can probably tell we had been shown a huge variety of different search needs and solutions using a range of technologies over the two days and it was clear to me that the BioSolr project is only a small first step towards improving the software available – we have applied for further funding and we hope to have good news soon! Working with life science data, often at significant scale, has been fascinating.

Most of the presentations are now available for download. Thanks to all the presenters (especially those who travelled from abroad), the EBI for kindly hosting the event and in particular to Dr Sameer Velankar who has been the driving force behind this project.

The post Better search for life sciences at the BioSolr Workshop, day 2 – Elasticsearch & others appeared first on Flax.

Our first speaker was Nick Burch who told us about a fascinating past project of his to source freely available postcode data for the UK. His team collected out-of-copyright maps, scanned them and created a website to crowd-source knowledge of the postcode of individual features (say a childhood home or a church). In 6-9 months they had a database of the first four characters of all UK postcodes (e.g CB1 1xx) and their locations – good enough for many location based services to take advantage of their free feeds. Shortly afterwards the UK’s Ordnance Survey released their own data for free – partly as a result of projects like Nick’s and pressure from the burgeoning Open Data movement. Nick suggested that the best way to approach projects such as this is to look for data similar to what you require, find a way to interest people on the Internet in it, provide an API for corrections and feedback and to release all your data using a permissive license.

Next up was Craig Mills, who provided some background on his past projects monitoring cod stocks (apparently it’s good to offer fisherman a £500 bounty for returning your tagging hardware!) and more recently on tools for monitoring and visualising ecology. He mentioned the open source Sphinx search engine and visualisation tool CartoDB as two key technologies, and talked about how a clickable map interface is often preferred to the traditional search box. An interesting technique was to crowd source photos from around the world and use an algorithm to spot the relative amount of ‘nature’ and ‘man-made’ textures in them – a potentially powerful way to measure how humans are changing the planet.

We finished as ever with beers, snacks and chat in the thankfully cooler downstairs bar. Thanks to both our speakers and all who came – next week we have a fantastic opportunity to join Grant Ingersoll on a free Apache Lucene/Solr hack day – do let us know if you’re coming as space is limited.

The post Cambridge Search Meetup: free postcodes from old maps & visualising fish appeared first on Flax.

We then gave a quick Introduction to Open Source Search, detailing the various packages available (from Apache Lucene/Solr to Xapian and Sphinx) and showing a quick Solr-powered demo we’d built to search some pages from the BBC Music website. Using the programmer’s first choice for an example query (the ever reliable ‘foo*’) we discovered the wonderfully named Original Rabbit Foot Spasm Band – which interestingly you can’t find via the BBC’s own site search engine due to lack of wildcard support.

Andrew Savory, Sirius’ CTO and Apache Foundation member, then gave a presentation on what an Apache project actually is and how best to engage with an open source community – very useful for those considering open source for the first time. The morning finished with a delicious barbeque on the riverbank provided by Sirius. We thought the event went very well and we’d love to confirm the rumour that this will become a regular event. Thanks to all at Sirius for organising and hosting the day and we look forward to returning.

The post An open day on open source search from Sirius & Flax appeared first on Flax.

What I find most interesting about these analyst reports is how various vendors are subdivided – either by target market, or by size, or by how ‘complex’ their platform is. Open source solutions don’t always fit the categories – for example Real Story Group list ‘Apache Project’ as a ‘specialised vendor’ – which it really isn’t. Perhaps it’s time for some new categories in these analyst reports – maybe a list of specialist open source integrators, linked with the available technologies such as Lucene, Xapian or Sphinx, combined with some data about likely costs.

The post Chalk and cheese – the difficulty of analysing open source options appeared first on Flax.