In my last blog post, I demonstrated how to set up and configure Solr to use the new XJoin search components we’ve developed for the BioSolr project, using an example from an e-commerce setting. This time, I’ll show how to use XJoin to make use of user click-through data to influence the score of products in searches.
I’ll step through things a bit quicker this time around and I’ll be using code from the last post so reading that first is highly recommended. I’ll assume that the prerequisites from last time have been installed and set up in the same directories.
The design
Suppose we have a web page for searching a collection of products, and when a user clicks on product listing in the result set (or perhaps, when they subsequently go on to buy that product – or both) we insert a record in an SQL database, storing the product id, the query terms they used, and an arbitrary weight value (which will depend on whether they merely clicked on a result, or if they went on to buy it, or some other behaviour such as mouse pointer tracking). We then want to use the click-through data stored in that database to boost products in searches that use those query terms again.
We could use the sum of the weights of all occurrences of a product id/query term combination as the product score boost, but then we might start to worry about a feedback process occurring. Alternatively, we might take the maximum or average weight across the occurrences. In the code below, we’ll use the maximum.
The advantage of this design over storing the click-through information in Solr is that you don’t have to update the Solr index every time there is user activity, which could become costly. An SQL database is much more suited to this task.
The external click-through API
Again, we’ll be using Python 3 (using the flask and sqlite3 modules) to implement the external API. I’ll be using this API to update the click-through database (by hand, for this example) as well as having Solr query it using XJoin. Here’s the code (partly based on code taken from here for caching the database connection in the Flask application context, and see here if you’re interested in more details about sqlite3’s support for full text search). Again, all the code written for this example is also available in the BioSolr GitHub repository:
from flask import Flask, request, g
import json
import sqlite3 as sql
# flask application context attribute for caching database connection
DB_APP_KEY = '_database'
# default weight for storing against queries
DEFAULT_WEIGHT = 1.0
app = Flask(__name__)
def get_db():
""" Obtain a (cached) DB connection and return a cursor for it.
"""
db = getattr(g, DB_APP_KEY, None)
if db is None:
db = sql.connect('click.db')
setattr(g, DB_APP_KEY, db)
c = db.cursor()
c.execute("CREATE VIRTUAL TABLE IF NOT EXISTS click USING fts4 ("
"id VARCHAR(256),"
"q VARCHAR(256),"
"weight FLOAT"
")")
c.close()
return db
@app.teardown_appcontext
def teardown_db(exception):
db = getattr(g, DB_APP_KEY, None)
if db is not None:
db.close()
@app.route('/')
def main():
return 'click-through API'
@app.route('/click/<path:id>', methods=["PUT"])
def click(id):
# validate request
if 'q' not in request.args:
return 'Missing q parameter', 400
q = request.args['q']
try:
w = float(request.args.get('weight', DEFAULT_WEIGHT))
except ValueError:
return 'Could not parse weight', 400
# do the DB update
db = get_db()
try:
c = db.cursor()
c.execute("INSERT INTO click (id, q, weight) VALUES (?, ?, ?)", (id, q, w))
db.commit()
return 'OK'
finally:
c.close()
@app.route('/ids')
def ids():
# validate request
if 'q' not in request.args:
return 'Missing q parameter', 400
q = request.args['q']
# do the DB lookup
try:
c = get_db().cursor()
c.execute("SELECT id, MAX(weight) FROM click WHERE q MATCH ? GROUP BY id", (q, ))
return json.dumps([{ 'id': id, 'weight': w } for id, w in c])
finally:
c.close()
if __name__ == "__main__":
app.run(port=8001, debug=True)
This web API exposes two end-points. First we have PUT /click/[id] which is used when we want to update the SQL database after a user click. For the purposes of this demonstration, we’ll be hitting this end-point by hand using curl to avoid having to write a web UI. The other end-point, GET /ids?[query terms], is used by our XJoin component and returns a JSON-formatted array of id/weight objects where the query terms from the database match those given in the query string.
Java glue code
Now we just need the Java glue code that sits between the XJoin component and our external API. Here’s an implementation of XJoinResultsFactory that does what we need:
package uk.co.flax.examples.xjoin;
import java.io.IOException;
import java.net.URLEncoder;
import java.util.HashMap;
import java.util.Map;
import javax.json.JsonArray;
import javax.json.JsonObject;
import javax.json.JsonValue;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.search.xjoin.XJoinResults;
import org.apache.solr.search.xjoin.XJoinResultsFactory;
public class ClickXJoinResultsFactory
implements XJoinResultsFactory {
private String url;
@Override
@SuppressWarnings("rawtypes")
public void init(NamedList args) {
url = (String)args.get("url");
}
/**
* Use 'click' REST API to fetch current click data.
*/
@Override
public XJoinResults getResults(SolrParams params)
throws IOException {
String q = URLEncoder.encode(params.get("q"), "UTF-8");
String apiUrl = url + "?q=" + q;
try (HttpConnection http = new HttpConnection(apiUrl)) {
JsonArray products = (JsonArray)http.getJson();
return new ClickResults(products);
}
}
public class ClickResults implements XJoinResults {
private Map<String, Click> clickMap;
public ClickResults(JsonArray products) {
clickMap = new HashMap<>();
for (JsonValue product : products) {
JsonObject object = (JsonObject)product;
String id = object.getString("id");
double weight = object.getJsonNumber("weight").doubleValue();
clickMap.put(id, new Click(id, weight));
}
}
public int getCount() {
return clickMap.size();
}
@Override
public Iterable getJoinIds() {
return clickMap.keySet();
}
@Override
public Object getResult(String id) {
return clickMap.get(id);
}
}
public class Click {
private String id;
private double weight;
public Click(String id, double weight) {
this.id = id;
this.weight = weight;
}
public String getId() {
return id;
}
public double getWeight() {
return weight;
}
}
}
Unlike the previous example, this time getResults() does depend on the SolrParams argument, so that the user’s query, q, is passed to the external API. Store this Java source in blog/java/uk/co/flax/examples/xjoin/ClickXJoinResultsFactory.java and compile into a JAR (again, we also need the HttpConnection class from the last blog post as well as javax.json-1.0.4.jar):
blog$ javac -sourcepath src/java -d bin -cp javax.json-1.0.4.jar:../lucene_solr_5_3/solr/dist/solr-solrj-5.3.2-SNAPSHOT.jar:../lucene_solr_5_3/solr/dist/solr-xjoin-5.3.2-SNAPSHOT.jar src/java/uk/co/flax/examples/xjoin/ClickXJoinResultsFactory.java blog$ jar cvf click.jar -C bin .
Solr configuration
Starting with a fresh version of solrconfig.xml, insert these lines near the start to import the XJoin and user JARs (substitute /XXX with the full path to the parent of the blog directory):
<lib dir="${solr.install.dir:../../../..}/contrib/xjoin/lib" regex=".*\.jar" />
<lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-xjoin-\d.*\.jar" />
<lib path="/XXX/blog/javax.json-1.0.4.jar" />
<lib path="/XXX/blog/click.jar" />
And our request handler configuration:
<queryParser name="xjoin" class="org.apache.solr.search.xjoin.XJoinQParserPlugin" />
<valueSourceParser name="weight" class="org.apache.solr.search.xjoin.XJoinValueSourceParser">
<str name="attribute">weight</str>
<double name="defaultValue">0.0</double>
</valueSourceParser>
<searchComponent name="x_click" class="org.apache.solr.search.xjoin.XJoinSearchComponent">
<str name="factoryClass">uk.co.flax.examples.xjoin.ClickXJoinResultsFactory</str>
<str name="joinField">id</str>
<lst name="external">
<str name="url">http://localhost:8001/ids</str>
</lst>
</searchComponent>
<requestHandler name="/xjoin" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="wt">json</str>
<str name="echoParams">none</str>
<str name="defType">edismax</str>
<str name="df">description</str>
<str name="fl">*</str>
<bool name="x_click">false</bool>
<str name="x_click.results">count</str>
<str name="x_click.fl">*</str>
</lst>
<arr name="first-components">
<str>x_click</str>
</arr>
<arr name="last-components">
<str>x_click</str>
</arr>
</requestHandler>
Reload the Solr core (products) to get the new config in place.
Putting the pieces together
The following query will verify our Solr setup (remembering to escape curly brackets):
blog$ curl 'http://localhost:8983/solr/products/xjoin?qq=excel&q=$\{qq\}&fl=id,name,score&rows=4' | jq .
I’ve used Solr parameter substitution with the q/qq parameters which will simplify later queries (this has been in Solr since 5.1). This query returns:
{
"responseHeader": {
"status": 0,
"QTime": 25
},
"response": {
"numFound": 21,
"start": 0,
"maxScore": 2.9939778,
"docs": [
{
"name": "individual software professor teaches excel and word",
"id": "http://www.google.com/base/feeds/snippets/13017887935047670097",
"score": 2.9939778
},
{
"name": "individual software prm-xw3 professor teaches excel & word",
"id": "http://www.google.com/base/feeds/snippets/7197668762339216420",
"score": 2.9939778
},
{
"name": "individual software prm-xw3 professor teaches excel & word",
"id": "http://www.google.com/base/feeds/snippets/16702106469790828707",
"score": 1.8712361
},
{
"name": "individual software prm-xw3 professor teaches excel & word",
"id": "http://www.google.com/base/feeds/snippets/9200068133591804002",
"score": 1.8712361
}
]
}
}
Some repeat products in the data, but so far, so good. Next, get the click-through API running:
blog$ python3 click.py
And check it’s working (this should return [] whatever query is chosen because the click-through database is empty):
curl localhost:8001/ids?q=software | jq .
Now, let’s populate the click-through database by simulating user activity. Suppose, given the above product results, the user goes on to click through to the fourth product (or even buy it). Then, the UI would update the click web API to indicate this has happened. Let’s do this by hand, specifying the product id, the user’s query, and a weight score (here, I’ll use the value 3, supposing the user bought the product in the end):
curl -XPUT 'localhost:8001/click/http://www.google.com/base/feeds/snippets/9200068133591804002?q=excel&weight=3'
Now, we can check the output that XJoin will see when using the click-through API:
blog$ curl localhost:8001/ids?q=excel | jq .
giving:
[
{
"weight": 3,
"id": "http://www.google.com/base/feeds/snippets/9200068133591804002"
}
]
Using the bf edismax parameter and the weight function set up in solrconfig.xml to extract the weight value from the external results stored in the x_click XJoin search component, we can boost product scores when they appear in the click-through database for the user’s query:
blog$ curl 'http://localhost:8983/solr/products/xjoin?qq=excel&q=$\{qq\}&x_click=true&x_click.external.q=$\{qq\}&bf=weight(x_click)^4&fl=id,name,score&rows=4' | jq .
which gives:
{
"responseHeader": {
"status": 0,
"QTime": 13
},
"response": {
"numFound": 21,
"start": 0,
"maxScore": 3.2224145,
"docs": [
{
"name": "individual software prm-xw3 professor teaches excel & word",
"id": "http://www.google.com/base/feeds/snippets/9200068133591804002",
"score": 3.2224145
},
{
"name": "individual software professor teaches excel and word",
"id": "http://www.google.com/base/feeds/snippets/13017887935047670097",
"score": 2.4895983
},
{
"name": "individual software prm-xw3 professor teaches excel & word",
"id": "http://www.google.com/base/feeds/snippets/7197668762339216420",
"score": 2.4895983
},
{
"name": "individual software prm-xw3 professor teaches excel & word",
"id": "http://www.google.com/base/feeds/snippets/16702106469790828707",
"score": 1.5559989
}
]
},
"x_click": {
"count": 1,
"external": [
{
"joinId": "http://www.google.com/base/feeds/snippets/9200068133591804002",
"doc": {
"id": "http://www.google.com/base/feeds/snippets/9200068133591804002",
"weight": 3
}
}
]
}
}
Lo and behold, the product the user clicked on now appears top of the Solr results for the that query. Have a play with the API, generate some more user activity and see how this effects subsequent queries. It will cope fine with multiple-word queries, for example, suppose a user searches for ‘games software’:
curl 'http://localhost:8983/solr/products/xjoin?qq=games+software&q=$\{qq\}&x_click=true&x_click.external.q=$\{qq\}&bf=weight(x_click)^4&fl=id,name,score&rows=4' | jq .
There being no relevant queries in the click-through database, this has the same results as for a query without the XJoin, and as we can see, the value of response.x_click.count is 0:
{
"responseHeader": {
"status": 0,
"QTime": 15
},
"response": {
"numFound": 1158,
"start": 0,
"maxScore": 0.91356516,
"docs": [
{
"name": "encore software 10568 - encore hoyle puzzle & board games 2005 - complete product - puzzle game - 1 user - complete product - standard - pc",
"id": "http://www.google.com/base/feeds/snippets/4998847858583359731",
"score": 0.91356516
},
{
"name": "encore software 11141 - fate sb cs by wild games",
"id": "http://www.google.com/base/feeds/snippets/826668451451666270",
"score": 0.8699497
},
{
"name": "encore software 10027 - hoyle board games (win 98 me 2000 xp)",
"id": "http://www.google.com/base/feeds/snippets/8664755713112971171",
"score": 0.85982025
},
{
"name": "encore software 11253 - brain food games: cranium collection 2006 sb cs by encore",
"id": "http://www.google.com/base/feeds/snippets/15401280256033043239",
"score": 0.78744644
}
]
},
"x_click": {
"count": 0,
"external": []
}
}
Now let’s simulate the same user clicking on the second product (with default weight):
blog$ curl -XPUT 'localhost:8001/click/http://www.google.com/base/feeds/snippets/826668451451666270?q=games+software'
Next, suppose another user then searches for just ‘games’:
blog$ curl 'http://localhost:8983/solr/products/xjoin?qq=games&q=$\{qq\}&x_click=true&x_click.external.q=$\{qq\}&bf=weight(x_click)^4&fl=id,name,score&rows=4' | jq .
In the results, we see the ‘wild games’ product boosted to the top:
{
"responseHeader": {
"status": 0,
"QTime": 60
},
"response": {
"numFound": 212,
"start": 0,
"maxScore": 1.3652229,
"docs": [
{
"name": "encore software 11141 - fate sb cs by wild games",
"id": "http://www.google.com/base/feeds/snippets/826668451451666270",
"score": 1.3652229
},
{
"name": "xbox 360: ddr universe",
"id": "http://www.google.com/base/feeds/snippets/16659259513615352372",
"score": 0.95894843
},
{
"name": "south park chef's luv shack",
"id": "http://www.google.com/base/feeds/snippets/11648097795915093399",
"score": 0.95894843
},
{
"name": "egames. inc casual games pack",
"id": "http://www.google.com/base/feeds/snippets/16700933768709687512",
"score": 0.89483213
}
]
},
"x_click": {
"count": 1,
"external": [
{
"joinId": "http://www.google.com/base/feeds/snippets/826668451451666270",
"doc": {
"id": "http://www.google.com/base/feeds/snippets/826668451451666270",
"weight": 1
}
}
]
}
}
Extensions
Of course, this approach can be extended to add in more sophisticated weighting and boosting strategies, or include more data about the user activity than just a simple weight score, which could be used to augment the display of the product in the UI (for example, “ten customers in the UK bought this product in the last month”).
The XJoin patch was developed as part of the BioSolr project but it is not specific to bioinformatics and can be used in any situation where you want to use data from an external source to influence the results of a Solr search. (Other joins, including cross-core joins, are available – but you need XJoin if the data you are joining against is not in Solr.). We’ll be talking about XJoin and the other features we’ve developed for both Solr and Elasticsearch, including powerful ontology indexing, at a workshop at the European Bioinformatics Institute next week.
