Popularity + most searched terms

In this post I will combine the popularity of a document with the most searched terms in order to boost the results based on past search issued against a particular index.

PUT blogposts

PUT statistics

PUT /blogposts/post/1
{
  "title":   "About popularity",
  "content": "In this post we will talk about...",
  "votes":   6
}

PUT /blogposts/post/2
{
  "title":   "About elasticsearch",
  "content": "In this post we will talk about...",
  "votes":   3
}

PUT /blogposts/post/3
{
  "title":   "About popularity",
  "content": "In this post we will talk about...",
  "votes":   7
}

PUT /statistics/queries/1
{
  "user_query":   "popularity"
}

PUT /statistics/queries/2
{
  "user_query":   "popularity in elasticsearch"
}

PUT /statistics/queries/3
{
  "user_query":   "boost"
}

PUT /statistics/queries/4
{
  "user_query":   "boost in elasticsearch"
}


PUT /statistics/queries/5
{
  "user_query":   "elasticsearch is the best search engine"
}

GET blogposts/post/_mapping

GET statistics/queries/_mapping

POST statistics/queries/_search
{
    "query" : {
        "match_all" : {}
    },
    "facets": {
        "keywords": {
            "terms": {
                "field": "user_query"
            }
        }
    }
}

POST blogposts/post/_search
{
 
    "sort" : [
        { "votes" : {"order" : "desc"}},
        "_score"
    ],
    "query" : {
        "match" : {
            "title":{
                "query":"elasticsearch popularity"
             
            }
        }
    }
}

[ElasticSearch] An example of using SetFetchSource

The SetFetchSource is a new feature of ElasticSearch1.1. This is a example of how to use it:

String [] excludes = {"field0"};
String [] includes = {"field1","field2"};

SearchResponse searcher = client.getClient().prepareSearch(INDEX).setFetchSource(includes, excludes).setQuery(qb).execute().actionGet();

If you do not include the "addFields()" command, you will not be able to iterate over the fields of the hits. However, you can iterate using "sourceAsMap()".


for (SearchHit hit : searcher.getHits().getHits()) {
    Map hits = hit.sourceAsMap();

    for (String key : hits.keySet()) {
        //do something
    }

    Object [] fieldValues = hits.values().toArray();

    for (Object fieldValue:fieldValues) {
        //do something
    }
}

As you will notice, the excluded fields at "excludes" will not show up. Also, if you try "hit.getFields()" it will be empty.



References:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-source-field.html

https://github.com/elasticsearch/elasticsearch/blob/master/src/test/java/org/elasticsearch/search/source/SourceFetchingTests.java

[ElasticSearch] QueryBuild vs. String: ElasticsearchParseException[Failed to derive xcontent from...

I had a problem while writing tests for use the new setFetchSource feature of ES 1.1. After some test, research and thinking (try and error). I finally realised that the problem rises when you are using a plain string query instead of a QueryBuilder.


org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query_fetch], all shards failed; shardFailures {[KMpFYBpxRECgm3gJC_1-uw][content][0]: SearchParseException[[content][0]: from[-1],size[10]: Parse Failure [Failed to parse source [{"size":10,"query_binary":"VGVzdHRleHQ=","_source":{"includes":["CONTENTID","URL"],"excludes":["CONTENT"]},"fields":["CONTENTID","URL"]}]]]; nested: ElasticsearchParseException[Failed to derive xcontent from (offset=0, length=8): [84, 101, 115, 116, 116, 101, 120, 116]]; }
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:272)
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:224)
    at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:307)
    at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:71)
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)


Makes sense, since the error says: "failed to derive xcontent from (offset". However, for a not native english speaker, took some time. Just change:

String query="";
        
SearchResponse response = client.prepareSearch(CONTENT_INDEX).setFetchSource(includes, excludes).setQuery(query).addFields(includes)
                    .setSize(10).execute().actionGet();

To:

String query="";

QueryStringQueryBuilder qb = QueryBuilders.queryString(query);

SearchResponse response = client.prepareSearch(CONTENT_INDEX).setFetchSource(includes, excludes).setQuery(qb).addFields(includes)
                    .setSize(10).execute().actionGet();

SecurityException: WifiService: Neither user 10082 nor current process has android.permission.ACCESS_WIFI_STATE

If you are trying to get scan the networks through a mobile device, first of all you have to add the permissions in your AndroidManifest.xml:








public class MainActivity ... {

 public void scanWiFi(View view) {
        mainWifi = (WifiManager) getSystemService(Context.WIFI_SERVICE);

        registerReceiver(receiverWifi, new IntentFilter(WifiManager.SCAN_RESULTS_AVAILABLE_ACTION));
        mainWifi.startScan();

}

    class WifiReceiver extends BroadcastReceiver {
        public void onReceive(Context c, Intent intent) {
            StringBuilder sb = new StringBuilder();
            wifiList = mainWifi.getScanResults();
            for (int i = 0; i < wifiList.size(); i++) {
                sb.append(new Integer(i + 1).toString() + ".");
                sb.append((wifiList.get(i)).SSID.toString());
                sb.append((wifiList.get(i)).frequency);
                sb.append((wifiList.get(i)).level);
                sb.append("\\n");
            }
            mainText.append(sb);
        }
    }

}

How to use the Path Hierarchy Tokenizer in ElasticSearch

How to use the Path Hierarchy Tokenizer:

0. If you and just to check the output of this tokenizer, you can run:

curl -XGET 'localhost:9200/_analyze?tokenizer=path_hierarchy&filters=lowercase' -d '/something/something/else'


1. Put a map:

$ curl -XPUT localhost:9200/index_files3/ -d '
{
  "settings":{
     "index":{
        "analysis":{
           "analyzer":{
              "analyzer_path_hierarchy":{
                 "tokenizer":"my_path_hierarchy_tokenizer",
                 "filter":"lowercase"
              }
           },
       "tokenizer" : {
              "my_path_hierarchy_tokenizer" : {
                  "type" : "path_hierarchy",
                  "delimiter" : "/",
                  "replacement" : "*",
                  "buffer_size" : "1024",
                  "reverse" : "true",
                  "skip" : "0"
               }
           }
        }
     }
  },
  "mappings":{
     "file":{
        "properties":{
           "path":{
              "analyzer":"analyzer_path_hierarchy",
              "type":"string"
           }
        }
     }
  }
}'


2. Get the map:

curl -XGET 'http://localhost:9200/index_files3/file/_mapping'


3. Add a new document:

$ curl -XPUT 'http://localhost:9200/index_files3/file/1' -d '{
    "name" : "c1",
    "text" : "t1",
    "path" : "/c1/c2/c3"
}'


4. Check if it is there:

$ curl -XGET 'http://localhost:9200/index_files3/file/1'


5. Search:

Fail: curl -XGET 'http://localhost:9200/index_files3/_search?q=path:c1'

Success: curl -XGET 'http://localhost:9200/index_files3/_search?q=path://c1'

Error: Could not find or load main class org.codehaus.classworlds.Launcher

$ which mvn

$ sudo apt-get install maven

$ mvn -version

The command apt-get install the Maven in /usr/share/maven, then

export M2_HOME=/usr/share/maven
export M2=%M2_HOME%\bin

PATH=$PATH:$JAVA_HOME
PATH=$PATH:$M2
export PATH


http://www.cubeia.com/2009/02/10-sure-signs-you-are-doing-maven-wrong/

Regular expression (regex) for JDBC connection string/URL

Regular expression (regex) for JDBC connection string/URL

String regex= "(^jdbc:"+databaseName.toLowerCase()+":){1,1}\\S*(:?)(//+)\\S*(/+)\\S+";

Good place to test:

http://java-regex-tester.appspot.com/