Showing posts with label Elasticsearch. Show all posts
Showing posts with label Elasticsearch. Show all posts

Thursday, 21 January 2016

Searching Elasticsearch index for matching data

The indexed document in Elasticsearch can be searched with matching criteria using various Query DSLs available. We can define the fields to be searched and the kind of data we require. There is variety of search options available with Elasticsearch which we will be discussing in upcoming topics.

In this post, let’s see how search works and how to do it using REST service and Java API.

Search index using REST API:

1. URI search

$ curl -XGET 'http://localhost:9200/simplyjava/user/_search?q=userName:steve’

The above search will retrieve all the users who have name “steve”. It will look up for complete word and partial matches will not be retrieved.

Response:
{
   "took": 7,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 1,
      "max_score": 1.4054651,
      "hits": [
         {
            "_index": "simplyjava",
            "_type": "user",
            "_id": "4",
            "_score": 1.4054651,
            "_source": {
               "userName": "Steve",
               "place": "Texas",
               "age": "29",
               "location": {
                  "lat": "80.234",
                  "lon": "-120.4"
               }
            }
         }
      ]
   }
}

2. Search with request body

The search can also be made by building request body using various query DSL statements in Elasticsearch. Below is an example of sending data as part of request body.

$ curl -XGET 'http://localhost:9200/simplyjava/user/_search –d’{
  "query": {
      "match ": {
         "userName": " steve"
      }
}
}’

Search index using Java API:

Below is the sample code snippet to build the search request and invoke the elasticsearch API to retrieve the results.

Code

QueryBuilder qb1 = QueryBuilders.matchQuery("userName", "steve");
SearchResponse response = client.prepareSearch("simplyjava")
                                         .setQuery(qb1).execute().actionGet();
for (SearchHit hit : response.getHits().getHits()) {
System.out.println(“Id : “ + hit.getId());
System.out.println(“Source : “+hit.getSourceAsString());
}

Output

Id : 4
Source : {
    "userName":"Steve",
    "place":"Texas",
    "age":"29",
    "location":{
        "lat":"80.234",
        "lon":"-120.4"
}

}


Note: We can also search for partial keywords as well. It will be covered in future topics.

Tuesday, 19 January 2016

Indexing and Searching Geo points using Elasticsearch

Elasticsearch allows users to index geo points as a part of document. The Elasticsearch provides an inbuilt datatype named ‘geo_point’ to get the latitude and longitude data indexed.

Let’s see how the geo points can be stored and searched with a simple example.

Mapping data:

curl -XPUT http://localhost:9200/simplyjava/user -d '{
                " user":{
                        "properties": {
                        "location":{
                             "type":" geo_point"
                        }
          }
    }
}’

Indexing data:

The data for geo_points can be indexed using the below code snippet.

curl -XPOST http://localhost:9200/simplyjava/user/testuser1 -d '{
"location":{
                                     "lat":"80.234",
                                     "lon":"-120.4"   
}
}’

Searching data:

Below code can be used to retrieve the list of people who are within a particular mile radius from a geo location.

curl -XGET 'http://localhost:9200/simplyjava/user/_search?pretty=true' -d '{
  "query": {
"filtered":{
                   "filter" : {
                         "geo_distance" : {
                              "distance" : "1000miles",
                                      "location" : {
                                     "lat" : 70.12,
                                      "lon" : -120.4
                                     }
                         }
            }
       }
  }
}'

Elasticsearch Response:

{
   "took": 131,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 1,
      "max_score": 1,
      "hits": [
         {
            "_index": "simplyjava",
            "_type": "user",
            "_id": "1",
            "_score": 1,
            "_source": {
               "location": {
                  "lat": "80.234",
                  "lon": "-120.4"
               }
            }
         }
      ]
   }

}

Sunday, 10 January 2016

Indexing a file in Elasticsearch using mapper attachment

If you are looking for ways to index a file in Elasticsearch and search through its contents then, this post is for you. Yes, Elasticsearch does allow us to index files of any type (e.g.doc,docx,pdf,ppt,xls). It is basically done using the Apache’s text extraction library Tika. 

The file needs to be encoded as base64 and stored in a field of mapper type “attachment”. The mapping type will not be available by default like String type, so we have to add it using an external plugin.

Below are the steps to index a file.

  Install plugin:

<ElasticsearchDirectory>/bin/plugin install elasticsearch/elasticsearch-mapper-attachments/<version>

The attachment mapper versions for corresponding Elasticsearch version are listed below:

es-1.7
2.7.1
es-1.6
2.6.0
es-1.5
2.5.0
es-1.4
2.4.3
es-1.3
2.3.2
es-1.2
2.2.1
es-1.1
2.0.0
es-1.0
2.0.0
es-0.90
1.9.0

Mapping field type
        curl -XPUT http://localhost:9200/simplyjava/resume -d '{
                " resume":{
                        "properties": {
                        "file":{
                             "type":"attachment"
                        }
          }
 }

Analyzers can also be added to the attachment type.


   Index file in Elasticsearch

    Using Script:

#!/bin/sh
encoded=`cat <filename> | perl -MMIME::Base64 -ne 'print encode_base64($_)'`
resume="{\"file\":\"${encoded}\"}"
echo "$resume" > resume.file
curl -X POST "localhost:9200/simplyjava/resume/user1 " -d @resume.file

   Using Java code:

              File file =new File(<filepath>);
        FileInputStream fis=new FileInputStream(file);
        int length=fis.available();
        byte[]byteArray=new byte[length];
        fis.read(byteArray);
        fis.close();
        BASE64Encoder encoder = new BASE64Encoder();
        base64= encoder.encode(byteArray);
        Map<String, Object> json = new HashMap<String, Object>();
        json.put("file",encodedFile);
       client.prepareIndex("simplyjava", "resume",”user1”).setSource(json).get();

The above script will store the file as an encoded string into the type “resume” under id “user1” with field name as “file”.

Tuesday, 29 December 2015

Update Index in Elasticsearch

Whenever there is a change to be done in an indexed document, it is not mandatory to replace the entire document, instead elasticsearch allows the partial update of required field in it.
The create index functionality will completely replace the existing details, whereas the update functionality will allow us to partially change the document.

REST Service to update the indexed document:
POST http://localhost:9200/index_name/type_name/document_Id/_update
{
    "doc":{
        "field_name":"new_value"
    }
}

Java API to update the indexed document:

UpdateRequest updateRequest = new UpdateRequest();
updateRequest.index(“index_Name”);
updateRequest.type(“type_Name”);
updateRequest.id(“document_id”);
Map<String, Object> json = new HashMap<String, Object>();
json.put(“field_name”,”field_Value”);
updateRequest.doc(json);
try {
            client.update(updateRequest).get();
} catch (InterruptedException | ExecutionException e) {
            e.printStackTrace();
}

Note :  More than one field can be updated using a single method call.


Thanks !! Meet you soon with next post !!

Sunday, 27 December 2015

Delete Index in Elasticsearch

An existing index/type/document in Elasticsearch can be deleted either by using REST service or by using the Java API invocation.
Below are the scripts to delete the index.

Delete complete index :

REST  Service:
$ curl -XDELETE 'http://localhost:9200/<indexname>'

Java API :
client.admin().indices().delete(new DeleteIndexRequest("<indexname>")).actionGet();

Delete a type in index   :

REST Service :
$ curl -XDELETE 'http://localhost:9200/<indexname>/<typename>'

Java API :
client.prepareDelete().setIndex("<indexname>").setType("<typename>").setId("*").execute().actionGet();

Delete a particular document :

REST Service :
$ curl -XDELETE 'http://localhost:9200/<indexname>/<typename>/<documentId>

Java API:
client.prepareDelete().setIndex("<indexname>").setType("<typename>").setId("<documentId>").execute().actionGet();

Note:

The delete operation cannot be rolled back at any point. So make sure to double check the delete command before execution.

Sunday, 20 December 2015

Create Elasticsearch index using RESTFul Service and Java API


Elasticsearch is used for faster retrieval of data from a stored index. An index can be created either by using the RESTFul service or Java APIs exposed by Elasticsearch. In this blog, let us see what an index is and how to create it.

Basically, an Elasticsearch index is more like a database with multiple types (tables) in it. In relational model, an index can be related to database and type can be related to a table in the database. All the documents that are indexed are similar to the rows in a database table.
An elastic search cluster can have more than one index.

The below code will create an index with name ‘simplyjava’ in the Elasticsearch cluster. We can add our own settings in the created index. By settings, I mean that the type of analyzers, number of shards and number of replicas to be used by index. This will be covered in upcoming posts.

Creating an index using Elasticsearch RESTFul service:

$ curl -XPUT 'http://localhost:9200/simplyjava/'

Creating an index using Elasticsearch Java API:

Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", "elasticsearch").build();
TransportClient transportClient = new TransportClient(settings);
transportClient = transportClient.addTransportAddress(new InetSocketTransportAddress("localhost", 9300));
Client  client = (Client) transportClient;
CreateIndexRequestBuilder createIndex = client.admin().indices().prepareCreate("simplyjava");
CreateIndexResponse response = createIndex.execute().actionGet();


Note : Even though both the codes do the same functionality you can see that the port value used is different. That is because, 9300 is the port where the TransportClient resides and 9200 is the http port configured in Elasticsearch.