Thursday 21 January 2016

Searching Elasticsearch index for matching data

The indexed document in Elasticsearch can be searched with matching criteria using various Query DSLs available. We can define the fields to be searched and the kind of data we require. There is variety of search options available with Elasticsearch which we will be discussing in upcoming topics.

In this post, let’s see how search works and how to do it using REST service and Java API.

Search index using REST API:

1. URI search

$ curl -XGET 'http://localhost:9200/simplyjava/user/_search?q=userName:steve’

The above search will retrieve all the users who have name “steve”. It will look up for complete word and partial matches will not be retrieved.

Response:
{
   "took": 7,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 1,
      "max_score": 1.4054651,
      "hits": [
         {
            "_index": "simplyjava",
            "_type": "user",
            "_id": "4",
            "_score": 1.4054651,
            "_source": {
               "userName": "Steve",
               "place": "Texas",
               "age": "29",
               "location": {
                  "lat": "80.234",
                  "lon": "-120.4"
               }
            }
         }
      ]
   }
}

2. Search with request body

The search can also be made by building request body using various query DSL statements in Elasticsearch. Below is an example of sending data as part of request body.

$ curl -XGET 'http://localhost:9200/simplyjava/user/_search –d’{
  "query": {
      "match ": {
         "userName": " steve"
      }
}
}’

Search index using Java API:

Below is the sample code snippet to build the search request and invoke the elasticsearch API to retrieve the results.

Code

QueryBuilder qb1 = QueryBuilders.matchQuery("userName", "steve");
SearchResponse response = client.prepareSearch("simplyjava")
                                         .setQuery(qb1).execute().actionGet();
for (SearchHit hit : response.getHits().getHits()) {
System.out.println(“Id : “ + hit.getId());
System.out.println(“Source : “+hit.getSourceAsString());
}

Output

Id : 4
Source : {
    "userName":"Steve",
    "place":"Texas",
    "age":"29",
    "location":{
        "lat":"80.234",
        "lon":"-120.4"
}

}


Note: We can also search for partial keywords as well. It will be covered in future topics.

Wednesday 20 January 2016

Keycloak Installation and Configuration with Wildfly

Let's discuss on how to install keycloak server and configure it in wildfly server.

Downloads

Download keycloak-1.6.0.Final.tar.gz from the belowurl
Download keycloak-wf9-adapter-dist-1.6.0.Final.tar.gz from the following url

Installation of Keycloak on the server

·         Extract the file keycloak-1.6.0.Final.tar.gz into the server path.
·         Modify the port numbers in Keycloak server(if the server is running in the same box as that of wildfly server)
·         Change the port numbers in the file standalone.xml present in
/usr/local/keycloak-1.6.0.Final/standalone/configuration

      The below screenshot contains the socket binding port numbers present in the default standalone.xml


The below screenshot contains the socket binding port number after modification




Mysql Database Creation and configuration in Keycloak server

·         Create a database named “keycloak” in the mysql database server.
·         Create a recursive folders mysql/main inside
/usr/local/keycloak-1.6.0.Final/modules/system/layers/base/com/mysql/main path
·         Create a module.xml file



·         Place the file mysql-connector-java-5.1.18-bin.jar inside this folder/usr/local/keycloak-1.6.0.Final/modules/system/layers/base/com/mysql/main
·         Modify jboss-cli.sh with the port number updated in standalone.xml file


Set JAVA_HOME path:/opt/jdk1.8.0_60

Starting Keycloak server

Locate the folder /usr/local/keycloak-1.6.0.Final/bin
– execute the shell script standalone.sh
./standalone.sh

Adding database driver and datasource to Keycloak server

·         Execute the file jboss-cli.sh present inside the folder /usr/local/keycloak-1.6.0.Final/bin.
·         Execute the commands given in the below screenshot. This command will create mysql driver and datasource in Keycloak server. (changes will appear in standalone.xml file)



Modify keycloak-server.json

·         modifyKeycloak-server.json file present inside configuration folder/usr/local/keycloak-1.6.0.Final/standalone/configuration.
·         connectionsJpafield should be updated with the new datasource created



Enabling SSL/HTTPS connection in Keycloak

·         Place the ssl certificate in the path /usr/local/keycloak-1.6.0.Final/standalone/configuration
·         Edit standalone/configuration/standalone.xml to enable SSL/HTTPS.
·         To the security-realms element add the below security-realm element:
·         Add the below element to <server name="default-server"> (it's a child element of <subsystem xmlns="urn:jboss:domain:undertow:2.0">):


           Now we can access the admin console of Keycloak server
http://<host_name>:<port_number>/auth/

Configuration changes to be done in wildfly server

Extract the adapter file into WILDFLY_HOME directory. Bin and modules folder will get updated with adapter files.
Add extension for Keycloak in standalone.xml of wildfly server under <extensions> element
Add security domain under <subsystem> – <security-domains>

Add subsystem Keycloak under <profile> tag

Troubleshooting

Problem

When invoking the admin console of Keycloak server for the first time, we get exception saying
WE'RE SORRY ...
HTTPS required

Solution


Enable HTTPS/SSL connection in Keycloak server(The same will be handled in the next posts)

Tuesday 19 January 2016

Indexing and Searching Geo points using Elasticsearch

Elasticsearch allows users to index geo points as a part of document. The Elasticsearch provides an inbuilt datatype named ‘geo_point’ to get the latitude and longitude data indexed.

Let’s see how the geo points can be stored and searched with a simple example.

Mapping data:

curl -XPUT http://localhost:9200/simplyjava/user -d '{
                " user":{
                        "properties": {
                        "location":{
                             "type":" geo_point"
                        }
          }
    }
}’

Indexing data:

The data for geo_points can be indexed using the below code snippet.

curl -XPOST http://localhost:9200/simplyjava/user/testuser1 -d '{
"location":{
                                     "lat":"80.234",
                                     "lon":"-120.4"   
}
}’

Searching data:

Below code can be used to retrieve the list of people who are within a particular mile radius from a geo location.

curl -XGET 'http://localhost:9200/simplyjava/user/_search?pretty=true' -d '{
  "query": {
"filtered":{
                   "filter" : {
                         "geo_distance" : {
                              "distance" : "1000miles",
                                      "location" : {
                                     "lat" : 70.12,
                                      "lon" : -120.4
                                     }
                         }
            }
       }
  }
}'

Elasticsearch Response:

{
   "took": 131,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 1,
      "max_score": 1,
      "hits": [
         {
            "_index": "simplyjava",
            "_type": "user",
            "_id": "1",
            "_score": 1,
            "_source": {
               "location": {
                  "lat": "80.234",
                  "lon": "-120.4"
               }
            }
         }
      ]
   }

}

Wednesday 13 January 2016

Creating users in keycloak using REST service

Keycloak allows us to create users through admin console. But in few scenarios, we might not want our users to know about the underlying authentication mechanisms. In those scenarios, we may need to create users from our application, rather than doing it from the admin console of keycloak.

In this post, let us discuss on how to create users in keycloak by using their REST services.

Code snippet


KeycloakSecurityContext session = (KeycloakSecurityContext)httpreq.getAttribute(KeycloakSecurityContext.class.getName());
String req = “http://localhost:8081/auth/admin/realms/<REALM_NAME>/users";
String jsonBody = "{\"username\":\""+request.getEmailId()+"\",\"enabled\":\"true\",\"firstName\":\""+request.getFirstName()+"\",\"lastName\":\""+request.getLastName()+"\",\"email\":\""+request.getEmailId()+"\,\"credentials\":[{\"type\":\"password\",\"value\":\"password\"}]}";
ClientRequest clientRequest = new ClientRequest(req);
 clientRequest.body("application/json", jsonBody);
clientRequest.accept("application/json");
clientRequest.header("Authorization", "Bearer " + session.getTokenString());
ClientResponse clientresponse = clientRequest.post(String.class);

jsonBody :   Contains the attributes of the user that needs to be created.

Username      -           Unique name(can be a name or an email id)
Enabled          -           this flag has to be set to true (otherwise , the user created will not be in use)
firstName       -           First name of the user
lastName        -           Last name of the user
email               -           Email id of the user
credentials     -           This is an object with value and type. Type implies the type of security to be used for the user login. Value implies the actual password to be set to the user

Tuesday 12 January 2016

How to pass xml data over HTTP post to an endpoint

A third party service can be invoked from any application using the HTTP protocols that they expose. The data that needs to be sent as part of the request, can be appended to the request through the outputstream. There are various types of data that can be sent over the HTTP Post method.

Below are the steps involved in sending xml data over HTTP Post.

      Setup the initial connection:
URL  serverAddress = new URL(endpoint_url);
            HttpURLConnection connection = null;
            connection = (HttpURLConnection) serverAddress.openConnection();

      endpoint_url – Extrenal URL to which the connection to be made.           
      
      Add headers to the request:

     The http method type and the content type can be set in the request header using the below code.
            connection.setRequestMethod("POST");
            connection.setRequestProperty("accept-charset", "UTF-8");
            connection.setRequestProperty("content-type", "text/xml; charset=utf-8");

      Include data in the request:
       
        Set whether the request has any output data and expects back any input data as part of the connection response.
             connection.setDoInput(true);
            connection.setDoOutput(true);
                
       Add the contents to be sent as part of the HTTP Post request. Make sure that the charset is added as required by the content.
wr = new OutputStreamWriter(connection.getOutputStream(),"UTF-8");
            DataOutputStream output = new DataOutputStream(connection.getOutputStream());
            System.out.println(xmlstr);
            output.writeBytes(xmlstr);
            output.flush();
            output.close();

      Connect and process the input sent as part of response:

        The connection response will have the response code and message based on which the status of the call can be evaluated. A response code 200 will be sent on successful processing of the request at the endpoint URL. The input sent from the URL can be obtained from the inputstream and can be parsed and used for further processing.
             connection.connect();
            int rc = connection.getResponseCode();
            System.out.println(rc);
            System.out.println(connection.getResponseMessage());
            rd = new BufferedReader(new InputStreamReader(connection.getInputStream()));
            sb = new StringBuilder();
            while ((line = rd.readLine()) != null) {
                 sb.append(line);
            }
            String resultStr = sb.toString();
            System.out.println(resultStr);

Note:  A 400 response code will be sent back when there is a malformation in the request data. Make sure that the content type is added as part of the request header in case of xml data. The xml data should be in correct format and the elements should be as expected by third party application

Monday 11 January 2016

Resetting password of a keycloak user using Rest Service

Not all the time, the users will want to reset the password the in keycloak admin console. Once the keycloak authentication is implemented in an application, there will be scenarios, where password reset has to be done from the third party application. 

Such scenarios can be handled either by keycloak java api or REST services exposed by keycloak.

In this post, let’s see how the password reset of a user in keycloak can be performed by using REST services.
From the code below, keycloak security context can be fetched from the http request.
KeycloakSecurityContext session = (KeycloakSecurityContext)httpreq.getAttribute(KeycloakSecurityContext.class.getName());
String req = “http://localhost:8081/auth/admin/realms/realm_name/users/userId/reset-password";
String jsonBody = "{\"type\":\"password\",\"value\":\"password\",\"temporary\":\"false\"}";
ClientRequest clientRequest = new ClientRequest(req);
clientRequest.body("application/json", jsonBody);
clientRequest.accept("application/json");
clientRequest.header("Authorization", "Bearer " + session.getTokenString());
ClientResponse clientresponse = clientRequest.put(String.class);
String resp = (String) clientresponse.getEntity();

Realm_name          -            Realm name in which the user is created
userId                       -            User id
jsonBody 
security           -           Type of security
value               -           Password to be set for the user
Temporary      -           false (makes sure that user need not change the password after logging into the application)
http://localhost:8081/auth - Keycloak admin console.

Sunday 10 January 2016

Indexing a file in Elasticsearch using mapper attachment

If you are looking for ways to index a file in Elasticsearch and search through its contents then, this post is for you. Yes, Elasticsearch does allow us to index files of any type (e.g.doc,docx,pdf,ppt,xls). It is basically done using the Apache’s text extraction library Tika. 

The file needs to be encoded as base64 and stored in a field of mapper type “attachment”. The mapping type will not be available by default like String type, so we have to add it using an external plugin.

Below are the steps to index a file.

  Install plugin:

<ElasticsearchDirectory>/bin/plugin install elasticsearch/elasticsearch-mapper-attachments/<version>

The attachment mapper versions for corresponding Elasticsearch version are listed below:

es-1.7
2.7.1
es-1.6
2.6.0
es-1.5
2.5.0
es-1.4
2.4.3
es-1.3
2.3.2
es-1.2
2.2.1
es-1.1
2.0.0
es-1.0
2.0.0
es-0.90
1.9.0

Mapping field type
        curl -XPUT http://localhost:9200/simplyjava/resume -d '{
                " resume":{
                        "properties": {
                        "file":{
                             "type":"attachment"
                        }
          }
 }

Analyzers can also be added to the attachment type.


   Index file in Elasticsearch

    Using Script:

#!/bin/sh
encoded=`cat <filename> | perl -MMIME::Base64 -ne 'print encode_base64($_)'`
resume="{\"file\":\"${encoded}\"}"
echo "$resume" > resume.file
curl -X POST "localhost:9200/simplyjava/resume/user1 " -d @resume.file

   Using Java code:

              File file =new File(<filepath>);
        FileInputStream fis=new FileInputStream(file);
        int length=fis.available();
        byte[]byteArray=new byte[length];
        fis.read(byteArray);
        fis.close();
        BASE64Encoder encoder = new BASE64Encoder();
        base64= encoder.encode(byteArray);
        Map<String, Object> json = new HashMap<String, Object>();
        json.put("file",encodedFile);
       client.prepareIndex("simplyjava", "resume",”user1”).setSource(json).get();

The above script will store the file as an encoded string into the type “resume” under id “user1” with field name as “file”.