Showing posts with label spring data. Show all posts
Showing posts with label spring data. Show all posts

Monday, June 25, 2012

Using Redis with Spring

As NoSQL solutions are getting more and more popular for many kind of problems, more often the modern projects consider to use some (or several) of NoSQLs instead (or side-by-side) of traditional RDBMS. I have already covered my experience with MongoDB in this, this and this posts. In this post I would like to switch gears a bit towards Redis, an advanced key-value store.

Aside from very rich key-value semantics, Redis also supports pub-sub messaging and transactions. In this post I am going just to touch the surface and demonstrate how simple it is to integrate Redis into your Spring application.

As always, we will start with Maven POM file for our project:


 4.0.0
 com.example.spring
 redis
 0.0.1-SNAPSHOT
 jar

 
  UTF-8
  3.1.0.RELEASE
 

 
  
   org.springframework.data
   spring-data-redis
   1.0.0.RELEASE
  

  
   cglib
   cglib-nodep
   2.2
  

  
   log4j
   log4j
   1.2.16
  

  
   redis.clients
   jedis
   2.0.0
   jar
  

  
   org.springframework
   spring-core
   ${spring.version}
  

  
   org.springframework
   spring-context
   ${spring.version}
  
 

Spring Data Redis is the another project under Spring Data umbrella which provides seamless injection of Redis into your application. The are several Redis clients for Java and I have chosen the Jedis as it is stable and recommended by Redis team at the moment of writing this post.

We will start with simple configuration and introduce the necessary components first. Then as we move forward, the configuration will be extended a bit to demonstrated pub-sub capabilities. Thanks to Java config support, we will create the configuration class and have all our dependencies strongly typed, no XML anymore:

package com.example.redis.config;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.jedis.JedisConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.GenericToStringSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;

@Configuration
public class AppConfig {
 @Bean
 JedisConnectionFactory jedisConnectionFactory() {
  return new JedisConnectionFactory();
 }

 @Bean
 RedisTemplate< String, Object > redisTemplate() {
  final RedisTemplate< String, Object > template =  new RedisTemplate< String, Object >();
  template.setConnectionFactory( jedisConnectionFactory() );
  template.setKeySerializer( new StringRedisSerializer() );
  template.setHashValueSerializer( new GenericToStringSerializer< Object >( Object.class ) );
  template.setValueSerializer( new GenericToStringSerializer< Object >( Object.class ) );
  return template;
 }
}
That's basically everything we need assuming we have single Redis server up and running on localhost with default configuration. Let's consider several common uses cases: setting a key to some value, storing the object and, finally, pub-sub implementation. Storing and retrieving a key/value pair is very simple:
@Autowired private RedisTemplate< String, Object > template;

public Object getValue( final String key ) {
    return template.opsForValue().get( key );
}

public void setValue( final String key, final String value ) {
    template.opsForValue().set( key, value );
}
Optionally, the key could be set to expire (yet another useful feature of Redis), f.e. let our keys expire in 1 second:
public void setValue( final String key, final String value ) {
    template.opsForValue().set( key, value );
    template.expire( key, 1, TimeUnit.SECONDS );
}
Arbitrary objects could be saved into Redis as hashes (maps), f.e. let save instance of some class User
public class User {
 private final Long id;
 private String name;
 private String email;
       
    // Setters and getters are omitted for simplicity
}
into Redis using key pattern "user:<id>":
public void setUser( final User user ) {
 final String key = String.format( "user:%s", user.getId() );
 final Map< String, Object > properties = new HashMap< String, Object >();

 properties.put( "id", user.getId() );
 properties.put( "name", user.getName() );
 properties.put( "email", user.getEmail() );

 template.opsForHash().putAll( key, properties);
}
Respectively, object could easily be inspected and retrieved using the id.
public User getUser( final Long id ) {
 final String key = String.format( "user:%s", id );

 final String name = ( String )template.opsForHash().get( key, "name" );
 final String email = ( String )template.opsForHash().get( key, "email" );

 return new User( id, name, email );
}
There are much, much more which could be done using Redis, I highly encourage to take a look on it. It surely is not a silver bullet but could solve many challenging problems very easy. Finally, let me show how to use a pub-sub messaging with Redis. Let's add a bit more configuration here (as part of AppConfig class):
@Bean
MessageListenerAdapter messageListener() {
 return new MessageListenerAdapter( new RedisMessageListener() );
}

@Bean
RedisMessageListenerContainer redisContainer() {
 final RedisMessageListenerContainer container = new RedisMessageListenerContainer();

 container.setConnectionFactory( jedisConnectionFactory() );
 container.addMessageListener( messageListener(), new ChannelTopic( "my-queue" ) );

 return container;
}
The style of message listener definition should look very familiar to Spring users: generally, the same approach we follow to define JMS message listeners. The missed piece is our RedisMessageListener class definition:
package com.example.redis.impl;

import org.springframework.data.redis.connection.Message;
import org.springframework.data.redis.connection.MessageListener;

public class RedisMessageListener implements MessageListener {
 @Override
 public void onMessage(Message message, byte[] paramArrayOfByte) {
  System.out.println( "Received by RedisMessageListener: " + message.toString() );
 }
}
Now, when we have our message listener, let see how we could push some messages into the queue using Redis. As always, it's pretty simple:
@Autowired private RedisTemplate< String, Object > template;

public void publish( final String message ) {
 template.execute(
  new RedisCallback< Long >() {
   @SuppressWarnings( "unchecked" )
   @Override
   public Long doInRedis( RedisConnection connection ) throws DataAccessException {
    return connection.publish(
     ( ( RedisSerializer< String > )template.getKeySerializer() ).serialize( "queue" ),
     ( ( RedisSerializer< Object > )template.getValueSerializer() ).serialize( message ) );
   }
  }
 );
}
That's basically it for very quick introduction but definitely enough to fall in love with Redis.

Wednesday, January 25, 2012

Storing hierarchical data in MongoDB

Continuing NoSQL journey with MongoDB, I would like to touch one specific use case which comes up very often: storing hierarchical document relations. MongoDB is awesome document data store but what if documents have parent-child relationships? Can we effectively store and query such document hierarchies? The answer, for sure, is yes, we can. MongoDB has several recommendations how to store Trees in MongoDB. The one solution described there as well and quite widely used is using materialized path.

Let me explain how it works by providing very simple examples. As in previous posts, we will build Spring application using recently released version 1.0 of Spring Data MongoDB project. Our POM file contains very basic dependencies, nothing more.


    4.0.0

    mongodb
    com.example.spring
    0.0.1-SNAPSHOT
    jar

    
        UTF-8
        3.0.7.RELEASE
    

    
        
            org.springframework.data
            spring-data-mongodb
            1.0.0.RELEASE
            
                
                    org.springframework
                    spring-beans
                
                
                    org.springframework
                    spring-expression
                
            
        

        
            cglib
            cglib-nodep
            2.2
        

        
            log4j
            log4j
            1.2.16
        

        
            org.mongodb
            mongo-java-driver
            2.7.2
        

        
            org.springframework
            spring-core
            ${spring.version}
        

        
            org.springframework
            spring-context
            ${spring.version}
        

        
            org.springframework
            spring-context-support
            ${spring.version}
        
    

    
        
            
                org.apache.maven.plugins
                maven-compiler-plugin
                2.3.2
                
                    1.6
                    1.6
                
            
        
    

To properly configure Spring context, I will use configuration approach utilizing Java classes. I am more and more advocating to use this style as it provides strong typed configuration and most of the mistakes could be caught on compilation time, no need to inspect your XML files anymore. Here how it looks like:

package com.example.mongodb.hierarchical;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.mongodb.core.MongoFactoryBean;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.data.mongodb.core.SimpleMongoDbFactory;

@Configuration
public class AppConfig {
    @Bean
    public MongoFactoryBean mongo() {
        final MongoFactoryBean factory = new MongoFactoryBean();
        factory.setHost( "localhost" );
        return factory;
    }

    @Bean
    public SimpleMongoDbFactory mongoDbFactory() throws Exception{
        return new SimpleMongoDbFactory( mongo().getObject(), "hierarchical" );
    }

    @Bean
    public MongoTemplate mongoTemplate() throws Exception {
        return new MongoTemplate( mongoDbFactory() );
    }

    @Bean
    public IDocumentHierarchyService documentHierarchyService() throws Exception {
        return new DocumentHierarchyService( mongoTemplate() );
    }
}

That's pretty nice and clear. Thanks, Spring guys! Now, all boilerplate stuff is ready. Let's move to interesting part: documents. Our database will contain 'documents' collection which stores documents of type SimpleDocument. We describe this using Spring Data MongoDB annotations for SimpleDocument POJO.

package com.example.mongodb.hierarchical;

import java.util.Collection;
import java.util.HashSet;

import org.springframework.data.annotation.Id;
import org.springframework.data.annotation.Transient;
import org.springframework.data.mongodb.core.mapping.Document;
import org.springframework.data.mongodb.core.mapping.Field;

@Document( collection = "documents" )
public class SimpleDocument {
    public static final String PATH_SEPARATOR = ".";

    @Id private String id;
    @Field private String name;
    @Field private String path;

    // We won't store this collection as part of document but will build it on demand
    @Transient private Collection< SimpleDocument > documents = new HashSet< SimpleDocument >();

    public SimpleDocument() {
    }

    public SimpleDocument( final String id, final String name ) {
        this.id = id;
        this.name = name;
        this.path = id;
    }

    public SimpleDocument( final String id, final String name, final SimpleDocument parent ) {
        this( id, name );
        this.path = parent.getPath() + PATH_SEPARATOR + id;
    }

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getPath() {
        return path;
    }

    public void setPath(String path) {
        this.path = path;
    }

    public Collection< SimpleDocument > getDocuments() {
        return documents;
    }
}

Let me explain few things here. First, magic property path: this is a key to construct and query through our hierarchy. Path contains identifiers of all document's parents, usually divided by some kind of separator, in our case just . (dot). Storing document hierarchical relationships in this way allows quickly build hierarchy, search and navigate. Second, notice transient documents collection: this non-persistent collection is constructed by persistent provider and contains all descendant documents (which, in case, also contain own descendants). Let see it in action by looking into find method implementation:

package com.example.mongodb.hierarchical;

import java.util.Arrays;
import java.util.Collection;
import java.util.HashMap;
import java.util.Map;

import org.springframework.data.mongodb.core.MongoOperations;
import org.springframework.data.mongodb.core.query.Criteria;
import org.springframework.data.mongodb.core.query.Query;

public class DocumentHierarchyService {
    private MongoOperations template;

    public DocumentHierarchyService( final MongoOperations template ) {
        this.template = template;
    }

    @Override
    public SimpleDocument find( final String id ) {
        final SimpleDocument document = template.findOne(
            Query.query( new Criteria( "id" ).is( id ) ),
            SimpleDocument.class
        );

        if( document == null ) {
            return document;
        }

        return build(
            document,
            template.find(
                Query.query( new Criteria( "path" ).regex( "^" + id + "[.]" ) ),
                SimpleDocument.class
            )
        );
    }

    private SimpleDocument build( final SimpleDocument root, final Collection< SimpleDocument > documents ) {
        final Map< String, SimpleDocument > map = new HashMap< String, SimpleDocument >();

        for( final SimpleDocument document: documents ) {
            map.put( document.getPath(), document );
        }

        for( final SimpleDocument document: documents ) {
            map.put( document.getPath(), document );

            final String path = document
                .getPath()
                .substring( 0, document.getPath().lastIndexOf( SimpleDocument.PATH_SEPARATOR ) );

            if( path.equals( root.getPath() ) ) {
                root.getDocuments().add( document );
            } else {
                final SimpleDocument parent = map.get( path );
                if( parent != null ) {
                    parent.getDocuments().add( document );
                }
            }
        }

        return root;
    }
}

As you can see, to get single document with a whole hierarchy we need to run just two queries (but more optimal algorithm could reduce it to just one single query). Here is a sample hierarchy and the the result of reading root document from MongoDB


template.dropCollection( SimpleDocument.class );

final SimpleDocument parent = new SimpleDocument( "1", "Parent 1" );
final SimpleDocument child1 = new SimpleDocument( "2", "Child 1.1", parent );
final SimpleDocument child11 = new SimpleDocument( "3", "Child 1.1.1", child1 );
final SimpleDocument child12 = new SimpleDocument( "4", "Child 1.1.2", child1 );
final SimpleDocument child121 = new SimpleDocument( "5", "Child 1.1.2.1", child12 );
final SimpleDocument child13 = new SimpleDocument( "6", "Child 1.1.3", child1 );
final SimpleDocument child2 = new SimpleDocument( "7", "Child 1.2", parent );

template.insertAll( Arrays.asList( parent, child1, child11, child12, child121, child13, child2 ) );

...

final ApplicationContext context = new AnnotationConfigApplicationContext( AppConfig.class );
final IDocumentHierarchyService service = context.getBean( IDocumentHierarchyService.class );

final SimpleDocument document = service.find( "1" );
//  Printing document show following hierarchy:
//
//  Parent 1
//   |-- Child 1.1
//     |-- Child 1.1.1
//     |-- Child 1.1.3
//     |-- Child 1.1.2
//       |-- Child 1.1.2.1
//   |-- Child 1.2

That's it. Simple a powerful concept. Sure, adding index on a path property will speed up query significantly. There are a plenty of improvements and optimizations but basic idea should be clear now.

Sunday, August 7, 2011

Exploiting MongoDB together with Spring Data project: advanced concepts

In previous post we had started discussion about MongoDB and Spring Data projects. In this post I would like to show some advanced features (which could be available in next Spring Data milestone or release as part of core functionality).

First of all, let us extend our MongoService with a method that counts documents in collection which match specific query.
package com.example.mongodb;

import java.util.Arrays;
import java.util.Collection;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.dao.DataAccessException;
import org.springframework.data.document.mongodb.CollectionCallback;
import org.springframework.data.document.mongodb.MongoOperations;
import org.springframework.data.document.mongodb.convert.MongoConverter;
import org.springframework.data.document.mongodb.query.Criteria;
import org.springframework.data.document.mongodb.query.Index;
import org.springframework.data.document.mongodb.query.Index.Duplicates;
import org.springframework.data.document.mongodb.query.Order;
import org.springframework.data.document.mongodb.query.Query;
import org.springframework.stereotype.Service;
import org.springframework.util.Assert;

import com.mongodb.BasicDBObject;
import com.mongodb.DBCollection;
import com.mongodb.MongoException;

@Service
public class MongoService {
    public long countDocuments( final String collection, final Query query ) {  
        return template.executeCommand( 
            "{ " +
                "\"count\" : \"" + collection + "\"," +
                "\"query\" : " + query.getQueryObject().toString() + 
            " }"  ).getLong( "n" );
    }
}
The approach for this particular functionality is to call native MongoDB command count passing the query as a parameter. The returning structure contains number of documents in n property.

Or, in more code-friendly way:
import org.springframework.dao.DataAccessException;
import org.springframework.data.document.mongodb.CollectionCallback;

import com.mongodb.DBCollection;
import com.mongodb.MongoException;

public long countDocuments( final String collection, final Query query ) {  
    return template.execute( collection,
        new CollectionCallback< Long >() {
            @Override
            public Long doInCollection( DBCollection collection ) 
                    throws MongoException, DataAccessException {
                return collection.count( q.getQueryObject() ) );
            }
        }
    );
}

Next useful feature is bulk inserts. Please note, that in current version of MongoDB 1.8.1, when there is a duplicate inside the collection of inserting documents, bulk insert stops on first duplicate and returns so all other documents won't be inserted. Be aware of such behavior. Before moving to code snippet, let me introduce simple class SimpleDocument which we will be persisting to MongoDB:
package com.example.mongodb;

import org.springframework.data.document.mongodb.mapping.Document;

@Document( collection = "documents" )
public class SimpleDocument {
    private String id;
    private String name;
    private String content;
 
    public SimpleDocument() { 
    }
 
    public SimpleDocument( final String id, final String name ) {
        this.id = id;
        this.name = name;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public String getContent() {
        return content;
    }

    public void setContent(String content) {
        this.content = content;
    }
}
Following method inserts all documents as single bulk update:
public void insert( final Collection< SimpleDocument > documents ) {  
    template.insert( documents, SimpleDocument.class );     
}
Another very cool and useful feature to explore is MongoDB's upserts (more about this here http://www.mongodb.org/display/DOCS/Updating): if document matching specific criteria exists, it will be updated, otherwise - new document will be inserted into collection. Here is a code snipped with demonstrates it by following use case: if SimpleDocument with such name exists, it will be updated, otherwise new document will be added to collection:
@Autowired private MongoConverter converter;

public void insertOrUpdate( final SimpleDocument document ) {
    final BasicDBObject dbDoc = new BasicDBObject();
    converter.write( document, dbDoc );
  
    template.execute( SimpleDocument.class, 
        new CollectionCallback< Object >() {
            public Object doInCollection( DBCollection collection ) 
                    throws MongoException, DataAccessException {
                collection.update( 
                    new Query()
                        .addCriteria( new Criteria( "name" ).is( document.getName() ) )
                        .getQueryObject(), 
                    dbDoc,  
                    true, 
                    false 
                );
     
                return null;
            }
        }
    );
}
Please notice usage of converter bean which helps to convert Java class to MongoDB's DBObject.

The last one I would like to show is findAndModify operation which does several things as one atomic sequence:
- find document matching criteria
- perform update
- return updated document (or old one, depending on what are your needs)
public void findAndModify( final Query query, final Update update ) {
    return template.execute( SimpleDocument.class,
        new CollectionCallback< SimpleDocument >() {
            @Override
            public SimpleDocument doInCollection( DBCollection collection ) 
                    throws MongoException, DataAccessException {
                return converter.read( SimpleDocument.class,       
                    collection.findAndModify( 
                        query.getQueryObject(), 
                        null,
                        null,
                        false,
                        update.getUpdateObject(),
                        true,
                        false
                    ) 
                );
            }
        }			
    );
}



For now, those are all interesting use cases I encountered. Honestly, I am very excited about MongoDB and strongly recommend it if it fits your application.

Wednesday, July 27, 2011

Exploiting MongoDB together with Spring Data project: basic concepts

All of us are observing the explosion of NoSql solutions these days. I get used to RDBMS but those are not a solution for all kind of challenges you might have. In my recent experience I got a chance to work with MongoDB - document database. In this post I intent to cover some basics (and some advanced features in next post) of using MongoDB together with Spring Data project. Before we start, small disclaimer: at the moment Spring Data is still in milestone phase so some classes / interfaces may change.

Before we start, please download and run MongoDB for your operating system. It's very simple so I won't spend time on this and let's start with simple POM file for our project:

    4.0.0

    mongodb
    com.example.spring
    0.0.1-SNAPSHOT
    jar

    
        UTF-8
        3.0.5.RELEASE
    

    
        
            org.springframework.data
            spring-data-mongodb
            1.0.0.M3
        

        
            log4j
            log4j
            1.2.16
        

        
            org.mongodb
            mongo-java-driver
            2.5.3
        

        
            org.springframework
            spring-core
            ${spring.version}
        

        
            org.springframework
            spring-context
            ${spring.version}
        
    

    
        
            springsource-milestone
            Spring Framework Milestone Repository
            http://maven.springframework.org/milestone
        
    

There are two key dependencies here:
- MongoDB java driver
- Spring Data for MongoDB

There are few ways to define MongoDB inside your Spring application context. Let me show a bit verbose but more flexible one:


    
    
  
    
 
    
        <constructor-arg index="0" ref="mongo" />
        <constructor-arg index="1" value="elements-db"/>
    

        
      

    
        <constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
        <constructor-arg name="mappingContext" ref="mappingContext" />         
     

    
        <constructor-arg name="mongoDbFactory" ref="mongoDbFactory"/>
        <constructor-arg name="mongoConverter" ref="converter" />        
        <property name="writeResultChecking" value="EXCEPTION" />
        <property name="writeConcern" value="NORMAL"/>
    
 
    
        
    

The role of each bean here:
  • mongo defines connection to MongoDB database (we rely on default settings, port 27027)
  • converter is used to convert Java classes to/from MongoDB's DBObject (== JSON)
  • mongoTemplate exposes operations we can do over MongoDB

So, we are ready to go!
Here are few code snippets to start with:
package com.example.mongodb;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.dao.DataAccessException;
import org.springframework.data.document.mongodb.CollectionCallback;
import org.springframework.data.document.mongodb.MongoOperations;
import org.springframework.data.document.mongodb.query.Index;
import org.springframework.data.document.mongodb.query.Index.Duplicates;
import org.springframework.data.document.mongodb.query.Order;
import org.springframework.stereotype.Service;

import com.mongodb.BasicDBObject;
import com.mongodb.DBCollection;
import com.mongodb.MongoException;

@Service
public class MongoService  {
    @Autowired private MongoOperations template;

    public void createCollection( final String name ) {
        template.createCollection( name  );
    }

    public void dropCollection( final String name ) {
        template.dropCollection( name  );
    }

    public void insert( final Object object, final String collection ) {
        template.insert( object, collection );
    }
   
    public void createIndex( final String name, final String collection ) {
        template.ensureIndex( 
            new Index()
                .on( name, Order.DESCENDING )
                .unique( Duplicates.DROP ), 
            collection  
        );
    }
 
    // Remove / save / ... operations here
}
That's it with basics. Next post will cover advanced features: using bulk inserts, update or insert operation and executing MongoDB commands. :)