Objective

This article will show you how to implement a full text search in Google App Engine using JDO. I tried my hand at this couple month ago, but after watching this presentation I decided to do it properly.

The Problem

In my first attempt I managed to get the search working, but after watching Brett Slatkin’s presentaion I realized where the problem is. In short deserializing a list of strings (which is our search index) is a very costly operation, but he presented with a solution. Bellow you will find my solution to this problem.

Data Model

For this example we will use such data model: we have Customer (name, contact, notes) which has a list of Addresses and Phones. We need ability to find customer by name, address or phone.

@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
public class Customer {
    @PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    private Long id;

    @Persistent
    private String name;
    @Persistent
    private String contactName;
    @Persistent
    private String comments;

    @Persistent(mappedBy = "customer")
    @Element(dependent = "true")
    private List<Address> addresses = new ArrayList<Address>();

    @Persistent(mappedBy = "customer")
    @Element(dependent = "true")
    private List<Phone> phones = new ArrayList<Phone>();

    @Persistent(dependent="true")
    private CustomerIndex index;

   // getters and setter go here....
}

@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
public class Address {
    @PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    private Key id;
    @Persistent
    private String type;
    @Persistent
    private String line1;
    @Persistent
    private String line2;
    @Persistent
    private String city;
    @Persistent
    private String state;
    @Persistent
    private String zip;
    @Persistent
    private Customer customer;

   // getters and setter go here....
}

@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
public class Phone {
   @PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    private Key id;
    @Persistent
    private String type;
    @Persistent
    private String phone;
    @Persistent
    private Customer customer;

   // getters and setter go here....
}

If you paid attention you notice that we have an interesting child in the Customer class called CustomerIndex. Here it is:

@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
public class CustomerIndex {
    @PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    private Key id;
    @Persistent
    private Set<String> index;

   // getters and setter go here....
}

Search Approach

Here is the theory of what we gonna do: Since deserializing of List properties is a very very costly operation (and we do not care abut the data it holds anyway) we move customer search index property into a Child object. We will perform a search on this Child and we will get only the keys of the child objects. This way we do not have to incur the penalty of deserializing our search index (the search happens on the index). Once we have our child object keys we will load Parent objects with those keys. We can do this because a child key is a composite key and always includes parent key.
To make our search more usable we will use Lucenen and SnowballAnalyzer for word stemming.
Here is the method that gives us the Set of words. We use it to generate the index of searchable words as well as search phrases.

protected Set<String> getIndex( String input, int maxTokens ) {
  Set<String> returnSet = new HashSet<String>();
  try {
    Analyzer analyzer =  new SnowballAnalyzer( org.apache.lucene.util.Version.LUCENE_30,"English", stopWords());
    TokenStream tokenStream = analyzer.tokenStream( "content", new StringReader(input) );
    while ( tokenStream.incrementToken() && (returnSet.size() < maxTokens) ) {
      if( tokenStream.hasAttribute( TermAttribute.class ) ) {
        TermAttribute attr = tokenStream.getAttribute( TermAttribute.class );
        logger.debug( attr.term() );
        returnSet.add( attr.term() );
      }
    }
  }catch( Exception exc ) {
    logger.equals(exc);
  }
  return returnSet;
}

Here is our search method:

public List<Customer> searchCustomers( String search1, Long entityId ) throws IOException {
  PersistenceManager pm = PMF.getManager();

  Set<String> search = getIndex(search1, 3);
  Query query = pm.newQuery("SELECT id FROM " + CustomerIndex.class.getName() );
  query.setFilter("index == param0");
  query.declareParameters("String param0");

  Query query2 = pm.newQuery(Customer.class);
  query2.setFilter("id == keyParam");
  query2.declareParameters("com.google.appengine.api.datastore.Key keyParam");

  List<Customer> custs = null;
  List<Key> keys;
  List<Key> parents = new ArrayList<Key>();

  try {
    keys = (List<Key>) query.execute( search );
    for( Key k : keys){
      parents.add( k.getParent() );
  }
  custs = (List<Customer>) query2.execute( parents );
  for( Customer cust : custs ) {
    for( Address addr : cust.getAddresses() )
      logger.debug( addr.getId() );
    for( Phone ph : cust.getPhones() )
      logger.debug( ph.getId() );
    }
  } catch ( Exception exc ) {
    logger.error(exc);
  } finally {
    query.closeAll();
    query2.closeAll();
    pm.close();
  }
  return custs;
}

You will notice that we walk the address and Phone lists for each customer to load them form Storage. We do that so we can ship them over the wire. UI in this case is a Flex client, so we do JSON serialization of the results.

Conclusion

Text searching can be implemented in GAE and to boot it can be implemented efficiently. Just remember before you store this Customer record you need to build out the CustomerIndex object with the set of words that we will search on. I just concat all the properties to one string and Lucene build the set for me by calling my getIndex().

Objective

This post will show you how to use Spring JDBC Template to perform real world CRUD operations. I hope I can show you that you do not need overweight Hibernate to interact with database of your choice.

The Problem Domain

Almost all application I have coded needed to store and get data of some sort. In most cases it was relational database. With hibernate being all the rage these days I have used it in multiple projects with varying degree of problems and success. In my personal opinion hibernate tries to solve a problem that really is not there. If you can not wire SQL you should not be using Hibernate or be doing any software development.

I advise the use of Spring JDBC templates. In essence Spring JDBC Template will offload all the mundane steps of database access from you shoulders only requiring you to write SQL and move data from result set into your model objects. Here is how Spring documentation describes responsibilities:

CRUD Example with Spring JDBC Template

For this example you will need spring-core, spring-tx and spring-aop components. We will use spring-tx to manage the transaction so we can get back the auto generated IDs of the persisted objects and aop to set up the transaction management. Here is meat for spring configuration file:

<bean id="propertyPlaceholder" 
      class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer" >
  <property name="location" value="/WEB-INF/config.properties" />
</bean>

<bean id="mysqlDataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" 
      destroy-method="close">
  <property name="driverClass" value="com.mysql.jdbc.Driver"/>
  <property name="jdbcUrl" value="${database.url}"/>
  <property name="user" value="${database.user}"/>
  <property name="password" value="${database.password}"/>
  <property name="initialPoolSize" value="${database.connections.start}"/>
  <property name="maxPoolSize" value="${database.connections.max}"/>  
  <property name="idleConnectionTestPeriod" value="270" />
</bean>

<bean id="txManager" 
      class="org.springframework.jdbc.datasource.DataSourceTransactionManager" >
  <property name="dataSource" ref="mysqlDataSource" />
</bean>

<tx:advice id="txAdvice" transaction-manager="txManager">
  <tx:attributes>
    <!-- all methods starting with 'get' are read-only -->
    <tx:method name="get*" read-only="true"/>
    <tx:method name="list*" read-only="true"/>
      <!-- other methods use the default transaction settings (see below) -->
      <tx:method name="*"/>
    </tx:attributes>
</tx:advice>

<aop:config>
  <aop:pointcut id="uOwnerDataSvcOperation" 
      expression="execution(* com.es.rto.dao.UOwnerDataServiceImpl.*(..))"/>
  <aop:advisor advice-ref="txAdvice" pointcut-ref="uOwnerDataSvcOperation"/>
</aop:config>

<bean id="namedTemplate" 
    class="org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate">
  <constructor-arg index="0" ref="mysqlDataSource"/>   
</bean>

<bean id="qandADao" class="com.es.rto.dao.QandADao" >
  <property name="jdbcTemplate" ref="namedTemplate" />
</bean>

<bean id="uOwnerDataSvc" class="com.es.rto.dao.UOwnerDataServiceImpl" >
  <property name="qandADao" ref="qandADao" />
</bean>

First thing we do we load our config.properties file where we have database configuration defined. Second we define a pooled data source to connect to mysql database. Third bean is a transaction manager that is watching our data source from the step two. Next we set up our transaction manager. We instruct transaction manager that methods starting with “get” and “list” are read only. This means that they will not be subject to transaction management. Then we define named jdbc template and pass in the data source as first constructor argument and then pass in the template into our own DAO bean.

Why do we need Transaction Management?

When we create a new row in a mysql table with autoincrement id value we need a way to get that auto generated id. To do so we need to query the database using same connection. When we use jdbcTemplate.update() or jdbcTemplate.query() it will check out a connection form the pool execute the query, process the results and return this connection to the pool. So if we do update() and then get() to query “SELECT LAST_INSERT_ID()” we will get another connection, which is a problem since we need to do this on the same connection to get the right id. This is where the transaction comes in. Instead of releasing connection back to the pool spring will hold on to it until we are done with it.

Java code side of the Spring JDBC template

We will use 3 beans to illustrate the scenario: question, answer and user.

public class Question {
        private long questionId;
        private long categoryId;
        private long userId;
        private long areaId;
        private String question;
        private String verifyKey;
        private Date created;
        private User user;
        private List<Answer> answers;
        private long answerCount;
        private String name;
        // getters and setters omited...
}

public class Answer {
        private long answerId;
        private long questionId;
        private long userId;
        private String answer;
        private String verifyKey;
        private String url;
        private Date created;
        private User user;
        private String name;
        // getters and setters omited...
}

public class User implements Serializable {
        private long id;
        private String email;
        private String retypeE;
        private String name;
        private String password;
        private String password2;
        private String verifyKey;
        private boolean buyer;
        private boolean seller;
        private boolean active;
        private boolean privacy;
        private int pageSize = 20;
        private String next;
        private boolean administrator;
        // getters and setters omited...
}

Here are the 2 classes that do all the work:

public class QandADao {

  private final Logger logger = Logger.getLogger(QandADao.class);
  protected NamedParameterJdbcTemplate jdbcTemplate = null;
  

  @SuppressWarnings("unchecked")
  public List<Question> getQuestions( String search, int page, int step ) {
    try {
      MapSqlParameterSource params = new MapSqlParameterSource( "search", "%"+search+"%" );
      String sql = "SELECT q.*, count( a.answer_id ) as answers, u.name, u.seller  FROM users u, questions q LEFT JOIN answers a "+
      "ON  a.question_id = q.question_id WHERE a.answer LIKE :search AND q.user_id = u.id "+
      "GROUP BY q.question_id, q.category_id, q.user_id, q.question, q.created, q.area_id, u.name, u.seller "+  
      "UNION "+
      "SELECT q.*, count( a.answer_id ) as answers, u.name, u.seller  FROM users u, questions q LEFT JOIN answers a "+ 
      "ON  a.question_id = q.question_id WHERE q.question LIKE :search AND q.user_id = u.id "+
      "GROUP BY q.question_id, q.category_id, q.user_id, q.question, q.created, q.area_id, u.name, u.seller "+
      "LIMIT "+(step*(page-1))+", "+step;
      logger.debug( sql );
      return jdbcTemplate.query(sql, params, new RowMappers.QuestionsMapper());
    } catch ( DataAccessException exc ) {
      logger.error("FAILED to get Question List", exc);
      return new ArrayList<Question>();
    }
  }

  public Question saveQuestion( Question question ) {
    try {
      BeanPropertySqlParameterSource namedParameters = new BeanPropertySqlParameterSource(question);
      String sql;
      if( question.getQuestionId() == 0 )
        sql = "INSERT INTO questions ( category_id, user_id, question, area_id, uname ) VALUES ( :categoryId, :userId, :question, :areaId, :name )";
      else
        sql = "UPDATE questions SET question=:question, category_id=:categoryId, area_id=:areaId WHERE question_id=:questionId";
      logger.debug( sql );
      jdbcTemplate.update(sql, namedParameters);
      if( question.getQuestionId() == 0 )
        question.setQuestionId( this.getInsertedID() );
        
    } catch ( DataAccessException exc ) {
      logger.error("SAVE answer FAILED!", exc);
      throw exc;
    }
    return question;
  }

  protected long getInsertedID() {
    return jdbcTemplate.queryForLong( "SELECT LAST_INSERT_ID()" , new MapSqlParameterSource());
  }
  
}

public class RowMappers {

  public static final class QuestionMapper implements RowMapper {
      public Object mapRow(ResultSet rs, int rowNum) throws SQLException {
        Question q = new Question();
        q.setQuestionId(rs.getLong( "question_id" ));
        q.setCategoryId(rs.getLong( "category_id" ));
        q.setUserId(rs.getLong( "user_id" ));
        q.setAreaId( rs.getLong("area_id"));
        q.setQuestion( rs.getString("question"));
        q.setCreated( rs.getTimestamp( "created" ) );
        q.setName( rs.getString("uname") );
        User u = new User();
        u.setName( rs.getString("name") );
        u.setSeller( rs.getBoolean("seller") );
        u.setId(q.getUserId());
        q.setUser(u);
      return q;
    }
  } 
}

So what is going on here? Lets look at the QuestionMapper. It implements one method – mapRow. Its hob is to process a single row of the result set. As you see here we create a Question bean and a User bean which is included in the Questions bean. It can not be more strait forward then this.

QandADao is the class where all the magic happens. First lets examine the getQuestions() method. First we define query parameter(s) using MapSqlParameterSource class. We simply put the values in the map and jdbc template will pull them out by the key. Next we write our sql query. In this case iti is pretty complex to pull out any questions where either question or the answer have the thing we are searching for. We use union to be able to pull out the questions without the answers. Lastly we call query method on the jdbc template and we pass in our query, parameters and the row mapper instance. Spring takes care of all the plumbing. What we get out of this call is a list of Question objects.

saveQuestion() method is not much different. In this case we use BeanPropertySqlParameterSource which maps bean properties to sql parameters. If you define a sql parameter “name” your bean should have “getName()” method. Instead of calling query() we call update() on the jdbc template which only takes sql and the parameters.

Conclusion

You may have noticed that the effort to use the Spring approach to JDCB is very simple, flexible and powerful. You have all the control in all the right places! You write the sql with all the performance hints and tweaks, you write code how to transfer results to the domain objects the way you need it or like it. I only mentioned about one tenth of the options you have with spring. Read The Fine Manual (RTFM) to explore the full potential.

Objective

This article will show you how to implement a simple search in Google App Engine using JDO engine including searching in child objects.

The Problem Domain

In my application I need to search for data and you probably need to do the same in yours. In Google App Engine you can not query for properties of the child objects. In SQL world this means you can not use “where” clause. So how we can not construct a query which looks at child objects. So how can we implement search in such restrictive environment?

The Solution

I came up with a very simple solution – I created an “index” property where I store all the data that I need to search on. Let’s say we have a Customer bean that has multiple addresses and phones. We want to be able to search on address, phone and customer name. Here are our beans:

@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
public class Customer implements Serializable {
    private static final long serialVersionUID = -3665292067832675548L;

    @PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    private Long id;
    @Persistent
    private String name;
    @Persistent
    private String contactName;
    @Persistent
    private String comments;

    @Persistent(mappedBy = "customer")
    @Element(dependent = "true")
    private List<Address> addresses = new ArrayList<Address>();

    @Persistent(mappedBy = "customer")
    @Element(dependent = "true")
    private List<Phone> phones = new ArrayList<Phone>();

    @Persistent
    private List<String> index;

    // Getters and Setter go here...
}

@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
public class Address {
    @PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    private Key id;
    @Persistent
    private String type;
    @Persistent
    private String name;
    @Persistent
    private String line1;
    @Persistent
    private String line2;
    @Persistent
    private String city;
    @Persistent
    private String state;
    @Persistent
    private String zip;
    @Persistent
    private Customer customer;

    // Getters and Setter go here...
}

@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
public class Phone {
    @PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    private Key id;
    @Persistent
    private String type;
    @Persistent
    private Number phone;
    @Persistent
    private Customer customer;

    // Getters and Setter go here...
}

In the Customer bean we defined an “index” property which is a String array. We will put all our searchable words int this index, so later we can construct a query. Before we save our customer we want to populate index property like this:

public void rebuildIndex() {
        index = new ArrayList<String>();
                
        Iterator<Address> it = this.addresses.iterator();
        while( it.hasNext() ) {
                Address ad = it.next();
                if( ad.getLine1() != null ){
                        String[] idx = ad.getLine1().split(" ");
                        for( int i = 0 ; i < idx.length ; i++ )
                                index.add( idx[i] );
                }
                if( ad.getLine2() != null ){
                        String[] idx = ad.getLine2().split(" ");
                        for( int i = 0 ; i < idx.length ; i++ )
                                index.add( idx[i] );
                }
        }

        Iterator<Phone> it2 = this.phones.iterator();
        while( it2.hasNext() ) {
                Phone ph = it2.next();
                if( ph.getPhone() != null )
                        index.add( ph.getPhone().toString() );
        }

        String[] idx = name.split(" ");
        for( int i = 0 ; i < idx.length ; i++ )
                index.add( idx[i] );
                
        idx = contactName.split(" ");
        for( int i = 0 ; i < idx.length ; i++ )
                index.add( idx[i] );
                
}

Now we have all our searchable words from child beans and parent bean in one property we can search on. Here is how my search method looks like:

@SuppressWarnings("unchecked")
@ModelAttribute("custs")
@RequestMapping(value = "/{search}", method = RequestMethod.GET)
public List<Customer> searchCustomers( @PathVariable String search, HttpSession session)
                                          throws IOException {
    PersistenceManager pm = pmf.getManager();
    Query query = pm.newQuery(Customer.class);
    query.setFilter("index == searchParam");
    query.declareParameters("String searchParam");
    List<Customer> custs;
    try {
        custs = (List<Customer>) query.execute( search );
    } finally {
        query.closeAll();
    }
    return custs;
}

Conclusion

We found an easy way around the big table JDO implementation limitation in Google App Engine. It is a simple and strait forward workaround, but it has couple drawbacks: we can only search on full words and we duplicate data using more storage space. In my book storage is cheap so that is not a big concern for me.

As I progress more with my projects on Google App Engine, I will surely revisit the search topic and post about other solutions to this challenge.

If you have a better solution please post it in the comments, I would love to give it a try!