Jackson JSON Deserializer using Groovy

This took me about half a day to do... Jackson's documentation in certain places is a scattered shot of different ways based on version # and usually ending in half complete sentences or small specific pages that aren't quite linked together in a relational sense. Jackson 2.0, I imagine, will further cludge the documentation due to refactors. But, by and large, it really is quite simple to write a deserializer or a serializer for that matter. It's just one class wrapped in a SimpleModule. And that's as easy as it should be.

The below deserializer is special because it allows you to pass in a map of aliases (name, class) and have the deserializer handle both untyped and typed objects. Which, apparently, is not supported by Jackson natively... not sure why. It ALSO supports nested untyped and typed objects allowing you to mix and match. If you look at the test cases down below, we pass in a root json object that is not typed and within that object is a typed object which becomes a fully fledged java object. Now, you could probably write the below examples in java (or maybe not since that's why it's not in Jackson by default), but groovys dynamic objects (add properties at runtime) allow this code to be straightforward. Performance hit, positively. But, sometimes you just have to tell the company to buy bigger servers. At least until java / groovy performance catches up to this type of scenario.


Deserializer:


Usage:




Mix-in:




References:

http://wiki.fasterxml.com/JacksonFeatureModules

http://wiki.fasterxml.com/JacksonHowToCustomDeserializers

http://wiki.fasterxml.com/JacksonInFiveMinutes

Using Amplify.js to Mock Service Calls

your-view-model.js


services.js


mock-services.js



What's happeing? 

We are simply overriding out service.js with mock-services.js. So... simply include the mock-service.js script AFTER you include the service.js script... And once you don't want to mock and more, simply comment out...

Resources

• http://api.jquery.com/jQuery.ajax/
• http://speakerdeck.com/u/dougneiner/p/streamline-your-ajax-requests-with-amplifyjs-and-jquery
• http://amplifyjs.com/api/request/#url_substitution/routing

Jackson Auto-Typing Objects

I'm using the Jackson JSON framework to do some (de)+serialization for my web app, but as we all know... JSON isn't typed. Here is a quick auto-typer that I created that runs when the app starts up (in tomcat). Look through the resources / references down below for more info.

First, create a basic mixin class.

package wookets.json;

import org.codehaus.jackson.annotate.JsonTypeInfo;
import org.codehaus.jackson.annotate.JsonTypeInfo.As;
import org.codehaus.jackson.annotate.JsonTypeInfo.Id;

@JsonTypeInfo(use=Id.NAME, include=As.PROPERTY, property="_type")
public class JacksonTyping {

}

Next, create a startup (bootstrap) in your app that runs this code when the app starts in the server. I'm assuming you're using the Spring Framework here.

def scanner = new ClassPathScanner()
    scanner.addIncludeFilter(new AnnotationTypeFilter(DomainModel.class))
    scanner.addIncludeFilter(new AnnotationTypeFilter(DataTransferObject.class))
    for(String path : paths) {
      for(Class c: scanner.getComponentClasses(path)) {
        log.debug "Adding ${c.simpleName} to json mapper"
        objectJsonMarshaller.getSerializationConfig().addMixInAnnotations(c, JacksonTyping.class) // outgoing
        objectJsonMarshaller.getDeserializationConfig().addMixInAnnotations(c, JacksonTyping.class) // incoming
        //objectJsonMarshaller.registerSubtypes(new NamedType(c, c.simpleName)) // old way
      }
    }

The ClassPathScanner is a nothing more than an extension of the spring class ClassPathScanningCandidateComponentProvider. It in essence will look through your java packages and return each class, which you can filter and inspect and do what have you...


Resources:

http://jackson.codehaus.org/

http://programmerbruce.blogspot.com/2011/05/deserialize-json-with-jackson-into.html
* this is the best reference...

http://stackoverflow.com/questions/5826928/how-can-i-prevent-jackson-from-serializing-a-polymorphic-types-annotation-prope

The Three Levels of Service

Back when I first started programming, everything was about 'The Three Tiers of server side applications.' Today, I find that not much has changed. Yes, the UI and top layers have been pushed to the client. If they haven't in your apps, they should. And the bottom layer, the data layer as been almost stripped away via the ORM and Active Record patterns. Which is good, because we are getting to the guts, the middle layer, the business layer.

A refresher (old school);

TOP - UI / Controller / View
This includes anything on the screen the user sees or clicks. And handling that click.

MIDDLE - Business / Manager / Service
This includes domain specific routing and logic and domain model manipulation.

BOTTOM - DAO / Data Access / Database
This includes transforming our middle layer data and logic paths into storage items.

New School 3-Tier; 

TOP - Web Service Gateway
This includes transforming JSON / XML / URL Encoded parameters into server language specified objects. I suppose you could throw in some SOAP, but let's try to REST instead.

MIDDLE - Business Service and Domain Logic
This includes taking objects and doing something with them, perhaps calling a remote service.

BOTTOM - Domain Validation and Data Storage
This includes securing data consistency for objects being stored and also writing or cascading domain model relationship contracts.

Example;
A web request comes in, say a login request.

url: /login

Our gateway will accept two possible entry points. First, url encoded params for the username and password. Second, a json submitted object which has two properties; username and password. The gateway is written to be smart enough so that either one works. After the gateway as accepted the call (remember to do validation on incoming params before doing anything else), we are tossed to the Service layer. Don't try to do too much in a gateway, save it for the service layer. The gateway is only for making sure we can accept a response and for quickly letting the user know if the params need to be changed.

For our purposes, every service layer method is wrapped with a security check. This security checks to see if the method we are invoking is allowed to run given the user asking for the request. In our gateway, we parsed a 'sessionToken' that we store in a request variable on the server. A request variable lives and dies on each request, so we can be sure we aren't mudding up a user's authority. If the service method is not marked as @Public, we verify that the user exists. We then verify that the user is also allowed to access this method, perhaps based on Role.

When the request gets passed security, we can now execute some code and do something with the request. In our case, the user wants to be authenticated. The service method will now send the username down to the dao layer. If a user is found that matches the requested username, we go on and validate password, then we validate that the user is still active, etc. If everything passes, we can create a 'session' for the user (which contains a 'sessionToken').

But, I've gotten ahead of myself. How does the dao layer work? Well, before we even get to the dao layer, we wrap it in an aspect, same as the security service wrapper. The dao wrapper could potentially do more validation, but more importantly will run more security. If we were creating an object (saving it from the UI) in our application, we wrapper the Crud.save method and ensure that, for instance, the object being saved is only being saved where the user is allowed to save said resource. This is done by comparing the UserSession (that we verified after the gateway and before the service layer) with the resource being saved and the 'context' of how the application is configured. If we want multi-tenancy, this works out great, because our service and our dao's know nothing about the tenant-state of the object, so we can at once remove multi-tenancy if needed.

The actual DAO layer (or active record if that is your thing) must only map objects to data storage. There should be no security (app / user wise) in your DAO. Make your dao simple and dumb and wrap it if you need more.

I'm sure none of the above makes much sense... Apologies...

A quick review;

Request -> Gateway -> Security Wrapper -> Service -> Dao Wrapper -> Dao -> Data storage

Response <- Gateway <- Service <- Post-Dao Wrapper <- Dao <- Data storage


You can also add a wrapper on the gateway for responses. If your app generates an exception, you'll need to write something nice back to the user. Otherwise, you'll pay for it with emails that say, "Internal Server Error"... And in reality, you should (upon exception) be sending your dev team a nice email with a stack trace or whatever you need when one occurs. Nothing better than pre-empting your users problems with "Oh, we saw that you had an issue and have already fixed it."

Amazon DynamoDB Groovy DAO Update

Here is a an update version of the DAO from the previous post. I added update methods (very useful)...

Next I will be working on some relationship type logic. Meaning, having a table that holds relationships and specifying you want those relationships (a list of ids) that will query that, then query the actual keys table and return the objects you want...

Sorry about not posting on Github. Maybe one day I'll get it up there.

package wookets.dynamo

import groovy.util.logging.Log4j

import org.springframework.stereotype.Repository

import wookets.util.DateUtil
import wookets.util.ResourceTypeUtil

import com.amazonaws.auth.AWSCredentials
import com.amazonaws.auth.PropertiesCredentials
import com.amazonaws.services.dynamodb.AmazonDynamoDBClient
import com.amazonaws.services.dynamodb.model.AttributeAction
import com.amazonaws.services.dynamodb.model.AttributeValue
import com.amazonaws.services.dynamodb.model.AttributeValueUpdate
import com.amazonaws.services.dynamodb.model.DeleteItemRequest
import com.amazonaws.services.dynamodb.model.DeleteItemResult
import com.amazonaws.services.dynamodb.model.GetItemRequest
import com.amazonaws.services.dynamodb.model.GetItemResult
import com.amazonaws.services.dynamodb.model.Key
import com.amazonaws.services.dynamodb.model.PutItemRequest
import com.amazonaws.services.dynamodb.model.PutItemResult
import com.amazonaws.services.dynamodb.model.ReturnValue
import com.amazonaws.services.dynamodb.model.UpdateItemRequest
import com.amazonaws.services.dynamodb.model.UpdateItemResult

@Log4j
@Repository
class DynamoDao {
  
  AmazonDynamoDBClient client
  
  DynamoDao() {
    InputStream credentialsAsStream = Thread.currentThread().getContextClassLoader().getResourceAsStream("AwsCredentials.properties");
    AWSCredentials credentials = new PropertiesCredentials(credentialsAsStream);
    client = new AmazonDynamoDBClient(credentials);
  }
  
  // read
  def load(Object resourceType, String resourceId) {
    log.debug "Loading ${resourceType}::${resourceId}"
    
    String tableName = ResourceTypeUtil.resolveType(resourceType)
    GetItemRequest request = new GetItemRequest(tableName, new Key(toDynamo(resourceId)))
    GetItemResult result = client.getItem(request)
    
    def obj = resourceType instanceof Class ? resourceType.newInstance() : [_type: tableName] // return typed or dynamic
    result.item.each { String prop, AttributeValue val ->
      obj[prop] = fromDynamo(val)
    }
    return obj
  }
  
  // save
  def save(Object resource) {
    log.debug "Saving ${resource}"
    
    def item = [:] // item to save
    if(resource.class == null) { // dynamic object
      resource.each { String prop, Object val ->
        if(prop == "_type") return; // strip out '_type'
        if(val == null) return; // dont waste space with null values
        item[prop] = toDynamo(val)
      }
    } else { // typed object
      resource.properties.each { String prop, Object val ->
        if(prop == "class") return; // strip out java added properties
        if(prop == "metaClass") return; // strip out groovy added properties
        if(val == null) return; // we don't support storing nulls
        item[prop] = toDynamo(val)
      }
    }
    
    String tableName = ResourceTypeUtil.resolveType(resource) // table name
    PutItemRequest request = new PutItemRequest(tableName, item)
    PutItemResult result = client.putItem(request)
  }
  
  // delete
  def delete(Object resourceType, String resourceId) {
    log.debug "Deleting ${resourceType}::${resourceId}"
    
    String tableName = ResourceTypeUtil.resolveType(resourceType)
    DeleteItemRequest request = new DeleteItemRequest(tableName, new Key(toDynamo(resourceId)))
    DeleteItemResult result = client.deleteItem(request)
  }
  
  // update
  def addToProperty(Object resourceType, String resourceId, String propertyName, Object propertyValue) {
    log.debug "Adding ${resourceType} - ${resourceId} - ${propertyName} - ${propertyValue}"
    
    String tableName = ResourceTypeUtil.resolveType(resourceType)
    Key key = new Key(toDynamo(resourceId))
    def updateItems = new HashMap()
    
    updateItems.put(propertyName, new AttributeValueUpdate(toDynamo(propertyValue), AttributeAction.ADD))
    UpdateItemRequest request = new UpdateItemRequest(tableName, key, updateItems);          
    UpdateItemResult result = client.updateItem(request);
  }
  
  def replaceProperty(Object resourceType, String resourceId, String propertyName, Object propertyValue) {
    log.debug "Replacing ${resourceType} - ${resourceId} - ${propertyName} - ${propertyValue}"
    
    String tableName = ResourceTypeUtil.resolveType(resourceType)
    Key key = new Key(toDynamo(resourceId))
    def updateItems = new HashMap()
    
    updateItems.put(propertyName, new AttributeValueUpdate(toDynamo(propertyValue), AttributeAction.PUT))
    UpdateItemRequest request = new UpdateItemRequest(tableName, key, updateItems);
    UpdateItemResult result = client.updateItem(request);
  }
  
  def removeFromProperty(Object resourceType, String resourceId, String propertyName, Object propertyValue) {
    log.debug "Removing ${resourceType} - ${resourceId} - ${propertyName} - ${propertyValue}"
    
    String tableName = ResourceTypeUtil.resolveType(resourceType)
    Key key = new Key(toDynamo(resourceId))
    def updateItems = new HashMap()
    
    updateItems.put(propertyName, new AttributeValueUpdate(toDynamo(propertyValue), AttributeAction.DELETE))
    UpdateItemRequest request = new UpdateItemRequest(tableName, key, updateItems)
    UpdateItemResult result = client.updateItem(request)
  }
  
  def removeProperty(Object resourceType, String resourceId, String propertyName) {
    log.debug "Removing ${resourceType} - ${resourceId} - ${propertyName}"
    
    String tableName = ResourceTypeUtil.resolveType(resourceType)
    String hashKey = resourceId
    
    def updateItems = new HashMap()
    Key key = new Key().withHashKeyElement(new AttributeValue().withS(hashKey))
    
    updateItems.put(propertyName, new AttributeValueUpdate().withAction(AttributeAction.DELETE));
    def updateItemRequest = new UpdateItemRequest().withTableName(tableName).withKey(key).withAttributeUpdates(updateItems);
    UpdateItemResult result = client.updateItem(updateItemRequest);
  }
  
  // private helper methods
  
  private AttributeValue toDynamo(Object value) {
    if(value instanceof Number) { // number support
      return new AttributeValue().withN(value)
    } else if(value instanceof String) { // string support
      return new AttributeValue().withS(value)
    } else if(value instanceof List) { // list support
      if(value[0] instanceof Number) { // number list support
        return new AttributeValue().withNS(value)
      } else if(value[0] instanceof String) { // string list support
        return new AttributeValue().withSS(value)
      }
    } else if(value instanceof Boolean) { // boolean support
      return new AttributeValue().withS(value ? "bool:true" : "bool:false")
    } else if(value instanceof Date) { // date support
      return new AttributeValue().withS("date:" + DateUtil.dateFormatter.format(value))
    } else {
      throw new RuntimeException("Unsupported data type for value ${value}")
    }
  }
  private Object fromDynamo(AttributeValue value) {
    if(value.getS() != null) {
      String val = value.getS()
      if(val == "bool:true") { // boolean support
        return true
      } else if(val == "bool:false") {
        return false
      } else if(val?.startsWith("date:")) { // date support
        return DateUtil.dateFormatter.parse(val.substring(5))
      } else { // string support
        return val
      }
    } else if(value.getN() != null) { // number support
      return value.getN()
    } else if(value.getSS() != null) { // set of strings support
      return value.getSS()
    } else if(value.getNS() != null) { // set of numbers support
      return value.getNS()
    }
  }
}

Groovy Amazon DynamoDB Simple Example

The following is a two class example of Amazon's DynamoDB. The first class is just a Dao (yes, I still prefer Daos over ARs).

package wookets.dynamo

import com.amazonaws.auth.AWSCredentials
import com.amazonaws.auth.PropertiesCredentials
import com.amazonaws.services.dynamodb.AmazonDynamoDBClient
import com.amazonaws.services.dynamodb.model.AttributeValue
import com.amazonaws.services.dynamodb.model.ComparisonOperator
import com.amazonaws.services.dynamodb.model.DeleteItemRequest
import com.amazonaws.services.dynamodb.model.DeleteItemResult
import com.amazonaws.services.dynamodb.model.GetItemRequest
import com.amazonaws.services.dynamodb.model.GetItemResult
import com.amazonaws.services.dynamodb.model.Key
import com.amazonaws.services.dynamodb.model.PutItemRequest
import com.amazonaws.services.dynamodb.model.PutItemResult

class DynamoDao {
  
  AmazonDynamoDBClient client
  
  DynamoDao() {
    InputStream credentialsAsStream = Thread.currentThread().getContextClassLoader().getResourceAsStream("AwsCredentials.properties");
    AWSCredentials credentials = new PropertiesCredentials(credentialsAsStream);
    client = new AmazonDynamoDBClient(credentials);
  }
  
  // save
  def save(Object resource) {
    String tableName = resource._type
    if(resource._type == null) {
      throw new RuntimeException("You need to specify a _type to be able to save data under")
    }
    
    def item = [:]
    resource.each { key, value ->
      if(key == "_type") return;
      if(value instanceof Number) {
        item.put(key, new AttributeValue().withN(value))
      } else if(value instanceof String) {
        item.put(key, new AttributeValue().withS(value))
      } else if(value instanceof List) {
        if(value[0] instanceof Number) {
          item.put(key, new AttributeValue().withNS(value))
        } else if(value[0] instanceof String) {
          item.put(key, new AttributeValue().withSS(value))
        }
      } else {
        throw new RuntimeException("Unsupported data type for property ${resource._type}.${key}")
      }
    }
    
    PutItemRequest putItemRequest = new PutItemRequest().withTableName(tableName).withItem(item)
    PutItemResult result = client.putItem(putItemRequest)
  }
  
  // read
  def load(String resourceType, String resourceId) {
    Key lookupKey = new Key().withHashKeyElement(new AttributeValue().withS(resourceId))
    GetItemRequest getItemRequest = new GetItemRequest().withTableName(resourceType).withKey(lookupKey)
    
    def item = [_type: resourceType, id: resourceId]
    GetItemResult result = client.getItem(getItemRequest);
    result.item.each { propName, value ->
      if(value.getS() != null) {
        item[propName] = value.getS()
      } else if(value.getN() != null) {
        item[propName] = value.getN()
      } else if(value.getSS() != null) {
        item[propName] = value.getSS()
      } else if(value.getNS() != null) {
        item[propName] = value.getNS()
      }
    }
    return item
  }
  
  // delete
  def delete(String resourceType, String resourceId) {
    Key key = new Key().withHashKeyElement(new AttributeValue().withS(resourceId));
    DeleteItemRequest deleteItemRequest = new DeleteItemRequest().withTableName(resourceType).withKey(key)
    DeleteItemResult result = client.deleteItem(deleteItemRequest);
  }
  
}

The next class is the test harness.

package test.dynamo;

import static org.junit.Assert.*

import java.text.SimpleDateFormat

import org.junit.Test

import wookets.dynamo.DynamoDao

class TestDynamoDao {
  
  DynamoDao dynamoDao = new DynamoDao()
  
  static SimpleDateFormat dateFormatter = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'");
  
  @Test
  void testSave() {
    def mock = [_type:"dynamomock", id: "MOCK1", name: "Namer", lister: ["Some1", "Like", "U"]]
    dynamoDao.save(mock)
  }
  
  @Test
  void testLoad() {
    def mock = dynamoDao.load("dynamomock", "MOCK1")
    mock.each { prop, value ->
      print "${prop}: ${value}, "
    }
    println ""
  }
  
  @Test 
  void testDelete() {
    dynamoDao.delete("dynamomock", "MOCK1")
  }
  
  @Test
  void testSaveAll() {
    dynamoDao.save([_type:"dynamomock", id: "MOCK1", name: "Namer"])
    dynamoDao.save([_type:"dynamomock", id: "MOCK2", name: "Namer2"])
    dynamoDao.save([_type:"dynamomock", id: "MOCK3", name: "Namer3"])
    dynamoDao.save([_type:"dynamomock", id: "MOCK4", name: "Namer4"])
    dynamoDao.save([_type:"dynamomock", id: "MOCK5", name: "Namer5"])
  }
  @Test
  void testLoadAll() {
    ["MOCK1", "MOCK2", "MOCK3", "MOCK4", "MOCK5"].each {
      def mock = dynamoDao.load("dynamomock", it)
      mock.each { prop, value ->
        print "${prop}: ${value}, "
      }
      println ""
    }
  }
}

Obviously this is a simple use case, but might help you in your travels...

Btw, the properties (credentials file) looks like...

secretKey=

accessKey=

Amazon S3 Website Hosting DNS Gotchya

Recently, while setting up a website on Amazon's S3 (see here), I was also attempting to set the mx records of the domain to point to google apps (see here). But, the mx records were not updating.

I thought this was a hover.com problem. Btw, I've never had a company answer my phone call faster than these guys. It is actually quite freaky and I was unprepared with my question. Literally took less than a second to reach and start talking to a human being. Amazing.

Anyway, I'm on the phone with hover and they tell me that the cname that is pointing to amazon is returning a faux mx record / screwing something up with the name server. I was a bit stunned about how this could be. Why would a cname screw up mx records. But it was true and my hover helper was right.

Apparently, if you have a cname record @ that points to Amazon's S3, any custom mx records never reach the DNS.

@ - domain - s3 website

can't be used with

@ - domain - mx records

Well, that solves that problem I guess. And I suppose this problem may not be isolated to amazon and is probably some unknown dns protocol that I attempted to violate. Wish it would warn me though.