IT Tips and Memory Joggers!

Friday, 29 April 2016

AEM Sling Servlet

To create an AEM Sling Servlet you need to make sure that you have either used the mvn archetype to have the correct pom.xml file or include a number of dependencies.

<dependency>
<groupId>org.apache.felix</groupId>
<artifactId>org.apache.felix.scr</artifactId>
<version>1.6.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.felix</groupId>
<artifactId>org.apache.felix.scr.annotations</artifactId>
<version>1.9.6</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax.jcr</groupId>
<artifactId>jcr</artifactId>
<version>2.0</version>
<scope>provided</scope>
</dependency>
   <dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
<version>2.4</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.sling</groupId>
<artifactId>org.apache.sling.models.api</artifactId>
<version>1.0.0</version>
<scope>provided</scope>
</dependency>

Create a simple servlet with the following code,

import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.http.HttpSession;

import org.apache.felix.scr.annotations.sling.SlingServlet;
import org.apache.sling.api.SlingHttpServletRequest;
import org.apache.sling.api.SlingHttpServletResponse;
import org.apache.sling.api.resource.Resource;
import org.apache.sling.api.servlets.SlingAllMethodsServlet;
import org.apache.sling.api.servlets.SlingSafeMethodsServlet;

/**
* Simple first servlet
*/
@SlingServlet(paths = "/bin/firstserlvet/test", methods = "GET")
public class MyFirstServlet extends SlingSafeMethodsServlet
{

/**
* The method which receives the GET request.
*/
@Override
protected void doGet(final SlingHttpServletRequest request, final SlingHttpServletResponse response) throws ServletException, IOException
{
final Resource resource = request.getResource();

// Get the path that was requested
response.getOutputStream().println(resource.toString());

// Output something
response.getOutputStream().println("Output from simple servlet");
}
}

The path here is defined in the @SlingServlet annotation. It starts with /bin/ which is one of the defined permitted paths configured in the SlingServletResolver. Any of the defaults can be used or new permitted values can be added. The current configuration can be seen by browsing to

http://localhost:4502/system/console/configMgr/org.apache.sling.servlets.resolver.SlingServletResolver

Or go

http://localhost:4502/system/console
OSGi - Configuration
Apache Sling Servlet/Script Resolver and Error Handler (click on it)

You can see here a list of the default permitted paths. This can be added to by just clicking the + on the right hand side.

Friday, 12 February 2016

Mongo Unit Testing - Fongo

Unit testing with any database can be a bit of a pain. However, with Mongo it is really easy thanks to a testing framework called Fake Mongo (fongo). The Fongo Client implements the MongoClient interface in memory so is perfect for unit testing.

https://github.com/fakemongo/fongo

Maven Dependency

<dependency>
<groupId>com.github.fakemongo</groupId>
<artifactId>fongo</artifactId>
<version>2.0.4</version>
<scope>test</scope>
</dependency>

Example

Now just inject an instance of Fongo anywhere that a MongoClient would have been used.

final Fongo fakeMongo = new Fongo("FakeMongo");
final MongoClient mongoClient = fakeMongo.getMongo();

Now just interact with the mongoClient as normal. Add documents, delete, read etc

XML to Json

Converting XML to JSON without knowing the xsd schema is a really useful trick for storing data into mongo db. I'm commenting this because although it is easy I don't want to forget!

Maven Dependency

Conversion

String data; // Xml as a string

final JSONObject jsonObject = XML.toJSONObject(data);
doc.put("data", BsonDocument.parse(jsonObject.toString()));
doc.put("rawData", data);

Mongo & Java

Doing a search with Mongo using the Java driver is very simple. Firstly use the mongo maven dependency in your pom.

Maven Dependency

<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongodb-driver</artifactId>
<version>3.2.0</version>
</dependency>

MongoClient

Then you need a MongoClient. This can just be created or you can inject it with spring.

final MongoClient mongoClient = new MongoClient(new ServerAddress(hostname, port));

or using spring....

<bean id="mongoClient" class="com.mongodb.MongoClient">
<constructor-arg>
<util:list id="mongoServerAddresses" value-type="com.mongodb.ServerAddress">
<bean class="com.mongodb.ServerAddress">
<constructor-arg value="localhost" />
<constructor-arg value="27017" />
</bean>
<bean class="com.mongodb.ServerAddress">
<constructor-arg value="localhost" />
<constructor-arg value="27018" />
</bean>
</util:list>
</constructor-arg>
</bean>

Writing

Create a Document object and populate it,

// Create the document to store in mongo.
final Document doc = new Document();
doc.put("timestamp", now.getTime());
doc.put("data", BsonDocument.parse(data.toString()));

Write the document to the database

try
{
final MongoCollection<Document> mongoCollection = mongoClient.getDatabase(databaseName).getCollection(collection);
mongoCollection.insertOne(doc);
  }
  catch (final Exception e)
  {
LOG.error("Error writing to mongodb", e);
}

Reading

final MongoCollection mongoCollection = mongoClient.getDatabase(database).getCollection(collection);

// Create a document to do the search with
Bson bson = Filters.and(Filters.gte("timestamp", fromTime), Filters.lte("timestamp", toTime));
bson = Filters.and(bson, Filters.eq("name", "fred"));

// Do the find request against the database.
final FindIterable<Document> find = mongoCollection.find(bson);
final MongoCursor<Document> cursor = find.iterator();

// The results
final List<Result> results = new ArrayList<>();

// Loop through the results and put them into the match values.
while (cursor.hasNext())
{
final Document doc = cursor.next();
final Result result = new Result();
result.setTimestamp(doc.getLong("timestamp"));
result.setDob(doc.getLong("dob"));
result.setSurname(doc.getString("surname"));
results.add(result);
}

Alternatively you can just return all the values in the document by doing

// Loop through the results and put them into the match values.
while (cursor.hasNext())
{
final Document doc = cursor.next();
final Result result = new Result();
result.setTimestamp(doc.getLong("timestamp"));
for (final Entry<String, Object> entry : doc.entrySet())
{
result.add(entry.getKey(), entry.getValue().toString());
}

}

Monday, 1 February 2016

JodaTime, XSD, XJC

For Java 1.7 and earlier JodaTime (http://www.joda.org/joda-time/) is a great replacement for the standard XML / XSD date time object that Java creates when using XJC.

From Java 1.8 onwards the date time support in Java is much better and is basically a JodaTime implementation (JSR310).

Specific Bindings

To use JodaTime create a bindings file which contains an XPath location to the correct element, such as,

<jxb:bindings namespace="http://www.me.com/my/message" schemaLocation="my-message.xsd">

<jxb:schemaBindings>

<jxb:package name="com.me.my.message.generated" />

</jxb:schemaBindings>

<jxb:bindings node="//xs:complexType[@name='Message']/xs:sequence/xs:element[@name='PublishDateTime']">

<xjc:javaType name="org.joda.time.DateTime" adapter="com.me.my.util.JodaDateTimeAdaptor" />

</jxb:bindings>

Adaptor

The JodaDateTimeAdaptor will do the actual conversion and can be reused.

public class JodaDateTimeAdaptor

extends XmlAdapter<String, DateTime>

{

/**

* Convert a DateTime to String.

* @param datetime DateTime to be converted.

* @return The value of datetime formatted as per ISO8601, or the empty string if date is null.

@Override

public String marshal(final DateTime datetime)

{

return datetime.toString();

}

/**

* Convert a String representation of a date into a Date object.

* @param str String representation of a date/time, in the ISO8601. May be null or empty.

* @return str converted to a Date object. If str is null or empty, null is returned.

@Override

public DateTime unmarshal(final String str)

{

if (str == null || str.length() == 0)

{

return null;

}

return DateTime.parse(str);

}

Global Bindings

Global bindings can also be used but these don't work when one XSD imports another and you want to use an adaptor on both. You end up with a class name clash because the automatically generated Adaptors appear in a

org.w3._2001.xmlschema.Adaptor1.java

org.w3._2001.xmlschema.Adaptor2.java

...

However, for a single xsd or single project this may work. To use the global bindings instead of the specific ones use

<jxb:globalBindings typesafeEnumMemberName="generateName" >

<jxb:serializable uid="1" />

<jxb:javaType name="org.joda.time.DateTime" xmlType="xs:date"

parseMethod="com.me.my.util.JodaDateAdaptor.unmarshal"

printMethod="com.me.my.util.JodaDateAdaptor.marshal"/>

<jxb:javaType name="org.joda.time.DateTime" xmlType="xs:dateTime"

parseMethod="com.me.my.util.JodaDateTimeAdaptor.unmarshal"

printMethod="com.me.my.util.JodaDateTimeAdaptor.marshal"/>

</jxb:globalBindings>

Friday, 29 January 2016

JSR303 & JSR349 Validation

Here is a simple memory jogger for doing the validation framework.

Annotations

@Null
@NotNull
@Min - Also creates a column check constraint
@Max - Also creates a column check constraint
@DecimalMin
@DecimalMax
@Future
@Past
@Size(max = , min = ) - used for strings, collections etc
@Pattern(regex = )
@AssertTrue
@AssertFalse

Test Case

For a Car class,

import javax.validation.constraints.NotNull;

/**
* The Car type.
*/
public class Car
{
/**
* The manufacturer field.
*/
@NotNull
private String manufacturer;

/**
* Constructs a new Car with the given parameters.
*/
public Car()
{

}

/**
* Gets the manufacturer value.
*
* @return the manufacturer
*/
public String getManufacturer()
{
return manufacturer;
}

/**
* Sets manufacturer to the given value.
*
* @param manufacturer the manufacturer to set
*/
public void setManufacturer(String manufacturer)
{
this.manufacturer = manufacturer;
}
}

A test to exercise the validation framework,

import java.util.Set;

import javax.validation.ConstraintViolation;
import javax.validation.Validation;
import javax.validation.Validator;
import javax.validation.ValidatorFactory;

import org.junit.Test;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;

/**
* Test class to check the validation framework.
*/
public class ValidationTest
{

/**
* Test class to print out the values of a failed validation check.
*/
@Test
public void testValidate()
{
// Arrange
final ValidatorFactory validatorFactory = Validation.buildDefaultValidatorFactory();
final Validator validator = validatorFactory.getValidator();

final Car car = new Car();

// Act
final Set<ConstraintViolation<Car>> violations = validator.validate(car);

// Assert
final ConstraintViolation constraintViolation = violations.iterator().next();

assertTrue(constraintViolation.getRootBean() instanceof Car);
assertEquals("com.package.to.validated.class.Car", constraintViolation.getRootBeanClass().getName());
assertEquals("manufacturer", constraintViolation.getPropertyPath().toString());
assertEquals("may not be null", constraintViolation.getMessage());
}
}

Alternatively you can use the ValidationFactory to query the actual annotations that are on a bean and test that way. For example,

final PropertyDescriptor propertyDescriptor = validator.getConstraintsForClass(Car.class).getConstraintsForProperty("manufacturer");

final Set<ConstraintDescriptor<?>> constraints = propertyDescriptor.getConstraintDescriptors();
final String message = descriptor.getMessageTemplate();
final Object annotationType = descriptor.getAnnotation().annotationType();
etc.

Maven

These are the maven dependencies that are required to make the validation classes available,

<dependency>
<groupId>javax.validation</groupId>
<artifactId>validation-api</artifactId>
<version>1.0.0.GA</version>
</dependency>

<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-validator</artifactId>
<version>4.3.1.Final</version>
</dependency>

BVal

BVal is an alternative JSR303 implementation from Apache. Because Hibernate is written by JBoss it doesn't necessarily play well with old JBoss AS instances and in that case using BVal can work well.

<dependency>
<groupId>javax.validation</groupId>
<artifactId>validation-api</artifactId>
<version>1.1.0.Final</version>
</dependency>

<dependency>
<groupId>org.apache.bval</groupId>
<artifactId>bval-jsr</artifactId>
<version>1.1.1</version>
</dependency>

References

http://docs.jboss.org/hibernate/validator/4.2/reference/en-US/pdf/hibernate_validator_reference.pdf

Thursday, 3 December 2015

Mongo DB Crib Sheet

General

Using school as the database and student as the collection.

use school; // Change to use the school database

show collections; // Show the collections which are available for this database

db.......pretty(); // Use pretty to make sure the output is formatted well

db.student.drop(); // Drop the collection

db.student.stats(); // Get the collection stats

db.student.totalIndexSize(); // Available from the stats but a short cut. It is important that the indexes fit into memory

Find

db.students.find();

db.students.findOne();

db.students.find({name:'bob'});

db.students.find({name:'bob'}).count(); // Return the number of matches for this find

db.students.find({age:{$gt:15}}); // Find students with age greater than 15

db.students.find({age:{$gt:15, $lt:18}}); // Find students with age greater than 15 and less than 18

db.students.find({name:{$regex : 'q'}, email : {$exists : true}}); // Find all the students whose name includes a 'q' and who have an email

db.students.find({$or:[{age:15}, {age:16}]}); // Students with age 15 or 16

db.students.find({pets:{$in:['dog','cat']}}); // Students with a pet dog or cat

db.students.find({pets:{$all:['dog','cat']}}); // Students with a pet dog and cat

db.students.find({age:{$gt:15}}, {"_id":0, name:1, age:1}); // Projection which changes the data fields that are returned.

db.students.find( {_id:4}, {pets: {$slice:2}}); // Return the first two elements of the pet array only

db.students.find().sort({age:1}); // Sort by age ascending

db.students.find().limit(20); // Return the first 20 records

var cursor = db.students.find();

cursor.next(); // A cursor is returned if the find query is allocated to a variable

db.students.find({$text : {$search : 'bob fred'}}); // Do a search on a text index for all the documents containing 'bob' or 'fred' (case insensitive)

db.students.find({address:{$near: [x,y]}}); // Returns values based on a 2d index in increasing distance often used with a limit(n)

db.students.find({address:{

$near: {

$geometry : {

type : "Point",

coordinates : [long, lat]},

$maxDistance : 2000

}

}); // Returns values based on a 2dspherical index in increasing distance often used with a limit(n). Searches for Lat & Long with a max distance

Insert

db.students.insert({name: 'Fred', age:17, results:[78.8, 89.9]});

Delete / Remove

db.students.remove({}); // Removes all documents from the collection

db.students.remove({name:'Fred'}); // Removes all Fred entries

db.students.remove({name:'Fred'}, {justOne:true}); // Removes the first 'Fred' entry found

Update

db.country.update({pop:{$gt:2000000}}, {$set: {large_country:'true'}}, {multi:true}); // Update all countries which have a population greater than 2Mil to have a new flag saying large_country=true

db.country.update({_id:'Japan'},{pop='30000000', lang='Japanese'}); // All other entries in the document are removed.

db.country.update({_id:'Japan'}, {$set : {countryCode:'JP'}}); // Add the countryCode = 'JP' to the document with _id = 'Japan'

db.country.update({_id:'Japan'}, {$unset : {countryCode:1}}); // Remove the countryCode key from the document with _id = 'Japan'

db.country.update({_id:'Japan'}, {$push : { languages: "Japanese"}}); // Adds 'Japanese' to the list of languages for 'Japan'

db.country.update({_id:'Japan'}, {$pop : { languages: -1}}); // Removes the first item from the list of languages for 'Japan'

db.country.update({_id:'Japan'}, {$addToSet : { languages: "English"}}); // Adds 'English' to the list of languages if it doesn't exist already

db.country.update({_id:'Japan'}, {$pushAll : { languages: ["Japanese", "English"]}}); // Adds 'Japanese' and 'English' to the list of languages even if they exist already
db.country.update({_id:'GB'}, {_id:'GB', languages:'English', pop:50000000}, {upsert:true}); // Looks to update the object but will insert it if it can't be found

Indexes

Index creation is run in the foreground by default and will lock the collection that is being indexed so that no read or write request will be published.

db.student.getIndexes();

db.student.createIndex({name:1, class:-1}); (1 = ascending, -1 = descending)

db.student.createIndex({name:1}, {unique:true}); // Create a unique index

db.student.createIndex({name:1}, {unique:true, sparse:true}); // Create a unique index which is also sparse. Used when there could be null / missing values from a document.

db.student.createIndex({name:1}, {background:true}); // Create this index in the background without locking the collection

db.student.createIndex({'name.result':1}); // Create an index into the array of result embedded documents

db.student.createIndex({'name':'text'}); // Create an index into the array of name embedded documents

db.student.createIndex({'location':'2d'}); // Create an index based on a 2d location where 'location' is an array of two coordinates

db.student.createIndex({'location':'2dsphere'}); // Create an index based on a 2d spherical location where 'location' is an array of two coordinates latitude and longitude.

db.student.dropIndex({name:1});

Explain

Use the explain option to see which indexes are being used and other data.

db.student.explain().find({name:1}); // Explain the information about this find request

db.student.explain("executionStats").find({name:1});

db.student.explain("allPlansExecution").find({name:1});

Aggregation

Use the aggregation frame work to do groupings, counts, sums, averages etc

$group
db.student.aggregate([{$group:{_id : "$surname", count:{$sum:1}}}]); // Count the number of students with the same surname

db.student.aggregate([{$group:{_id : {name : "$name", surname : "$surname"}, exam_avg : {$avg:"$result"}}}; // Use an _id as a document to identify the student. Then average the exam marks

db.products.aggregate([{$group:{_id: $manufacturer, categories:{$addToSet:"$category"}}}]); // Build an array of the categories which each manufacturer has. This builds a set so doesn't create duplicates in the array

db.products.aggregate([{$group:{_id: $manufacturer, categories:{$push:"$category"}}}]); // Build an array of the categories which each manufacturer has. $push allows duplicates in the array.

db.products.aggregate([{$group:{_id: $manufacturer, categories:{$max:"$price"}}}]); // Find the max price of a manufacturer product.

db.products.aggregate([{$group:{_id: $manufacturer, categories:{$min:"$price"}}}]); // Find the min price of a manufacturer product.

db.students.aggregate([
{$group:{_id:{class_id:"$class_id", student_id:"$student_id"}, student_avg : {$avg:"$score"}}},
{$group:{_id:"$_id.class_id", class_avg: {$avg:"$student_avg"}}}]); // Average each students marks in a class, then average all the single

db.student.aggregate([{$sort: {category : 1, price : 1}}, {$group:{ _id:"$category", cheapest :{$first: "$price"}}}]); // Sort by category & price and return the first value = cheapest per category

$project
Allows you to 'reshape' a document eg add keys, remove keys. To keep a key put $keyName:1 otherwise the key will be removed - with the exception of _id which needs to be explicitly excluded.

db.products.aggregate([{$project :{
_id:0,
'maker':{$toLower:$manufacturer}, // Create a new key with the Lower Case of the manufacturer
'details':{category:"$category",
price:{$multiply : [$price, 10]}}, // Create new document of category and 10xprice
item: $name // Keep the name as a new key called 'item'
}}]);

$match
db.student.aggregate([{$match:{category : "tablet"}}]); // Match values - very similar to find()

$sort
db.student.aggregate([{$sort:{price: 1}}]); // Sort by price - very similar to sort()

$skip
db.student.aggregate([{$sort:{price: 1}, $skip : 10}]); // Skip the first 10 records - only use with a sort first

$limit
db.student.aggregate([{$sort:{price: 1}, $limit : 5 }]); // Only return 5 records - only use with a sort first

$unwind

db.student.aggregate([{$unwind : "$classes"}]); // Unwind the classes that the students have done

Profiling

With profiling on the db.system collection gets profiling information about queries which take longer than the threshold specified.

db.getProfilingLevel();

db.getProfilingStatus();

db.setProfilingLevel(1, 4); // 1 is the level and 4 is the threshold ms for queries

db.system.profile.find(); // Find queries which have been profiled because they took too long.

db.system.profile.find({millis:{$gt:1000}}).sort({ts:-1}); // Find all the queries that took longer than 1 second sorted in timestamp descending order

mongostat is useful and is similar to iostat on a linux box.

mongotop # // samples the database every # seconds and prints out an overview of where mongo is slow.

Replication

General
rs.status();
rs.conf(); // Show the current configuration of the replica set
rs.slaveOk(); // Allow queries on a secondary in a replica set
rs.help(); // Get help on the replica set commands

Create
Create a replication set called "rs1" with a particular data path and listening on a particular port.
mongod --replSet rs1 --logpath "1.log" --dbpath /data/rs1 --port 27017 --fork

config = { _id : "rs1" , members:[
{_id : 0, host : "192.168.1.1.27017"},
{_id : 1, host : "192.168.1.2.27017"}, ]}
rs.initiate(config);