How to intercept and modify Java stacktraces

This ticket was triggered by a “simple” requirement: “Change all the package names in the logs of a Java application (especially the stacktraces) from ‘abc.efg’ (put here whatever you want as name) to ‘hij.klm’ (put here also whatever you want as name) “. The first idea that popped in my mind was to change the packages names at the code level, but this was not feasible because of (rather) big codebase, the use of the (Java) reflexion and the tight timeline.

In the following lines, I will discuss possible solutions to implement this (weird) requirement.

 

Extend the log4j ThrowableRenderer

If the project is using log4j1x as log library, then a solution would be to create your own throwable renderer by extending the org.apache.log4j.spi.ThrowableRenderer. The (log4j) renderers are used to render instances of java.lang.Throwable (exceptions and errors) into a string representation.

The custom renderer that replaces the packages starting with “org.github.cituadrian” by “xxx.yyy” will look like this:

package org.github.cituadrian.stacktraceinterceptor.log4j;

import org.apache.log4j.DefaultThrowableRenderer;
import org.apache.log4j.spi.ThrowableRenderer;

public class CustomThrowableRenderer implements ThrowableRenderer {
    private final DefaultThrowableRenderer defaultRenderer =  
                   new DefaultThrowableRenderer(); 
    
    @Override 
    public String[] doRender(Throwable t) {
      String[] initialResult = defaultRenderer.doRender(t); 
      for (int i = 0; i < initialResult.length; i++) { 
        String line = initialResult[i]; 
        if (line.contains("org.github.cituadrian")) { 
           initialResult[i] = line.replaceAll("org.github.cituadrian", "xxx.yyy"); 
        } 
       } 
      return initialResult; 
    }
}

Basically, the custom renderer is delegating the task of creating a String from a Throwable to a DefaultThrowableRenderer and then it checks and replace the desired package names.

In order to be used, the renderer should be defined in the log4j.xml file:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration debug="true"
    xmlns:log4j='http://jakarta.apache.org/log4j/'>
 
  <throwableRenderer class= 
     "org.github.cituadrian.stacktraceinterceptor.log4j.CustomThrowableRenderer"/>
...

Use a log4j2 pattern layout

If your project is using log4j2 as logging library, then you can use a (log4j2) layout pattern.  The layout pattern will look like:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
 <Appenders>
 <Console name="STDOUT" target="SYSTEM_OUT">
 <PatternLayout pattern=
  "%replace{%class %log %msg %ex}{org\.github\.cituadrian}{xxx\.yyy}"/>
 </Console>
...

 

Modify (a.k.a. Weaving) the java.lang.StackTraceElement class with AOP

Before even explaining what it really means, I have to warn you that weaving JDK classes is rarely necessary (and usually a bad idea) even if it’s possible using an AOP framework like AspectJ.

For this case I used the AspectJ as AOP framwork because the weaver (aop compiler) is able to do binary weaving, meaning the weaver takes classes and aspects in .class form and weaves them together to produce binary-compatible .class files that run in any Java VM. The command line to obtain a weaved jar is the following one:

ajc -inpath rt.jar Aspect.java -outjar weavedrt.jar

In the case of weaving JDK classes one extra step is necessary in order to make the application work; we must create a new version of the rt.jar file  or create just a small JAR file with the JDK woven classes which then must be appended to the boot-classpath of the JDK/JRE when firing up the target application. The command line to execute the target application is the following one:

java -Xbootclasspath/<path to weavedrt.jar>;<path to aspectjrt.jar> TargetApplication

If you don’t want to worry about all the technical details of weaving and executing the application and you are using Maven then you can use the (marvelous) SO_AJ_MavenWeaveJDK project from gitHub (that handles everything using Maven)

The aspect that will modify the stacktrace packages looks like:

package org.github.cituadrian.stacktraceinterceptor.app;

import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;
@Aspectpublic class StackTraceInterceptorAspect {
    @Pointcut("execution(String java.lang.StackTraceElement.getClassName()) "
            + "&& !within(StackTraceInterceptorAspect)")     
    public void executeStackTraceElementGetClassNamePointcut() {}        
    
    @Around("executeStackTraceElementGetClassNamePointcut()")    
    public Object executeStackTraceElementGetClassNameAdvice(    
                   final ProceedingJoinPoint pjp) throws Throwable {        
        Object initialResponse =  pjp.proceed();         
        if (initialResponse instanceof String 
               && ((String)initialResponse).startsWith("org.github.cituadrian")) {     
                 return ((String)initialResponse).replaceFirst("org.github.cituadrian", "xxx.zzz"); 
        }        
        return initialResponse;    
    }
 }

In a nutshell, the StackTraceInterceptorAspect will intercept all the calls to the java.lang.StackTraceElement#getClassName method and it will change the returned result of the method if the class name contains the string “org.github.cituadrian”.

If you are interested to learn more about AspectJ I really recommend you to buy a copy of the AspectJ in action (second edition) book.

 

Modify and shadow the java.lang.StackTraceElement class

 Using AOP just to intercept and modify a single method of a single class is a little bit over-killing. In this case there is another solution; the solution would be create a custom version of the java.lang.StackTraceElement class and add this custom class in the boot-classpath of the JDK/JRE when firing up the target application, so the initial version will be shadowed by the custom version.

An implementation of StacktraceElement class can be found here. So you can modify by hand the java.lang.StackTraceElement#getClassName method or the java.lang.StackTraceElement#toString method.

 To execute the target application, you must create a jar with the modified class and add it into the boot-classpath (something similar to the AspectJ solution):

java -Xbootclasspath/<path to custom class.jar> TargetApplication

 

 

How to remotely connect to an in-memory HSQLDB database

hsqldbVery often in-memory instances of HSQLDB are used in the context of unit tests; the unit test starts a database instance (eventually on a random port), provision the database with some data, run the test against the database end then stop it.

Now, from time to time you need to debug the unit tests and sometimes you also need to run manually some queries on the in-memory database (using HSQLDB manager or any other software).

So here are the steps in order to be able to connect to an in-memory HSQLDB instance:

  • Start the DB in the “remote open db” mode.This is driven by the “server.remote_open” property that is false by default (you can look to the code of org.hsqldb.server.ServerProperties class to see other properties that might be interesting). The code that starts a dataabse instacne in remote open mode will look something like:
HsqlProperties props = new HsqlProperties();
props.setProperty("server.remote_open", true);
....add more properties here
Server server = new Server();
server.setProperties(props);
server.start();
  • Connect to the data base instance. Now that the database has started the next step is to connect to data base instance. Because we started an in-memory instance the connection url is a memory database url that looks like jdbc:hsqldb:mem:instanceName You will not be able to connect using this url because neither the host and the port are available, instead you should use a server database url that looks like jdbc:hsqldb:hsql://host:port/instanceName

 

How to find (buggy) calls of java.lang.Object.equals() with incompatible types

Introductionjavabug

All this started from a change (and a mistake) done by one of my co-workers that did a small change in the code with rather dramatic consequences;

So, imagine we have a (java) class T having a private field f of type String:

public class T {
    private Boolean f;

    public void setF(final Boolean f) {
        this.f = f;
    }
    public Boolean getF() {
        return f;
    }
    .... 
}

Now, imagine (again) that the type of f field will change from String to something else, let’s say Boolean; the compiler will help you to find the places where the setF method is called with the wrong parameter and also where the getF method do not comply with the new signature.

But there is at least one case where the compiler cannot help you; it is the case of the call of java.lang.Object.equals method. So something like “aString”.equals(t.getF()) in the case when the f field is a boolean will always return false because we try to check the equality for incompatible types.

The goal of this ticket is to find solutions to catch this kind of problem at runtime or (ideally) at compile time. Anyway, if you want to catch this kind of problem in your code and you do not want to reinvent the wheel (as I will do in this ticket), just use the FindBugs. FindBugs is capable to catch this kind of error (EC_UNRELATED_TYPES) and many, many more (see FindBugs Bug Descriptions).

How to find the buggy calls at runtime

The only way that I see to catch the buggy equals calls in a generic way is to use AOP. The basic idea is to write an aspect that will intercept all the calls to the equals methods and check that the caller and the callee are of the same type.

For the implementation I choose to use AspectJ framework (for a very good introduction to AspectJ I strongly recommend the AspectJ in action book) and the code it looks like this:

import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Before;
import org.aspectj.lang.annotation.Pointcut;
import org.github.adriancitu.equals.WrongUseOfEqualsException;

@Aspect
public class EqualsCheckerAspect {
    @Pointcut(
            "execution(public boolean Object+.equals(Object)) "
            + "&& target(target) "
            + "&& args(methodArgument)"
            + "&& !within(EqualsCheckerAspect)")
    public void executeEqualsPointcut(
            final Object target, 
            final Object methodArgument) {
    }
    
    @Before("executeEqualsPointcut(target,methodArgument)")
    public void executeEqualsAdvice(
            final Object target, 
            final Object methodArgument) throws Exception {
       
        if (target != null && !target.getClass().equals(methodArgument.getClass())) {
            throw new WrongUseOfEqualsException("Tried to call the equals methods on different classes types");
        }
    }
}

I will try briefly to explain what the previous code is doing; this is not a tutorial about AspectJ or AOP so, some of the definitions will be a little bit approximative, so AOP purists please forgive me.

The aspect (which is a Java class annotated with the @Aspect annotation) contains 2 parts; the pointcut (the method annotated with the @Pointcut annotation) and the advice (the method annotated with the @Before annotation).

The pointcut represents the parts of program flow that we want to intercept; in our case we want to intercept all the calls to the method having the following signature boolean equals(Object) on any instance of java.lang.Object class and on any subclasses (that’s the meaning of the + from the ..boolean Object+.equals). The  target and args are just AspectJ structures that helps to capture the caller and the callee of the equals method.

The advice represents the code that will be executed when a poincut is intercepted. In our  case we would like to do the verification that the caller and the callee are of the same type before the execution of the equals method (this is the meaning of the @Before annotation). The advice code represented by the executeEqualsAdvice method is quite simple, it just retrieve the caller and the callee objects (already captured by the target and args) and check that are instances of the same class.

This solution will fix the problem but it have some drawbacks:

  • the check is done at runtime
  • you have a dependency on AspectJ

How to find the buggy calls at compile time

It would be really nice to be able to catch this problem directly at compile time and eventually (not mandatory) having no other technical dependency.

Our salvation is coming from the javac plug-in mechanism which is new in Java8. The javac plug-in mechanism allows to a user to specify one or more plug-ins on the javac command line, to be started soon after the compilation has begun. There is no official tutorial about how to craft and use a javac plug-in, the only information that I’ve found is this Javadoc link and this blog ticket.

The javac plug-in mechanism gives access to the abstract syntax tree of the compiled program by implementing the visitor pattern so we will use this mechanism to catch and check the executions of the equals methods. The entire code of the plug-in can be found on this GitHub repository.

The steps of crafting a (javac) plug-in are the following ones:

  1. The entry point of the plug-in should implement the com.sun.source.util.Plugin interface. The code of the class can be found here : RuntimeEqualsCheckPlugin.java
  2. Create a class that implements the com.sun.source.util.TaskListener interface in order to perform additional behavior after the type-checking phase. The code of the class can be found here: EqualsMethodCallTaskListener.java
  3. Create an abstract syntax tree visitor by extending the com.sun.source.util.TreePathScanner<R,P>
    end extend the behavior for the method invocation. The code for this class can be found here: CodePatternTreeVisitor2.java

The most important part of CodePatternTreeVisitor2.java class is the overridden visitMethodInvocation method. I will show you the code but the most important lesson that I learned is that in the code you have to work with trees, everything is a subtype of the Tree interface.

public Void visitMethodInvocation(MethodInvocationTree methodInvocationTree, Void aVoid) {
  final List<? extends ExpressionTree> arguments = methodInvocationTree.getArguments();
  final ExpressionTree methodSelect = methodInvocationTree.getMethodSelect();
  switch (methodInvocationTree.getKind()) {
      case METHOD_INVOCATION:
        Tree.Kind methodSelectKind = methodSelect.getKind();
        switch (methodSelect.getKind()) {
            case MEMBER_SELECT:
             //t1.equals
             //or
             //field.equals
             MemberSelectTree memberSelectTree = (MemberSelectTree) methodSelect;
             //it's a equals method invocation
             if (isEqualsCall(
                  new TreePath(getCurrentPath(), methodSelect),
                  methodInvocationTree.getArguments() != null ?
                       methodInvocationTree.getArguments().size() : 0)) {
                //t1
                //or
                //field
                ExpressionTree expression = memberSelectTree.getExpression();
                TypeMirror callerType =
                  trees.getTypeMirror(
                            new TreePath(getCurrentPath(), expression));
                Optional argumentType =
                            omputeArgumentType(methodInvocationTree.getArguments());
                if (argumentType.isPresent() 
                      && !callerType.equals(argumentType.get())) {
                      System.err.println("Try to call equals on different parameters at line "
                          + getLineNumber(methodInvocationTree)
                          + " of file " +
                          currCompUnit.getSourceFile().getName()
                          + "; this is a bug!"
                                );
                 }
              }
          }
        }
        return super.visitMethodInvocation(methodInvocationTree, aVoid);
    }

The steps to execute the (javac) plug-in are the following ones:

  1. Set up a file called com.sun.source.util.Plugin located in META-INF/services/ of your plug-in code because the plug-ins is located via java.util.ServiceLoader. The content of the file should contain the name of the plug-in (com.github.adriancitu.equals.com.github.adriancitu.equals.runtime.RuntimeEqualsCheckPlugin in this case).
  2. Create a jar with all the plug-in files and the com.sun.source.util.Plugin file.
  3. Execute your plug-in against the javac using the  -processorpath flag to indicates the path where the plug-in JAR file is located and -Xplugin flag to indicate the name of the plug-in to run. In our case the command will be something like :
    javac -processorpath plugIn.jar
    -Xplugin:RuntimeEqualsCheckPlugin ./NoDependenciesTest.java

So finally was possible to have a compile time solution, the only drawback that I see for the plug-in solution is that the API is not very user friendly.

A Java implementation of CSRF mitigation using “double submit cookie” pattern

Goal of this articlecsrf

The goal of this article is to present an implementation of the “double submit cookie” pattern used to mitigate the Cross Site Request Forgery (CSRF) attacks. The proposed implementation is a Java filter plus a few auxiliary classes and it is (obviously) suitable for projects using the Java language as back-end technology.

Definition of CSRF and possible mitigations

In the case of a CSRF attack, the browser is tricked into making unauthorized requests on the victim’s behalf, without the victim’s knowledge. The general attack scenario contains the following steps:

  1. the victim connects to the vulnerable web-site, so it have a real, authenticated session.
  2. the hacker force the victim (usually using a spam/fishing email) to navigate to another (evil) web-site containing the CSRF attack.
  3. when the victim browser execute the (evil) web-site page, the browser will execute a (fraudulent) request to the vulnerable web-site using the user authenticated session. The user is not aware at all of the fact that navigating on the (evil) web-site will trigger an action on the vulnerable web-site.

For deeper explanations I strongly recommend  to read chapter 5 of Iron-Clad Java: Building Secure Applications book and/or the OWASP Cross-Site Request Forgery (CSRF) Prevention Cheat Sheet.

Definition of “double submit cookie” pattern

When a user authenticates to a site, the site should generate a (cryptographically strong) pseudo-random value and set it as a cookie on the user’s machine separate from the session id. The server does not have to save this value in any way, that’s why this patterns is sometimes also called Stateless CSRF Defense.

The site then requires that every transaction request include this random value as a hidden form value (or other request parameter). A cross origin attacker cannot read any data sent from the server or modify cookie values, per the same-origin policy.

In the case of this mitigation technique the job of the client is very simple; just retrieve the CSRF cookie from the response and add it into a special header to all the requests:

csrfclient

Client workflow

The job of the server is a little more complex; create the CSRF cookie and for each request asking for a protected resource, check that the CSRF cookie and the CSRF header of the request are matching:

csrfserver

Server workflow

Note that some JavaScript frameworks like AngularJS implements the client worflow out of the box; see Cross Site Request Forgery (XSRF) Protection

Java implementation of “double submit cookie” pattern

The proposed implementation is on the form of a (Java) Servlet filter and can be found here: GenericCSRFFilter GitHub.

In order to use the filter, you must define it into you web.xml file:

<filter>
   <filter-name>CSRFFilter</filter-name>
   <filter-class>com.github.adriancitu.csrf.GenericCSRFStatelessFilter</filter-class>
<filter>

<filter-mapping>
   <filter-name>CSRFFilter</filter-name>
   <url-pattern>/*</url-pattern>
</filter-mapping>

 

The filter can have 2 optional initialization parameters: csrfCookieName representing the name of the cookie that will store the CSRF token and csrfHeaderName representing the name of the HTTP header that will be also contains the CSRF token.

The default values for these parameters are “XSRF-TOKEN” for the csrfCookieName and “X-XSRF-TOKEN” for the csrhHeaderName, both of them being the default values that AngularJS is expecting to have in order to implement the CSRF protection.

By default the filter have the following features:

  • works with AngularJS.
  • the CSRF token will be a random UUID.
  • all the resources that are NOT accessed through a GET request method will be CSRF protected.
  • the CSRF cookie is replaced after each non GET request method.

How it’s working under the hood

The most of the functionality is in the GenericCSRFStatelessFilter#doFilter method; here is the sequence diagram that explains what’s happening in this method:

doFilter method sequence diagram

doFilter method sequence diagram

The doFilter method is executed on each HTTP request:

  1. The filter creates an instance of ExecutionContext class; this class is a simple POJO containing the initial HTTP request, the HTTP response, the CSRF cookies (if more than one cookie with the csrfCookieName is present) and implementation of the ResourceCheckerHook , TokenBuilderHook and ResponseBuilderHook .(see the next section for the meaning of this classes).
  2. The filter check the status of the HTTP resource, that status can be:MUST_NOT_BE_PROTECTED, MUST_BE_PROTECTED_BUT_NO_COOKIE_ATTACHED,MUST_BE_PROTECTED_AND_COOKIE_ATTACHED (see ResourceStatus enum) using an instance of ResourceCheckerHook.
  3. If the resource status is ResourceStatus#MUST_NOT_BE_PROTECTED
    ResourceStatus#MUST_BE_PROTECTED_BUT_NO_COOKIE_ATTACHED then
    the filter creates a CSRF cookie having as token the token generated by an instance of TokenBuilderHook.
  4. if the resource status ResourceStatus#MUST_BE_PROTECTED_AND_COOKIE_ATTACHED
    then compute the CSRFStatus of the resource and then use an instance of ResponseBuilderHook to return the response to the client.

How to extend the default behavior

It is possible to extend or overwrite the default behavior by implementing the hooks interfaces. All the hooks implementations must be thread safe.

  1. The ResourceCheckerHook is used to check the status of a requested resource. The default implementation is DefaultResourceCheckerHookImpl and it will return ResourceStatus#MUST_NOT_BE_PROTECTED for any HTTP GET method, for all the other request types, it will return [email protected] ResourceStatus#MUST_BE_PROTECTED_BUT_NO_COOKIE_ATTACHED if any CSRF cookie is present in the query or ResourceStatus#MUST_BE_PROTECTED_BUT_NO_COOKIE_ATTACHED otherwise.The interface signature is the following one:
    public interface ResourceCheckerHook extends Closeable {
        ResourceStatus checkResourceStatus(ExecutionContext executionContext);
    }  
  2. The TokenBuilderHook hook is used to generate the token that will be used to create the CSRF cookie. The default implementation  is DefaultTokenBuilderHookImpl and it uses a call to UUID.randomUUID to fetch a token. The interface signature is the following one:
    public interface TokenBuilderHook extends Closeable {
        String buildToken(ExecutionContext executionContext);
    }
  3. The ResponseBuilderHook is used to generate the response to the client depending of the CSRFStatus of the resource. The default implementation is DefaultResponseBuilderHookImpl and it throws a SecurityException if the CSRF status is CSRFStatus#COOKIE_NOT_PRESENT, CSRFStatus#HEADER_TOKEN_NOT_PRESENT or CSRFStatus#COOKIE_TOKEN_AND_HEADER_TOKEN_MISMATCH. If the CSRF status is CSRFStatus#COOKIE_TOKEN_AND_HEADER_TOKEN_MATCH then the old CSRF cookies are deleted and a new CSRF cookie is created. The interface signature is the following one:
    public interface ResponseBuilderHook extends Closeable {
        ServletResponse buildResponse(ExecutionContext executionContext,
                                      CSRFStatus status);
    }
    

The hooks are instantiated inside the GenericCSRFStatelessFilter#init method using the ServiceLoader Java 6 loading facility. So if you want to use your implementation of one of the hooks then you have to create a  META-INF/services directory that contains a text file whose name matches the fully-qualified interface class name of the hook that you want to replace.

Here is the sequence diagram representing the hooks initializations:

initmethod

ElasticSearch:How to enable scripting on the embedded server

The goalelastic

In this ticket I will present all the required steps in order to enable the scripting on an embedded elasticsearch server. I will not explain how to create and use an embedded server but if you need more infos you can read this ticket Embedded Elasticsearch Server for Tests.

If you execute a script within the embedded server the following exception will be thrown;

Caused by: java.lang.IllegalArgumentException: script_lang not supported [xxxxx]
    at  org.elasticsearch.script.ScriptService.getScriptEngineServiceForLang(ScriptService.java:211)

 Code changes

 The easiest way to create an embedded server is the following one:

 Node node = NodeBuilder.nodeBuilder()
              .local (true)
              .settings(
               //add settings here
               )
              .node ();

The problem with this approach is that the NodeBuilder.node() method it will create a node class with an empty list of plug-ins; see the NodeBuilder.node code:

   /*
    * Builds the node without starting it.
    */
    public Node build() {
        return new Node(settings.build());
    }
    //Node(Settings) constructor
     /**
     * Constructs a node with the given settings.
     *
     * @param preparedSettings Base settings to configure the node with
     */
     public Node(Settings preparedSettings) {
        this(InternalSettingsPreparer
                .prepareEnvironment(preparedSettings, null),
             Version.CURRENT,
             Collections.<Class<? extends Plugin>>emptyList());
     }

In order to create a node having a list of desired plug-ins the second constructor of the Node class should be used:

protected Node(
                Environment tmpEnv, 
                Version version, 
                Collection<Class<? extends Plugin>> classpathPlugins)

The problem with this (second) constructor is that it is protected so we will need to extend the Node class to be able to use the protected constructor and we will need to extend also the NodeBuilder class in order to use the new Node class.

And here is the code:

  //PluginNode class 
  private static class PluginNode extends Node {
      public PluginNode(Settings preparedSettings, List<Class<? extends Plugin>> plugins) {
         super(InternalSettingsPreparer.prepareEnvironment(preparedSettings, null), Version.CURRENT, plugins);
      }
  }

  //PluginNodeBuilder class
  public class PluginNodeBuilder extends NodeBuilder {
        private List<Class<? extends Plugin>> plugins = new ArrayList<>();
        
        public PluginNodeBuilder() {
            super();
        }
        //new method to add plug-ins       
        public PluginNodeBuilder addPlugin(Class<? extends Plugin> plugin) {
            plugins.add(plugin);
            return this;
        }

        @Override
        //use the new PluginNode to build a node 
        public Node build() {
            return new PluginNode(settings().build(), plugins);
        }
    };

And how to use the new node builder:

 Node node = new PluginNodeBuilder()
                .addPlugin(ExpressionPlugin.class)
                .addPlugin(GroovyPlugin.class)
                .local (true)
                .settings(
                 //add settings here
                 )
                .node ();

Enable scripting feature

Using the previous Java code it will not be sufficient to execute the scripts; you must also use the right settings at the server creation and add the (Maven) dependencies to your classpath.

Server settings

 Node node = new PluginNodeBuilder()
                .addPlugin(ExpressionPlugin.class)
                .addPlugin(GroovyPlugin.class)
                .local (true)
                .settings (Settings.settingsBuilder ()
                     ...
                     //this is common for all languages
                     .put ("script.inline", true)
                     //this are the settings for the groovy language
                     .put ("script.engine.groovy.inline.mapping", true)
                     .put ("script.engine.groovy.inline.search", true)
                     .put ("script.engine.groovy.inline.update", true)
                     .put ("script.engine.groovy.inline.plugin", true)
                     .put ("script.engine.groovy.inline.aggs", true)
                      
                     //this are the settings for the expression language
                     .put ("script.engine.expression.inline.mapping", true)
                     .put ("script.engine.expression.inline.search", true)
                     .put ("script.engine.expression.inline.update", true)
                     .put ("script.engine.expression.inline.plugin", true)
                     .put ("script.engine.expression.inline.aggs", true)

If you more details about all the possible settings options you can look to the ElasticSearch Scripting documentation.

Maven dependencies

For the expression language:

       <dependency>
            <groupId>org.apache.lucene</groupId>
            <artifactId>lucene-expressions</artifactId>
            <version>5.5.2</version>
            <scope>compile</scope>
        </dependency>

       <dependency>
            <groupId>org.elasticsearch.module</groupId>
            <artifactId>lang-expression</artifactId>
            <version>2.2.0</version>
            <scope>compile</scope>
        </dependency>

For the groovy language:

<dependency>
    <groupId>org.elasticsearch.module</groupId>
    <artifactId>lang-groovy</artifactId>
    <version>2.3.2</version>
</dependency>

Book review: Iron-Clad Java Building Secure Web Applications

This a review of the Iron-Clad Java: Building Secure Web Applications book.

(My) Conclusion

I will start with the conclusion because it’s maybe the most important part of this review.

For me this is a must read book if you want to write more robust (web and non web) applications in Java, it covers a very large panel of topics from the basics of securing a web application using HTTP/S response headers to handling the encryption of sensitive informations in the right way.

Chapter 1: Web Application Security BasicsironCladJavaBook

This chapter is an introduction to the security of web application and it can be split in 2 different types of items.

The first type of items is what I would call “low-hanging fruits” or how you could improve the security  of your application with a very small effort:

  • The use of HTTP/S POST request method is advised over the use of HTTP/S GET. In the case of POST the parameters are embedded in the request body, so the parameters are not stored and visible in the browser history and if used over HTTPS are opaques to a possible attacker.
  • The use of the HTTP/S response headers:
    • Cache-Control – directive that instructs the browser how should cache the data.
    • Strict-Transport-Security – response header that force the browser to alway use the HTTPS protocol. This it can protect against protocol downgrade attacks and cookie hijacking.
    • X-Frame-Options – response header that can be used to indicate whether or not a browser should be allowed to render a page in a <frame>, <iframe> or <object> . Sites can use this to avoid clickjacking attacks, by ensuring that their content is not embedded into other sites.
    • X-XSS-Protection – response header that can help stop some XSS atacks (this is implemented and recognized only by Microsoft IE products).

 The second types of items are more complex topics like the input validation and security controls. For this items the authors just scratch the surface because all of this items will be treated in more details in the future chapters of the book.

Chapter 2: Authentication and Session Management

This chapter is about how a secure authentication feature should work; under the authentication topic is included the login process, session management, password storage and the identity federation.

The first part is presenting the general workflow of login and session management (see next picture) and for every step of the workflow  some dos and don’t are described.

login and session management workflow

login and session management workflow

The second part of the chapter is about common attacks on the authentication and for each kind of attack a solution to mitigated is also presented. This part of the chapter is strongly inspired from the OWASP Session Management Cheat Sheet which is rather normal because one of the authors (Jim Manico) is the project manager of the OWASP Cheat Sheet Series.

If you want to have a quick view of this chapter you can take a look to the presentation Authentication and Session Management done by Jim.

Even if you are not implementing an authentication framework for you application, you could still find good advices that can be applied to other web applications; like the use of the use of the secured and http-only attributes for cookies and the increase of the session ID length.

Chapter 3: Access Control

The chapter is about the advantages and pitfalls of implementing an authorization framework and can be split in three parts.

The first part describes the goal of an authorization framework and defines some core terms:

  • subject : the service making the request
  • subject attributes : the attributes that defines the service making the request.
  • group : basic organizational structure
  • role : a functional abstraction that uniquely describe system collaborators with similar or unique duties.
  • object : data being operating on.
  • object attributes : the attributes that defines the type of object being operating on.
  • access control rules : decision that need to be made to determine if a subject is allowed to access an object.
  • policy enforcement point : the place in code where the access control check is made.
  • policy decision point : the engine that takes the subject, subject attributes, object, object attributes and evaluates them to make an access control decision.
  • policy administration point : administrative entry in the access control system.

The second part of the chapter describes some access control (positive) patterns and anti-patterns.

Some of the (positive) access control patterns: have a centralized policy enforcement point  and policy decision point (not spread through the entire code),  all the authorization decisions should be taken on server-side only (never trust the client), the changes in the access control rules should be done dynamically (should not be necessary to recompile or restart/redeploy the application).

For the anti-patterns, some of then are just opposite of the (positive) patterns : hard-coded policy (opposite of “changes in the access control rules should be done dynamically”), adding the access control manually to every endpoint (opposite of have a centralized policy enforcement point  and policy decision point)

Others anti-patterns are around the idea of never trusting the client: do not use request data to make access control policy decisions and fail open (the control access framework should cope with wrong or missing parameters coming from the client).

The third part of the chapter is about different approaches (actually two) to implement an access control framework. The most used approach is RBAC (Role Based Access Control) and is implemented by some well knows Java access control frameworks like Apache Shiro and Spring Security. The most important limitation of RBAC is the difficulty of implementing data-specific/contextual access control. The use of ABAC (Attribute Based Access Control) paradigm can solve the data-specific/contextual access control but there are no mature frameworks on the market implementing this.

Chapter 4: Cross-Site Scripting Defense

This chapter is about the most common vulnerability found across the web and have two parts; the presentation of different types of cross-site scripting (XSS) and the way to defend against it.

XSS is a type of attack that consists in including untrusted data into the victim (web) browser. There are three types of XSS:

  • reflected XSS (non persistent) – the attacker tampers the HTTP request to submit malicious JavaScript code. Reflected attacks are delivered to victims via e-mail messages, or on some other web site. When a user clicks on a malicious link, submitting a specially crafted form the injected code travels to the vulnerable web site, which reflects the attack back to the user’s browser. The browser then executes the code because it came from a “trusted” server.
  • stored XSS (persistent XSS) – the malicious script is stored on the server hosting the vulnerable web application (usually in the database) and it is served later to other users of the web application when the users are loading the vulnerable page. In this case the victim does not require to take any attacker-initiated action.
  • DOM-based XSS – the attack payload is executed as a result of modifying the DOM “environment” in the victim’s browser.

For the defense techniques the big picture is that the input validation and output encoding should fix (almost) all the problems but very often various factors needs to be considered when deciding the defense technique.

Some projects are presented for the input validation (OWASP Java Encoder Project) and output encoding (OWASP HTML Sanitizer, OWSP AntiSamy).

Chapter 5: Cross-Site Request Forgery Defense and Clickjacking

Chapter 6: Protecting Sensitive Data

This chapter is articulated around three topics; how to protect (sensitive) data in transit, how to protect (sensitive) data at rest and the generation of secure random numbers.

How to protect the data in transit

The standard way to protect data in transit is by use of cryptographic protocol Transport Layer Security (TLS). In the case of web applications all the low level details are handled by the web server/application server and by the client browser but if you need a secure communications channel programmatically you can use the Java Secure Sockets Extension (JSSE). The authors recommendations for the cipher suites is to use the JSSE defaults.

Another topic treated by the authors is about the certificate and key management in Java. The notions of trustore and keystore are very well explained and examples about how to use the keytool tool are provided. Last but not least examples about how to manipulate the trustores and keystores programmatically are also provided.

How to protect data at rest

The goal is how to securely store the data but in a reversible way, so the data must be wrapped in protection when is stored and the protection must be unwrapped later when it is used.

For this kind of scenarios, the authors are focusing on Keyczar which is a (open source) framework created by Google Security Team having as goal to make it easier and safer the use cryptography for the developers. The developers should not be able to inadvertently expose key material, use weak key lengths or deprecated algorithms, or improperly use cryptographic modes.

Examples are provided about how to use Keyczar for encryption (symmetric and asymmetric) and for signing purposes.

 Generation of secure random numbers

Last topic of the chapter is about the Java support for the generation of secure random numbers like the optimal way of using the java.security.SecureRandom (knowing that the implementation depends on the underlying platform) and the new cryptographic features of Java8 (enhance of the revocation certificate checking, TLS Server name indication extension).

Chapter 7: SQL Injection and other injection attacks

This chapter is dedicated to the injections attacks; the sql injection being treated in more details that the other types of injection.

SQL injection

The sql injection mechanism and the usual defenses are very well explained. What is interesting is that the authors are proposing solutions to limit the impact of SQL injections when the “classical” solution of query parametrization cannot be applied (in the case of legacy applications for example): the use of input validation, the use of database permissions and the verification of the number of results.

Other types of injections

XML injection, JSON-Based injection and command injection are very briefly presented and the main takeaways are the following ones:

  • use a safe parser (like JSON.parse) when parsing untrusted JSON
  • when received untrusted XML, an XML schema should be applied to ensure proper XML structure.
  • when XML query language (XPath) is intermixed with untrusted data, query parametrization or encoding is necessary.

Chapter 8: Safe File Upload and File I/O

The chapter speaks about how to safety treat files coming from external sources and to protect against attacks like file path injection, null byte injection, quota overloaded Dos.

The main takeaways are the following ones: validate the filenames (reject filenames containing dangerous characters like “/” or “\”), setting a per-user upload quota, save the file to a non-accessible directory, create a filename reference map to link the actual file name to a machine generated name and use this generated file name.

Chapter 9: Logging, Error Handling and Intrusion Detection

Logging

What should be be logged: what happened, who did it, when it happened, what data have been modified, and what should not be logged: sensitive information like sessions IDs, personal informations.

Some logging frameworks for security are presented like OWASP ESAPI Logging and Logback. If you are interested in more details about the security logging you can check OWASP Logging Cheat Sheet.

Error Handling

On the error handling the main idea is to not leak to the external world stacktraces that could give valuable information about your application/infrastructure to an attacker. It is possible to prevent this by registering to the application level static pages for each type of error code or by exception type.

Intrusion Detection

The last part of the chapter is about techniques to help monitor end detect  and defend against different types of attacks. Besides the “craft yourself” solutions, the authors also re presenting the OWASP AppSensor application.

Chapter 10: Secure Software Development Lifecycle

The last chapter is about the SSDLC (Secure Software Development Life Cycle) and how the security could be included in each steps of development cycle. For me this chapter is not the best one but if you are interested about this topic I highly recommend the Software Security: Building Security in book (you can read my own review of the book here, here and here).

Book review: Continuous Enterprise Development in Java

This is a very short review of the Continuous Enterprise Development in Java book.

The book can be easily split in two parts.

The first part of the book from chapter 1 to chapter 4 javaArguillian contains general information about the difficulty of testing the JEE applications, about the software development cycles, the types of testing and some more technical details about the testing frameworks (JUnit and TestNG), about build tools like Maven and (JBoss) Forge, about version control (only Git deserves a paragraph) and (finally) about the Arquillian which is presented as “an innovative and highly extensible testing platform for the JVM”.

A very nice introduction is done to the ShrinkWrap  which is an API to create programmatically deployable JEE archives (jars, wars, ears).

An entire chapter (Chapter 3) is dedicated to write and deploy some business code and the associated Arquillian tests. Almost all the tools used in this chapter are JBoss or RedHat tools; Forge for the build of the application, JBoss Application Server to deploy the application, JBoss Developper Studio to deploy on the (Red Hat) OpenShift cloud service.

The second part of the book from chapter 4 to chapter 12 contains the implementation of the http://geekseek.continuousdev.org/app/root/show application, which is the JEE application. Every chapter is treating one aspect of the application: chapter 5  treats the persistence layer, chapter 6 the integration with NoSql data bases, chapter 7 the services layer, chapter 8 the REST services, chapter 9 the security, chapter 10 the user interface and chapter 11 the deployment on live.

Every chapter follow the same pattern, it starts with an introduction to the technology that will be used within the chapter, then the use cases and the business requirements are presented then it follows the implementation of the requirements and lastly the testing of the implementation using Arquillian.

I will conclude my ticket with a few points about what i like and what i don’t like about this book.

What i like about this book:

  • The author clearly masters the different JEE components; the technology introduction paragraphs of each chapter of the second part of the book are very clear and easy to understand.
  • The author knows the Arquillian framework inside out; all the examples are well explained and the introduction to ShrinkWrap is very well done.
  • Some of the chapters contains very valuable external references, like PicketLink for the security or Richardson Maturity Model for REST.

What i do not like about this book:

  • Too much marketing of the RedHat/JBoss products; I would have preferred to have more vendor agnostic examples of use for the Arquillian framework.
  • The subtitle of the book is “Testable Solutions with Arquillian” so it suppose to focus more on the testing part of the applications. Unfortunately, for me the book is not focusing on testing the applications but rather tries to  present how to continuously develop (JEE) applications and integration testing is only one part of this.
  • Nothing is said about the integration of Arquillian with other Java (non JEE) projects/frameworks like Spring and Guice and how Arquillian could ease (if it it can) the testing of the applications using these frameworks.