How to intercept and modify Java stacktraces

This ticket was triggered by a “simple” requirement: “Change all the package names in the logs of a Java application (especially the stacktraces) from ‘abc.efg’ (put here whatever you want as name) to ‘hij.klm’ (put here also whatever you want as name) “. The first idea that popped in my mind was to change the packages names at the code level, but this was not feasible because of (rather) big codebase, the use of the (Java) reflexion and the tight timeline.

In the following lines, I will discuss possible solutions to implement this (weird) requirement.

 

Extend the log4j ThrowableRenderer

If the project is using log4j1x as log library, then a solution would be to create your own throwable renderer by extending the org.apache.log4j.spi.ThrowableRenderer. The (log4j) renderers are used to render instances of java.lang.Throwable (exceptions and errors) into a string representation.

The custom renderer that replaces the packages starting with “org.github.cituadrian” by “xxx.yyy” will look like this:

package org.github.cituadrian.stacktraceinterceptor.log4j;

import org.apache.log4j.DefaultThrowableRenderer;
import org.apache.log4j.spi.ThrowableRenderer;

public class CustomThrowableRenderer implements ThrowableRenderer {
    private final DefaultThrowableRenderer defaultRenderer =  
                   new DefaultThrowableRenderer(); 
    
    @Override 
    public String[] doRender(Throwable t) {
      String[] initialResult = defaultRenderer.doRender(t); 
      for (int i = 0; i < initialResult.length; i++) { 
        String line = initialResult[i]; 
        if (line.contains("org.github.cituadrian")) { 
           initialResult[i] = line.replaceAll("org.github.cituadrian", "xxx.yyy"); 
        } 
       } 
      return initialResult; 
    }
}

Basically, the custom renderer is delegating the task of creating a String from a Throwable to a DefaultThrowableRenderer and then it checks and replace the desired package names.

In order to be used, the renderer should be defined in the log4j.xml file:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration debug="true"
    xmlns:log4j='http://jakarta.apache.org/log4j/'>
 
  <throwableRenderer class= 
     "org.github.cituadrian.stacktraceinterceptor.log4j.CustomThrowableRenderer"/>
...

Use a log4j2 pattern layout

If your project is using log4j2 as logging library, then you can use a (log4j2) layout pattern.  The layout pattern will look like:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
 <Appenders>
 <Console name="STDOUT" target="SYSTEM_OUT">
 <PatternLayout pattern=
  "%replace{%class %log %msg %ex}{org\.github\.cituadrian}{xxx\.yyy}"/>
 </Console>
...

 

Modify (a.k.a. Weaving) the java.lang.StackTraceElement class with AOP

Before even explaining what it really means, I have to warn you that weaving JDK classes is rarely necessary (and usually a bad idea) even if it’s possible using an AOP framework like AspectJ.

For this case I used the AspectJ as AOP framwork because the weaver (aop compiler) is able to do binary weaving, meaning the weaver takes classes and aspects in .class form and weaves them together to produce binary-compatible .class files that run in any Java VM. The command line to obtain a weaved jar is the following one:

ajc -inpath rt.jar Aspect.java -outjar weavedrt.jar

In the case of weaving JDK classes one extra step is necessary in order to make the application work; we must create a new version of the rt.jar file  or create just a small JAR file with the JDK woven classes which then must be appended to the boot-classpath of the JDK/JRE when firing up the target application. The command line to execute the target application is the following one:

java -Xbootclasspath/<path to weavedrt.jar>;<path to aspectjrt.jar> TargetApplication

If you don’t want to worry about all the technical details of weaving and executing the application and you are using Maven then you can use the (marvelous) SO_AJ_MavenWeaveJDK project from gitHub (that handles everything using Maven)

The aspect that will modify the stacktrace packages looks like:

package org.github.cituadrian.stacktraceinterceptor.app;

import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;
@Aspectpublic class StackTraceInterceptorAspect {
    @Pointcut("execution(String java.lang.StackTraceElement.getClassName()) "
            + "&& !within(StackTraceInterceptorAspect)")     
    public void executeStackTraceElementGetClassNamePointcut() {}        
    
    @Around("executeStackTraceElementGetClassNamePointcut()")    
    public Object executeStackTraceElementGetClassNameAdvice(    
                   final ProceedingJoinPoint pjp) throws Throwable {        
        Object initialResponse =  pjp.proceed();         
        if (initialResponse instanceof String 
               && ((String)initialResponse).startsWith("org.github.cituadrian")) {     
                 return ((String)initialResponse).replaceFirst("org.github.cituadrian", "xxx.zzz"); 
        }        
        return initialResponse;    
    }
 }

In a nutshell, the StackTraceInterceptorAspect will intercept all the calls to the java.lang.StackTraceElement#getClassName method and it will change the returned result of the method if the class name contains the string “org.github.cituadrian”.

If you are interested to learn more about AspectJ I really recommend you to buy a copy of the AspectJ in action (second edition) book.

 

Modify and shadow the java.lang.StackTraceElement class

 Using AOP just to intercept and modify a single method of a single class is a little bit over-killing. In this case there is another solution; the solution would be create a custom version of the java.lang.StackTraceElement class and add this custom class in the boot-classpath of the JDK/JRE when firing up the target application, so the initial version will be shadowed by the custom version.

An implementation of StacktraceElement class can be found here. So you can modify by hand the java.lang.StackTraceElement#getClassName method or the java.lang.StackTraceElement#toString method.

 To execute the target application, you must create a jar with the modified class and add it into the boot-classpath (something similar to the AspectJ solution):

java -Xbootclasspath/<path to custom class.jar> TargetApplication

 

 

Advertisement

How to find (buggy) calls of java.lang.Object.equals() with incompatible types

Introductionjavabug

All this started from a change (and a mistake) done by one of my co-workers that did a small change in the code with rather dramatic consequences;

So, imagine we have a (java) class T having a private field f of type String:

public class T {
    private Boolean f;

    public void setF(final Boolean f) {
        this.f = f;
    }
    public Boolean getF() {
        return f;
    }
    .... 
}

Now, imagine (again) that the type of f field will change from String to something else, let’s say Boolean; the compiler will help you to find the places where the setF method is called with the wrong parameter and also where the getF method do not comply with the new signature.

But there is at least one case where the compiler cannot help you; it is the case of the call of java.lang.Object.equals method. So something like “aString”.equals(t.getF()) in the case when the f field is a boolean will always return false because we try to check the equality for incompatible types.

The goal of this ticket is to find solutions to catch this kind of problem at runtime or (ideally) at compile time. Anyway, if you want to catch this kind of problem in your code and you do not want to reinvent the wheel (as I will do in this ticket), just use the FindBugs. FindBugs is capable to catch this kind of error (EC_UNRELATED_TYPES) and many, many more (see FindBugs Bug Descriptions).

How to find the buggy calls at runtime

The only way that I see to catch the buggy equals calls in a generic way is to use AOP. The basic idea is to write an aspect that will intercept all the calls to the equals methods and check that the caller and the callee are of the same type.

For the implementation I choose to use AspectJ framework (for a very good introduction to AspectJ I strongly recommend the AspectJ in action book) and the code it looks like this:

import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Before;
import org.aspectj.lang.annotation.Pointcut;
import org.github.adriancitu.equals.WrongUseOfEqualsException;

@Aspect
public class EqualsCheckerAspect {
    @Pointcut(
            "execution(public boolean Object+.equals(Object)) "
            + "&& target(target) "
            + "&& args(methodArgument)"
            + "&& !within(EqualsCheckerAspect)")
    public void executeEqualsPointcut(
            final Object target, 
            final Object methodArgument) {
    }
    
    @Before("executeEqualsPointcut(target,methodArgument)")
    public void executeEqualsAdvice(
            final Object target, 
            final Object methodArgument) throws Exception {
       
        if (target != null && !target.getClass().equals(methodArgument.getClass())) {
            throw new WrongUseOfEqualsException("Tried to call the equals methods on different classes types");
        }
    }
}

I will try briefly to explain what the previous code is doing; this is not a tutorial about AspectJ or AOP so, some of the definitions will be a little bit approximative, so AOP purists please forgive me.

The aspect (which is a Java class annotated with the @Aspect annotation) contains 2 parts; the pointcut (the method annotated with the @Pointcut annotation) and the advice (the method annotated with the @Before annotation).

The pointcut represents the parts of program flow that we want to intercept; in our case we want to intercept all the calls to the method having the following signature boolean equals(Object) on any instance of java.lang.Object class and on any subclasses (that’s the meaning of the + from the ..boolean Object+.equals). The  target and args are just AspectJ structures that helps to capture the caller and the callee of the equals method.

The advice represents the code that will be executed when a poincut is intercepted. In our  case we would like to do the verification that the caller and the callee are of the same type before the execution of the equals method (this is the meaning of the @Before annotation). The advice code represented by the executeEqualsAdvice method is quite simple, it just retrieve the caller and the callee objects (already captured by the target and args) and check that are instances of the same class.

This solution will fix the problem but it have some drawbacks:

  • the check is done at runtime
  • you have a dependency on AspectJ

How to find the buggy calls at compile time

It would be really nice to be able to catch this problem directly at compile time and eventually (not mandatory) having no other technical dependency.

Our salvation is coming from the javac plug-in mechanism which is new in Java8. The javac plug-in mechanism allows to a user to specify one or more plug-ins on the javac command line, to be started soon after the compilation has begun. There is no official tutorial about how to craft and use a javac plug-in, the only information that I’ve found is this Javadoc link and this blog ticket.

The javac plug-in mechanism gives access to the abstract syntax tree of the compiled program by implementing the visitor pattern so we will use this mechanism to catch and check the executions of the equals methods. The entire code of the plug-in can be found on this GitHub repository.

The steps of crafting a (javac) plug-in are the following ones:

  1. The entry point of the plug-in should implement the com.sun.source.util.Plugin interface. The code of the class can be found here : RuntimeEqualsCheckPlugin.java
  2. Create a class that implements the com.sun.source.util.TaskListener interface in order to perform additional behavior after the type-checking phase. The code of the class can be found here: EqualsMethodCallTaskListener.java
  3. Create an abstract syntax tree visitor by extending the com.sun.source.util.TreePathScanner<R,P>
    end extend the behavior for the method invocation. The code for this class can be found here: CodePatternTreeVisitor2.java

The most important part of CodePatternTreeVisitor2.java class is the overridden visitMethodInvocation method. I will show you the code but the most important lesson that I learned is that in the code you have to work with trees, everything is a subtype of the Tree interface.

public Void visitMethodInvocation(MethodInvocationTree methodInvocationTree, Void aVoid) {
  final List<? extends ExpressionTree> arguments = methodInvocationTree.getArguments();
  final ExpressionTree methodSelect = methodInvocationTree.getMethodSelect();
  switch (methodInvocationTree.getKind()) {
      case METHOD_INVOCATION:
        Tree.Kind methodSelectKind = methodSelect.getKind();
        switch (methodSelect.getKind()) {
            case MEMBER_SELECT:
             //t1.equals
             //or
             //field.equals
             MemberSelectTree memberSelectTree = (MemberSelectTree) methodSelect;
             //it's a equals method invocation
             if (isEqualsCall(
                  new TreePath(getCurrentPath(), methodSelect),
                  methodInvocationTree.getArguments() != null ?
                       methodInvocationTree.getArguments().size() : 0)) {
                //t1
                //or
                //field
                ExpressionTree expression = memberSelectTree.getExpression();
                TypeMirror callerType =
                  trees.getTypeMirror(
                            new TreePath(getCurrentPath(), expression));
                Optional argumentType =
                            omputeArgumentType(methodInvocationTree.getArguments());
                if (argumentType.isPresent() 
                      && !callerType.equals(argumentType.get())) {
                      System.err.println("Try to call equals on different parameters at line "
                          + getLineNumber(methodInvocationTree)
                          + " of file " +
                          currCompUnit.getSourceFile().getName()
                          + "; this is a bug!"
                                );
                 }
              }
          }
        }
        return super.visitMethodInvocation(methodInvocationTree, aVoid);
    }

The steps to execute the (javac) plug-in are the following ones:

  1. Set up a file called com.sun.source.util.Plugin located in META-INF/services/ of your plug-in code because the plug-ins is located via java.util.ServiceLoader. The content of the file should contain the name of the plug-in (com.github.adriancitu.equals.com.github.adriancitu.equals.runtime.RuntimeEqualsCheckPlugin in this case).
  2. Create a jar with all the plug-in files and the com.sun.source.util.Plugin file.
  3. Execute your plug-in against the javac using the  -processorpath flag to indicates the path where the plug-in JAR file is located and -Xplugin flag to indicate the name of the plug-in to run. In our case the command will be something like :
    javac -processorpath plugIn.jar
    -Xplugin:RuntimeEqualsCheckPlugin ./NoDependenciesTest.java

So finally was possible to have a compile time solution, the only drawback that I see for the plug-in solution is that the API is not very user friendly.

Book review: AspectJ in action – Second edition (Part 1 Understanding AOP and AspectJ)

Introduction and … conclusion

Since I’m a fan of AOP and especially of AspectJ, I participated to the MEAP (Manning Early Access Program) of the AspectJ in action – Second edition book. The review that I will done will be of the MEAP edition, so maybe the final version will be slightly different. So, my conclusion about this book is that it is the bible of AspectJ; it gives a very clear introduction to AOP and a tremendous view of the AspectJ functionalities.

Chapter 1:  Discovering AOP

The chapter explain what is AOP and how it can be integrated into the today IT systems. Then the author presents the basic ingredients of an AOP language(join point, pointcut, advice and aspect) and how an AOP language can help to improve the modularity of a complex system.

Chapter 2:  Introducing AspectJ

The chapter is a very gentle introduction to the AspectJ5 framework containing the classic “Hello world” application.For every basic block of an AOP language enumerated in the first chapter, the equivalent AspectJ5 structure is presented using some simple examples. Beside the traditional AspectJ syntax, the new syntax using the annotations is presented (this new language syntax is called @AspectJ) and a paragraph is dedicated to the integration of AspectJ with the Spring framework.

Chapter 3:  Understanding the Join Point Model

This chapter takes us in the heart of the AspectJ framework by presenting the join point model. The joint points are the parts of a Java class (and or) object that can be intercepted by the framework. The joint points exposed by AspectJ are: methods, constructors, field access (read and write access), exception processing, static initialization of classes and objects and AspectJ advices. For every joint point type the author gives very detailed explanations with many examples . For some join point types, a small comparison is made with the Spring AOP framework.

Chapter 4:  Modifying Behavior with Dynamic Crosscutting

The goal of the dynamic crosscutting is to modify the execution behavior of the program. The construct that is offered by AspectJ for dynamic crosscutting is the advice.
This chapter deeply explains the notion of advice. An advice is a structure that contains the code to be executed when a join point is intercepted. The reflection support in AspectJ also is present.The reflection support is represented by a small number of interfaces that provides access to the join point’s static and dynamic information.

Chapter 5:  Modifying Structure with Static Crosscutting

The goal of static crosscutting is to modify the static structure of the Java elements (classes, interfaces). AspectJ offers the following features:

  • Member introduction. Introduction of new fields(final and non-final) and methods to any Java class. There is no need to have the source codes of the modified classes for using member introduction.
  • Type hierarchy modifications. With AspectJ you can modify the inheritance hierarchy of existing classes to declare a superclass and interfaces of an existing class. Of course the AspectJ cannot break the rules of the Java language; cannot declare multiple inheritance or cannot declare a class being the parent of an interface.
  • Compile-time errors and warnings. This is similar to the #error and #warning preprocessor directives from the C/C++ world. A typical use of this feature will be to enforce rules, such as prohibiting calls to certain unsupported methods or issuing a warning about such calls.
  • Softening checked exception. Exception softening allows treating checked exceptions thrown by specified pointcuts as unchecked ones and thus eliminated the need to deal wit it. Well, for me this constructs is a real danger and it should never be used. Even the author, raise a warning about overusing this feature because it can lead to masking off some checked exceptions that should be handle in the normal way.

Chapter 6:  Aspect: Putting it all together

As the name of the chapter it said, all the previous elements (joinpoints and advices) are put together into a same structure that is called “aspect”. The aspects have some similarities with the Java classes:

  • can be abstract
  • have data and members
  • have access specification
  • implement or extend other types
  • be nested inside other types

The main differences with aspects and Java classes:

  • aspects cannot be explicitly created (the AspectJ framework controls the aspect initialization)
  • aspects can inherit only from abstract aspects and not concrete aspects
  • aspects can have access to the private members of the classes (using the access specifier “privileged”)

So, as it was already said, the user cannot directly instantiate the aspects but it can instruct the framework how and when to instantiate the aspects. AspectJ offers 4 kinds of aspect associations:

  • singleton (default). A single instance of an aspect is created for the entire application.
  • per object. A new aspect instance will be created when executing a join point of a matching object for the first time.
  • per control-flow. A new instance will be associated with each control flow associated with a join point
  • per type Similar as the per object association with the difference that the aspect will be associated to the class instead of objects.

The last part of the chapter explains the aspects precedence or how to control the order on which advices sharing the same joint point are executed. For aspect precedence, the AspectJ framework introduced the “precedence” keyword.

Example : declare precedence : Aspect1, Aspect2; // Aspect1 have the precedence over the Aspect2

For the case of advices from the same aspect sharing the same joint point, the lexical order is applied (the advice declared first have the precedence over the second one, etc…).

Chapter 7:  Diving into the @AspectJ Syntax

The AspectJ 5 (the last version of AspectJ) introduced support for the Java5 language features. The introductions of annotations in Java5 permitted to the AspectJ team to offer an alternative syntax to the “classical” AspectJ syntax. The newly syntax called @AspectJ offer a choice to compiling source code with a plain Java compiler because all the crosscutting information is kept in annotations on Java elements:

  • aspects are normal Java classes annotated with @Aspect annotation.
  • advices are Java methods annotated with the following annotations @Around, @After, @AfterReturning, @AfterThrowing, @Before.
  • pointcuts are Java methods annotated with the @Pointcut annotation
  • modifications of type hierarchy is done by annotating Java fields with @DeclareParents annotation.
  • error and warning declarations are done annotating Java fields with @DeclareError and @DeclareWarning annotations.

Unfortunately, not all the AspectJ feature are available using the @AspectJ syntax:

  • the member introduction; a workaround for this limitation is to use modification of type hierarchy by declaring as parent an interface using the @DeclareParents annotation with the attribute “defaultImpl” containing a class that implements the interface.
  • softening exceptions.
  • privileged aspects.

Chapter 8:  AspectJ Weaving Models

The AspectJ framework offers two types of weaving:

  • build-time weaving; the classes and aspects are weaved during the build process using the ajc compiler. The ajc compiler can take as input source files, class files and jar files each of which may contains classes and aspects.
  • load-time weaving (LTW);the aspects are weave into the classes as the classes are loaded by the classloader. The LTW uses another Java5 feature which is Java Virtual Machine Tool Interface (JVMTI). JVMTI allows a JVMTI agent to imtercept the loading of a class. The load-time weaver is a JVMTI agent that use this functionality to weave the classes as the VM loads them. If the @AspectJ syntax is used then the LTW must be used. The load-time weaver needs some additional information to decide which aspects to weave; this information should be in a XML format into the file META-INF/aop.xml (alternatives to aop.xml are also META-INF/aop-ajc.xml or org/aspectj/aop.xml file).

Basically, the aop.xml file should contains the list of aspects to weave, the list of aspects to exclude from weaving, the list of the packages of the classes to be modified by the aspects and the list of packages of classes that should not be weaved.

Chapter 9:  Integration with Spring

This chapter presents more than the integration of AspectJ into Spring framework; the chapter presents the Spring AOP framework.

Prior to Spring 2.0 version the only Spring AOP offered “schema-style” AOP support that consists into writing crosscutting information in the XML form (for more details about “schema style” aop support please see this: http://static.springframework.org/spring/docs/1.2.x/reference/aop.html). The “schema style” support is suitable for projects that use version of Spring prior to Spring 2.0 or your project don’t use Java5.
Since Spring 2.0, the Spinng AOP is capable to understand the @AspectJ syntax.
The Spring AOP is a proxy-based framework, Spring creates proxy classes for each bean that matches the criterias specified in pointcuts. The proxy classes will intercept the calls to the original object. Under the hood, Spring uses dynamic proxies mechanism for intercepting calls to a method of an interface or it uses the CGLIB-proxies if the method to intercept is not part of an interface.

Some drawbacks of the Spring AOP linked to the proxy-based nature:

  • method execution-only join points;the proxies can intercept only methods.
  • cannot intercept/advise final methods; since Java prevents overriding final methods, the dynamically created proxies cannot advise final methods.
  • bean-only crosscutting; the  proxy creation requires that Spring bean factory create the objects that needs to be advised, so Spring AOP works only on Spring beans.

Due to the proxy-based mechanism it is not possible to use the full AspectJ power; the following pointcuts are not available: call(),
initialization(), preinitialization(), staticinitialization(), get(), set(), handler(), adviceexecution(), withincode(), cflow(), cflowbelow(),
if(), @this(), @withincode().

On the other side, Spring AOP (since version 2.5)introduce a new pointcut for selecting Spring beans. The pointcut is bean(<name-pattern>) and the name-pattern follows the AspectJ matching rules for a name pattern with ‘*’ being the only allowed wildcard.

aoplib4j version 0.0.3 is out !

A new version of the aoplib4j is out. Since the last version 2 types of aspects have been added :

Beside the new version some other small changes happend :