How to programmatically set-up a (HTTP) proxy for a Selenium test

Context

In the context of a (Java) Selenium test it was needed to set-up a http proxy at the level of the browser. What I wanted to achieve it was exactly what is shown in the next picture but programmatically. In this specific case the proxy was BurpPro proxy but the same workflow can be applied for any kind of (http) proxy.

Solution

I know this is not really rocket science but I didn’t found elsewhere any clear explanation about how to do it. In my code the proxy url is injected via a (Java) system property called “proxy.url“.

And the  code looks like this:

String proxyUrl = System.getProperty("proxy.url");
if (proxyUrl != null) {
    Proxy proxy = new Proxy();
    proxy.setHttpProxy(proxyUrl);

    FirefoxOptions options = new FirefoxOptions();
    options.setProxy(proxy);
    
    driver = new FirefoxDriver(options);
} else {
    driver = new FirefoxDriver();
}

Book Review: Clean Architecture

This is the review of the Clean Architecture (A Craftsman’s Guide to Software Structure and Design) book.

(My) Conclusion

I personally have mixed feelings about this book; the first 4 parts of the book that presents the paradigms and different design principles are quite good (for me it contains all the theory that you need in order to tackle the IT architectural problems). You start reading from the first chapter and gradually you build knowledge on top of previous chapter/s.

On the other side, the part 5 and 6 of the book (which are representing the backbone of the book) have a different cognitive structure; the chapters are not really linked together, you cannot read and build on top of previous chapter/s because there is no coherency between chapters (some of the chapters are extended versions of blog tickets from https://8thlight.com/blog/).

The book explains very well the rules and patterns to apply in order to build an application easy to extend and test but the subjects like the scalability, availability and security that are qualities that an (every) application should have, are not treated at all.

Part I Introduction

The author tries to express the fact that good software design and (good) software architecture are intimately linked and that is very important to invest time and resources in having a good software design even if it looks like the project it advances slower.

The quality of the (software) design will influence the overall quality of the software product and to prove this the author comes with some figures/numbers (unfortunately there are reference to the source of this figures).

Part II Starting with the bricks: Programming Paradigms

The following programming paradigms are explained: Structured ProgrammingObject Oriented Programming and Functional Programming.

For each paradigm a brief history is done and also the author expresses how each paradigm characteristics can help and impact the software architecture.

  • the immutability characteristic of Functional Programming can help to simplify the design in respect of concurrency issues.
  • the polymorhism characteristic of Object Oriented Programming  can help the design to not care about the implementation details of the used components.
  • the Structured Programming helped us to decompose a (big) problem in smaller problems that can be then handled independently.

Part III Design Principles

This part is about the SOLID design principles; each one of these design principles are clearly explained using sometimes UML diagrams. The solid design principles are (usually) applied by software developers to write clean(er) code but  the author also explains how these principles can be applied to an architecture level:

  • SRP (Single Responsabilty Principle)  for a software developer is “A class should have only one reason to change.” but for an architect became “A module should be responsible to one, and only one author”.
  • OCP (Open-Closed principle) is translated in architectural terms by replacing the classes with high level components the goal being to arrange those components into a hierarchy that protects higher-level components from changes in lower-level components.
  • LSP (Liskov Substitution Principle) is translated in architectural terms by extending the interface concept from a programming language structure to gateways that different system components are using to communicate. The violation of substitutability of these gateways (interfaces) are causing the system architecture to be poluted.
  • ISP (Interface Segregation Principle) is translated in architectural terms by stating that generally is harmful that your systems depends on frameworks that has more features that you need.
  • DIP (Dependency Inversion Principle) is used to create architectural boundaries between different system components.

Part IV Component Principles

The components principles are categorized in two types: (component) cohesion and coupling.

The component cohesion principles are :

  • (REP) The Reuse/Release Equivalence Principle : This principle states that “The unit of reuse is the unit of release”. Classes and modules that are formed into a component must belong to a cohesive group and should be released together.
  • (CCP) The Common Closure Principle: This principle is actually the Single Responsibility Principle for components. The principle states that should gather into same component classes that changes for the same reason at the same time.
  • (CRP) The Common Reuse Principle: This principle states that “should not depend on things that you don’t need it”. This principle rather tell which classes should not be put together in the same module; classes that are not tightly bound to each other should not be in the sane component.

This principles are linked together and applying them could be contradictory. The following diagram express this contradiction; each edge express the cost hat it must be payed to abandon the principle for the opposite vertex.

The component coupling principles are:

  • (ADP) The Acyclic Dependencies Principle: The principle states that should have no cycle into the component dependency graph, the dependency graph should be a DAG (Directed Acyclic Graph). Solutions to eliminate dependencies cycles are: apply the Dependencies Injection Principle (DIP) or create a new component that will contain the classes that other components are depending on.
  • (SDP) The Stable Dependencies Principle: This principle states that modules that are intended to be easy to change should not be dependent on by modules that are harder to change. The component stability metric, called I (for instability) is computed in the following way: I = Incoming dependencies / (Incoming dependencies + Outgoing dependencies). So SDP can be restated as :the  I metric of a component should be larger than the I metric of the components that it depends on, a component should depend on more stable components only.
  • (SAP)The Stable Abstraction Principle: For this principle, the author introduces a new metric called abstractness which is defined as follow: A = Number of classes in the component / Number of abstract classes and interfaces in the component.  A value of 0 implies that the component have no abstract classes, a value of 1 implies that the component contains only abstract classes. The SAP principle sets up a relationship between stability (I) and abstractness (A) that have the form of a graph:

Part V Architecture

This part of the book is made of 14 chapters (almost 120 pages) and treats different aspects of a good architecture: how to define appropriate boundaries and layers (“Boundary Anatomy” chapter, “Partial Boundaries” chapter, “Layers and Boundaries” chapter, “The Test Boundary”), how to make a system that is easy to understand, develop (“The Clean Architecture” chapter, “Presenters and Humble Objects” chapter), maintain and deploy, how to organize components and services (“Screaming Architecture” chapter).

It would be very difficult to resume 120 pages in few phrases but the most important take-away would be the characteristics of a system produced by a good architecture:

  • independent of any frameworks – must see the (technical) frameworks as tools and the architecture should not depend of this frameworks (“Screaming Architecture” chapter develops and argued more about this topic).
  • testable – the business rules of the system should be testable without any external element.
  • independent of the UI – the UI can change without affecting the use cases of the system.
  • independent of the database – the business rules/ use cases should not be bounded to any database.

Clean Architecture

The golden rule for a clean architecture is: Source code dependencies must point only inward toward higher-level policies; any item from a circle should know nothing about the items from outer circle/s. (see the following image).

For more information for the earlier concept of Clean architecture you can check the Uncle Bob initial blog post: The Clean Architecture.

Part VI Details

The last part of the book tries to explain why some of the (technological) items used in it projects like the database, the UI technology or (technical) frameworks should not influence/contaminate the system architecture and it should always be positioned at the outer circle (see the previous image). This part also has a case study on which some of the rules and thoughts about architecture are put together and applied.

 

How to intercept and modify Java stacktraces

This ticket was triggered by a “simple” requirement: “Change all the package names in the logs of a Java application (especially the stacktraces) from ‘abc.efg’ (put here whatever you want as name) to ‘hij.klm’ (put here also whatever you want as name) “. The first idea that popped in my mind was to change the packages names at the code level, but this was not feasible because of (rather) big codebase, the use of the (Java) reflexion and the tight timeline.

In the following lines, I will discuss possible solutions to implement this (weird) requirement.

 

Extend the log4j ThrowableRenderer

If the project is using log4j1x as log library, then a solution would be to create your own throwable renderer by extending the org.apache.log4j.spi.ThrowableRenderer. The (log4j) renderers are used to render instances of java.lang.Throwable (exceptions and errors) into a string representation.

The custom renderer that replaces the packages starting with “org.github.cituadrian” by “xxx.yyy” will look like this:

package org.github.cituadrian.stacktraceinterceptor.log4j;

import org.apache.log4j.DefaultThrowableRenderer;
import org.apache.log4j.spi.ThrowableRenderer;

public class CustomThrowableRenderer implements ThrowableRenderer {
    private final DefaultThrowableRenderer defaultRenderer =  
                   new DefaultThrowableRenderer(); 
    
    @Override 
    public String[] doRender(Throwable t) {
      String[] initialResult = defaultRenderer.doRender(t); 
      for (int i = 0; i < initialResult.length; i++) { 
        String line = initialResult[i]; 
        if (line.contains("org.github.cituadrian")) { 
           initialResult[i] = line.replaceAll("org.github.cituadrian", "xxx.yyy"); 
        } 
       } 
      return initialResult; 
    }
}

Basically, the custom renderer is delegating the task of creating a String from a Throwable to a DefaultThrowableRenderer and then it checks and replace the desired package names.

In order to be used, the renderer should be defined in the log4j.xml file:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration debug="true"
    xmlns:log4j='http://jakarta.apache.org/log4j/'>
 
  <throwableRenderer class= 
     "org.github.cituadrian.stacktraceinterceptor.log4j.CustomThrowableRenderer"/>
...

Use a log4j2 pattern layout

If your project is using log4j2 as logging library, then you can use a (log4j2) layout pattern.  The layout pattern will look like:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
 <Appenders>
 <Console name="STDOUT" target="SYSTEM_OUT">
 <PatternLayout pattern=
  "%replace{%class %log %msg %ex}{org\.github\.cituadrian}{xxx\.yyy}"/>
 </Console>
...

 

Modify (a.k.a. Weaving) the java.lang.StackTraceElement class with AOP

Before even explaining what it really means, I have to warn you that weaving JDK classes is rarely necessary (and usually a bad idea) even if it’s possible using an AOP framework like AspectJ.

For this case I used the AspectJ as AOP framwork because the weaver (aop compiler) is able to do binary weaving, meaning the weaver takes classes and aspects in .class form and weaves them together to produce binary-compatible .class files that run in any Java VM. The command line to obtain a weaved jar is the following one:

ajc -inpath rt.jar Aspect.java -outjar weavedrt.jar

In the case of weaving JDK classes one extra step is necessary in order to make the application work; we must create a new version of the rt.jar file  or create just a small JAR file with the JDK woven classes which then must be appended to the boot-classpath of the JDK/JRE when firing up the target application. The command line to execute the target application is the following one:

java -Xbootclasspath/<path to weavedrt.jar>;<path to aspectjrt.jar> TargetApplication

If you don’t want to worry about all the technical details of weaving and executing the application and you are using Maven then you can use the (marvelous) SO_AJ_MavenWeaveJDK project from gitHub (that handles everything using Maven)

The aspect that will modify the stacktrace packages looks like:

package org.github.cituadrian.stacktraceinterceptor.app;

import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;
@Aspectpublic class StackTraceInterceptorAspect {
    @Pointcut("execution(String java.lang.StackTraceElement.getClassName()) "
            + "&& !within(StackTraceInterceptorAspect)")     
    public void executeStackTraceElementGetClassNamePointcut() {}        
    
    @Around("executeStackTraceElementGetClassNamePointcut()")    
    public Object executeStackTraceElementGetClassNameAdvice(    
                   final ProceedingJoinPoint pjp) throws Throwable {        
        Object initialResponse =  pjp.proceed();         
        if (initialResponse instanceof String 
               && ((String)initialResponse).startsWith("org.github.cituadrian")) {     
                 return ((String)initialResponse).replaceFirst("org.github.cituadrian", "xxx.zzz"); 
        }        
        return initialResponse;    
    }
 }

In a nutshell, the StackTraceInterceptorAspect will intercept all the calls to the java.lang.StackTraceElement#getClassName method and it will change the returned result of the method if the class name contains the string “org.github.cituadrian”.

If you are interested to learn more about AspectJ I really recommend you to buy a copy of the AspectJ in action (second edition) book.

 

Modify and shadow the java.lang.StackTraceElement class

 Using AOP just to intercept and modify a single method of a single class is a little bit over-killing. In this case there is another solution; the solution would be create a custom version of the java.lang.StackTraceElement class and add this custom class in the boot-classpath of the JDK/JRE when firing up the target application, so the initial version will be shadowed by the custom version.

An implementation of StacktraceElement class can be found here. So you can modify by hand the java.lang.StackTraceElement#getClassName method or the java.lang.StackTraceElement#toString method.

 To execute the target application, you must create a jar with the modified class and add it into the boot-classpath (something similar to the AspectJ solution):

java -Xbootclasspath/<path to custom class.jar> TargetApplication

 

 

5 (software) security books that every (software) developer should read

I must admit that the title is a little bit catchy; a better title would have been “5 software security books that every developer should be aware of“. Depending on your interest you might want to read entirely these books or you could just know that they exists. There must be tons of software security books on the market but this is my short list of books about software security that I think that each developer that is interested in software security should be aware of.

Hacking – the art of exploitation This book explains the basics of different hacking techniques, especially the non-web hacking techniques: how to find vulnerabilities (and defend against)  like buffer overflow or stack-based buffer overflow , how to write shellcodes, some basic concepts on cryptography and attacks linked to the cryptography like the man-in-the-middle attack of an SSL connection. The author tried to make the text easy for non-technical peoples but some programming experience is required (ideally C/C++) in order to get the best of this book. You can see my full review of the book here.

Iron-Clad Java: Building secure web applications This book presents the hacking techniques and the countermeasures for the web applications; you can see this books as complementary of the previous one; the first one contains the non-web hacking techniques, this one contains (only) web hacking techniques; XSS, CSRF, how to protect data at rest, SQL injection and other types of injections attacks. In order to get the most of the book some Java knowledge is required. You can see my full review of the book here.

Software Security-Building security in  This books explains how to introduce the security into the SDLC; how to introduce abuse cases and security requirements in the requirements phase, how to introduce risk analysis (also known as Threat Modeling) in the design phase and software qualification phase. I really think that each software developer should at least read the first chapter of the book where the authors explains why the old way of securing application (seeing the software applications as “black boxes” than can be protected using firewalls and IDS/IPS) it cannot work anymore in the today software landscape. You can see my full review of the book here: Part 1, Part 2 and Part 3.

The Tangled Web: A Guide to Securing Modern Web Applications This is another technical book about security on which you will not see a single line of code (the Software Security-Building security in is another one) but it highly instructive especially if you are a web developer. The book presents all the “bricks” of the today Internet: HTTP, WWW, HTML, Cookies, Scripting languages, how these bricks are implemented in different browsers and especially how the browsers are implementing the security mechanism against rogue applications. You can see my full review of the book here.

Threat modeling – designing for security Threat modeling techniques (also known as Architectural Risk Analysis) were around for some time but what it has changed in the last years is the accessibility of these technique for the software developers.  This book is one of the reasons for which the threat modeling is accessible to the developers. The book is very dense but it  suppose that you have no knowledge about the subject. If you are interested in the threat modeling topic you can check this ticket: threat modeling for mere mortals.

A Java implementation of CSRF mitigation using “double submit cookie” pattern

Goal of this articlecsrf

The goal of this article is to present an implementation of the “double submit cookie” pattern used to mitigate the Cross Site Request Forgery (CSRF) attacks. The proposed implementation is a Java filter plus a few auxiliary classes and it is (obviously) suitable for projects using the Java language as back-end technology.

Definition of CSRF and possible mitigations

In the case of a CSRF attack, the browser is tricked into making unauthorized requests on the victim’s behalf, without the victim’s knowledge. The general attack scenario contains the following steps:

  1. the victim connects to the vulnerable web-site, so it have a real, authenticated session.
  2. the hacker force the victim (usually using a spam/fishing email) to navigate to another (evil) web-site containing the CSRF attack.
  3. when the victim browser execute the (evil) web-site page, the browser will execute a (fraudulent) request to the vulnerable web-site using the user authenticated session. The user is not aware at all of the fact that navigating on the (evil) web-site will trigger an action on the vulnerable web-site.

For deeper explanations I strongly recommend  to read chapter 5 of Iron-Clad Java: Building Secure Applications book and/or the OWASP Cross-Site Request Forgery (CSRF) Prevention Cheat Sheet.

Definition of “double submit cookie” pattern

When a user authenticates to a site, the site should generate a (cryptographically strong) pseudo-random value and set it as a cookie on the user’s machine separate from the session id. The server does not have to save this value in any way, that’s why this patterns is sometimes also called Stateless CSRF Defense.

The site then requires that every transaction request include this random value as a hidden form value (or other request parameter). A cross origin attacker cannot read any data sent from the server or modify cookie values, per the same-origin policy.

In the case of this mitigation technique the job of the client is very simple; just retrieve the CSRF cookie from the response and add it into a special header to all the requests:

csrfclient
Client workflow

The job of the server is a little more complex; create the CSRF cookie and for each request asking for a protected resource, check that the CSRF cookie and the CSRF header of the request are matching:

csrfserver
Server workflow

Note that some JavaScript frameworks like AngularJS implements the client worflow out of the box; see Cross Site Request Forgery (XSRF) Protection

Java implementation of “double submit cookie” pattern

The proposed implementation is on the form of a (Java) Servlet filter and can be found here: GenericCSRFFilter GitHub.

In order to use the filter, you must define it into you web.xml file:

<filter>
   <filter-name>CSRFFilter</filter-name>
   <filter-class>com.github.adriancitu.csrf.GenericCSRFStatelessFilter</filter-class>
<filter>

<filter-mapping>
   <filter-name>CSRFFilter</filter-name>
   <url-pattern>/*</url-pattern>
</filter-mapping>

 

The filter can have 2 optional initialization parameters: csrfCookieName representing the name of the cookie that will store the CSRF token and csrfHeaderName representing the name of the HTTP header that will be also contains the CSRF token.

The default values for these parameters are “XSRF-TOKEN” for the csrfCookieName and “X-XSRF-TOKEN” for the csrhHeaderName, both of them being the default values that AngularJS is expecting to have in order to implement the CSRF protection.

By default the filter have the following features:

  • works with AngularJS.
  • the CSRF token will be a random UUID.
  • all the resources that are NOT accessed through a GET request method will be CSRF protected.
  • the CSRF cookie is replaced after each non GET request method.

How it’s working under the hood

The most of the functionality is in the GenericCSRFStatelessFilter#doFilter method; here is the sequence diagram that explains what’s happening in this method:

doFilter method sequence diagram
doFilter method sequence diagram

The doFilter method is executed on each HTTP request:

  1. The filter creates an instance of ExecutionContext class; this class is a simple POJO containing the initial HTTP request, the HTTP response, the CSRF cookies (if more than one cookie with the csrfCookieName is present) and implementation of the ResourceCheckerHook , TokenBuilderHook and ResponseBuilderHook .(see the next section for the meaning of this classes).
  2. The filter check the status of the HTTP resource, that status can be:MUST_NOT_BE_PROTECTED, MUST_BE_PROTECTED_BUT_NO_COOKIE_ATTACHED,MUST_BE_PROTECTED_AND_COOKIE_ATTACHED (see ResourceStatus enum) using an instance of ResourceCheckerHook.
  3. If the resource status is ResourceStatus#MUST_NOT_BE_PROTECTED
    ResourceStatus#MUST_BE_PROTECTED_BUT_NO_COOKIE_ATTACHED then
    the filter creates a CSRF cookie having as token the token generated by an instance of TokenBuilderHook.
  4. if the resource status ResourceStatus#MUST_BE_PROTECTED_AND_COOKIE_ATTACHED
    then compute the CSRFStatus of the resource and then use an instance of ResponseBuilderHook to return the response to the client.

How to extend the default behavior

It is possible to extend or overwrite the default behavior by implementing the hooks interfaces. All the hooks implementations must be thread safe.

  1. The ResourceCheckerHook is used to check the status of a requested resource. The default implementation is DefaultResourceCheckerHookImpl and it will return ResourceStatus#MUST_NOT_BE_PROTECTED for any HTTP GET method, for all the other request types, it will return {@link ResourceStatus#MUST_BE_PROTECTED_BUT_NO_COOKIE_ATTACHED if any CSRF cookie is present in the query or ResourceStatus#MUST_BE_PROTECTED_BUT_NO_COOKIE_ATTACHED otherwise.The interface signature is the following one:
    public interface ResourceCheckerHook extends Closeable {
        ResourceStatus checkResourceStatus(ExecutionContext executionContext);
    }  
  2. The TokenBuilderHook hook is used to generate the token that will be used to create the CSRF cookie. The default implementation  is DefaultTokenBuilderHookImpl and it uses a call to UUID.randomUUID to fetch a token. The interface signature is the following one:
    public interface TokenBuilderHook extends Closeable {
        String buildToken(ExecutionContext executionContext);
    }
  3. The ResponseBuilderHook is used to generate the response to the client depending of the CSRFStatus of the resource. The default implementation is DefaultResponseBuilderHookImpl and it throws a SecurityException if the CSRF status is CSRFStatus#COOKIE_NOT_PRESENT, CSRFStatus#HEADER_TOKEN_NOT_PRESENT or CSRFStatus#COOKIE_TOKEN_AND_HEADER_TOKEN_MISMATCH. If the CSRF status is CSRFStatus#COOKIE_TOKEN_AND_HEADER_TOKEN_MATCH then the old CSRF cookies are deleted and a new CSRF cookie is created. The interface signature is the following one:
    public interface ResponseBuilderHook extends Closeable {
        ServletResponse buildResponse(ExecutionContext executionContext,
                                      CSRFStatus status);
    }
    

The hooks are instantiated inside the GenericCSRFStatelessFilter#init method using the ServiceLoader Java 6 loading facility. So if you want to use your implementation of one of the hooks then you have to create a  META-INF/services directory that contains a text file whose name matches the fully-qualified interface class name of the hook that you want to replace.

Here is the sequence diagram representing the hooks initializations:

initmethod