Integrating applications, data, and automated workflow presents a uniquely entrenched set of dependencies for software developers. The need for an endpoint to exist in order to integrate with it seems basic and undeniable. Additionally, the data that an endpoint produces or consumes needs be available for the integration to work. It turns out that these constraints are in fact the goal of integration.

So given that integration the endpoints must be available and functioning in order to send and receive data, and that they are external to one another and middleware, how then can we develop the integration code without these dependencies being in place at all times? Furthermore, how do we test messaging without having actual messages, and how do we isolate our routes from the things that they integrate?

These questions seem rhetorical and yet there is a very real “chicken-egg” problem inherent in the general domain of integration. How to develop integration, without the things you need to integrate?

The concepts for good developer practices are ubiquitous. Developing under test with a set of specifications describing integration logic is in reality no different from any other type of software development.

In this article, I am going to walk you through exactly how to do this in a simple series of concepts and steps that will enable you to fly as an integration developer. But first, we need to level our knowledge regarding the following concerns:

1. Conceptual Ingredients
2. Architectural Patterns
3. Framework Facilities
4. Testing Idioms

Designing applications to be testable

Applications in general must be designed with affordances that enable them to be automatically tested. The ease or difficulty of testing an application is most directly impacted by how it’s architected. In order to achieve the benefit of designing an application to be tested, the best approach is to design it under test from the very beginning, ala Test-Driven-Development.

By writing a test for a required facility before creating the facility, we guarantee that it will be testable. Seems too obvious and yet, many developers skip straight to the implementation, writing the required code, only to find that it’s design is difficult to cleanly test. Integration code is actually no different.

Just because it uses facilities external to itself, doesn’t mean that it cannot be designed with clean seams and isolatable subsystems. To this end, TDD of integration code can yield excellent results, not to mention let you go fast, get it right quickly, and ensure that the design is not a snarled up ball of mud, deeply coupled to a network, external servers, databases, the web, or other external things. Finally, there is an advantage of having the tests run as part of your build and Continuous Integration pipeline. This is like having living armor to ensure that as you continue to develop and evolve the application, you have a set of automated alarm bells that will go off when problems are unintentionally introduced. When tests fail, you are alerted immediately and are presented with the opportunity to fix the cause fast. Much of what I’ve said here, is just common wisdom related to test-first mentality, however the point is that integration applications are just applications in that they run, process, and change the state of data. This makes them ideal for TDD, which allows you to focus on calibrating the internal logic and design directly to your requirements.

The key to successful test-driven integration development and reaping its benefits is to understand what facilities exist within your application framework for design and testing. Spring has most of the architectural concerns already thought out for you. Let’s take a quick survey of what’s important.

Object lifecycles

Singleton – created only once in the application runtime
Prototype – a new version is created everytime a class is
Scoped – new versions of a given class are created and destroyed based on an orthagonal scope, e.g. WebRequest

Managed versus Unmanaged objects

Managed Objects – objects whose lifecycle has been delegated to the application framework, e.g. Spring Framework. Note that the default lifecycle in Spring is Singleton. This is by design in order to encourage an architecture that instantiates an object facility once and then reuses it throughout the running of the app.

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:amq="http://activemq.apache.org/schema/core" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">


   <bean id="xmlProcessingService" class="com.mycompany.integration.project.services.XmlProcessingService"></bean>
 

</beans>

Unmanaged Objects – objects whose lifecycle will be handled by the developer

import com.mycompany.integration.project.services.XmlProcessingService;

XmlProcessingService xmlProcessingService = new XmlProcessingService();

Patterns of intent are key to determining what lifecycle any given object in your domain should have. If there are class-level fields that are intended to change on a per usage basis, then that bean should be declared as a prototype. However, this does not mean that a bean with fields cannot be a singleton. It should be singleton only if those fields are set once and not changed again. This promotes clean state sharing across concurrency boundaries within the system. We don’t want to be changing state out from under an object shared between threads, nor do we want to have to deal with locking, mutexes, or potential race conditions. By simply being mindful of what the lifecycle of an object should be, we are safe to use it within that scope.

Domain Facilities

Domain Services –
represent transactional concerns in an application
reponsibilities are narrowly focused
promote composability and reusability
generally are managed objects (usually singletons)

Utilities –
represent small reusable concerns
generally not managed objects

Routes –
represent transition of data
represent transactional pipelines
a pipeline is composed of routing steps
routing steps are usages of facilities of the camel framework

The ingredients of a Camel Route

• components
• processors
• beans (managed)

Now lets have a look at our example application. We will use it to demonstrate the architecture. Note that testing is purposefully kept minimalistic in order to highlight the salient concepts and reduce distractions.

Example Application

To setup the context of our environment, let’s have a look at what a reasonable project structure looks like. In our case, we will be using a Maven-based project archetype, with the Fuse nature enabled in Eclipse. We’ll use good-old JUnit for our work here. The good news is that Spring and Camel together provide excellent test support classes. These will make our testing convenient and straight forward.

Models

This app will integrate data related to Customers and Orders, using a sample Xml payload and corresponding model classes. These have been generated from an Xml Schema Definition (xsd) available here. I have pregenerated the model classes from the xsd.

They have been filed into a namespace dedicated to models.

package com.mycompany.integration.project.models;

Domain Services

There are 2 domain services.

XmlProcessingService – responsible for processing message body contents as XmlDocument.
ModelProcessingService – repsonsible for processing message body contents as an Object Graph.

They have been organized into a package dedicated to services.

package com.mycompany.integration.project.services;

Utilities

There are a handful of simple utilities in this application.

ModelBuilder – helper for serialization/deserialization concerns.
FileUtils – helper for interacting with filesystems.
Perf – helper for timing execution of code.
StackTraceInfo – helper for inferring the context of code execution at runtime.

And as you would guess utilities have been organized into a package dedicated to them.

package com.mycompany.integration.project.utils;

The project structure looks like this.

It is generally a good idea to organize your ApplicationContext into logically similar units as well. I prefer a role-based approach. When designing the composition model, I ask myself, “what kind of object is this that I’m declaring as a bean?”. The answer to this question usually yields a clear answer to the following concerns:

1. Is there already a Spring context file for this kind of thing?
2. Do I need to create a new context file for this kind of thing and if so what is the category of this kind of thing? The name of the context file should align to the category.

Have a look at the composition model of the context files in our project.

Unit Testing

There is a world of highly-opinionated approaches to testing, and they are all right. In this article, I want to focus on the nuts and bolts. Specifically, we are going to focus on unit testing the 3 categories of things of we discussed earlier, Utilities, Domain Services, and Routes.

Testing Utilities

Example:

package com.mycompany.integration.project.tests.utils;

import org.junit.Test;

import com.mycompany.integration.project.models.*;
import com.mycompany.integration.project.utils.*;

public class ModelBuilderTests {

	String xmlFilePath = "src/exemplar/CustomersOrders-v.1.0.0.xml";

	@Test
	public void test_can_fast_deserialize() throws Exception {
		// arrange
		String xmlString = FileUtils.getFileString(xmlFilePath);

		// act
		CustomersOrders customersOrders = CustomersOrders.class
				.cast(ModelBuilder.fastDeserialize(xmlString,
						CustomersOrders.class));

		// assert
		assert (customersOrders != null);
	}

	@Test
	public void test_can_deserialize() throws Exception {
		// arrange
		String axmlString = FileUtils.getFileString(xmlFilePath);

		// act
		CustomersOrders customersOrders = CustomersOrders.class
				.cast(ModelBuilder.deserialize(axmlString,
						CustomersOrders.class));

		// assert
		assert (customersOrders != null);
	}
}

Testing utilities can be quite easy. Since the purpose of utilities is to function as stand-alone units of functionality, isolating them as a SUT (System Under Test) is not difficult. Standard black-box testing of input and output apply.

Testing Domain Services

Domain services usually represent transactional components. They are generally stateless and provide simple facilities to handle single or related sets of responsibilities. Collaborators are objects used by a given domain service. They are often other domain services, e.g. a PurchasingService collaborates with an OrderingService. When unit testing a single domain service collaborators are usually mocked to isolate the service as a SUT. We will look at mocking in detail later.

Example:

package com.mycompany.integration.project.tests.services;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.annotation.DirtiesContext;
import org.springframework.test.annotation.DirtiesContext.ClassMode;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import com.mycompany.integration.project.services.XmlProcessingService;
import com.mycompany.integration.project.utils.FileUtils;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations={"classpath:META-INF/spring/domain.xml"})
@DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)
public class XmlProcessingServiceTests {

	String xmlFilePath = "src/exemplar/CustomersOrders-v.1.0.0.xml";
	
	@Autowired
    private XmlProcessingService xmlProcessingService;
	
	@Test
	public void test_service_gets_injected() throws Exception {
		assert(xmlProcessingService != null);
	}

	@Test
	public void test_process_an_xml_transaction() throws Exception {
		// Arrange
		String xml = FileUtils.getFileString(xmlFilePath);
		
		// Act
		Boolean result = xmlProcessingService.processTransaction(xml);
		
		// Assert
		assert(result);
	}
}

Note that we are instructing JUnit to run the tests in this class with:

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations={“classpath:META-INF/spring/domain.xml“})
@DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)

These annotations tell JUnit to load the ApplicationContext file domain.xml from the classpath and to reset the context to its default state after each run. The later ensures that we don’t bleed state between tests. We don’t even need a setUp() method in the class because of these annotations.

Now, because this class is Spring aware and Spring managed, the XmlProcessingService instance gets automatically @Autowired into it, through another simple annotation. This facility allows complex composition of Domain Services and their Collaborators to be handled by your Spring configuration, while you the developer just pull in what you need and test it.

A final important distinction of domain services is that they should always be Singleton Managed Objects. This means that our application framework (Spring), will be the creator and custodian of these objects. Whenever we need one, we’ll either ask the Spring ApplicationContext via service lookup or have it injected as a dependency. Composing an application in Spring is actually quite straight forward, but is outside the scope of our present study. If you want to know more about it, read up on it here.

Testing Camel Routes Expressed In The Spring DSL

It’s important to remember when testing a CamelContext that the SUT we are interested in is the Route. Well what is the route comprised of? It’s a set of steps, specifically step-wise treatments that are applied to messages as they traverse the route. Thus, what we are interested in testing are that these steps happen, that they are correctly sequenced, and that together they produce the desired result. The state that we examine in routing tests is the Message itself, and sometimes the Camel Routing Exchange.

The killer trick for testing a CamelContext with routes declared in Spring DSL is this:

In your src/test/resources You need to create a test-camel-context.xml that imports your real camel-context.xml from the classpath. Then in the test-camel-context.xml file you add the InterceptSendToMockEndpointStrategy bean to “mock all endpoints”, and you add a DirectComponent to override your activemq broker bean definition from the real camel-context.xml.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">

	<!-- the Camel route is defined in another XML file -->
	<import resource="classpath:META-INF/spring/camel-context.xml"></import>
	
	<!-- mock endpoint and override activemq bean with a noop stub (DirectComponent) -->
	<bean id="mockAllEndpoints" class="org.apache.camel.impl.InterceptSendToMockEndpointStrategy"></bean>
    <bean id="activemq" class="org.apache.camel.component.direct.DirectComponent"></bean>
</beans>

This in effect mocks all endpoints and provides a fake broker bean, so that you don’t have to have an instance of ActiveMQ actually available. This is what I mean by isolating your tests away from their integration points. This Spring ApplicationContext can now provide UnitTesting in a vacuum.

Route Testing Examples

Before diving in and showing off the code, its worth taking a step back here and asking, “what is it that we are going to want to know to determine if our routes are functioning as intended?”. The answer is again kind of straight down the middle of standard unit testing’s goals. We have state that should be pushed through a pipeline of routing steps. It travels in the form of a Camel Message. The structure of a message is just like anything a courier would carry for us. It has a payload (the body), and meta-data describing the payload in the form of the message’s headers.

{ 
  message:
    headers:
      status: "SUCCESS"
      foo: "bar"
    body:
      "Hi, I am a message payload..."
}

These are simple key-value pair data structures that help both the developer and Camel get the message to the right place, and ensure that everything went as expected. Thus, the idea of having expectations about correctness of a message at various stages in the route is at the heart of route testing. Luckily Camel includes the CamelSpringTestSupport class, which gives us an api with expectation-based semantics. With it driving our routing tests, we simply tell the test framework what expectations we have about the message and then feed the route an example message we want it to process. If all of the expectations are met, then the test passes. Otherwise the framework tells us which ones were not met..

Example camel-context.xml:

<!-- Configures the Camel Context -->
<camelContext id="customers_and_orders_processing" xmlns="http://camel.apache.org/schema/spring">
	<route id="process_messages_as_models">
		<from uri="file:src/data1" />
		<process ref="customersOrdersModelProcessor" id="process_as_model" />
		<to uri="file:target/output1" />
	</route>
	<route id="process_messages_as_xml">
		<from uri="file:src/data2" />
		<process ref="customersOrdersXmlDocumentProcessor" id="process_as_xml" />
		<to uri="file:target/output2" />
	</route>
	<route id="process_jetty_messages_as_xml">
		<from uri="jetty:http://0.0.0.0:8888/myapp/myservice/?sessionSupport=true" />
		<process ref="customersOrdersXmlDocumentProcessor" id="process_jetty_input_as_xml" />
		<to uri="file:target/output3" />
		<transform>
			<simple>
				OK
			</simple>
		</transform>
	</route>
</camelContext>

Below is the test class for this Camel Context. Note the naming convention alignment of route Id to test method name. Observing this convention will make clear which tests are targeting which routes. This is nice because the test output in your build reports will be easy to read and understand.

Example Route 1: process_messages_as_models

process_messages_as_models_test() -> process_messages_as_models
It expects the route to run File to File through the Model Deserialization Processor.

Example Route 2: process_messages_as_xml

process_messages_as_xml_test() -> process_messages_as_xml
It expects the route to run File to File through the XmlDocument manipulation Processor.

Example Route 3: process_jetty_messages_as_xml

Expects Route to Be Http to File through XmlDocument manipulation Processor
process_jetty_messages_as_xml_test() -> process_jetty_messages_as_xml

Example RoutTester.java:

package com.mycompany.integration.project.tests.routes;

import org.apache.camel.CamelContext;
import org.apache.camel.ConsumerTemplate;
import org.apache.camel.ProducerTemplate;
import org.apache.camel.component.mock.MockEndpoint;
import org.apache.camel.spring.SpringCamelContext;
import org.apache.camel.test.spring.CamelSpringTestSupport;
import org.junit.Before;
import org.junit.Ignore;
import org.junit.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.support.AbstractXmlApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

import com.mycompany.integration.project.models.*;
import com.mycompany.integration.project.utils.FileUtils;
import com.mycompany.integration.project.utils.StackTraceInfo;

public class RouteTester extends CamelSpringTestSupport {

	public String testXmlContextPath = "/test-camel-context.xml";
	
	
	@Autowired
	protected CamelContext camelContext;

	@Override
	public String isMockEndpoints() {
		// override this method and return the pattern for which endpoints to
		// mock.
		// use * to indicate all
		return "*";
	}

	private ProducerTemplate producer;
	private ConsumerTemplate consumer;

	protected CamelContext getCamelContext() throws Exception {
		applicationContext = createApplicationContext();
		return SpringCamelContext.springCamelContext(applicationContext);
	}

	@Override
	protected AbstractXmlApplicationContext createApplicationContext() {
		return new ClassPathXmlApplicationContext(testXmlContextPath);
	}

	
	String inputExemplarFilePath = "src/exemplar/CustomersOrders-v.1.0.0.xml";
	String inputExemplar;
	
	String outputExemplarFilePath = "src/exemplar/CustomersOrders-v.1.0.0-transformed.xml";
	String outputExemplar;
	
	@Before
	public void setUp() throws Exception {

		System.out.println("Calling setUp");
		
		// load i/o exemplars
		inputExemplar = FileUtils.getFileString(inputExemplarFilePath);
		outputExemplar = FileUtils.getFileString(outputExemplarFilePath);
		
		camelContext = getCamelContext();

		camelContext.start();

		producer = camelContext.createProducerTemplate();
		consumer = camelContext.createConsumerTemplate();

	}

	@Test
	public void process_messages_as_models_test() throws Exception {

		System.out.println("Calling " + StackTraceInfo.getCurrentMethodName());
		
		String inputUri = "file:src/data1";
		String outputUri = "file:target/output1";
		
		// Set expectations
		int SEND_COUNT = 1;
		
		MockEndpoint mockOutput = camelContext.getEndpoint("mock:" + outputUri, MockEndpoint.class);
		//mockOutput.expectedBodiesReceived(outputExemplar);
		mockOutput.expectedHeaderReceived("status", "SUCCESS");
		mockOutput.expectedMessageCount(SEND_COUNT);
		

		// Perform Test

		for (int i = 0; i < SEND_COUNT; i++) {
			System.out.println("sending message.");

			// do send/receive, aka. run the route end-to-end
			producer.sendBody(inputUri, inputExemplar); 
			String output = consumer.receiveBody(outputUri, String.class); 
		}

	
		// ensure that the order got through to the mock endpoint
		mockOutput.setResultWaitTime(10000);
		mockOutput.assertIsSatisfied();
	}
	
	@Test
	public void process_messages_as_xml_test() throws Exception {

		System.out.println("Calling " + StackTraceInfo.getCurrentMethodName());
		
		// Set expectations
		int SEND_COUNT = 1;

		String inputUri = "file:src/data2";
		String outputUri = "file:target/output2";
		
		MockEndpoint mockOutput = camelContext.getEndpoint("mock:" + outputUri, MockEndpoint.class);
		//mockOutput.expectedBodiesReceived(outputExemplar);
		mockOutput.expectedHeaderReceived("status", "SUCCESS");
		mockOutput.expectedMessageCount(SEND_COUNT);

		// Perform Test

		for (int i = 0; i < SEND_COUNT; i++) {
			System.out.println("sending message.");

			// do send/receive, aka. run the route end-to-end
			producer.sendBody(inputUri, inputExemplar); 
			String output = consumer.receiveBody(outputUri, String.class); 
		}

		// ensure that the order got through to the mock endpoint
		mockOutput.setResultWaitTime(100000);
		mockOutput.assertIsSatisfied();
	}

	
	@Test
	public void process_jetty_messages_as_xml_test() throws Exception {

		System.out.println("Calling " + StackTraceInfo.getCurrentMethodName());
		
		// Set expectations
		int SEND_COUNT = 1;

		String inputUri = "jetty:http://0.0.0.0:8888/myapp/myservice/?sessionSupport=true";
		String outputUri = "file:target/output3";
		
		MockEndpoint mockOutput = camelContext.getEndpoint("mock:" + outputUri, MockEndpoint.class);
		mockOutput.expectedBodiesReceived(inputExemplar);
		mockOutput.expectedHeaderReceived("status", "SUCCESS");
		mockOutput.expectedMessageCount(SEND_COUNT);

		// Perform Test

		for (int i = 0; i < SEND_COUNT; i++) {
			System.out.println("sending message.");

			// do send/receive, aka. run the route end-to-end
			String result = producer.requestBody(inputUri, inputExemplar, String.class); 
			String output = consumer.receiveBody(outputUri, String.class); 
			
			assertEquals("OK", result);
		}

		// ensure that the order got through to the mock endpoint
		mockOutput.setResultWaitTime(10000);
		mockOutput.assertIsSatisfied();
	}
}

Discussion of the RouteTester class

The important thing to know about my RouteTester.java example is this. It extends CamelSpringTestSupport, which requires you to override its createApplicationContext() method. This method tells it where to find the Spring ApplicationContext you want it to test. In our case that context, is a Camel Context. Thus I’ve set the path to “/test-camel-context.xml”. This basically boots up the Camel Context and now we can run its routes from inside our @Test methods.

Furthermore, there is a VERY IMPORTANT and VERY SIMPLE convention you need to understand in order to use the mocking framework. It is this:

If you want to mock an endpoint, say “file:src/data1”, the syntax to mock it will look like this “mock:file:src/data1”.

That’s it… Once you understand this, you see how easy it is to wrap your endpoints, whether they be producers or consumers in mocks that will prevent them from actually running or needing to be there, and instead provide you with programmatic access to both feed them and receive from them in your tests. That said, the expectations based semantics the mocks give you is pretty awesome. It just makes sense to human brains.

For example in the process_jetty_messages_as_xml_test() test, we tell the “output routing step”, file:target/output3, to expect the following to be the case:

mockOutput.expectedBodiesReceived(inputExemplar);
mockOutput.expectedHeaderReceived("status", "SUCCESS");
mockOutput.expectedMessageCount(SEND_COUNT);

Basically, we told it to expect that the output matches the exemplar payload we want, as well as the “status” header should be set to “SUCCESS”, and the send count should be what we set it to.

If any of these expectations are not met, then the test will fail and we’ll get a comprehensible message from the framework. Here’s an example of when the “status” header doesn’t meet the expectation.

This is great! We know exactly why it failed and can quickly investigate, fix, and retest to ensure that we fix the bug.

We did not need a real jetty server or a http client to call it, nor did we need a filesystem to put the file into. More importantly, we found out that there was a problem while our hands were on the keyboard during dev time, not production, or some heavy manual regression testing. Best of all is that this test helped us now, and will continue to run every time we build. This means Jenkins or whatever other CI tooling you’re using will also provide you with elegant, automatic, and perpetual test running. So that in the future, when you accidentally break something indirectly related to this route, perhaps a change to one of the Collaborating Domain Services, you get an email telling exactly what’s wrong.

So, we’ve gone through a lot of content here, touching on a number of topics that are all directly or indirectly related to getting a high-quality test capability in place for your integration code. With the CamelSpringTestSupport, the excellent Apache Camel framework showcases just how powerful testing within it can be. Given that Camel is a mature and widely-used integration solution, it has evolved to accommodate good testing practices. Developers only need to be aware and in command of the testing layer of the framework to put it to work in their daily practices.

In this article, I distilled down what are some of the more important design and testing concepts and showed you how to apply them with the tooling. Mind you, these concepts are not specific to tools, platforms, or any specific flavor of integration technology. In fact, they are generally applicable to good development practices. Going forward I would encourage you to investigate Domain-Driven-Design, the Spring Framework, Apache Camel, and the Arrange Act Assert and Behavior-Based unit testing paradigms. The rabbit hole is deep and one could spend a career learning and applying these things to their professional work. However, this is a well understood area of Software Architecture and the best stuff to be had is near the surface. My hope is that you’ve found this work insightful and that it finds it’s way into your thought process and key strokes in the future.

If you would like to contact me with questions and discussion, I’m available via twitter and the comments of this article. The code in this article can be found here. If you’ve made it this far, you are ready to grab it and begin using these techniques yourself. Good luck and enjoy.