Test execution order – JUnit 4.X

I want to execute test methods which are annotated by @Test in specific order.

For example:

public class MyTest{
    @Test public void create(){System.out.println("create");}
    @Test public void update(){System.out.println("update");}
    @Test public void delete(){System.out.println("delete");}
}

I want to ensure to run create() before update() and updatebefore delete each time I run MyTest.

The test execution order i would like to get is,

    create
    update
    delete

Test execution order

By design, JUnit does not specify the execution order of test method invocations. Until now, the methods were simply invoked in the order returned by the reflection API. However, using the JVM order is unwise since the Java platform does not specify any particular order, and in fact JDK 7 returns a more or less random order. Of course, well-written test code would not assume any order, but some does, and a predictable failure is better than a random failure on certain platforms.

From version 4.11, JUnit will by default use a deterministic, but not predictable, order (MethodSorters.DEFAULT). To change the test execution order simply annotate your test class using @FixMethodOrder and specify one of the available MethodSorters:

@FixMethodOrder(MethodSorters.JVM): Leaves the test methods in the order returned by the JVM. This order may vary from run to run.

@FixMethodOrder(MethodSorters.NAME_ASCENDING): Sorts the test methods by method name, in lexicographic order.

package com.vel.qa.tests.detailedtests;

import org.junit.FixMethodOrder;
import org.junit.Test;
import org.junit.runners.MethodSorters;

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class MyTest {
	@Test
	public void aCreate() {
		System.out.println("create");
	}

	@Test
	public void bUpdate() {
		System.out.println("update");
	}

	@Test
	public void cDelete() {
		System.out.println("delete");
	}
}

This would produce the output as i wanted.

Advertisements

Result Aggregation using JUnit- Rule (Testwatcher)

Hi All,

I was asked to create a custom JUnit test result file which only has test case name and pass/failure message.

result.txt should contain like mentioned below.

testmethod1 : success
testmethod2 : failed
testmethod3 : success
testmethod4 : success
etc..

1. Result Aggregation:

The challenge is , we can not log the test results within the test method itself because at this time we will not be able to know whether the test will pass or fail. You may think ,we can write a log statement at the end of the test method but that will log only the success test methods, what about theFailed/Ignored/Errors test methods? As soon as assert failed , AssertionError will be thrown by the Junit and we will not be having any control to log this anywhere.. Still you may think, you can catch this exception and write a log statement and then fail the test methods. Yeah , this would work but it has some drawbacks.

  1. we will not be able to capture the ignored tests.
  2. we will not be able to clearly classify Errors vs Failures (these are completely different)
  3. we need to write a try/catch block in all of our tests. More customization required.

The above functionlity can be achieved easily using TestWatchmen Rule provided by JUnit-Framework.

JUnit-4.9 introduced new features called "Rule" which can be used to keep a log of each passing, ignored, failing tests. Click here for more information

With this approach we need to do no modification to the existing code still we can do our result aggregation.

Step 1: Create a class which implements TestRule Interface.

 
package com.vel.qa.tests.detailedtests;

import org.junit.rules.TestWatcher;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;

import com.vel.qa.helpers.LoggerUtility;

public class ResultAggregator extends TestWatcher {

	private static String resultLog = "";
	private LoggerUtility loggerUtil = null;

	ResultAggregator() {
		// TODO need to refactor this constructor
		loggerUtil = LoggerUtility.getInstance();
	}

	@Override
	public Statement apply(Statement base, Description description) {
		return super.apply(base, description);
	}

	@Override
	protected void succeeded(Description description) {
		resultLog = description.getMethodName() + ": " + "SUCCESS \n";
		loggerUtil.getLogger("result").info(resultLog);
	}

	@Override
	protected void failed(Throwable e, Description description) {
		resultLog = description.getMethodName() + ":  " + "FAILED " + "\n";
		loggerUtil.getLogger("result").info(resultLog);
	}

	@Override
	protected void starting(Description description) {
		super.starting(description);
	}

	@Override
	protected void finished(Description description) {
		super.finished(description);

	}

}

This class will keep a log of each passing and failing test:

Step 2: Create test class and add @Rule annotations.

  
package com.numerify.qa.tests.detailedtests;

import static org.junit.Assert.fail;

import org.junit.Rule;
import org.junit.Test;

public class ResultAggregatorTest {

	@Rule
	public ResultAggregator log = new ResultAggregator();

	@Test
	public void testmethod1() {

	}

	@Test
	public void testmethod2() {
		fail();
	}

}

If you execute the above test class you will get an output as shown below.

testmethod1 : SUCCESS

testmethod1 : FAILED

Note: LoggerUtil is the class which is an custom defined logger for logging different files using log4j.

2. Repeat test with different input files.

We could achieve this by extending Junit-Core Runners [Junit-4X runners , Junit3X runners will be compatible with annotations] and iterate it with our files. I understood that this runner [Parameterized.java] are provided by JUnit itself in the recent releases. Click here to know more about Parameterized-tests

Dynamic number of tests – JUnit4

In our project I have several JUnit tests that e.g. take every file from a directory and run a test on it. If I implement a testEveryFileInDirectory method in the TestCase this shows up as only one test that may fail or succeed. But I am interested in the results on each individual file. How can I write a TestCase / TestSuite such that each file shows up as a separate test e.g. in the graphical testrunner of eclipse? (Coding an explicit test method for each file is not an option.)

package com.vel.qa.tests.detailedtests;

import java.io.File;
import java.util.ArrayList;
import java.util.Collection;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

@RunWith(Parameterized.class)
public class ParameterizedTest {

	@Before
	public void method() {
		System.out.println("Set Up method  ");
	}

	@Parameters(name = "{0}")
	public static Collection<Object[]> data() {
		// load the files as you want from the directory
		System.out.println("Parameters");
		Collection<Object[]> data = new ArrayList<Object[]>();
		for (int i = 0; i < 10; i++) {
			File file = new File("file_name_" + i);
			Object[] fileArg1 = new Object[] { file };
			data.add(fileArg1);
		}
		return data;
	}

	private File fInput;

	// Constructor with arguments
	public ParameterizedTest(File input) {
		System.out.println("constructor");
		this.fInput = input;
	}

	// This test will be called multiple times depends on size of the parameters
	// set at the @paramenters method.
	@Test
	public void test() {
		System.out.println("test called  ");
		// do more validation based on your requirements.
	}

	@After
	public void tearDown() {

	}
}
  • 1. Junit will call this static method (@Parameters) first to get all the collection of data
  • 2. For Each array in the collection , Junit will create an object of this class by calling the constructor(argument) you provided and then call set up method(@Before) and then call the test method (@Test) that you have written.
  • 3. In that way if size of the collection is 50, then 50 times test method (@Test)will be called by JUnit.
  • The results of the above program looks like as shown below.

    junit_graphical_results_eclipse

    junit_graphical_results_eclipse

    running-junit-tests-in-parallel

    I’m using JUnit 4.4 and Maven and I have a large number of long-running integration tests.

    When it comes to parallelizing test suites there are a few solutions that allow me to run each test method in a single test-class in parallel. But all of these require that I change the tests in one way or another.

    I really think it would be a much cleaner solution to run X different test classes in X threads in parallel. I have hundreds of tests so I don’t really care about threading individual test-classes.

    I opted for Junit-toolbox JUnit-Tool-Box

    1. add this dependency to your POM.xml.

    <dependency>
        <groupId>com.googlecode.junit-toolbox</groupId>
        <artifactId>junit-toolbox</artifactId>
        <version>1.5</version>
    </dependency>

    2. Write a Junit test classes.

    package com.numerify.qa.tests.detailedtests;
    
    import org.junit.After;
    import org.junit.Before;
    import org.junit.Test;
    import org.junit.runner.RunWith;
    
    import com.googlecode.junittoolbox.ParallelRunner;
    
    @RunWith(ParallelRunner.class)
    public class ParallelClassTest {
    	@Before
    	public void setup() {
    
    	}
    
    	@Test
    	public void test1() {
    		// TODO Auto-generated method stub
    	}
    
    	@Test
    	public void test2() {
    		// TODO Auto-generated method stub
    	}
    
    	@Test
    	public void test3() {
    		// TODO Auto-generated method stub
    	}
    
    	@After
    	public void tearDown() {
    
    	}
    }
    
    

    ParallelRunner — Executes all @Test methods as well as the calls to @Theory methods with different parameter assignments concurrently using several worker threads.

    Download files from bitbucket

    I was asked to download a file (raw) from bitbucket repository and wrote an automation script using those files. Bitbucket exposed set of REST Web services for their repository using which we can integrate our application with bitbucket.

    Steps to download files from bitbucket:

    1. Need a HttpClient library , I opted for Apache CXF Client.

    <dependency>
    <groupId>org.apache.cxf</groupId>
    <artifactId>cxf-rt-rs-client</artifactId>
    <version>3.0.0-milestone1</version>
    </dependency>
    

    2. REST API documents . File Resource

    public class GitClient {
    	static String url = "https://bitbucket.org/api/1.0/repositories/myrepository/mr/raw/";
    	static String branch = "master";
    	static String source = "/MavenWeb/src/main/resources/";
    	String test = "/src/test/resources/";
    	static String user = "username";
    	static String password = "password";
    
    	public static void usingWebClient() throws FileNotFoundException,IOException {
    		downloadFileFromBitBucket("provisioner-components-setup.ini");
    		downloadFileFromBitBucket("provisioner-properties-setup.csv");
    		downloadFileFromBitBucket("provisioner-resources-setup.zip");
    		downloadFileFromBitBucket("provisioner-system-setup.ini");
    	}
    
    	private static void downloadFileFromBitBucket(String resource)
    			throws IOException, FileNotFoundException {
    		WebClient client = WebClient.create(url + branch + source + resource,user, password, null);
    		Response r = client.accept("text/plain").get();
    		System.out.println("reponse" + r.getStatus());
    		IOUtils.copyLarge((InputStream) r.getEntity(), new FileOutputStream(new File("C:\\folder" + resource)));
    	}
    
    	public static void main(String args[]) throws FileNotFoundException,IOException {
    		usingWebClient();
    	}
    }