Fail Hard, with Grace

Summary:

  • We shall continue with the test execution even after an issue is found or some checking fails, till we cannot, or make no sense, to go further.
  • For failed cases, we shall take screenshot at each moment when an error/exception we found or fun into.
  • We can decorate Junit’s ErrorCollector class to collect errors and take screenshot at the same time

When shall we fail the case?
A typical test case contains many steps, in each step you do something, and then verify something:

Test case “testxxxx”:
Step 1: do something, then verify something
Step 2: Do something again, and verify some other thing
[….. More steps ……]

Naturally, when we automate test cases, we don’t want to quit the case when the first assertion fails, unless that failure is fatal and prevent us from going further.

We shall continue with the test execution even after an issue is found or some checking fails, till we cannot, or make no sense, to go further. For example, if we have a case checking the validation of all input fields on the page, we don’t want to fail the case right way when tiny issue is found agains certain field, instead, we want to ‘remember’ that error, and keep checking the other fields, and fail the case after we done checking all the fields on the page.

Then when the execution is done, we want to let the case pass or fail based on the errors/exceptions we found, also, we want to record all necessary information along the way of execution for debugging purpose.

What information shall automation collect upon failure?
The following are a ‘must’:

  • Console Logging
  • screenshot at the moment when an error/exception we found or fun into.
    Screenshot provides us many more information that a console logging cannot give, and it is much more direct and straight-forward. And since we might find more than one error or run into more than one exception, one failed case could come up with multiple screenshots recording what happens on the screen at all those moments when bad thing happens.

    The screenshot name shall be unique each time you take it and the name shall give you some hint on what test case it is associated with. Usually I name screenshot using test case name + time Stamps to ensure the name uniqueness.

how to implement this?
This can be achieved with either Junit or TestNG. Here I will show how we can decorate Junit’s ErrorCollector class to:

  • execute test case till we cannot go further and record all the errors we found along the way
  • take screenshot for each error we found

No more Assert: With Junit Assert the test will terminate when the very first assertion fails, so Assert will not help us here.

Instead, we will use Junit Verify-like feature: we can utilize @Rule and ErrorCollector class to add errors/exceptions along the way of test execution and then fail the case when the execution is done. The code is like this:

public class TestWithErrorCollector{
    @Rule
    public ErrorCollector errorCollector= new ErrorCollector();    
  
    @Before
    public void init( )throws Exception
    {        // [...Do whatever you need to do here...]
    }

    @Test
    public void demoErrorCollector( ) throws Exception
   {
        // [ ... do something here ...]
       errorCollector.addError(new Throwable("the first exception."));
       // [ .... do something else... ]
      errorCollector.checkThat(a, equalTo(b));
      //        .....
   }

    [.... other methods like @after, @aferClass etc.....]
}

All the errors and exceptions are stored in errorCollector, and at the end, test will fail if errorCollector is not empty. In the generated test report, all errors/exceptions will be displayed.

But what about taking screenshot along the way for each error? To achieve that, we need to decorate ErrorCollector class, and just add ‘take screenshot’ function to those critical moments:

//import section is skipped
//code documentation is skipped. 
public class ErrorCollectorDecorator extends ErrorCollector{
    protected ErrorCollector decoratedErrorCollector;
    public ErrorCollectorDecorator(ErrorCollector decoratedErrorCollector){
       this.decoratedErrorCollector = decoratedErrorCollector;
    }
    public void addError( Throwable error, WebDriver driver,  String screenshotName  )
    {
       decoratedErrorCollector.addError(error);
       takeScreenshot(driver, screenshotName);
    }
   
    public <T> void checkThat(final T value, final Matcher<T> matcher,  WebDriver driver, String screenshotName) {
       checkThat("", value, matcher, driver, screenshotName );
       
    }
  
    public <T> void checkThat(final String reason, final T value, final Matcher<T> matcher , WebDriver driver,  String screenshotName) {
        checkSucceeds((new Callable<Object>() {
            public Object call() throws Exception {
                assertThat(reason, value, matcher);
                return value;
            }
        }), driver, screenshotName);
    }
   
    public Object checkSucceeds(Callable<Object> callable, WebDriver driver,  String screenshotName) {
        try {
             return callable.call();
         } catch (Throwable e) {
             addError(e, driver, screenshotName);
             return null;
         }
    }
   
    public void takeScreenshot(WebDriver driver, String  screenShotName)
    {
       try {
             File screenshot = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
            FileUtils.copyFile(screenshot, new File(screenShotName));
        } catch(Exception ex) {
            System.out.println(ex.toString());
            System.out.println(ex.getMessage());
        }
    }
}

Our test runner class will be like this:

public class TestWithErrorCollectorDecorator{
    private ErrorCollectorDecorator errorCollectorDecorator ;
    @Rule public ErrorCollector errorCollector= new ErrorCollector();
    @Rule public TestName name = new TestName();
    @Before
    public void init( )throws Exception 
   {
       errorCollectorDecorator = new ErrorCollectorDecorator(errorCollector);
       screenshot_prefix = name.getMethodName();
             //[...Do whatever else you need to do here...]
   }

   @Test
   public void demoErrorCollector( ) throws Exception 
   {
           // [ ... do something here ...]
       errorCollectorDecorator.addError(new Throwable("the first exception."), driver, generateScreenshotName() );
          // [ .... do something else... ]
       errorCollectorDecorator.checkThat(a, equalTo(b), driver, generateScreenshotName());
               //.....
  }

   //Get current time in Milliseconds
   private long getCurrentTime()
   {
      Date now = new Date();
      return now.getTime();
  }

   private String generateScreenshotName()
  {
     return screenshot_prefix + "_" + getCurrentTime() + ".jpg";
   }

    //[.... other methods like @after, @aferClass etc.....]
}

In summary:

  • We shall continue with the test execution even after an issue is found or some checking fails, till we cannot, or make no sense, to go further.
  • For failed cases, we shall take screenshot at each moment when an error/exception we found or fun into.
  • We can decorate Junit’s ErrorCollector class to collect errors and take screenshot at the same time

Leave a Reply

Your email address will not be published. Required fields are marked *