CI and Testing, Part 2: Fieldin CI and Development Cycle

In part #1 of this two posts series, I wrote about the QA philosophy at Fieldin and now it is time to get into more details about our CI and development cycle.

Before we jump into the details, I want to start with how we test our code and to introduce you to a startup called Testim.IO.

In most cases, your developers will create and maintain unit tests while a different team, commonly the automation QA team will maintain integration tests, usually using Selenium or any other similar automation framework. That means that you need to have developers in your QA team that know how to code for Selenium and have them fix and maintain tests each time a new scenario or feature is introduced. Your company probably has some sort of dashboard or reporting system to follow all the tests and you also have to maintain this dashboard somehow.

This is where Testim.IO comes into play as they have super easy to maintain and use Selenium framework as SAAS (there are several other companies providing the same solution, we found them to be the easiest to use and integrate, but you are welcome to check others of course). There are also free Selenium extensions out there to record the tests, but nothing even remotely comes with the simplicity and full framework like Testim.IO or similar SAAS companies. You can run tests on their grid or locally, you can monitor results and you can record tests in a couple of seconds. The tests are also very stable and usually do not break.

Fieldin Testing Philosophy — Anti-pattern of the Pyramid of Tests

Many integration tests, very few focused unit tests.

In one sentence, this is our testing philosophy. Using Testim.IO we can create, maintain and deploy a huge amount of integration tests (a very long full-feature check test) that can run on every step of the way (when the developer finishes working on her task, before QA, during QA, before production, etc.). If the test fails, we get a very detailed report describing where and why it failed. True, it will not be as pinpoint as a unit test, but to us the benefit of multiple tests is more important (right now we have around 500 integration tests in the system and each week we add more).

The following diagrams show how we save 30% development time:

On the right, the ideal by the book software testing pyramid, on the left the software ice-cream cone anti pattern. Notice that at Fieldin we do almost no manual testing as almost everything is covered by easy to maintain UI tests. Since the developers are not busy writing a lot of unit tests we can easily increase the development time.

Unit Tests

We still do unit tests but we are only performing them on complex logic or functions that can receive very diverse and different inputs which we want to make sure still work after every change. We also do a lot of unit tests on offline services of course that are not covered by the automation UI tests.

The Cycle

The cycle of development — QA — Deploy is very simple.

A developer is assigned with a task (we are using JIRA as our task management system) and creates a new feature branch in the GIT repository. We have scripted everything together so with a simple CLI command the developer can create a new branch and move the task to “in progress”. Once the developer finishes the task, she will create pinpoint unit tests and then move the task to the selenium grid (powered with Testim.IO). If one of the tests failed, the task automatically returns to the developer and she will get notified. If all goes well, the task is moved to QA for deeper observation and usually the QA will create new tests using the Testim.IO framework. Once approved, the task will be merged into the main branch and will be deployed to production in our weekly releases.

We usually do weekly releases and aim for 1 day releases, but we can do a release whenever we want as the main branch is always ready to be released.

Conclusion

The main advantages of our CI is how fast we can move and deploy mature code to production. We manage to deploy new features and fixes very quickly and move very fast.

The logic behind our way is that developers need to do what they do best — do more code and less QA.

We know this is standing in contradiction with many other development traditions like TDD and using the developers to write their own tests, but we believe that the combination of the Testim.IO framework and our ability to use dedicated QA engineers to generate a huge amount of tests makes the path we took worthwhile — at least for now, we are checking ourselves all the time. The results speak for themselves as we manage to move and react very fast to customers’ needs.