CI and Testing, Part 1

Should developers do their own QA?

The question of should developers do their own QA has risen in the past couple of years, especially with the increase of CD processes which try to minimize the time to production. It is enough to check Google autocomplete to see that this is a major interest in the industry.

Unfortunately for the confused R&D manager or team leader, there is no definite answer. The decision if your developer should do QA (and if so, how much?) should come after reviewing a few factors like the stage of the company (bootstrap vs corporate), the number of developers in the team, their skill, the R&D budget and other factors.

Having taken all of the above into consideration, to me, an early stage startup up to a medium-sized company should really reduce the QA their developers do and let it be handled by QA professionals. Why? Well money of course.

By the way, for a big company or corporate it is a more complex question which also depends on the product stage, team etc.

Let’s start by looking on the two main pros for developers to QA their own code:

  • Owning the task — the developer knows his code best. QAing the task and giving it the final stamp of approval makes the developer responsible for her work instead of sending the task to be QAed by someone else.
  • Reduce the time from development to production — this is the holy grail for all companies as they want to ship new features and bug fixes to production as fast as possible.

Of course, no one will argue against the first one — we all want our developers (and all other employees in the company) to be responsible for their work. However, as many developers noticed, it is very hard to test your own work — even if you have full tests suites covering your code. You are still limited by what you are not thinking to test. There are a lot of methods to overcome this mental block, but those are out of the scope of this post.

But wait, we assumed that the project has full (or almost full) tests coverage. In real life, reaching the level where there is enough tests coverage of the code might take time, and without it, you don’t really want your developers to deploy their code to production. In addition, many companies (mainly B2B) just don’t have enough users to really do gradual deploy for limited users at first and then full roll out so you are going to push the code for all users without knowing if you have covered every scenario.

Now let’s talk a little about money which to me is somehow always pushed aside in the discussion — maybe because most team leaders or R&D managers don’t get involved in the day to day operation and budget of the company. This is a big issue as developers might choose the wrong path or technology because it is better for development but not better for the company. I will try to discuss this topic in a different post in the future.

The average time spent by developers on testing their code is estimated to be around 30% of the working time. According to Glassdoor, the average salary difference between QA and a developer is 2/3. That means that if your team is more than 2 or 3 developers, you should really think about letting your developers do what they are being paid to do and bring a QA to do what they know to do.

Can you imagine the face of your product manager if you tell her, “listen we can work 30% better and write 30% more code which means more features and tasks done in the product”.

Of course, there is a downside. Your deployment will be reduced from immediate by the developer to once a week (or any other timeframe you choose. I recommend once a week although in Fieldin we aim for 1-day delivery) in order to let the QA team test everything and we all want to be agile and lean and to deliver. However, this once a week version will have less bugs and more features so it should be overall ok money-wise.

Wait, money again? Yes money. Your product manager is not writing feature tickets as a hobby. Those features provide better product and thus increase the value of the product and company — that equals money even if sometimes it is hard to quantify it. 30% more coding is a lot of money for the long run of a company. For most companies, there is no real difference between shipping code every day or once a week — if you can achieve better code quality and less bugs, to me the choice is obvious.

Having written all of that, it is still important to understand that:

1. You still need CI and tests for large scale product. You should develop a CI method that tests your code and features. It is just that you want your QA to oversee most of it.

2. Your developers can still write tests and unit tests are always welcome. Just don’t reach 30% of the time of the developer.

3. The right person to the right job. From my experience, good QA and good developers are just different types of people. The average developer will not be able to be as thorough as the average QA (and the opposite of course) so why put such expectations on your team?

At Fieldin we are practicing the above which allows us to deliver new features in a rapid pace and still be very flexible and adaptive to changes.

We have had our CI system for almost 3 years now and it’s still relevant and supports us as we grow with developers and QA. Our developers do unit testing and are responsible for their work. We always try to reduce bugs and talk about responsibility, but we are not letting them write full test cases and other stuff that can be handled by experts in their field.

In the second post of this two-post series, I will go into details on how we implement the vision behind our CI.

Read the second part, CI and Testing, Part 2: Fieldin CI and Development Cycle.