I recently came across this paper: “When, How, and Why Developers (Do Not) Test in Their IDEs” written by Moritz Beller, Georgios Gousios, Annibale Panichella, and Andy Zaidman.

Caveat: It doesn’t appear to be super recent ¹, the sample size is not large (< 500 developers), and their data is gathered from Eclipse ², so the conclusions drawn in the paper may need to be taken with a grain of salt. Having said that, I think the results are really interesting.

I recommend reading the paper, but I’ve gathered some of the interesting parts below (with my commentary):

developers rarely run their tests in the IDE

Woah, no beating around the bush here.

TDD is not widely practiced. Programmers who claim to do it, neither follow it strictly nor for all their modifications.

Seems obvious… TDD always seemed like a bit of a farce to me. There’s a good idea in there somewhere, but I think people take it way too far.

software developers only spend a quarter of their work time engineering tests, whereas they think they test half of their time

This one’s interesting - I always thought it was a 1:1 ratio. That is, if I spend 4 hours writing production code, I spend 4 hours writing tests. I’ll have to find a way to get more stats on this for my team.

Observed behavior often contradicted survey answers.

This is not at all surprising - this is why I believe in measuring real data, and not trusting that people can remember what they did.

The majority of projects and users do not practice testing actively.

Not surprising, but damning.

Developers largely do not run tests in the IDE. However, when they do, they do it heftily.

This doesn’t seem to be a bad thing though:

One reason why some developers do not execute tests in the IDE is that the tests would render their machine unusable, for example during the execution of UI tests in the Eclipse Platform UI project. The Eclipse developers push their untested changes to the Gerrit review tool and rely on it to trigger the execution of the tests on the continuous integration server. In such a scenario, the changes only become part of the “holy repository” if the tests execute successfully. Otherwise, the developer is notified via email. Despite the tool overhead and a possibly slower reaction time, our low results on test executions in the IDE suggest that developers increasingly prefer such more complex setups to manually executing their tests in the IDE. IDE creators could improve the continuous integration server support in future releases to facilitate this new workflow of developers.

Tests run in the IDE take a very short amount of time.

we found that testing in the IDE happens fast-paced. Most tests finish within five seconds, or less.

This is definitely a problem I’ve faced before. Nobody likes waiting on a computer.

Tests and production code do not co-evolve gracefully.

Not surprising, but pretty damning.

Developers frequently select a specific set of tests to run in the IDE. In most cases, developers execute one test.

This resonates with me. We use gradle to build, and IntelliJ as our IDE. Unfortunately they don’t play very nice with each other and occasionally, I can’t run tests from IntelliJ. This usually means I don’t run tests at all, because running one test from gradle is a pain.

Conclusion

Bottom line, this paper paints a pretty horrifying picture of testing in software development. I can’t say I’m surprised by this. Most code I’ve encountered isn’t tested properly, and yet most of it seems to mostly work. This isn’t to say that I think testing isn’t a worthwhile activity. I still do believe it’s a valuable tool. However, it is perhaps also worthwhile to think about what degree of testing is actually required instead of being dogmatic about TDD, BDD, etc.