JUnit and TestNG
Cedric Beust from Google was here to talk about a personal project of his, a JUnit alternative testing framework called TestNG. The session was useful as a critique of JUnit, and also as an eye-opener to how our ideas of what unit tests should be have been shaped (and limited) by JUnit.
He jumped pretty quickly into an overview of the limitations of JUnit. JUnit re-instantiates test classes before testing each method call, which can be inefficient if you're doing a lot of setUp() or tearDown() work each time. You can work around the problem with static methods, but as Beust put it, then you're just trading one limitation for another one. Another problem with JUnit is that you can't easily test individual methods, at least outside of an IDE. Also, you can't enable or disable test suites without recompiling code. And, if nothing else, JUnit hasn't been updated in years, so it's not necessarily up-to-date with changes in JDK 1.4, 5, etc.
Mostly, Beust's complaints seemed to center around the fact that JUnit is a very static programming model, and that it imposes strict naming conventions around its configuration extensions. His product, TestNG separates business logic of the actual test from the run-time model of the test execution.
TestNG supports test groups which can categorize and aggregate common test types (database, front-end, etc). TestNG supports parameters into test methods, and method dependencies. Dependent methods report failure differently if it results from a missing dependency; it reports SKIP instead of FAIL. He refers to this feature as "partial failure." These business logic features of TestNG are all annotation-based. The run-time model is driven from a configuration file called testng.xml.
A lot of the features of TestNG could be viewed as requirements for a new revision of JUnit. Fundamentally, they are very different tools, but I'm not sure how much credibility and longevity this TestNG tool will have. Instead, I see its value more for NI as a way to identify gaps in JUnit so we can do our best to mitigate them.
One thing I did really like about TestNG was that its annotations include support for timeout and invocation count aspects. Wouldn't it be nice to be able to set timeout rules on our tests to enforce performance requirements? Or invocation counts to load test within the unit test framework?
Overall, honestly, TestNG seems like a far superior tool. My hesitation for now is based on the lack of confidence I have in the longevity of TestNG. JUnit has been around a while, and I am not sure what the value proposition is for migrating to TestNG. It sounds like JUnit has lots of room for improvement, but I just don't know that this is enough to warrant breaking apart from such a de facto standard. I would be very curious to see how TestNG adoption increases over this year. If it starts being seen as a viable, stable alternative to JUnit, it may be worth looking at.
He jumped pretty quickly into an overview of the limitations of JUnit. JUnit re-instantiates test classes before testing each method call, which can be inefficient if you're doing a lot of setUp() or tearDown() work each time. You can work around the problem with static methods, but as Beust put it, then you're just trading one limitation for another one. Another problem with JUnit is that you can't easily test individual methods, at least outside of an IDE. Also, you can't enable or disable test suites without recompiling code. And, if nothing else, JUnit hasn't been updated in years, so it's not necessarily up-to-date with changes in JDK 1.4, 5, etc.
Mostly, Beust's complaints seemed to center around the fact that JUnit is a very static programming model, and that it imposes strict naming conventions around its configuration extensions. His product, TestNG separates business logic of the actual test from the run-time model of the test execution.
TestNG supports test groups which can categorize and aggregate common test types (database, front-end, etc). TestNG supports parameters into test methods, and method dependencies. Dependent methods report failure differently if it results from a missing dependency; it reports SKIP instead of FAIL. He refers to this feature as "partial failure." These business logic features of TestNG are all annotation-based. The run-time model is driven from a configuration file called testng.xml.
A lot of the features of TestNG could be viewed as requirements for a new revision of JUnit. Fundamentally, they are very different tools, but I'm not sure how much credibility and longevity this TestNG tool will have. Instead, I see its value more for NI as a way to identify gaps in JUnit so we can do our best to mitigate them.
One thing I did really like about TestNG was that its annotations include support for timeout and invocation count aspects. Wouldn't it be nice to be able to set timeout rules on our tests to enforce performance requirements? Or invocation counts to load test within the unit test framework?
Overall, honestly, TestNG seems like a far superior tool. My hesitation for now is based on the lack of confidence I have in the longevity of TestNG. JUnit has been around a while, and I am not sure what the value proposition is for migrating to TestNG. It sounds like JUnit has lots of room for improvement, but I just don't know that this is enough to warrant breaking apart from such a de facto standard. I would be very curious to see how TestNG adoption increases over this year. If it starts being seen as a viable, stable alternative to JUnit, it may be worth looking at.
posted by Anonymous at 1:20 PM
0 Comments:
Post a Comment
<< Home