Test-driven Pyweek

mauve on 2012/04/25 19:43 of Mauvesoft Wasabi

Since the last Pyweek I've become seriously disciplined at test-driven development in my day job. If it doesn't has tests, how can I know it works? Now I'm wondering whether it would be valuable for Pyweek.

Six months ago I would have thought it was a crazy idea to write tests for a Pyweek game. Pyweek is all about coding as fast as possible, right? And tests are just a waste of time I could be spending adding more features! But seriously, the amount of time I spend in a Pyweek manually testing my own game is massive, and it's always possible to leave some crasher in, something that worked once but got broken in the headlong rush to move features. I've read articles suggesting TDD is just always faster.

And anyway, how can you test that such and such a particle effect looks just so? Well, you're much more likely to break the code so that the particle effect just crashes with some exception, or is never started, or emits them off screen or something. How it looks is hard to test, but it's not the most important thing to test. I love having to not have to deal with that kind of crap any more.

So what do we think? Has anyone done a substantial amount of TDD in a Pyweek? Is it an approach that can work?

(log in to comment)

Comments

bjorn on 2012/04/26 00:08:

I did a bit of tdd during pyweek a couple years ago, but was part of a team and not everyone was writing unit tests, and the code moves so quickly that keeping the tests up to date and relevant was pretty fruitless and I abandoned it pretty quickly.

But if you are comfortable with tdd and are the only one working on the code (or everyone on your team is doing TDD) then I say go for it.  I've heard estimates that it usually takes about 3 weeks on a new project before TDD starts paying off and being faster than writing the code with no tests, so pyweek may not be a prime candidate for that here, but it probably depends what you are comfortable with, and it really can save you a lot of debugging so I say go for it. I'd be interested to hear how it goes. 

If I find time to participate this time, there are definately pieces I would TDD, and others I'd probably just hack away at to get the feel and look right.

blakeohare on 2012/04/26 00:09:

For me, personally, I usually generate prototype-quality code during PyWeek. I plan and implement the core design and keep things tidy-through Wednesday, and then let carefully-placed spaghetti take over through Saturday. This means that core components can sometimes change fundamentally and I would imagine this would lead to tons of false positives in any tests I have. By the time I get to the point where I tickle the code on one side and it laughs on the other, it's already Friday and nearly all features have been implemented and only minor bug fixes remain.

I usually code alone but when I don't [insert meme here], I partition the code in such a way that all coders don't really have to touch anyone else's code with the exception of one or two functions. One of the biggest payouts of unit testing is a safeguard of one coder violating the assumptions of code they didn't write but in isolated coding, this doesn't have as much value.

And for everything else, there's good ol' fashioned black-box manual testing which usually is far more efficient for the "particle effect looks sorta funny" class of bugs.

tartley on 2012/04/27 13:14:

TDD is always faster, but only if *how* to test something is evident, and any infrastructure you'll need to do that testing is already present.

The difficulty with full TDD for games is writing acceptance/functional/system tests (they all mean the same thing, to me) which verify the end-to-end behaviour of the whole system. For example, writing a test which fires up your game and 'plays' it, walking the whole things through to game completion. (or a series of such tests, one for each level, or however else you choose to partition it.)

Without such a system-level test, you aren't doing proper TDD, and obviously writing such a test, in the general case, is difficult, because it requires the test to look at the screen and analyse the bitmap currently being displayed, and then decide on what inputs to provide. For some games (e.g. roguelikes) this is relatively easy, for others (e.g. openGL FPS) this is considerably harder.

Aside from that caveat, doing semi-TDD (just using unit tests) will, IMHO, be faster & produce better designed & implemented code than not using TDD, provided you have experience working in a TDD style.

tartley on 2012/04/27 13:23:

For particle effects: That's a good illustrative example. If the magic is happening on the GPU, then writing unit tests is hard: I don't know how to unit test shader code (but I understand some people are doing it.) Similarly, writing acceptance tests is hard, for reasons similar to those I outlined above. Writing a test that specifies 'it looks right' is fiddly to quantify without being overly strict (comparing to a canned bitmap would do it, but would be too brittle, breaking every time you tweaked some appearance parameter, which violates the rule-of-thumb that you shouldn't be testing the value of constants.)

I think all you can usefully do for particle effects is have unit tests which assert that you end up calling your rendering API with the correct parameters. If these tests are somewhat immune to changes in values of constants like colors, "density", etc, then you can tweak the appearance somewhat without breaking all the tests.

mauve on 2012/04/30 22:18:

"Without such a system-level test, you aren't doing proper TDD"

I disagree. I don't think there's anything about TDD that mandates that tests must cover every level of abstraction. It's just about writing tests first to verify that what you are about to write will be correct. TDD is valuable whatever the level of tests you write

CrazyPyro on 2012/05/02 05:03:

Where tests would be valuable is around the classes that implement your game's actors and their interactions: "Does the plane go down when you decrease throttle?  Does it blow up when it hits the ground or a missile?  Does the game end when you are out of lives?"  Etc.  NOT "Does it look just so...?"  This is why webapps test the Models and Controllers but not the Views - checking your logic is very worthwhile, but trying to test how something looks is so difficult per benefit that it's pretty much a waste of time, IMHO.

Woodwolf on 2012/05/02 07:23:

Um... I'm interested but I've never done any TDD. Is there a good read I could grab to learn about all this?

MarkR on 2012/05/06 14:57:

Hi, I can't really see that TDD is very helpful in a PyWeek. It is quite difficult to auto-test game anyway (except for trivial units which are probably working pretty well anyway), so I really don't get it.

I don't think it'll give you any benefit, except possibly learning about techniques for auto-testing games. At least it won't make your PyWeek game get done any sooner.

Mark

mauve on 2012/05/09 06:46:

A quick update on this - so far I've written unit tests, but only for the lowest level algorithmic stuff. Much of the game is plumbing together UI components where I'd have to invest too much work to be able to test - for example, mocking the OpenGL modelview matric operations, etc.

However, for the algorithmic stuff, it's been very useful - got some complicated stuff working really very fast.

superjoe on 2012/05/09 06:49:

Thanks for the update. I was curious whether this would work or not. I have to admit I was skeptical about it, but it does make sense for the algorithmic stuff.