How is beta testing done?

I'm finally coming in to the home straight with my game, Tank Universal.

But before release there's still a lot of hills to scale, one of which is getting some testing done by people with different systems and points of view.

I can release another demo (which I plan to do), but I also need the full programme tested.

So my question is, how does one go about this? And by 'go about it', I mean how do I get testers who;

If they have a gameplay or technical problem or suggestion, can write about it in a reasonably coherent fashion.

Will spend time trying to 'break' the game.

Will make reasonable suggestions rather than suggestions that would require a rewrite, such as 'I think the tank should be able to fly around and shoot things because that would be cool!'

Won't distribute it to everyone on the net.

GFK - how do you go about getting your releases tested?

You need two different types of tester. First you need a group of people who know what they are doing, and these lot will try and break it, and comment on how badly programmed it is, and how they could have done better etc, and you will understand what they mean.
And secondly you need a group of people who dont know what they are doning, and they will probably break it worse, and give more useful comments, but you wont understand what they mean.

The second gruop is the most important. And that group could just be friends and family. They dont need to know what they are doing.

Whenever I have a "foolproof" piece of code, I let my brother/dad play on it, and they break it in no time flat!

Good advice - unfortunately I don't really have the second group available - my immediate friends and family (particularly my brother and father) aren't interested in computers - don't have the hardware even.

>I'm finally coming in to the home straight with my game,
>Tank Universal.

Yay! It wasn't long ago that you were talking about a new demo, and now you're done! I still have the old demo and actually still play it sometimes.

>I can release another demo (which I plan to do), but I
>also need the full programme tested.

Demo please!

>If they have a gameplay or technical problem or
>suggestion, can write about it in a reasonably coherent
>fashion.

Is that really so dificult?

>Will spend time trying to 'break' the game.

You might want to consider asking different people to focus on different parts. Either make special compiles of that part(level counter variable always equals level 2 etc, but remember to remove the other level code as it is otherwise an easy job to change the variable contents.), or maybe a savegame function that enables them to focus on that part.

Ask people with time on their hands to test.

>Will make reasonable suggestions rather than suggestions
>that would require a rewrite, such as 'I think the tank
>should be able to fly around and shoot things because that
>would be cool!'

ask people to focus on the part of the game they have been assigned.

>Won't distribute it to everyone on the net.

NDA?!

As for testers not warezing it, I think you should work out a scheme to make individual versions for each person. This can be scripted if you need to provide many downloads. Keep the full version in as few hands as possible and NEVER let go of the source. In the end, it's likely that someone at the CD production company will beat your most morally questionable tester to warezing the game.

What specs are you aiming at?

Finally, make a NDA/contract that includes what should be tested and how. Set up some benchmarks and define how to reach them.

Then you need to set up some kind of tracking system, for the testers to report bugs, you to fix them and for the testers to test your bugfixes.

http://en.wikipedia.org/wiki/Game_tester
http://www.sloperama.com/advice/lesson17.html
http://students.washington.edu/emedina/Testing.html

As I said here http://www.blitzbasic.com/Community/posts.php?topic=22875#235410 I want to be the first to buy it.

I'll be happy to test a demo, but a full-time job+ and low spec hardware may make that unattractive to you.


Andy

Andy

'Yay! It wasn't long ago that you were talking about a new demo, and now you're done! '

(cough) I wouldn't say 'I'm done' - still got a long list of hurdles to clear before release ;-)

But in the meantime, here's a compile of the latest code for level4 (the Training level). I'd be really interested in how you find it vs the last version. I think you'll find it plays quite a bit better.

http://homepages.ihug.co.nz/~pajones/TU_DEMO.zip
(45meg file, zipped moleboxed exe file)
note: run it at 800x600 or 1024x768 - there's interface issues at 1280x960.

WASD control, RMB stops the turret

There's still the odd bug, but it's fairly stable.

In terms of final spec's, I need to do more optimising. I can say they're going to be higher than your average puzzle game spec's though.

Cheers for the other suggestions and info too.

>But in the meantime, here's a compile of the latest code
>for level4

Oh, you tease, You! :)


Andy

I really like the universe and the style of the game. It's also come along way from the first demo. It feels much more optimized and smooth.

The meshes are nice, and the sound, graphics and effects seems to be coming together nicely.

However, the control of the turret is very sluggish and unintuitive. I am running it on a P3 1.4GHz with a Radeon 9800Pro.

I've only encountered one issue so far. When approaching the enemy base, the program shut down and displayed a 'memory access violation'.


Andy

Found the "you need people who don't know what they're doing" comment quite funny. This is precisely what you DON'T need.

The primary quality that your 'testers' will need, is time to actually do some testing. Five minutes followed by a "works OK here!!" post is nowhere near enough.

You need to draw up test plans covering all aspects of the game you want to have tested. Give these to your testers and get them to check things off as they test them. You know where you game is most likely to break so you can get testers to concentrate on those areas if you want.

You'll also want your testers to play the game sans test plan, as they may come across issues you hadn't thought of.

You also need to give your testers some means of reporting bugs clearly and consistently. Taking the above post as an example; "When approaching the enemy base, the program shut down and displayed a 'memory access violation'", I'd want to know:

1. Which base if there's more than one?... and...
2. ...does it happen approaching all bases?
3. What else was going on at the time?
4. How many times (out of, say, 5) does this happen?

...and so on.

Also, way up your priority list should be compatibility tests. There is absolutely no point at all harvesting testers from here, since most have higher-end systems which Average Joe simply does not have.

And there's the stumbling block. People generally will not give you their time, and assuming that you aren't in a position to pay people a wage, you'll have to make the best of your available resources; i.e. friends/family etc.

I can't pay people either. I have a handful of people I occasionally get to look at the game and give me an honest opinion. Other than that, I have family that test my game, plus I do most of the structured testing myself. Other than me, there's probably about 8 or 9 people I know I can rely on to give the game a good hammering when I need them to.

H&K's suggestion is spot on!

I would include some form of debug-logging with your beta release, at least to give YOU soe idea of what crashes and where, even if your testers are unable to identify the exact reason. Have them email you the debuglogs or something in event of crashes/errors.

Prior to a Beta release, you should have held an Alpha Testing session, (mainly just yourself, other developers or people from H&K's "first group").
The Alpha session will ensure the game is pretty free of most crashing and code errors, while the Beta test is to evaluate gameplay, compatibility issues, exploits etc. finally, tweaking.

Having played the demo many more times, I love the whole universe and ideas, but I still hate the controls. I know you hate the FPS controls people were asking for after the last demo, but the controls seem unresponsive and imprecise.

Maybe it's just me or my computer, but it really kills it that I can't shoot the enemies with any degree of accuracy.

Is the steering smooth and responsive to everyone else?

Andy

GFK

Thanks very much for the suggestions - the feedback framework idea is one I've definitely noted. And as you suggest, I'll also try to locate some people locally whom I can distribute to and can trust and know will give intelligent feedback.

There's also something to be said for releasing test levels here on the forum I guess - both as a way of testing and raising interest.

Andy
Will release a test demo relatively soon with many more options for controls. Also, P3 1.4GHz with a Radeon 9800Pro is kind of on the low side for when there's lots of tanks/explosions/particle fx going on.