hunting the bugs

28 07 2006

The other day I was chatting with Jade Lily a little bit about the Preview Grid and how it’s not that much used as it might be better for Second Life. So she was thinking about what one can do about that fact so that new versions are hopefully better tested.

Now I must admit that I am also rarely in the preview grid. Usually I follow an announcement, see some cool feature which is announced in there and I go and check that out (like flexible prims, lighting, new UI stuff and so on). Eventually I discover some bug which I then report.
But once I have done that I am also quickly out there again. And I started thinking about it and came up with the following reasons:

  • Usually nobody is around in there and as communicating is key it get’s rather boring there
  • Building stuff does not really make much sense as you cannot take it with you to the main grid. This is especially true for bigger builds.
  • Same with scripting, although you might copy your scripts to some textfile and upload them again later. But then again scripting usually also goes with building.
  • Every time I plan to go there some version might have changed and I have to download another client. Even more problematic with 2 previews open.
  • For many people testing is boring, especially if it’s doing the same thing over and over again in every version. I am probably one of them šŸ˜‰

So what does that actually mean for user-driven testing?

With no building going on you will rather not find bugs in that area, same with scripting
Most of the bugs might have something to do with interacting with the world or other people. If we e.g. look at the group enhancement in 1.12 then testing this would make sense with lots of people who are part of some group. These people are not really present and thus not that much testing is going on
With not many people in the preview also testing load-related bugs might be hard to discover
People might not come back into preview in order for testing if some bugfix by LL is really working for them.

So in general it means not very well tested version (at least regarding the resident side of testing). Another problem might also that nobody knows what actually is done on Linden Lab’s side on testing. So I wouldn’t know which tests are handled by them anyway, like all the repetitive ones, like the unit test boxes at Morris. Moreover, as the bug tracker is not publically visible, you don’t know which bugs are fixed or still being checked in upcoming versions and thus it’s hard to test these again. Also for people reporting bugs it is not always clear what the status of that bug is, whether it’s fixed or not.

So what can be done to improve the situation? Basically the incentive to use the preview grid needs to be improved. We had some pile-on tests someday with Vektor Linden which have been lots of fun not only for the residents but also for Vektor. And there have been about 100 people in the preview at that time. So this and some other points:

– Make being in preview more fun. Do events there with some Lindens around to delegate testing. Make it a community fun event.
– Make it easier to keep up with preview versions by trying to update the version more easily
– Eventually make it possible to copy stuff back from preview to main grid
– Open up the bug tracker for the non-expoitable stuff and enable people to see progress

I think Philip was talking about being able someday in the future to have different versions of software running on the same grid. No idea how that might work e.g. regarding different feature sets but from a technically viewpoint everything is doable just depending on how much work you want to put into it šŸ˜‰

I know these points are not easy to do but making preview more fun at least should work. Jade actually also was proposing maybe some contests like a building contest. This might work and at least some experiment might be nice to have. In the end having residents do the testing saves quite some money from which a fraction is enough to put into some contest.

So maybe other people have more ideas on that. If so, please leave a comment.

Technorati Tags: , , , , , , , , ,




2 responses

28 07 2006
Torley Linden

You do this, I’ll support you.

And if we can accommodate on the Linden side of things, let me know. šŸ™‚

29 07 2006

Well, I do think that this is a job for somebody at Linden Lab as one needs to guide people through the tests (if it’s a pile-on test) and residents don’t know what’s important to test at the individual version.

Moreover (as also pointed out) it helps the QA for Second Life which should be in the interest of Linden Lab. Thus it makes sense to e.g. hire somebody for doing that.

Of course there can be some volunteering program for QA as well. But the problem is again that you need to be more integrated into the process. Thus what Linden Lab can do is to open up this process which can be started be opening up the bugtracker somewhat.

And if there’s contests being held in preview to e.g. catch building bugs then here, too, Linden Lab is asked in e.g. giving the price money.

All in all I don’t know what is done regarding QA at Linden Lab at all, e.g. which tests are done, how automated are they (are they?) and so on. So if I know that certain stuff is tested anyway already then residents don’t need to do that again.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: