ADMIN: disqualifiying for using certain libraries

richard on 2008/04/13 22:03

Several entries (my own included) used libraries that were released very close to the challenge or were even being developed during the challenge. It is quite fair to disqualify those entries on this basis if you wish to. I've always left this up to each individual to decide whether to do so (hence it's part of the ratings.)

Did a library author gain advantage by releasing their library late? In my mind the slightly-late pyglet 1.1 alpha release did not confer a great advantage because the library was well-documented - and many entries used the library.

Did a library author gain disadvantage by using an in-development library? In my case I would say yes. Using the SVN versions of both pyglet and Cocos meant that I lost about a day and a half's worth of development time sorting through bugs in either library.

On the other hand, I have realised that I did not disclose my library use and that some judges may not know. I've done so now, but I also encourage others who used rule-breaking libraries to 'fess up now please :)

(log in to comment)

Comments

riq on 2008/04/13 23:46:

We (the 'Robo-T2' team ) also broke the same rule by using cocos and pyglet from the SVN.

richard on 2008/04/14 00:03:

Personally I don't believe it's fair to disqualify anyone using pyglet 1.1 alpha releases.

Both of the two pyglet developers were using pyglet SVN - anyone using an alpha release is gaining no advantage from doing so (well, apart from using the coolness that is pyglet 1.1 :)

Martin on 2008/04/14 10:28:

Robot Underground also uses pyglet SVN.

kfields on 2008/04/14 10:44:

Guilty. I was wondering about the newness of it at the time before the contest, but if people look back at the posts about using it, I got the feeling that it was acceptable to do so, therefore I did. Guilty with pleasure. :)

richard on 2008/04/14 11:41:

You people are crazy! :)

Tigga on 2008/04/14 12:44:

Given a lot of people want to use PyWeek as a testbed for new libraries, and people are ignoring the one month rule, shouldn't a change be made to the rules?

Can I propose this change to the rules?:
* Lower the 1 month to 1 week for updated libraries (ie. not new, just minor updates)

This will allow people to use up to date versions of popular libraries without allowing people to release a massive personal library shortly before the deadline.

Thoughts?

Cosmologicon on 2008/04/14 14:10:

Yeah, if you don't want a rule to be enforced, you should change it. Maybe in addition to the general one-month rule, you can have a list of "approved" libraries that people can use any versions of, no matter how new.

HanClinto on 2008/04/14 15:04:

I've generally thought that the one-month rule really hindered development of Python libraries rather than helped it. The SDAFF team that I helped out with a little bit in Pyweek4 created their own custom binding to Irrlicht before the competition, but because it wasn't within the one-month boundary, they trashed all their code and re-did it from scratch by hand during the first two days of the contest. I'm not sure I see how the one-month-rule helps quality libraries get developed (at least in that specific case).

dthulson on 2008/04/14 15:39:

Our team used Cocos and Rabbyt. We did not use Cocos 0.2 because it required the Pyglet alpha and we weren't sure that would be allowed. Unfortunately, now that I look at the release dates more closely, it appears that all versions of Cocos were released after the library deadline (see the Cocos download list). Also, the most recent Rabbyt (version 0.8.1) was released on March 3rd. We clearly did not look carefully enough at our dependencies and broke the library release rule.

As far as any changes to the rules go, it does seem like it would be nice to allow entrants to use libraries that they are not involved in. That is, none of us influenced the release of any of our dependencies (at least as far as I know!). Perhaps like Tigga suggested, changes could be made to encourage the use of the most recent versions of common libraries but discourage the use of personal code bases.

humitos on 2008/04/14 21:09:

I think that one have your own method to evaluate competition and I think that libraries used do influence.
For me, if I have to install too many dependences gain disadvantage in my vote. I can't play some games for this reason...
(sorry for my english)

richard on 2008/04/14 21:58:

The rule exists to prevent someone putting in considerable pre-challenge time developing a large library and then announcing it right before the challenge starts. No-one else would be willing to use such an unknown library, so the person creating it potentially gains a huge advantage.

Clearly the devs of Cocos, rabbyt and pyglet (even using the SVN versions) gained little advantage during this challenge.

This does require some further thought.

gcewing on 2008/04/14 23:49:

Perhaps the rule should say that a later version of the library can be used as long as the API is not substantially changed or enlarged since the last admissable release.

HanClinto on 2008/04/15 00:09:

@gcewing: I like that.

alex on 2008/04/15 00:36:

Seeing as disqualification is by vote anyway, why the need for a hard-and-fast rule? How about some suggestions for disqualification such as
  • If you feel this game was not developed in the spirit of PyWeek, for example by making use of a large personal code base, consider voting to disqualify it.
In the past, even with the 1-month rule being clearly violated, and advertised (for example, in my previous pyglet-based entries), there were very few disqualifications actually given out -- so apparently people are already exercising their own judgement anyway.

RB[0] on 2008/04/15 00:46:

gcewing - from what I have seen - pyglet 1.1 is much enlarged from 1.0 ...

Personally - I think a simple:
"Any new libraries must be released at least one month before pyweek to be usable."
And...
"A previously existing and valid library may have new releases at any time and be valid - unless they break backwards compatibility in a significant way."

I'm not sure if that would be any easier to enforce - but it would certainly clear up the whole current mess of pygame, pyglet, cocos and rabbyt IMHO.

gcewing on 2008/04/16 07:23:

Even if it doesn't break backwards compatibility, if it has substantial new features, that could confer an unfair advantage on its author.

Tigga on 2008/04/16 10:05:

"Seeing as disqualification is by vote anyway, why the need for a hard-and-fast rule?"

I think the point is, especially with things like pyglet, is that people judging don't know when the library was releasd.

RB[0] on 2008/04/16 18:21:

gcewing - then what will we define "substantial new features"?

Because, IMO, both pyglet and pygame had pretty substantial new features in their possibly-too-close-to-contest releases.

I think the biggest reason to have any limits at all are for those personal libraries people want to use - so they are announcing them so everyone knows they aren't cheating.

Again, IMO, I would say it limits the "fun" feeling of the competition to have a hard and fast rule that, in effect, would be the same as we now have - except it would exempt minor bug-fixes or changes.

A good part of the fun for me, and others, is the month leading up to pyweek - being able to work on our own libraries up to that point would be nice.
Otherwise we need to just say something like "no new releases or developments after the one-month-before period will be allowed."

And then we are back to square one :/

scav on 2008/04/16 19:49:

I always use PyWeek as a test-bed for my own not-particularly-good library, tdgl. I suspect we will get another high % of DNW this time, underscoring the point that it is really NOT an advantage :-(

Having said that, I always use a "fair" more-than-1-month-old release of tdgl, and monkey-patch as needed to cover bugs I find during writing the game. Then the patches find their way back into SVN, and eventually another release.

If I were going to DQ an entry, it would be if the game didn't work without the latest less-than-1-month-old version of a library. And I won't even bother voting for a game than needs something I can't get working very easily on Debian "testing".

richard on 2008/04/19 07:03:

One entry did not include source. This is a pretty clear violation of the rules, which state "Your entry must include all source code."

Interestingly, not many people are disqualifying it.

ServalKatze on 2008/04/19 09:14:

I suppose many didn't notice. ^^

adam on 2008/04/19 23:02:

I don't think the way Pyweek is presented at the moment makes it clear that responsibility for enforcing the rules lies with all of the judges. It's easy to assume that matters of fact such as "does this entry include source code?" aren't part of the game-judging process, whereas actually if judges neglect this a game can get through even if it demonstrably violates the rules.

nihilocrat on 2008/04/22 02:02:

I believe the cutoff date for libraries was Feburary 28th... I used Panda3D 1.5.0, which was actually released sometime mid-March; I didn't even realize this until I saw this post of yours. Sorry for 'cheating'! To be honest, I didn't use any of the new features added since 1.4.2.