ADMIN: judging irregularities

richard on 2008/04/20 00:29

It's pretty clear from the judging results that two users (from the same team) have gone through and given identical (word for word in most cases) ratings to many teams.

Please, if you're not going to spend the time and effort play and rate a game yourself then don't bother at all.

(log in to comment)

Comments

Cosmologicon on 2008/04/20 01:02:

I notice that the Disqualify percentages seem underreported. My game is listed as having 0% even though one person (out of 23) voted to DQ for not having a robot (?!?!). In SeaWar (not to pick on it, but I just noticed it), there were 2 out of 13 DQs, and the percentage is listed as 1%.

richard on 2008/04/20 01:34:

DQ is calculated as percentage of *all participants* not just those who rated.

Cosmologicon on 2008/04/20 01:44:

Oh okay, but it says "1% of respondents wished to disqualify the entry" just like it says for DNW. That's why I was confused.

And you need 50% of all entrants to vote DQ before you're disqualified? Man, that's never gonna happen.... Which is fine, I'm just saying.

Tigga on 2008/04/20 20:55:

"It's pretty clear from the judging results that two users (from the same team) have gone through and given identical (word for word in most cases) ratings to many teams."

And many of these results were 1-1-1 which suggests that it may have been tactical as well. Not what we want in PyWeek. I have to say I'm a bit annoyed, as 2/14 of the people whose marks counted for my game (18 respondants, 4 DNW (oops, sorry!)) were these guys, and I'd like to feel I got the score I deserved - instead I just feel a little cheated.

I also agree with cosmo about the DQ - I don't know how many voters there were, but I doubt there were more than a handful (if that) who actually had 50% reply at all.

RB[0] on 2008/04/21 00:27:

Yeah, with the top respondent count, abg4 got 49 voters.
But, even though one guy DQ'd the game, it says 0% wanted to DQ, thus there had to have been an awefully large number of participants.
Thus, if say, we had 150 individual participants, team or solo, then you would have to have a full 75 people vote to DQ a game.
But when only half that number even votes it is impossible....

bjorn on 2008/04/21 00:45:

I think it may help if we can improve the judging interface wrt DNW especially. I know I found it confusing in my first pyweek, and still annoyed now that I have to choose ratings for a game even when I'm marking it DNW. The comment should be the only required element in these cases (and normal ratings should be made unavailable.).

I was a bit annoyed when reading through the ratings my team recieved to see this:

1 1 1 	It didn't work: IOError: [Errno 2] No such file or directory: 'data/story01.png'

Ok, for one thing, this person obviously didn't check the diary entries or the wiki for the fix, which was easy (and a fairly standard case error bug to be found in games that didn't get a final test on linux). But fair enough, it didn't work out of the box so I think I DNW entry is OK . . . except that they didn't mark it DNW! Just gave it 1 1 1. Makes me a little grumpy to see something like that. I'm hoping an interface improvement could help prevent stuff like this, but I'm not sure if it would.

richard on 2008/04/21 01:35:

I've added an idea for a more interactive, more clear interface for ratings. http://code.google.com/p/pyweek/wiki/Improvements

RB[0] on 2008/04/21 15:58:

I think you should have a rule where you must justify your ranking if you give one or more 1's - because a 1 should not really happen, except maybe for a pure clone, but a lot of games have gotten those this time (1/*/* etc.).

The most frustrating thing for me is when they give a really bad rating, but no reason, or they say it didn't work or broke the rules, yet didn't give a DNW/DQ.

Also, I think that a DQ rating should also not count towards the total score of a game, otherwise a DNW shouldn't. Because if someone DQ'd a game, it shouldn't matter if it was awesome in there minds, a lot of games outside of a competition are better than our top games, and if you cheat you are outside of the compo IMO.

Finally, it would be nice to have a little area where you can say how long you played the game, ie 15/30 mins, 2-6 hours etc.

keep up the good work :)

Cosmologicon on 2008/04/21 16:33:

I noticed a couple judges would split their comments up, ie, put "fun", "innovation", and "production" sections into their comment. You could have three separate comment fields, to encourage things like that and add structure to the comments. It would also make it easier to enforce RB[0]'s suggestion of justifying 1's. You could require a comment for any category you rate a 1 in. I don't know if that's something you'd want, but it's something to consider.

nihilocrat on 2008/04/22 01:38:

Yeah, I'd have to say the only really frustrating responses I saw were the "game didn't work no I didn't bother actually reading instructions" and the "I am not going to even bother downloading your choice of library, so I'm voting you 1/1/1 and DNW". In cases where I couldn't get the game to work because I couldn't configure the prerequisite libraries properly, I had the decency to not leave a rating. I even felt like changing a line or two of code in some cases to get things working (because, you know, I can program). It makes me really angry, but I guess I'll just have to get over it. Please, at least give a half-assed rating about the actual content of the game like "not fun, crappy graphics".

Also I noticed someone seems to have "voted twice". I saw ratings like this in many games, complete with the funky single quote. Apparently they don't have the brain cells to be sneaky about things.

1 	1 	1 	 	yes 	The game don´t work, but i don´t know why! I am a graphic artist.
1 	1 	1 		yes 	The game don´t work. But i dont know why!

Being a graphic artist should not preclude you from being able to read instructions or operate a computer. Thanks for worsening my opinion of the "I CAN DOES ARTS BUT HOW DID I PRORGAM GUYS?!?" crowd.

I guess it's just very difficult to have an open, detailed, systematic way of rating things that people are not going to cheat with and always use in the way that's intended (even then, who's to say what's "the way that's intended"?).

gcewing on 2008/04/24 03:01:

It would also help to mention that html markup can be used in the comments (if that's a deliberate feature -- it seems to work anyway). Many people run all their comments together, probably because they don't know you can put paragraph breaks in, etc.

RB[0] on 2008/04/24 14:20:

Actually - I tried that - and it removed all my comment after a link I tried to put in - so at least some html will cause your rating to look, confused ;)

gcewing on 2008/04/26 23:08:

Actually - I tried that - and it removed all my comment after a link I tried to put in

Hmmm, so it seems to be an accidental feature. It would be good to make the use of at least some html official, since long comments that are all run together are hard to read.