Does wasabi2d==1.2.0 bend/break rule no. 2 ?

tizilogic on 2019/10/02 09:22
— edited on 2019/10/02 10:41

While testing/judging, I noticed 2 games in the challenge are using:

wasabi2d==1.2.0

in their respective requirements.txt

Because I've not yet heard of that engine, I went to take a look pypi.org, and to my surprise I saw there, that v1.0.0, the first release on pypi, was released exactly on 22nd of September 2019 and the required version stated above was actually released on the 29th of September. Also the documentation of wasabi2d appeared first on September 21st, when the rules state the following: "You are allowed to use existing libraries that have been available for at least one month before the challenge (and are well documented)".

This is my first PyWeek and I'm no expert on those rules or how to interpret them accurately, let alone on how PyWeek entrants usually judge such circumstances. For what it's worth, I feel like the 2nd and 3rd paragraph of rule 2 are being bent if not outright broken by using this library, first released on pypi on the first day of PyWeek 28 and the final version used, released on the very last day of PyWeek 28. And also the part of "and are well documented" don't meet the requirements. If I had found this library 1 month before the competition, I dare say, I would've not considered using it, mostly because of a lack of documentation at that point.

I'm looking forward to see what other contestants have to say to this, before I decide whether to tick that box or not...

(edit: added also missing/sparse documentation until the first day of the challenge)

(log in to comment)

Comments

tundish on 2019/10/02 18:17:

Just commenting here to bump the thread.

I support anyone who creates libraries for others to use. But clarification on this from those authors would be very welcome.

mauve on 2019/10/02 18:48:

Documentation was included in the GitHub README; it only moved to Read The Docs a couple of weeks ago.

Releasing on PyPI isn't a requirement. For example, Cosmologicon's pygame-text, used in many PyWeeks (including this one?), has never been released there.

But in addition, wasabi2d has been linked from the PyWeek Handbook Libraries page since a month before the competition also. (Anyone can contribute to this page by making a PR.)

So, I'm very sure no rules were broken. But the real evidence is that another entry used the library for a completely different game, and submitted some issues, which I fixed for them. Clearly this is not a personal codebase created for private advantage.

tizilogic on 2019/10/02 19:34:

Thank you for the clarification. I didn't mention that I thought this to be a private codebase to use for personal advantage. It's actually more the highlighted/bold part of the documentation or rather the lack thereof I was concerned about. And if releasing it would be the criteria, then you're obviously right that you posted this correctly at the right place.

What is still a concern is, that you state that the documentation was there in the README, just moved to readthedocs. The documentation about this library on the 23rd of August was in this state: https://github.com/lordmauve/wasabi2d/tree/414aa6f42908dfebec11ca6d0184dba7736b04de and this is what I meant by "I would've not considered using it".

The documentation clearly was extended a great deal, as was the library in the upcoming month to this competition and looking at it now, I'm much more intrigued to trying it out.

I don't want to be a stickler of the rules and I don't think I've earned the right to interpret how those rules should or shouldn't be enforced. So for me, the decision is definitive to not check that box about Cheating. I very much liked both games made with it and it definitely has me interested into digging into wasabi2d sometime in the near future.

This also served as clarification for myself, since I decided against using a library I started to write (opensource and freely available) but it mine is lacking a ton of documentation and isn't even close to be in a usable state by others... So my decision was clear... Now I have an idea on how far along I have to be, one month prior to the next PyWeek and also where to post a link to my library at that point.

Thanks for the great Game Jam!! I loved it and am looking forward to the next one.

rdb on 2019/10/02 20:17:

I noticed this too this morning when I couldn't run one of those games against the wasabi2d version I had installed (which was an August 22 pre-release version built from GitHub, exactly one month before the challenge). For some reason I was also under the impression that this was illegal, so I was already internally debating whether to mark it DNW (because it doesn't work with that version) or to disqualify it (because it relies on what appeared to be an illegal version). I'm honestly a little surprised to learn that this is considered OK. I'm sure I've asked before whether I could release updates to a library I maintain during the challenge itself, and was told that I shouldn't.

Because the rules are a little vague, I suppose that the most lenient possible interpretation is that as long as the code exists somewhere a month before the challenge, and you try your best not to sabotage users of it, it's OK to make any changes at any time. But if you ask me, that interpretation is inconsistent with the spirit of the rules in more than one way:

Others need to have "reasonable opportunity" to use the library. But, if the library is still very immature, or if major features are added the week before PyWeek, then others don't have "reasonable opportunity" to familiarise themselves those features, or seriously consider it in their choice of library. To take this to the logical extreme, what prevents someone from creating a very minimal game support library up on GitHub a month before the challenge, and then continuing to flesh it out right up until the moment of the challenge? Or, instead of releasing a new library, just adding new functionality to an existing library they happen to have somewhere?
The rules use the word "release" and "release a version", whereas no actual release was cut and published, neither on GitHub nor on PyPI, or announced on this board. It can be argued that a GitHub commit hash is a "released version", but it's a bit of a stretch to me. I would apply the same standard to pygame-text (it was at least announced on this board, but no release was cut).

If you ask me, we should clarify the rules with the following points, which I had personally already been sticking to for my libraries:

The library needs to have had an actual release, preferably on PyPI—if not, it should be announced on this board.
The game has to work on a version of the API released a month before the challenge, meaning bugs may still be fixed. This prevents relying on new features or major overhauls done after the 1-month deadline, but allows making use of fixes for bugs discovered after that deadline.

To be clear, I have nothing personal against the entries in question (which look great), or the entrants (who I'm sure acted in good faith), or the library (which looks like something I would definitely consider using in the future). In fact, one of the goals of PyWeek is to "increase the body of Python game tools, code and expertise", so we should encourage the creation of new libraries. I just really think we need to be clear on the rules to ensure a level playing field, so that I don't have to feel restricted from creating updates to my own libraries either.

As for the documentation, it does appear that most of the documentation was added after the Aug 22 deadline, but "well documented" is really open to interpretation. It can be argued that if the other user of wasabi2d has been able to produce a game with it without significant direct support from the author, I would say that the documentation was apparently good enough.

mauve on 2019/10/03 07:31:

I agree we should be clear on the rules, but I would change the rules in the opposite direction. The tools and discipline of maintaining open source libraries has changed a lot since the rules were written (in 2005).

There are a lot of usable libraries on GitHub, which have never been released to PyPI. Keeping documentation in your README is ubiquitous. For example, in What The Frog, I used earcut-python, which only exists on GitHub.

Some people use git submodules for version management.

So I would suggest

The library needs to be publicly available on a versioned service such as GitHub or PyPI, at least 30 days before the competition.
The documentation must be published 30 days before the competition, and must include an overview and API docs.
Rely on versioning to ensure stability for users. Entrants can contribute to libraries during the competition period and use releases made as a result.
Libraries must have an OSI-approved license in order to ensure a level playing field.
As before, libraries may not include gameplay logic.

tundish on 2019/10/03 19:10:

Not relevant to the specific issue regarding wasabi2d, but I'd like to argue for high standards regarding the packaging of libraries.

Throwing code on Github is not sufficient if you intend to distribute and support a library. If you want the user experience to be happy and solid, you must pay attention to the details of proper packaging and documentation.

And no, git submodules are the worst idea ever. We need to educate people in how namespace packages work.

Sorry, this is something of a rant. But right now I see Python as being the best option to deploy code to every OS and up into server land and cloud.

The only way to eliminate issues around installation/deployment is by being fastidious in producing well-formed Python distributions of your library package.

I'm using you in this context to mean we. Not just you, obviously :-)

rdb on 2019/10/03 19:35:

I agree with tundish that if the goal is to encourage a healthy ecosystem of game tools and libraries, it would serve the community better to set reasonable standards. PyPI is (1) the standard repository for Python packages, (2) really easy to publish a package to, if you're followed a few basic guidelines for packaging, and (3) makes the package visible and easy to install for PyWeek users. There's really no good reason not to put the package up on PyPI if you're making a package intended for the Python community at large to benefit from.

I also don't think submodules are a great way to pull in a thirdparty package. If we allow people to refer to an arbitrary Git commit in their code, we just encourage people being sloppy rather than to make a library that maximises its usefulness to the community.

I also think we need some language to prevent someone from slapping a single-function library up on GitHub somewhere and actually making it useful in the week before PyWeek. I still argue to require games to be theoretically runnable against the month-old version of the API (minus the presence of bugs and optimizations), but I might be persuaded that weaker language or a shorter period suffices for that.

As for requiring libraries to have an OSI-approved license, I recall that this sort of issue was discussed before, and at least Richard considered it OK to use closed-source libraries. I'm a little on the fence; I think they should definitely be discouraged, and using them increases the risk of receiving DNWs, but I'm not sure that it should be banned altogether. I think the goal of PyWeek is to promote the Python ecosystem primarily, and open-source only secondarily (if at all).

mauve on 2019/10/03 20:38:

I agree with all you say about PyPI, git submodules etc. There is a certain way of doing things that maximises utility to most Python users.

However, I don't think the rules can codify that. Despite the drawbacks to users, people do publish their libraries purely on GitHub. I think this is even bigger in gamedev than in other applications; gamedev is either Unity, web, or custom stacks, so people publish libraries pretty agnostically. GitHub has a life of its own, beyond the regular Python community, and PyWeek is too little to stand in the way of the these global trends. We roll with them or die.

Plus, it isn't just the Python community. PyWeek is not limited to Python - at least not exclusively. I believe the rule is that entries must be mostly in Python, ie. >50%. But that leaves up to 50% of each entry able to be written in any language - with JavaScript and C/C++ strong contenders. We cannot mandate that library code be published to PyPI if that library code includes JavaScript and pure C++ libraries.

I even think it is basically Ok for someone to find a JavaScript library on GitHub and port it to Python during the competition. Why would this not be Ok if it has met the discoverability test?

On OSI licenses the requirement arises from the "level playing field" principle. I can imagine very few proprietary licenses that guarantee this. A library might be completely free to use, but unless the license is irrevocable and open to absolutely anyone for any purpose, it is still potentially unavailable to some entrant while it is to others.

tundish on 2019/10/03 21:32:

As I see it, there are two worlds.

The first is the world of now. The world of this PyWeek. Not the last one, this one.

The 'game dev' environment has very diverse and heterogeneous stacks, which are changing all the time. When you pile in all the combinations of OpenGL (etc), Unity (etc), Web Canvas API, Javascript libraries and browser compatibility there's an imperative to be up-to-date. Because what's happening right now might actually be the only thing that works.

That world has no future. I can explain why. Wistfully, I went back recently to play a previous PyWeek entry I made in Renpy.

I can't get it to work any more. I wrote it in Python 2 on MacOSX 10.5 and it is now lost to me.

OK, so I'm the only person in the world to miss that particular Renpy game. But are the contestants of PyWeek 28 aware that what they've made might not be playable in 3, 6 or 12 months' time?

The other world, (the second world) is stable, based on open-standards. It might be conservative, dull, technically unambitious. But it will be the bedrock of games written in the Python language.

We need to match the expectations of the current crop of game developers. But balance that with a more staunch effort to provide a solid platform which means Python games written now will still be playable later.

This is not an easy thing to achieve, I know. I don't have any answers, really.

I'd like to know if these comments resonate with anyone?

tizilogic on 2019/10/04 10:19:

@tundish: I write again, just to let you know, that this resonates with me a great deal.

also @rdb makes a good point about the hypothetical case of someone creating and documenting a stub library and then filling in the blanks in the week leading up to PW. I think there's no reason why someone shouldn't be allowed to write or port their own library during PW, but in no way should it be allowed to do what @rdb lined out in his second post, third paragraph.

On a sidenote, I also would be interested in hearing more detail about what you mentioned in a previous post about namespace packages. If I understood you correctly, you are referring to what pep 420 lines out and found its way into Python 3.3 and newer. But this feels like going off topic and probably should be discussed in its own thread.

mauve on 2019/10/04 12:38:

the hypothetical case of someone creating and documenting a stub library and then filling in the blanks in the week leading up to PW. I think there's no reason why someone shouldn't be allowed to write or port their own library during PW, but in no way should it be allowed to do [that]

Yep, rdb mentioned this on IRC. It's a great question. Consider some scenarios:

An established, years old project starts a 2-month project to add comprehensive support for platform games, which is released on the day before of the competition, and the maintainer uses it in PyWeek.
Someone creates a placeholder GitHub library just containing the function vec_to_polar() and its docstring. They push thousands of lines of crudely documented code for platform games on the first day of the competition.
The maintainer of an existing library releases a bunch of new minor features during PyWeek specifically for their own PyWeek entry.

In law there's the test of mens rea, ie. intention. Let's assume the key rule is Rule 2.1 You can’t have a personal “library” codebase that you use during the competition, rule 2.2 is met, and rule 2.3 is met through semver.

I feel that

Case 1 is allowed because it appears that the intention was to deliver the functionality in time for PyWeek, not to create an advantage in PyWeek
Case 2 not allowed because it appears the intention was to get an advantage in PyWeek while complying with the library rules
Case 3 is allowed only if the work was actually done during PyWeek. If it was then PyWeek has merely inspired or highlighted the need for features. If not then they have kept an personal codebase by way of unpublished changes. You could maybe guess which based on the repo history/timestamps, and nature and scale of the changes.

Because we can infer the intention of the author from their actions, we can distinguish between these cases. The argument that if we allow new library releases we have to allow people trying to give themselves an advantage, is a slippery slope argument.

rdb on 2019/10/05 11:15:

Just yet we were talking about how the intent was to create a level playing field. Now we seem to be talking about whether people intend to give themselves an advantage or not. That is definitely not what I am arguing. Whether they intend to or not is irrelevant to whether or not their actions make the playing field uneven. If I accidentally were to make an illegal move in a chess tournament, would the referee really say "Hmm, you didn't intend to, so I'll allow the move"? ¹

Not to mention the fact that intent will in many real-world cases be impossible to guess with absolute certainty and surely inspire many long-winded discussion threads similar to this one. Your constructed scenarios appear to have obvious answers because they are extremes, and real-world examples are likely to be much more intricate and ambiguous.

So, I strongly disagree that case 1 should be allowed. The person who is in some sense adding a new "sub-library" to their library has created an unfair advantage for themselves, whether they intended to or not (which I maintain is impossible to determine, assuming the author isn't blatantly forthright). This disadvantages others in the challenge who were not given a chance to prepare to use this library with these features. (The argument that the intent was to deliver the functionality in time, ergo not to advantage themselves, boggles my mind; surely this hypothetical maintainer is doing this so that it can be used during PyWeek, and they should be aware that releasing it a day before the challenge does not afford others an equal chance to familiarise themselves with it?)

I would contrast this to, say, an example where the maintainer has released a version that fixes a crash in their library that occurs on Windows. In doing so, they have improved the library for all entrants who use it without specifically advantaging themselves; there is no new functionality to learn, after all. There is the risk that the library contains a regression, of course, but entrants can pin their requirements.txt to an older version if this is the case without having to change API calls or missing out on new features.

Anyway, the only way we can reliably enforce this is if we set a clear, binding standard for what we consider acceptable changes that do not leave much to the imagination. (I thought that the existing rules had already clearly set that guideline by recommending that changes are held off until after the challenge, but since it's written only as a recommendation, it is not effective.) I also think it serves us well to err on the cautious side in this regard, although I could be convinced that fewer than 30 days are necessary.

As for modifying a library during the challenge, I'm on the fence. On the one hand, everyone has a chance to modify this library, which only deducts from time spent hacking on their entry. On the other, not everyone has the chance to upload a new version of said library that they can easily reference in their game.

¹ Yes, the analogy is flawed, but it's a better analogy than comparing it to something used to determine a punishment for a crime. The point of disqualification isn't primarily to punish, but to ensure that players have an equal chance, ceteris paribus. Similar to how in a chess game, the outcome is only valid if all players have access to the same moves.

tundish on 2019/10/07 22:29:

Just chiming in again. This time to say; it's never been my objective to win PyWeek.

It seems to me that this discussion is focused around regulating the environment so that no one has an advantage going into the competition.

I don't look at it that way. I think the objective of PyWeek ought to be to share the excellence of those who managed to create good work. And thereby to incrementally improve the opportunities to entrants in PyWeek n+1.

What I'm seeing though is a need to foster a proper understanding around how to create, invigorate and preserve that innovation in order that it can carry over from one competition to another.

Slightly related to that, I created a thread about packaging, etc.

Oh my god. It's bedtime once more.