The Red Spider Project

A place to discuss the implementation and style of computer programs.

Moderators: phlip, Moderators General, Prelates

Rpgamer12
Posts: 8
Joined: Mon Jan 02, 2012 3:33 pm UTC

Re: Idea for a fun joint effort: xkcd hacking project

Postby Rpgamer12 » Mon Mar 26, 2012 10:25 pm UTC

Using an adventure type game as inspiration:
We could make a new web browser, except it'll be a web-ADVENTURER! You could make a pretty cool game to 'browse' the web with; websites would be like buildings, pages would be like rooms, pictures as wall art. It's something that would be cool/fun to make.

User avatar
DavidH
Posts: 80
Joined: Mon Aug 03, 2009 3:53 am UTC

Re: Idea for a fun joint effort: xkcd hacking project

Postby DavidH » Mon Mar 26, 2012 10:54 pm UTC

Jplus wrote:
qubital wrote:Updated the name of the repo.
Although this is going to be a "heterogenous" project, it still needs to be coherent. We definitely need a real plan and timetable moving forward, as Mat suggested.

I kind of agree that some level of coherence is a good idea, but I think that should be possible without a timetable. We have a stable branch which can take care of the coherence; when people make additions to the project, they simply base it on the stable branch. When done, they merge it back into the stable branch so other people can base their work on that as well.

Besides, I see no reason why everyone should be working on the same thing at the same time. Why can't Nautilus and Mat write a code obfuscator on one branch while you write a 2d game together with some other people on another branch?


I agree. This was Jplus's idea, so correct me if I'm wrong, but this is how I interpreted it. I envisioned this as something anyone can add anything to, with the exception that it can't be malicious (obviously), and shouldn't change anything outside of its directory.

For example, if I wanted to add something that played a random song. I'd make a subfolder with my program, and I'd change the code in the main program to accept a "song" command, which would then fork and run my little code. My idea was that everything we each do should be encapsulated (with exceptions. We should be able to make small modifications to each others code if done tastefully) in our own sub directory, and the only changes to the main program would be adding the execv functionality or what have you. Did I understand this correctly?

Unrelated, I assume we're not doing this all in interpreted code, right? I'd like to have a general make file (or cross platform equivalent) that simply calls the make file in each sub directory for us who'd like to do things in compiled code like C.

Edit: Maybe even have the python shell we have written call the make file itself upon starting. Stream line things a bit.

qubital
Posts: 25
Joined: Fri Mar 02, 2012 12:13 am UTC

Re: Idea for a fun joint effort: xkcd hacking project

Postby qubital » Mon Mar 26, 2012 11:35 pm UTC

scarecrovv wrote:
qubital wrote:
scarecrovv wrote:
zed0 wrote:can I suggest that we change the-xkcd-community to be an organisation and then add active people to it, this would allow for easier pushing to the main branch as multiple people would be able to do it without re-logging in as the user.
You can change the user into an organisation at https://github.com/settings/organizations

That sounds like a very good idea to me.

Me too, although someone else will need to do it since I don't have a separate account available to adminster it. I think that's necessary, not sure though.

Done. zed0, ahammel, and I are the owners. We can add people as necessary. qubital's email is still the official one. Do we in general or qubital in particular care about this?

Nope, you can change that too.

User avatar
scarecrovv
It's pronounced 'double u'
Posts: 674
Joined: Wed Jul 30, 2008 4:09 pm UTC
Location: California

Re: Idea for a fun joint effort: xkcd hacking project

Postby scarecrovv » Tue Mar 27, 2012 3:27 am UTC

I'm working on an xkcd comic fetcher (see my "xkcd-fetch" branch). It caches comics and all the metadata (number, title, image, title-text, date, and transcript) in the "work/xkcd-fetch" directory. I'm working on commenting it better. It could use improvement. Eventually I'll include options for outputting the data properly instead of simply leaving some files around, but one thing at a time.

edit: I've added some comments, and done some testing. It seems to work fairly well so far. It also gives output now, but the format could probably use some work (I just borrowed the format I made up for the cache file). The glaring omission at this point is that it looks for and/or creates the "work/xkcd-fetch" directory in "../work/xkcd-fetch" instead of a correct absolute path. I think I need that "first command" Jplus mentioned in the first post, in order to determine the correct location. I might start thinking about it tomorrow.

User avatar
Mat
Posts: 414
Joined: Fri Apr 21, 2006 8:19 pm UTC

Re: Idea for a fun joint effort: xkcd hacking project

Postby Mat » Tue Mar 27, 2012 8:09 am UTC

zed0 wrote:I started on an adventure shell, it's in my forked project at: https://github.com/zed0/the-red-spider-project
It's written in Python which I've never used before so the code is probably pretty horrible.


Cool, I might add to it this weekend, if I don't get distracted by other things. I can help with the python part at least :)

User avatar
scarecrovv
It's pronounced 'double u'
Posts: 674
Joined: Wed Jul 30, 2008 4:09 pm UTC
Location: California

Re: Idea for a fun joint effort: xkcd hacking project

Postby scarecrovv » Tue Mar 27, 2012 1:56 pm UTC

Ok, I've written initial implementations that I'm reasonably satisfied with for the first command, and a comic fetcher. They're the rscall and xkcd-fetch branches respectively. I've pushed those branches to the-xkcd-community. If you guys want to try them out, and like them, we can merge them into the master branch.

User avatar
Jplus
Posts: 1721
Joined: Wed Apr 21, 2010 12:29 pm UTC
Location: Netherlands

Re: Idea for a fun joint effort: xkcd hacking project

Postby Jplus » Tue Mar 27, 2012 4:08 pm UTC

I love you folks. So good to see the project take form. Especially because so far, I haven't done much by myself yet to realise my idea. I can't wait until I can start contributing. :)

qubital wrote:I've changed the name to "the-xkcd-community" to help ease your fears: https://github.com/the-xkcd-community

Thanks. I think it's acceptable to call ourselves "the xkcd community" as long as any xkcd fan will be able to join in and perhaps launch other repositories. I'm also open to another name change if that's at all possible. Something like "Real Programmers"?

scarecrovv wrote:One thing I think we should work out now, before too many people contribute, is licensing. How about this? It's the MIT license, with the first line tailored appropriately (I think):

Spoiler:
All work is copyright (C) 2012 by the respective authors

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Will that work? I'm not a lawyer, so I'm not 100% sure. Also is everybody happy with it (particularly Jplus since it's your idea, and qubital, zed0, and ahammel, since you've already started writing)?

I'm fine with a flat MIT license. It's already proven itself so we can safely assume that it will work. One comment though: if the license is provided in a separate file (like License.txt in the project root directory) then it's customary to leave out the copyright notice. Instead each source file contains its own copyright notice together with a statement that the license can be found in /License.txt. The top line and the middle paragraph will then need some adjustment (adopted from the Boost license, which has also proven itself):
Spoiler:
The Red Spider Project License - 27. March 2012

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The copyright notices in the Software and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
We can do some non-intrusive additions to make it more xkcd-like (spot the differences):
Spoiler:
The Red Spider Project License - 27. March 2012

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify (with or without emacs C-x M-c M-butterfly), merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The copyright notices in the Software and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, SAFETY FROM VELOCIRAPTORS AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
I'm not entirely sure that the latter is safe, so we should probably check somebody who's into copyright before adopting that one.

scarecrovv wrote:
zed0 wrote:can I suggest that we change the-xkcd-community to be an organisation and then add active people to it, this would allow for easier pushing to the main branch as multiple people would be able to do it without re-logging in as the user.

Done. zed0, ahammel, and I are the owners. We can add people as necessary. qubital's email is still the official one. Do we in general or qubital in particular care about this?

Good idea, well done, and I don't care about the email address either.

Rpgamer12 wrote:Using an adventure type game as inspiration:
We could make a new web browser, except it'll be a web-ADVENTURER! You could make a pretty cool game to 'browse' the web with; websites would be like buildings, pages would be like rooms, pictures as wall art. It's something that would be cool/fun to make.

Yes. Seems hard but could definitely be fun. Probably easiest to start with a text-based "browser" that does simple HTML processing with regexes (you can make it prettier later on). If you actually want to render pages you can use the system's default browser for the time being. Go for it!

DavidH wrote:
Jplus wrote:
qubital wrote:Although this is going to be a "heterogenous" project, it still needs to be coherent. [...]
I kind of agree that some level of coherence is a good idea, but [...]

Besides, I see no reason why everyone should be working on the same thing at the same time. [...]

I agree. This was Jplus's idea, so correct me if I'm wrong, but this is how I interpreted it. I envisioned this as something anyone can add anything to, with the exception that it can't be malicious (obviously), and shouldn't change anything outside of its directory.

Correct.

DavidH wrote:For example, if I wanted to add something that played a random song. I'd make a subfolder with my program, and I'd change the code in the main program to accept a "song" command, which would then fork and run my little code. My idea was that everything we each do should be encapsulated (with exceptions. We should be able to make small modifications to each others code if done tastefully) in our own sub directory, and the only changes to the main program would be adding the execv functionality or what have you. Did I understand this correctly?

Actually I was proposing to pool all of our code in the same directories, unless some subproject really takes lots of files. The idea was to encourage each other to read our code in that way. Also, that "main program" that forks in order to spawn processes for the subprojects is not exactly what I meant. I was proposing to make a "first command" which creates a suitable environment for the other commands, but after that the other commands should be able to run by themselves. (It seems that scarecrovv did the fork thing though, which make me think maybe my idea was not so realistic. In that case perhaps we should make the first command call the system shell so the effect will be the same.) Other than that, everything you said about making tasteful modifications to programs from others and adding functionality to the "first command" matches my idea.

DavidH wrote:Unrelated, I assume we're not doing this all in interpreted code, right? I'd like to have a general make file (or cross platform equivalent) that simply calls the make file in each sub directory for us who'd like to do things in compiled code like C.

This is along the lines I proposed. Write in Fortran or Haskell if you like. Anything that is portable across our "supported platforms" will be added to the general makefile (I've suggested to use CMake but we could also choose SCons, Perforce Jam or something else).

Come to speak of it: we should probably agree on some initial list of supported platforms. I guess it should at least include Windows XP SP3+, Mac OS X 10.5+ and any recent variants of Debian Linux (such as Ubuntu 10.04+). Those platforms are most popular and we can probably manage to test them.

DavidH wrote:Edit: Maybe even have the python shell we have written call the make file itself upon starting. Stream line things a bit.

I'm not sure. What do others think?

scarecrovv wrote:Ok, I've written initial implementations that I'm reasonably satisfied with for the first command, and a comic fetcher. They're the rscall and xkcd-fetch branches respectively. I've pushed those branches to the-xkcd-community. If you guys want to try them out, and like them, we can merge them into the master branch.

Great! We definitely needed that. I guess we should call it "rscall" instead of "first command" from now on. Keep up the good work.
"There are only two hard problems in computer science: cache coherence, naming things, and off-by-one errors." (Phil Karlton and Leon Bambrick)

coding and xkcd combined

(Julian/Julian's)

User avatar
scarecrovv
It's pronounced 'double u'
Posts: 674
Joined: Wed Jul 30, 2008 4:09 pm UTC
Location: California

Re: Idea for a fun joint effort: xkcd hacking project

Postby scarecrovv » Tue Mar 27, 2012 6:39 pm UTC

Jplus wrote:I'm fine with a flat MIT license. It's already proven itself so we can safely assume that it will work. One comment though: if the license is provided in a separate file (like License.txt in the project root directory) then it's customary to leave out the copyright notice. Instead each source file contains its own copyright notice together with a statement that the license can be found in /License.txt. The top line and the middle paragraph will then need some adjustment (adopted from the Boost license, which has also proven itself):
Spoiler:
The Red Spider Project License - 27. March 2012

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The copyright notices in the Software and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
We can do some non-intrusive additions to make it more xkcd-like (spot the differences):
Spoiler:
The Red Spider Project License - 27. March 2012

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify (with or without emacs C-x M-c M-butterfly), merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The copyright notices in the Software and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, SAFETY FROM VELOCIRAPTORS AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
I'm not entirely sure that the latter is safe, so we should probably check somebody who's into copyright before adopting that one.

I like it. I think I'll add the first version to the repository (we can change it to the second if somebody can assure us it's safe).

Jplus wrote:Also, that "main program" that forks in order to spawn processes for the subprojects is not exactly what I meant. I was proposing to make a "first command" which creates a suitable environment for the other commands, but after that the other commands should be able to run by themselves. (It seems that scarecrovv did the fork thing though, which make me think maybe my idea was not so realistic. In that case perhaps we should make the first command call the system shell so the effect will be the same.) Other than that, everything you said about making tasteful modifications to programs from others and adding functionality to the "first command" matches my idea.

I actually like it your way better, I just couldn't figure out a way to make it cross platform. I think rscall (red spider call) is an ugly hack, and if you can figure out something better to replace it, please do. I just wanted something temporary to let me get work done on xkcd-fetch.

Chaosboye
Posts: 5
Joined: Mon May 09, 2011 2:32 am UTC

Re: Idea for a fun joint effort: xkcd hacking project

Postby Chaosboye » Tue Mar 27, 2012 7:46 pm UTC

Just an idea for something to add to the XKCD comic retriever program:

What if you add tags to the comics? Then, if you want a comic about lesbians, you'll get #507, #575, #717, etc. If you want a comic about python, you'll get all the comics with the "python" tag, and so on and so forth.

I'm a terrible coder myself at the moment, so I would be of no help, but I thought this would be a cool way to add to your work.

P.S. I love the Red Spider Project overall and I look forward to seeing what everyone comes up with.

User avatar
Mat
Posts: 414
Joined: Fri Apr 21, 2006 8:19 pm UTC

Re: Idea for a fun joint effort: xkcd hacking project

Postby Mat » Tue Mar 27, 2012 8:12 pm UTC

Jplus wrote:I love you folks. So good to see the project take form. Especially because so far, I haven't done much by myself yet to realise my idea. I can't wait until I can start contributing. :)

qubital wrote:I've changed the name to "the-xkcd-community" to help ease your fears: https://github.com/the-xkcd-community

Thanks. I think it's acceptable to call ourselves "the xkcd community" as long as any xkcd fan will be able to join in and perhaps launch other repositories. I'm also open to another name change if that's at all possible. Something like "Real Programmers"?

scarecrovv wrote:One thing I think we should work out now, before too many people contribute, is licensing. How about this? It's the MIT license, with the first line tailored appropriately (I think):

Spoiler:
All work is copyright (C) 2012 by the respective authors

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Will that work? I'm not a lawyer, so I'm not 100% sure. Also is everybody happy with it (particularly Jplus since it's your idea, and qubital, zed0, and ahammel, since you've already started writing)?

I'm fine with a flat MIT license. It's already proven itself so we can safely assume that it will work. One comment though: if the license is provided in a separate file (like License.txt in the project root directory) then it's customary to leave out the copyright notice. Instead each source file contains its own copyright notice together with a statement that the license can be found in /License.txt. The top line and the middle paragraph will then need some adjustment (adopted from the Boost license, which has also proven itself):
Spoiler:
The Red Spider Project License - 27. March 2012

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The copyright notices in the Software and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
We can do some non-intrusive additions to make it more xkcd-like (spot the differences):
Spoiler:
The Red Spider Project License - 27. March 2012

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify (with or without emacs C-x M-c M-butterfly), merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The copyright notices in the Software and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, SAFETY FROM VELOCIRAPTORS AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
I'm not entirely sure that the latter is safe, so we should probably check somebody who's into copyright before adopting that one.

There's always the WTFPL http://sam.zoy.org/wtfpl/

DavidH wrote:For example, if I wanted to add something that played a random song. I'd make a subfolder with my program, and I'd change the code in the main program to accept a "song" command, which would then fork and run my little code. My idea was that everything we each do should be encapsulated (with exceptions. We should be able to make small modifications to each others code if done tastefully) in our own sub directory, and the only changes to the main program would be adding the execv functionality or what have you. Did I understand this correctly?

Actually I was proposing to pool all of our code in the same directories, unless some subproject really takes lots of files. The idea was to encourage each other to read our code in that way. Also, that "main program" that forks in order to spawn processes for the subprojects is not exactly what I meant. I was proposing to make a "first command" which creates a suitable environment for the other commands, but after that the other commands should be able to run by themselves. (It seems that scarecrovv did the fork thing though, which make me think maybe my idea was not so realistic. In that case perhaps we should make the first command call the system shell so the effect will be the same.) Other than that, everything you said about making tasteful modifications to programs from others and adding functionality to the "first command" matches my idea.

It sounds like it's going to be messy if we have everything into one or two directories. Can't we have subdirectories under src for different things instead of using src/ and include/?

When you say "creates a suitable environment", is this a shell that runs the other commands?

DavidH wrote:Unrelated, I assume we're not doing this all in interpreted code, right? I'd like to have a general make file (or cross platform equivalent) that simply calls the make file in each sub directory for us who'd like to do things in compiled code like C.

This is along the lines I proposed. Write in Fortran or Haskell if you like. Anything that is portable across our "supported platforms" will be added to the general makefile (I've suggested to use CMake but we could also choose SCons, Perforce Jam or something else).

Come to speak of it: we should probably agree on some initial list of supported platforms. I guess it should at least include Windows XP SP3+, Mac OS X 10.5+ and any recent variants of Debian Linux (such as Ubuntu 10.04+). Those platforms are most popular and we can probably manage to test them.

Makes sense to me. Speaking of testing, shouldn't we have a test/ (with the same layout as src/)?

User avatar
Nautilus
Posts: 105
Joined: Thu Feb 25, 2010 1:19 am UTC
Location: That's a strange place to put a piano

Re: Idea for a fun joint effort: xkcd hacking project

Postby Nautilus » Tue Mar 27, 2012 8:48 pm UTC

Forked, I'll add something this weekend, when I will be moderately less busy.
We should update Readme.md, to keep up with changes in the thread. Actually, that's something that I can do, gimme a day or two

Mat wrote:When you say "creates a suitable environment", is this a shell that runs the other commands?

I think "suitable environment" is meant in terms of creating/updating environment variables- as of yet we're only setting the variable $RED_SPIDER_ROOT, but we'll probably be doing more in the future. We want to make sure that a) system stuff will behave consistently and b) our stuff will be able to access certain information it needs consistently.

Also, can we add the agreed-upon directory structure to master? Does github not save folders unless they have stuff in them?
My flagella bring all the boys to the yard

User avatar
Jplus
Posts: 1721
Joined: Wed Apr 21, 2010 12:29 pm UTC
Location: Netherlands

Re: Idea for a fun joint effort: xkcd hacking project

Postby Jplus » Tue Mar 27, 2012 9:56 pm UTC

Ideally, the copyright notice in the individual source files should be a bit more explicit on where to find the license. So, instead of

Code: Select all

# See License.txt
rather something like

Code: Select all

# See RED_SPIDER_ROOT/License.txt for licensing information.
or

Code: Select all

# Licensed under the Red Spider Project License.
# See the License.txt that shipped with your copy of this software for details.


scarecrovv wrote:
Jplus wrote:Also, that "main program" that forks in order to spawn processes for the subprojects is not exactly what I meant. I was proposing to make a "first command" which creates a suitable environment for the other commands, but after that the other commands should be able to run by themselves. (It seems that scarecrovv did the fork thing though, which make me think maybe my idea was not so realistic. In that case perhaps we should make the first command call the system shell so the effect will be the same.)

I actually like it your way better, I just couldn't figure out a way to make it cross platform. I think rscall (red spider call) is an ugly hack, and if you can figure out something better to replace it, please do. I just wanted something temporary to let me get work done on xkcd-fetch.

I guessed that you didn't get my idea to work (yet), but for me that was a reason to think that perhaps it simply isn't possible. If there's still a reason to try it when I join in (probably next week), I'll see what I can do. Did you find a way to do it on one platform, and if so, could you publish that on a new branch?

On a not entirely unrelated note: the way we've talked about it so far, if something isn't on the master branch (which appears to be our sacred/stable branch) then users shouldn't expect it to work, so it doesn't matter if it isn't portable.

Chaosboye wrote:Just an idea for something to add to the XKCD comic retriever program:

What if you add tags to the comics? Then, if you want a comic about lesbians, you'll get #507, #575, #717, etc. If you want a comic about python, you'll get all the comics with the "python" tag, and so on and so forth.

That seems like a fun and very useful idea! So many times I racked my brains over which comic was the one with the computer that complains about segfaults. To make it slightly more modular you could make this a separate command that just outputs a bunch of comic numbers on a given keyword. Let's call it "key-xkcd" for now. The numbers from key-xkcd then could, but need not, be fed into xkcd-fetch to display the matching comics.
(For the generics maniacs among us: we should probably add a "foreach" command at some point.)

Chaosboye wrote:I'm a terrible coder myself at the moment, so I would be of no help, but I thought this would be a cool way to add to your work.

You could also consider it a nice way to improve your coding skills. You'll have to learn to use git as well, but that's a very useful skill too. You can take as much time as you want, because nobody's life is depending on this project (let alone on key-xkcd), and you don't have to worry that others might be hindered by your unfinished code because that's what the branches are for. We can also help you if you like.

For general coding help there's always How Do I Learn About [n]? and the rest of the Coding forum.

Chaosboye wrote:P.S. I love the Red Spider Project overall and I look forward to seeing what everyone comes up with.

That's good to hear.

Mat wrote:There's always the WTFPL http://sam.zoy.org/wtfpl/

What do the others think? Personally I prefer the current license as it's more complete, especially the xkcd-ified variant which is more xkcd-like and which I estimate should be at least as safe in court as the WTFPL.

Mat wrote:It sounds like it's going to be messy if we have everything into one or two directories. Can't we have subdirectories under src for different things instead of using src/ and include/?

I don't think we really disagree here. There is no point in making separate subdirectories (under src/ or include/) if a program consists of only one or two files. I suspect such small programs will form a majority, especially in the beginning. If on the other hand the program consists of a hundred files, then a dedicated subdirectory is a necessity. Where to draw the line, that is the question, but the answer will be kind of arbitrary. We can always discuss if necessary.

Mat wrote:Speaking of testing, shouldn't we have a test/ (with the same layout as src/)?

Definitely. Note that many programs won't have a separate test program, though. In most cases it will be enough to run the program itself as a means of having it tested before merging it back into master.

Nautilus wrote:Forked, I'll add something this weekend, when I will be moderately less busy.
We should update Readme.md, to keep up with changes in the thread. Actually, that's something that I can do, gimme a day or two

Cool.

Nautilus wrote:
Mat wrote:When you say "creates a suitable environment", is this a shell that runs the other commands?

I think "suitable environment" is meant in terms of creating/updating environment variables- as of yet we're only setting the variable $RED_SPIDER_ROOT, but we'll probably be doing more in the future. We want to make sure that a) system stuff will behave consistently and b) our stuff will be able to access certain information it needs consistently.

This. Though I'm thinking that one way to achieve this might be by forking a new shell.

One additional environment variable that might be useful to update now already is PYTHONPATH.

Nautilus wrote:Also, can we add the agreed-upon directory structure to master? Does github not save folders unless they have stuff in them?

Git doesn't seem to like empty trees in general, but I don't know whether that's the reason the other directories haven't showed up yet. If somebody finds a way to make them show up, please do.
"There are only two hard problems in computer science: cache coherence, naming things, and off-by-one errors." (Phil Karlton and Leon Bambrick)

coding and xkcd combined

(Julian/Julian's)

EvanED
Posts: 4331
Joined: Mon Aug 07, 2006 6:28 am UTC
Location: Madison, WI
Contact:

Re: Idea for a fun joint effort: xkcd hacking project

Postby EvanED » Tue Mar 27, 2012 10:10 pm UTC

Git flat out ignores empty directories. The canonical "solution" is to put some empty file in it with a dot name.

User avatar
scarecrovv
It's pronounced 'double u'
Posts: 674
Joined: Wed Jul 30, 2008 4:09 pm UTC
Location: California

Re: Idea for a fun joint effort: xkcd hacking project

Postby scarecrovv » Tue Mar 27, 2012 10:32 pm UTC

Jplus wrote:
Nautilus wrote:Also, can we add the agreed-upon directory structure to master? Does github not save folders unless they have stuff in them?

Git doesn't seem to like empty trees in general, but I don't know whether that's the reason the other directories haven't showed up yet. If somebody finds a way to make them show up, please do.

EvanED wrote:Git flat out ignores empty directories. The canonical "solution" is to put some empty file in it with a dot name.

Done.

Jplus wrote:Ideally, the copyright notice in the individual source files should be a bit more explicit on where to find the license. So, instead of

Code: Select all

# See License.txt
rather something like

Code: Select all

# See RED_SPIDER_ROOT/License.txt for licensing information.
or

Code: Select all

# Licensed under the Red Spider Project License.
# See the License.txt that shipped with your copy of this software for details.

Ok. What about who the copyright should be assigned to? I just picked myself somewhat arbitrarily, but can/should I assign it to "The Red Spider Project" or something like that?

Jplus wrote:
scarecrovv wrote:I actually like it your way better, I just couldn't figure out a way to make it cross platform. I think rscall (red spider call) is an ugly hack, and if you can figure out something better to replace it, please do. I just wanted something temporary to let me get work done on xkcd-fetch.

I guessed that you didn't get my idea to work (yet), but for me that was a reason to think that perhaps it simply isn't possible. If there's still a reason to try it when I join in (probably next week), I'll see what I can do. Did you find a way to do it on one platform, and if so, could you publish that on a new branch?

I had some vauge ideas, but they all failed the cross-platform test at first glance, so I didn't pursue them very far. I'll look into it.

Jplus wrote:On a not entirely unrelated note: the way we've talked about it so far, if something isn't on the master branch (which appears to be our sacred/stable branch) then users shouldn't expect it to work, so it doesn't matter if it isn't portable.

Cool.

Jplus wrote:
Chaosboye wrote:Just an idea for something to add to the XKCD comic retriever program:

What if you add tags to the comics? Then, if you want a comic about lesbians, you'll get #507, #575, #717, etc. If you want a comic about python, you'll get all the comics with the "python" tag, and so on and so forth.

That seems like a fun and very useful idea! So many times I racked my brains over which comic was the one with the computer that complains about segfaults. To make it slightly more modular you could make this a separate command that just outputs a bunch of comic numbers on a given keyword. Let's call it "key-xkcd" for now. The numbers from key-xkcd then could, but need not, be fed into xkcd-fetch to display the matching comics.
(For the generics maniacs among us: we should probably add a "foreach" command at some point.)

Nice idea. If key-xkcd is going to be seperate from xkcd-fetch, I propose that the tags be stored in a seperate file from comic-data.txt, so that changes to key-xkcd don't affect xkcd-fetch.

User avatar
Jplus
Posts: 1721
Joined: Wed Apr 21, 2010 12:29 pm UTC
Location: Netherlands

Re: Idea for a fun joint effort: xkcd hacking project

Postby Jplus » Wed Mar 28, 2012 12:24 am UTC

scarecrovv wrote:
EvanED wrote:Git flat out ignores empty directories. The canonical "solution" is to put some empty file in it with a dot name.

Done.

Thanks, it looks better this way. How about bin/, lib/, test/ and work/?

scarecrovv wrote:What about who the copyright should be assigned to? I just picked myself somewhat arbitrarily, but can/should I assign it to "The Red Spider Project" or something like that?

You did the right thing. (How is picking yourself arbitrary?) Copyright by default always belongs to the person(s) who created the thing in question. If you'd assign it to the red spider project instead, that would seem to suggest you transfer your rights to the work, which is kind of risky because it's not very clear who's "The Red Spider Project" (in fact I'm not entirely sure whether the red spider project can hold copyright at all since it isn't legally registered anywhere). It might for example be qubital because they created the GitHub account, or some organisation that we don't know about that happens to have "The Red Spider Project" as a name. Regardless, there's no reason why you shouldn't give yourself the credit you deserve; after all, we're not some kind of organisation that needs to make money*.

You might want to also refer to the red spider project in each source file. You could do that with an additional comment line somewhere near the start of your file. I don't mind whether and how you do that (the copyright and license information is much more important), but if people feel there is a need we can agree on some kind of uniform file header format.

______
*) There's a catch with organisations: if you plan to develop code for the red spider project at work, first make sure that your employer is ok with it and that they won't claim copyright over it. Because in some cases your contract might entitle them to do such a thing.
"There are only two hard problems in computer science: cache coherence, naming things, and off-by-one errors." (Phil Karlton and Leon Bambrick)

coding and xkcd combined

(Julian/Julian's)

User avatar
scarecrovv
It's pronounced 'double u'
Posts: 674
Joined: Wed Jul 30, 2008 4:09 pm UTC
Location: California

Re: Idea for a fun joint effort: xkcd hacking project

Postby scarecrovv » Wed Mar 28, 2012 12:57 am UTC

Jplus wrote:Thanks, it looks better this way. How about bin/, lib/, test/ and work/?

I forgot test/. However it seemed to me that the files in bin/, lib/, and work/ would be generated by running the software in the red spider project, and not tracked by git. Perhaps that's a faulty assumption, in which case we can change it.

Jplus wrote:You did the right thing. (How is picking yourself arbitrary?) Copyright by default always belongs to the person(s) who created the thing in question. If you'd assign it to the red spider project instead, that would seem to suggest you transfer your rights to the work, which is kind of risky because it's not very clear who's "The Red Spider Project" (in fact I'm not entirely sure whether the red spider project can hold copyright at all since it isn't legally registered anywhere). It might for example be qubital because they created the GitHub account, or some organisation that we don't know about that happens to have "The Red Spider Project" as a name. Regardless, there's no reason why you shouldn't give yourself the credit you deserve; after all, we're not some kind of organisation that needs to make money*.

You might want to also refer to the red spider project in each source file. You could do that with an additional comment line somewhere near the start of your file. I don't mind whether and how you do that (the copyright and license information is much more important), but if people feel there is a need we can agree on some kind of uniform file header format.

______
*) There's a catch with organisations: if you plan to develop code for the red spider project at work, first make sure that your employer is ok with it and that they won't claim copyright over it. Because in some cases your contract might entitle them to do such a thing.

Fair enough. I just figured that if everybody who ever changes a file added themselves to a list of authors at the top it might get a bit unweildy after a while, and that we ought to have some way to abbreviate that. What if we had a file like "Authors.txt" in the project root, and listed everybody there? If you commit for the first time, add yourself to that file. If there's ever a question about who added what, git keeps a record.

So for header format, how about this (adjust comments to language):

Code: Select all

# Copyright YYYY by authors listed in the Authors.txt that shipped with your copy of this software.
# Licensed under the Red Spider Project License.
# See the License.txt that shipped with your copy of this software for details.


I've also made a new branch rsshell with an alternative to rscall. Instead of calling a command specified as an argument, it starts another shell. Now you have a shell running in a shell, but the inner shell has the correct environment.

User avatar
Jplus
Posts: 1721
Joined: Wed Apr 21, 2010 12:29 pm UTC
Location: Netherlands

Re: Idea for a fun joint effort: xkcd hacking project

Postby Jplus » Wed Mar 28, 2012 10:07 am UTC

scarecrovv wrote:
Jplus wrote:Thanks, it looks better this way. How about bin/, lib/, test/ and work/?

I forgot test/. However it seemed to me that the files in bin/, lib/, and work/ would be generated by running the software in the red spider project, and not tracked by git. Perhaps that's a faulty assumption, in which case we can change it.

Yeah, you're right about bin/, lib/ and work/. And build/, which I forgot to mention.

scarecrovv wrote:Fair enough. I just figured that if everybody who ever changes a file added themselves to a list of authors at the top it might get a bit unweildy after a while,
Don't worry about that. Most of the time we can comfortably fit four (!) author names on a single line, and we can always add a second copyright statement if that seems appropriate. We can also treat authors who did the actual coding differently from authors who made only small contributions (such as typos or hard-coded file renames).
scarecrovv wrote:and that we ought to have some way to abbreviate that. What if we had a file like "Authors.txt" in the project root, and listed everybody there? If you commit for the first time, add yourself to that file. If there's ever a question about who added what, git keeps a record.
I like the Authors.txt, but I think it shouldn't be a catch-all. If you wrote parts of the code in some program, then your name should appear at the top. If somebody wants to find out who holds the rights to a work they shouldn't have to search through GitHub or git-log (besides, that possibility might not exist because anyone may modify and re-distribute the project).

With that said, I'd propose a slightly different header format:

Code: Select all

# Copyright YYYY __authors_of_major_contributions__ ,
# with minor contributions by other authors;
# please refer to the Authors.txt that shipped with your copy of this software.
# Licensed under the Red Spider Project License.
# See the License.txt that shipped with your copy of this software for details.

# Acknowledgements: X provided idea A, Y provided idea B.
The minor contributions clause and the acknowledgements are optional. When somebody makes a contribution to a source file they can decide for themselves whether they should be on the first line or in Authors.txt, though I think a guiding question should be: "Did I contribute to the program logic?".

We could also move the minor contributions line to the acknowledgements:

Code: Select all

# Copyright YYYY __authors_of_major_contributions__
# Licensed under the Red Spider Project License.
# See the License.txt that shipped with your copy of this software for details.

# Acknowledgements: X provided idea A, Y provided idea B.
# Minor contributions were made by Z/by several authors;
# please refer to the Authors.txt that shipped with your copy of this software.

Another optional addition is to prepend a comment with the filename.

scarecrovv wrote:I've also made a new branch rsshell with an alternative to rscall. Instead of calling a command specified as an argument, it starts another shell. Now you have a shell running in a shell, but the inner shell has the correct environment.

Cool. To comment on the "double exit quirk": don't worry about it. In this respect rsshell is just like any interactive terminal program. :)
"There are only two hard problems in computer science: cache coherence, naming things, and off-by-one errors." (Phil Karlton and Leon Bambrick)

coding and xkcd combined

(Julian/Julian's)

User avatar
scarecrovv
It's pronounced 'double u'
Posts: 674
Joined: Wed Jul 30, 2008 4:09 pm UTC
Location: California

Re: Idea for a fun joint effort: xkcd hacking project

Postby scarecrovv » Wed Mar 28, 2012 2:12 pm UTC

Looks good to me. I'll change the headers appropriately. I don't know what zed0's real name is, or even if zed0 wants it attached, so I'll hold off on adding the header to the adventure shell.

edit: My apologies to zed0 (see next post). I wasn't thinking.
Last edited by scarecrovv on Wed Mar 28, 2012 4:08 pm UTC, edited 2 times in total.

User avatar
Jplus
Posts: 1721
Joined: Wed Apr 21, 2010 12:29 pm UTC
Location: Netherlands

Re: Idea for a fun joint effort: xkcd hacking project

Postby Jplus » Wed Mar 28, 2012 3:54 pm UTC

scarecrovv wrote:I don't know what zed0's real name is, or even if he wants it attached, so I'll leave it to him to add the header to the adventure shell.

Not intending to nag or to criticise you, but are you sure that zed0 is a "he"? :)

(I realise the probability of a he is more than 50%, but I was just thinking that if we don't assume by convenience that all participants are hes, then perhaps shes might feel a little bit more welcome in the long run.)
"There are only two hard problems in computer science: cache coherence, naming things, and off-by-one errors." (Phil Karlton and Leon Bambrick)

coding and xkcd combined

(Julian/Julian's)

User avatar
scarecrovv
It's pronounced 'double u'
Posts: 674
Joined: Wed Jul 30, 2008 4:09 pm UTC
Location: California

Re: Idea for a fun joint effort: xkcd hacking project

Postby scarecrovv » Wed Mar 28, 2012 4:18 pm UTC

Thanks Jplus. I wasn't thinking.

Also, xkcd-search allows you to run regular expressions against the xkcd-fetch database. It looks at comic titles, title texts, and transcripts.

Code: Select all

neil@milligan ~/m/t/src> ./rsshell.py
neil@milligan ~/m/t/src> git checkout xkcd-fetch
Switched to branch 'xkcd-fetch'
neil@milligan ~/m/t/src> ./xkcd-search.py 'Stallman'
198
225
256
344
345
521
neil@milligan ~/m/t/src> ./xkcd-search.py 'Stallman' | ./xkcd-fetch.py
198
Perspective
perspective.png
I wonder what I was dreaming to prompt that.  I hope it wasn't the Richard Stallman Cirque de Soleil thing again.
2006-12-18
<transcript>
Narrator: Sometimes, when I first wake up, I am caught in the horrible grip of perspective.
[[A person sitting up in bed]]
Person: It may be a jewel of open source, but Firefox is JUST A BROWSER. It shows WEBPAGES. What the hell is WRONG with us?
Narrator: Fortunately, this subsides quickly.
{{alt text: I wonder what I was dreaming to prompt that. I hope it wasn't the Richard Stallman Cirque de Soleil thing again.}}
</transcript>

225
Open Source
(the rest of the output snipped)

This should facilitate the implementation of key-xkcd (which I think I'll let someone else write).

User avatar
Jplus
Posts: 1721
Joined: Wed Apr 21, 2010 12:29 pm UTC
Location: Netherlands

Re: Idea for a fun joint effort: xkcd hacking project

Postby Jplus » Wed Mar 28, 2012 5:22 pm UTC

scarecrovv wrote:xkcd-search allows you to run regular expressions against the xkcd-fetch database. It looks at comic titles, title texts, and transcripts.

Cool!

By the way, where's that database you've been talking about? I couldn't find it on GitHub, but perhaps I just wasn't searching well enough.

Also: I've changed the topic title, because we now have a proper name.
"There are only two hard problems in computer science: cache coherence, naming things, and off-by-one errors." (Phil Karlton and Leon Bambrick)

coding and xkcd combined

(Julian/Julian's)

User avatar
scarecrovv
It's pronounced 'double u'
Posts: 674
Joined: Wed Jul 30, 2008 4:09 pm UTC
Location: California

Re: Idea for a fun joint effort: xkcd hacking project

Postby scarecrovv » Wed Mar 28, 2012 6:45 pm UTC

Jplus wrote:By the way, where's that database you've been talking about? I couldn't find it on GitHub, but perhaps I just wasn't searching well enough.

The cache is generated on the fly by xkcd-fetch. If it doesn't exist it will be created. If it does exist, but it doesn't contain the comic you ask for, that comic will be downloaded and cached so you've got it the next time you want it. The -a option will cause it to download all the comics.

By the way, comic 826 is weird, and not handled correctly by xkcd-fetch. It was written by Zach Weiner (the smbc guy) during guest week, and is full of javascript. I'm not even sure what correct behavior would look like for that, so I've just let it fail somewhat gracefully by not adding it to the database when the regexes I'm using to parse comic pages fail to match.

It also doesn't fetch large images even if they're available (like comic 657). I might fix that eventually. Maybe.

---editing so as to not double post---

I'm having a minor issue with GitHub, perhaps someone more experienced can help. I just used GitHub's web interface to merge Nautilus's pull request (thanks a bunch!) into the-xkcd-community. It should have been a fast-forward, but that's not the way GitHub chose to do it. It's hard to describe exactly what happened in words, so I'll just link you to the network graph. See where it forked and then imidiately merged again with no intervening commits? That's what I'm talking about. What the hell? How do I make it not do that, aside from doing the merge locally?

User avatar
Jplus
Posts: 1721
Joined: Wed Apr 21, 2010 12:29 pm UTC
Location: Netherlands

Re: Idea for a fun joint effort: red spider project

Postby Jplus » Thu Mar 29, 2012 7:57 am UTC

Thanks for the explanation of the cache of xkcd-fetch. You're doing a great job.

Nautilus: nice subtle update of the Readme.

scarecrovv wrote:---editing so as to not double post---

I'm having a minor issue with GitHub, perhaps someone more experienced can help. I just used GitHub's web interface to merge Nautilus's pull request (thanks a bunch!) into the-xkcd-community. It should have been a fast-forward, but that's not the way GitHub chose to do it. It's hard to describe exactly what happened in words, so I'll just link you to the network graph. See where it forked and then imidiately merged again with no intervening commits? That's what I'm talking about. What the hell? How do I make it not do that, aside from doing the merge locally?

I'm not experienced enough with GitHub to help you, but I do know that other tools enforce true merges too, e.g. git-flow. The rationale seems to be that true merges can be undone more easily and that they provide more detailed information of what happened. I totally agree with you that a fast-forward would have been better in this case and that users should have a choice, though. Hopefully somebody else can tell us how to get a fast-forward next time, if that's possible at all with GitHub.
"There are only two hard problems in computer science: cache coherence, naming things, and off-by-one errors." (Phil Karlton and Leon Bambrick)

coding and xkcd combined

(Julian/Julian's)

User avatar
PM 2Ring
Posts: 3713
Joined: Mon Jan 26, 2009 3:19 pm UTC
Location: Sydney, Australia

Re: Idea for a fun joint effort: xkcd hacking project

Postby PM 2Ring » Thu Mar 29, 2012 8:13 am UTC

scarecrovv wrote:I'm working on an xkcd comic fetcher (see my "xkcd-fetch" branch). It caches comics and all the metadata (number, title, image, title-text, date, and transcript) in the "work/xkcd-fetch" directory. I'm working on commenting it better. It could use improvement.


I just had a very brief look at your comic-fetching code and I see that you're scraping the comic page itself. It'd be more efficient (and friendlier on the server) to use the JSON feed.

Here's an example I did in JavaScript, from the Getting the most recent comic sticky thread. It's quite easy to do in Python if you use the json module. I can post example Python code here, if you like.

To actually display the comic image my Python comic-fetcher uses a module that I wrote using gtk to display an image (or GIF anim) directly from the URL (i.e., it doesn't cache the fetched image as a file). My image displayer isn't very fancy, but it loads images from a URL or a local filesystem; it can read quite a few file formats, thanks to gtk; it uses a scrolled window for display, and allows zooming on still images.

gtk is cross-platform, but I suppose there may be objections to using it for the Red Spider Project, since it's not part of the standard Windows Python package.

User avatar
Jplus
Posts: 1721
Joined: Wed Apr 21, 2010 12:29 pm UTC
Location: Netherlands

Re: Idea for a fun joint effort: xkcd hacking project

Postby Jplus » Thu Mar 29, 2012 12:32 pm UTC

PM 2Ring wrote:To actually display the comic image my Python comic-fetcher uses a module that I wrote using gtk to display an image (or GIF anim) directly from the URL (i.e., it doesn't cache the fetched image as a file). My image displayer isn't very fancy, but it loads images from a URL or a local filesystem; it can read quite a few file formats, thanks to gtk; it uses a scrolled window for display, and allows zooming on still images.

gtk is cross-platform, but I suppose there may be objections to using it for the Red Spider Project, since it's not part of the standard Windows Python package.

I can think of several possible solutions for this issue, and there are probably more of them:
  1. move the displaying of comics to a separate command (e.g. "xkcd-show") which isn't "installed" by default, so gtk will be an optional dependency and users can decide for themselves whether to install it or not;
  2. omit gtk and use the system's default image viewer instead (they can always be called from the command line, you just have to do some OS detection which is quite easy with Python);
  3. simply have the system default browser display the comic (either from disk or from the website -- this also requires some OS detection);
  4. don't display the comic afterall.
"There are only two hard problems in computer science: cache coherence, naming things, and off-by-one errors." (Phil Karlton and Leon Bambrick)

coding and xkcd combined

(Julian/Julian's)

User avatar
scarecrovv
It's pronounced 'double u'
Posts: 674
Joined: Wed Jul 30, 2008 4:09 pm UTC
Location: California

Re: Idea for a fun joint effort: xkcd hacking project

Postby scarecrovv » Thu Mar 29, 2012 2:06 pm UTC

PM 2Ring wrote:I just had a very brief look at your comic-fetching code and I see that you're scraping the comic page itself. It'd be more efficient (and friendlier on the server) to use the JSON feed.

Here's an example I did in JavaScript, from the Getting the most recent comic sticky thread. It's quite easy to do in Python if you use the json module. I can post example Python code here, if you like.

Oh, very nice. If I'd heard of JSON before, I would have used that. I'll work on it. Thanks for the tip!

edit:

It now uses the JSON feed. I also changed the cache and output format to use JSON too. The fields aren't all named the same as the official one, but whatever. Now we can download and search news too! Sadly, this change means you'll have to clear your cache with "rm -rf work/xkcd-fetch". It will be worth it.

I'm still open to improvements in the output format for xkcd-fetch.

edit:

PM 2Ring, how would you like to be credited for the JSON idea? I just said "PM 2Ring suggested JSON", but if you want your real name there (or would like to remain anonymous), feel free to say so, or change it yourself.

User avatar
PM 2Ring
Posts: 3713
Joined: Mon Jan 26, 2009 3:19 pm UTC
Location: Sydney, Australia

Re: Idea for a fun joint effort: xkcd hacking project

Postby PM 2Ring » Fri Mar 30, 2012 7:41 am UTC

scarecrovv wrote:PM 2Ring, how would you like to be credited for the JSON idea? I just said "PM 2Ring suggested JSON", but if you want your real name there (or would like to remain anonymous), feel free to say so, or change it yourself.

Don't thank me, thank davean. :)

There are a few ancient things on the net under my real name, but "PM 2Ring" is my Web identity, and I prefer to keep it that way.

BombSite_A
Posts: 5
Joined: Sun Apr 01, 2012 10:08 pm UTC

Re: Idea for a fun joint effort: red spider project

Postby BombSite_A » Sun Apr 01, 2012 10:10 pm UTC

I love this idea. I'm pretty good at C# and I'd love to help out.

User avatar
Jplus
Posts: 1721
Joined: Wed Apr 21, 2010 12:29 pm UTC
Location: Netherlands

Re: Idea for a fun joint effort: red spider project

Postby Jplus » Mon Apr 02, 2012 10:07 pm UTC

Nice! You can jump right in if you like.

Edit: I learnt about two other project today, which seem to have a similar level of fine-grained modularity to what we're going to build here. Both seem to be Windows-only but it's fun to know about them anyway.
Appifyer
Mango (written in AutoHotkey)
"There are only two hard problems in computer science: cache coherence, naming things, and off-by-one errors." (Phil Karlton and Leon Bambrick)

coding and xkcd combined

(Julian/Julian's)

User avatar
zed0
Posts: 179
Joined: Sun Dec 17, 2006 11:00 pm UTC

Re: Idea for a fun joint effort: red spider project

Postby zed0 » Tue Apr 03, 2012 11:35 am UTC

I started writing an adventure web browser (see advbrowser branch).
At the moment it just gets the html for a page and prints it out.
Currently my idea is that you can move around via links on the page described as doors and collect items (images, flash videos etc) which will have varying value and allow you to do various things.
Where I'm really stuck is how to describe the page you're currently on. I could either try and render an ASCII version of the page or I could describe each div as a different part of the chamber you are currently in and allow you to look at different parts of the page.
Any ideas would be appreciated.

User avatar
Jplus
Posts: 1721
Joined: Wed Apr 21, 2010 12:29 pm UTC
Location: Netherlands

Re: Idea for a fun joint effort: red spider project

Postby Jplus » Tue Apr 03, 2012 5:57 pm UTC

I think the idea of describing divs (or block level elements in general) of the page as parts of the room makes most sense. A rendering of the page will usually not remind the player of a room, which I assume is your intention. Perhaps you could restrict yourself to the divs (block level elements) with an id attribute, in which case you could use the id as the name of the part of the room. Let's say the parts are actually walls, and the hyperlinks are doors in those walls.


On a totally unrelated note: I think we have a problem with our branches. So far, when somebody started a new topic branch they usually merged in some other topic branches as well because those other topic branches contained something the author of the new topic branch wanted to use. For example, both xkcd-fetch and advbrowser incorporate commits (and hence, files) from rscall in addition to those that they inherited from master. This doesn't mix well with our policy that everything merged into master must be stable and portable.

To see why, consider the advbrowser branch (this just happens to be the most dramatic example at the time of writing). Apart from advbrowser.php, it also contains advshell.py, rscall.py, rsshell.py, xkcd-fetch.py and xkcd-search.py. The consequence is that advbrowser cannot be merged back into master until all of these programs are stable and portable (they don't need to be stable and portable at the same time, but each of them must have a stable+portable version and those stable versions must be the ones that are included in the advbrowser branch in order to have it merged into master). This will get worse with every additional branch that is merged into advbrowser. We must stop merging branches into each other like this, otherwise at some point it will be impossible to make any stable branch at all.

To turn the tide, we have to disentangle our current branches a bit and then adopt better habits for the future.

To disentangle our branches, the minimum we should do is to have rscall and rsshell merged into master as soon as we can, from the rsshell branch (it appears that these command are already stable and portable). xkcd-fetch will then be free of cross-dependencies (it can use rscall from master from that point onwards) and advbrowser will have one cross-dependency left instead of two. If we just don't create any more cross-dependencies after that, I think we should be OK in the long run.

Both the xkcd-fetch branch and the advbrowser branch then still have a double purpose, though. xkcd-fetch concerns both the xkcd-fetch command and the xkcd-search command while advbrowser concerns both the advbrowser command and the advshell command. I think the most pragmatic thing we can do, is to accept that at some point we'll introduce some unstable code into the master branch. We don't have a build system yet nor do we install anything into bin/ or lib/, and even if we did we could just exclude the unstable parts of the merged code from installation. So for the time being we can make an exception to our rule that the master branch shouldn't contain unstable code.

In conclusion, this is what I'm proposing to do in addition to the minimum I mentioned above. I take the xkcd-fetch branch as an example but you can apply analogous steps to advbrowser.
  1. Make either xkcd-search or xkcd-fetch stable and portable. It doesn't need to do anything meaningful, the program should just run on the major platforms without crashing (and the user must be able to quit the program normally).
  2. Merge the branch into master, then fork two new, separate branches from master: one for xkcd-fetch and one for xkcd-search.
  3. Don't merge those two branches into each other anymore. When one of them reaches a new stable state, merge it into master first and then merge master into the other. If necessary.

As better habits for the future, I propose the following. Please discuss.
  • As a rule of thumb, we only merge branches if either of them was forked directly from the other.
  • If you want to use code from another branch that you can't merge into your own because of the previous rule, do the minimum required so you can cascade that code through all intermediate branches. For example, if master is the parent of both branches, get the code that you want to import to run crash-free on the major platforms and then transfer it through master.
  • Branches other than master should only work on a single subproject. They can use and even edit code from other subprojects, but they should only do so if the subproject under focus depends on those subprojects.
  • These rules only apply to work that you publish. Locally on your computer you can do anything you want.
"There are only two hard problems in computer science: cache coherence, naming things, and off-by-one errors." (Phil Karlton and Leon Bambrick)

coding and xkcd combined

(Julian/Julian's)

User avatar
scarecrovv
It's pronounced 'double u'
Posts: 674
Joined: Wed Jul 30, 2008 4:09 pm UTC
Location: California

Re: Idea for a fun joint effort: red spider project

Postby scarecrovv » Tue Apr 03, 2012 6:48 pm UTC

I agree wholeheartedly with your sentiments about merging everything into everything being a bad plan, and the need to clean things up. I just wasn't sure what the right way to go about it would be.

About rscall and rsshell: they're pretty simple, and I think that since those branches have been merged into master, the branches can be discontinued. If somebody wants to develop them further they should make a new branch from master, and keep the new branch clean. One thing I'm unsure about is whether rsshell will work on windows, since it depends on the SHELL environment variable pointing to your preferred shell (maybe windows does this, maybe not, I don't know).

About xkcd-fetch: I think the xkcd-fetch branch is pretty stable. I don't have any further plans for it at the moment, though if somebody else has an idea, they should feel free to code it up. Pending some testing on non-unixy platforms, I think it should be merged into master and discontinued. Further development should happen on a new branch.

User avatar
Jplus
Posts: 1721
Joined: Wed Apr 21, 2010 12:29 pm UTC
Location: Netherlands

Re: Idea for a fun joint effort: red spider project

Postby Jplus » Tue Apr 03, 2012 7:50 pm UTC

Ah right, the SHELL variable. Totally forgot about that one.

Seems like we have quite a few infrastructure jobs that need to be done. I might work on a few of these tomorrow. I'm also planning to write a new Readme.
  • Make sure that rsshell is Windows-friendly
  • Establish our supported platforms
  • Add an issue on github to request testing of xkcd-fetch on Windows
  • Start a build system (this will be hard)

By the way: thanks.
"There are only two hard problems in computer science: cache coherence, naming things, and off-by-one errors." (Phil Karlton and Leon Bambrick)

coding and xkcd combined

(Julian/Julian's)

BombSite_A
Posts: 5
Joined: Sun Apr 01, 2012 10:08 pm UTC

Re: Idea for a fun joint effort: red spider project

Postby BombSite_A » Wed Apr 04, 2012 3:43 am UTC

Is .net alright? I'm worried that I might force users to use windows...

User avatar
Jplus
Posts: 1721
Joined: Wed Apr 21, 2010 12:29 pm UTC
Location: Netherlands

Re: Idea for a fun joint effort: red spider project

Postby Jplus » Wed Apr 04, 2012 10:07 am UTC

If it works well with Mono, it's alright. So C# should be OK, while PowerShell seems to be a no-go. There is a list of languages that you may expect to work fairly well.
"There are only two hard problems in computer science: cache coherence, naming things, and off-by-one errors." (Phil Karlton and Leon Bambrick)

coding and xkcd combined

(Julian/Julian's)

tidbit
Posts: 3
Joined: Thu Apr 05, 2012 2:42 am UTC

Re: Idea for a fun joint effort: red spider project

Postby tidbit » Thu Apr 05, 2012 2:57 am UTC

Hello Jplus and everyone!

I'm the author of Mango, as posted above. I have also made a vector logo:
(this is a temporary host. it can be re-hosted on someones GIT or whatever, if you decide to use it.)
http://imagebin.org/206572 (.png preview)

User avatar
scarecrovv
It's pronounced 'double u'
Posts: 674
Joined: Wed Jul 30, 2008 4:09 pm UTC
Location: California

Re: Idea for a fun joint effort: red spider project

Postby scarecrovv » Thu Apr 05, 2012 2:17 pm UTC


User avatar
Jplus
Posts: 1721
Joined: Wed Apr 21, 2010 12:29 pm UTC
Location: Netherlands

Re: Idea for a fun joint effort: red spider project

Postby Jplus » Thu Apr 05, 2012 2:38 pm UTC

For the record, I invited tidbit to show his drawing here so everyone could express their opinions. Welcome tidbit! And thanks for taking the effort. Personally, I like the drawing.


Unrelated: I've written a Platforms.md file and submitted a testing request for xkcd-fetch. Do we have any Windows testers on board, by the way?
"There are only two hard problems in computer science: cache coherence, naming things, and off-by-one errors." (Phil Karlton and Leon Bambrick)

coding and xkcd combined

(Julian/Julian's)

User avatar
scarecrovv
It's pronounced 'double u'
Posts: 674
Joined: Wed Jul 30, 2008 4:09 pm UTC
Location: California

Re: Idea for a fun joint effort: red spider project

Postby scarecrovv » Thu Apr 05, 2012 5:03 pm UTC

I like the drawing too. If it were reworked to look a bit more like the red spiders (6 long black legs that bend flexibly), I would support it's use as a logo.

tidbit
Posts: 3
Joined: Thu Apr 05, 2012 2:42 am UTC

Re: Idea for a fun joint effort: red spider project

Postby tidbit » Fri Apr 06, 2012 11:58 pm UTC

noodle-ified the legs and burnt them to a crispy black texture, added another eye (as shown in one of the images).
http://imagebin.org/207051


Return to “Coding”

Who is online

Users browsing this forum: No registered users and 13 guests