Lurking tool: ConceptArt.org image scrapper! (Edit: Version 2.2!)

[fil_razorback]

Hi all,
I'm a bad artist but a long time CA.org lurker. I've just spent a few hours making a tool that will make my lurking easier. I find the message board a bit cluttered and hard to navigate on a small screen, so this page is my solution.
It displays the most recent images posted on the forum and loads more as you scroll down. I know the tool is a bit slow, it's ok for me but if there's any interest I'll see what I can do to optimize it.

http://www.antoinegersant.com/misc/lurk/


EDIT: Version 2.1 is up and running!

-- Changelog Version 2.2 --
· Added tooltips with thread names

-- Changelog Version 2.1 --
· Filtering can now be turned ON or OFF (default: OFF) and its defaults setting filters everything out

-- Changelog Version 2.0 --
· Much faster
· Filter pictures by forum
· Keeps one month worth of pictures
· Updated every hour

[Wingal]

This is so neat man! Thanks for sharing and please if you can work on it more!

[ceddo]

This is so awesome. Thanks Fil I've already found a few really interesting threads I probably wouldn't have found otherwise.

[Januz]

Nice! That's a pretty good tool for discovering new threads. I think it'd be much be better if it was part of ca.org, or at least hosted in these servers. That way ca doesn't have to spend bandwidth on it.

Btw, you could try caching the first set of images every hour so it loads faster initially.
Great idea razorback

[carlosranna]

Very cool!

[Shorinji_Knight]

That's great! Yeah I would like to see what you could do to optimize it. Sometimes its better just to look at the pictures and not be distracted by all of the text.

[Bowlin]

I love it. I wonder if it would be possible to make it just for "Finally Finished" or "Sketchbook" sections?

[fil_razorback]

Thanks for your feedback! I'm working on an updated version =)

[Syle]

DUDE! Awesome!

I love it, and will be checking back to see what you can do to optimize it. Good luck, and thank you!

[Kuroyue]

This looks like something that could be a major addition to CA. If this was a part of the site (with some more improvements like faster loading time, etc.) I could possibly be more active here. Hmm.

[Jason Manley]

Hey Razor...Thanks for making that. Plz check your PM's when you get a chance. Awesome.


JM

[fil_razorback]

@Jason Manley > Replied
@Others > Thanks!

Version 2 is up and running!
Changelog:
- Much faster
- Filter pictures by forum
- Keeps one month worth of pictures
- Updated every hour

I hope you like it, I certainly had fun making it. Here it is.

[ConnyNordlund]

Very awesome!
I can also sense some work coming your way, huh ? That's great, man!

I'm looking forward to seeing updates with this program/site. Very useful!

[-Tragedy-]

Brilliant work!
2.0 is definitely faster than the previous version.

The filtering probably needs a bit of work IMO.

Example:
When page is opened, all forums are selected as one checkbox.
Along with all the other forums/sub-forums available deselected.
Enabling people to select what they want to see specifically without having to uncheck all the checkboxes.

Then, for example if this does get included into the site, have it so that upon login it can remember what forums you visit so once the page is opened it goes straight to your "favorites" instead of all selected.

Just my two-cents.

Again, great work!

[-Tragedy-]

Brilliant work!
2.0 is definitely faster than the previous version.

The filtering probably needs a bit of work IMO.

Example:
When page is opened, all forums are selected as one checkbox.
Along with all the other forums/sub-forums available deselected.
Enabling people to select what they want to see specifically without having to uncheck all the checkboxes.

Then, for example if this does get included into the site, have it so that upon login it can remember what forums you visit so once the page is opened it goes straight to your "favorites" instead of all selected.

Just my two-cents.

Again, great work!

[fil_razorback]

Thanks for the suggestion, I think it's a lot more convenient now.
Version 2.1

[Bowlin]

I just noticed the "Filter" option. That's amazing! Thanks for this.

[BADOMENS]

Wow, awesome. This is really handy since you can pick and choose quickly. It would be great if somehow you could see the thread name by hovering over the picture, which could quickly put the picture into context.

[Januz]

Great update. Feels snappier now

[fil_razorback]

Thanks all! Having actual users make me really happy =)
@Shirobon > Done! It's not *exactly* the thread names (because I haven't been crawling them) but a readable version of their URL. The difference is subtle so I think it will do.

[Syle]

I'm speechless! This is truly badass.

I like the "hover" title aspect. I don't want to see a bunch of title text over all of these images as I am scrolling through; only the ones I hover over. The whole appeal of this site is that it cuts out all the clutter of text and brings you just the images.

Again, truly awesome.

[Syle]

I'm speechless! This is truly badass.

I like the "hover" title aspect. I don't want to see a bunch of title text over all of these images as I am scrolling through; only the ones I hover over. The whole appeal of this site is that it cuts out all the clutter of text and brings you just the images.

Again, truly awesome.

[Syle]

PS: if there is a way to extend the backlog of images to more than a month, that would be great. I was just lurking the Tutorial section, and it looks pretty skimpy because not a lot has been uploaded in the last month.

Maybe you can keep the bigger subforums (Sketchbook, IFF, WIP, etc) at one month, but increase the smaller/less visited forums?

[Syle]

PS: if there is a way to extend the backlog of images to more than a month, that would be great. I was just lurking the Tutorial section, and it looks pretty skimpy because not a lot has been uploaded in the last month.

Maybe you can keep the bigger subforums (Sketchbook, IFF, WIP, etc) at one month, but increase the smaller/less visited forums?

[Zenith_]

Wow! This is awesome!
The only recommendation I have is maybe you could have a "search-by-keyword" function as well as the filters.

I know that CA has an image search function but I prefer the nice, clean interface you've created to browse pictures
Can't wait to see how this progresses!

[fil_razorback]

@Syle > I've adjusted the settings, the tool will now keep pictures for 6 months (that will occupy about 1GB on my server, just with thumbnails). However, I cannot crawl data older than a month (because the activity stream doesn't go this far) so it will fill gradually from now on. While I was at it I also started crawling actual thread names: pictures that are already indexed now display their slug (URL friendly name) on rollover but future ones will have their name exactly as it appears on the forum.
@The-Idle-Elk > Thanks. I'm not too sure about a search function…It is doable but it feels a bit out of scope and I'm not sure I could improve a lot on the experience you get from the original CA.org there.

[Shorinji_Knight]

Its working great. You've got the speed issue pretty well fixed. Now all I have to do is scan through the thumbnails for something interesting. Before you had to dig through all of the threads and you never knew if you were missing out on something cool. Thanks!

[Veneq]

Yeah I'm not surprised Jason hopped on this... this would be really cool to see integrated into the site somehow. There are portions of this forum that I am hardly aware of and I rarely click through more than a few sketchbooks. Very cool!

[snatti]

awesomeness, thanks alot!

[jpacer]

This is fantastic!!