Does anyone 'crawl' the net?

I'm kind of interested in internet 'crawling' - ie instead of using something like Google, you do it yourself.

Anyone use any actual 'Crawler' software?

I know, for example, you can get software that searches for pictures of your choice.

I just thought to myself, it would be nice to have a piece of software that would go happily on its way obtaining as much media that it can get it's hands on.

Out there in the internet are 3d models that are hidden away on some obscure site. For example, lots of people post downloads on this site - those links go to archived files that contain media. There are lots of programming sites on the internet with tons and tons of media posted on them.

Googling for 3d models is a slow process - searching sites for links that people have put in for their game demo is an impossible process.

I'd like to set a 'crawler' on the net to do it all for me - I could just leave it running all day grabbing everything in its path - ie, set it to search for 'medieval inn/tavern .3ds, .x,' etc. Off it goes searching the web for every model it can get it's paws on of that type.


Anyone know of such software?

I'm not after free stuff.

I want a crawler that is going grab what it can find - hunting through directories and archived folders/files of anything connected to the net.

Basically, I want to steal the stuff.

oh you mean you want to go to prision. fancy yourself a man slag huh? Dirty man.

He he.
Me too, when everybody's asleep - strip their harddisks.

Can the net fit on 20gig hardisk?

Well, you can buy crawlers that search for images. I just want one to search for 3D media.

Puki, you are aware that just because something is uploaded on the web it doesn't mean it's free to use.

Also if you do sucessfully find a crawler it won't be downloading all the image files for textures etc. I would suspect most people would zip up 3D models to keep the textures etc together. So really you're not looking for 3D media you're looking for Zip and Rar files which there will be at least 7 or 8 on the internet.

>So really you're not looking for 3D media you're looking
>for Zip and Rar files which there will be at least 7 or 8
>on the internet.

Give or take!

Andy

Puki: Why dont you just buy a content pack.
So you dont get any trouble when you release a comercial product.

puki: why don't you just admit to yourself that you're never going to finish your RPG, and stop wasting money?

considering that the Internet Archive [ http://www.archive.org ] hooked up a full petabyte of storage capacity a year ago, I'd say that's a resounding "No".

Press release: http://www.capricorn-tech.com/press_rel/pr050622.html

(A petabyte is a thousand terabyte, a million gigabyte, or a billion megabyte)

Actually, let me rephrase that: "$@#^* no!"

Puki's making a RPG? Is it the Quest for the legendary Bacon?