Welcome to Gaia! :: downloading millions of files.

You would need the permission of the authors to do this

please explain and expound on this.

here are a couple points that i see.

A.)just by viewing the stories i am downloading them to my computer.

B.)programs already exist that will download stories. http://www.deffnetizer.com/ is one that i would use except that it has no support for batch downloads. why are they not getting hounded for not asking permission to permanently archive the stories for when no internet connection is available.

if nothing else ill see if i can write something up in VB 08. but i was hopping to be able to at least partially automate the process by using a ready made downloader.

EDIT:the only problem i could see is being seen as a DDoS attack but i was told by a couple others in here that it should not be a problem as this is only one computer.

That's a really specialized need. I don't think there are many (if any) downloaders that are designed for something quite like that. The few I know of can copy files from a given directory, but not multiple directories. That might be why no one has responded though - no one has anything to offer.

thats entirely too understandable. the main reason for my my rant in the bump was because most of my threads around here go unanswered for a couple pages of bumps because if it was a simple fix i would have never made a thread.

and im thinking about trying this when i get a chance.
http://www.sharewareconnection.com/file-downloader.htm

i have no clue if it supports telling it what directory to put the file in though. if it does i can set VB to auto generate the batch file that would start the downloads.

i cant try it atm because im running off of a livecd due to my desktop not having an OS and my laptop's power cord frying in half. and while UBCD4WIN is great it has slightly limited support for installing stuff.

http://www.httrack.com/

I have actually seen at least two that are designed for (more or less) that specific purpose.

Brute-force downloading of web-pages is useful for a certain type of security audit... you spend enough time in the CompSec field and you'll see just about anything and everything.

Unfortunately, I don't remember either of the two downloaders and even if I did they'd be complicated and difficult to use. You'll better off with a bash script and wget forming a list of URLs and downloading them.