Welcome to Gaia! :: downloading millions of files.

Log In

Forgot your login?

Sign Up

Register
 
GST
 Post Reply
Gaia Forums » Entertainment » Computers & Technology
downloading millions of files.
Tags: downloading  millions  files 
Export this topic to other sites using WildFire!
forum:43, topic:40251259
1 2 >
Post: 40251259_1 created on Sat May 10, 2008 4:56 pmPosted: Sat May 10, 2008 4:56 pm
ok i am downloading a ton of stories off of fanfiction.net.

i need some type of download manager that can take a list of urls and download them.

however they can not all go to the same spot.

the urls are in the form of http://www.fanfiction.net/s/NumericStoryID/chapternumber/

what i need is to download each story to its own folder. i can generate the urls however they are needed. right now all i have are the story id's and number of chapters for each story pending a download manager that can do what i need.
 
     

when life gives you lemons your are supposed to make lemonade. well what do you do when theres no damn sugar.
Report this post
Post: 40251259_2 created on Sat May 10, 2008 5:07 pmPosted: Sat May 10, 2008 5:07 pm
 
You would need the permission of the authors to do this
     
TheNinja-RPG.com
El33t
Get Your License
Report this post
Post: 40251259_3 created on Sat May 10, 2008 8:07 pmPosted: Sat May 10, 2008 8:07 pm
If legality is not an issue i should be able to write a program that does this.
 
     
Report this post
Post: 40251259_4 created on Mon May 12, 2008 12:48 amPosted: Mon May 12, 2008 12:48 am
 
"El33t"
You would need the permission of the authors to do this
please explain and expound on this.

here are a couple points that i see.

A.)just by viewing the stories i am downloading them to my computer.

B.)programs already exist that will download stories. http://www.deffnetizer.com/ is one that i would use except that it has no support for batch downloads. why are they not getting hounded for not asking permission to permanently archive the stories for when no internet connection is available.

if nothing else ill see if i can write something up in VB 08. but i was hopping to be able to at least partially automate the process by using a ready made downloader.

EDIT:the only problem i could see is being seen as a DDoS attack but i was told by a couple others in here that it should not be a problem as this is only one computer.
     

when life gives you lemons your are supposed to make lemonade. well what do you do when theres no damn sugar.
Report this post
Post: 40251259_5 created on Mon May 12, 2008 5:28 amPosted: Mon May 12, 2008 5:28 am
birdman*** takes a pitchfork and waves it threateningly at the thread.


*mumbles*Why am i always forced to bump threads around here? it is a pain dealing with the idiots that say i can't even though they never look at the time stamps.
 
     

when life gives you lemons your are supposed to make lemonade. well what do you do when theres no damn sugar.
Report this post
Post: 40251259_6 created on Mon May 12, 2008 6:22 amPosted: Mon May 12, 2008 6:22 am
 
That's a really specialized need. I don't think there are many (if any) downloaders that are designed for something quite like that. The few I know of can copy files from a given directory, but not multiple directories. That might be why no one has responded though - no one has anything to offer.
     
if you can resize your active drive while using it
I'll boil your hard drive and have it with a side of fries
- EDs Ryuichi Sakuma
Report this post
Post: 40251259_7 created on Mon May 12, 2008 6:35 amPosted: Mon May 12, 2008 6:35 am
try IDM
 
     
s**t on you Dazaii

Etc123
Get Your License
Report this post
Post: 40251259_8 created on Mon May 12, 2008 6:38 amPosted: Mon May 12, 2008 6:38 am
 
"Minion4Hire"
That's a really specialized need. I don't think there are many (if any) downloaders that are designed for something quite like that. The few I know of can copy files from a given directory, but not multiple directories. That might be why no one has responded though - no one has anything to offer.
thats entirely too understandable. the main reason for my my rant in the bump was because most of my threads around here go unanswered for a couple pages of bumps because if it was a simple fix i would have never made a thread.

and im thinking about trying this when i get a chance.
http://www.sharewareconnection.com/file-downloader.htm

i have no clue if it supports telling it what directory to put the file in though. if it does i can set VB to auto generate the batch file that would start the downloads.

i cant try it atm because im running off of a livecd due to my desktop not having an OS and my laptop's power cord frying in half. and while UBCD4WIN is great it has slightly limited support for installing stuff.
     

when life gives you lemons your are supposed to make lemonade. well what do you do when theres no damn sugar.
Report this post
Post: 40251259_9 created on Mon May 12, 2008 11:27 amPosted: Mon May 12, 2008 11:27 am
My offer still stands, but i'm not writing this if you're not interested and you didn't seem to be so far.

Though it would likely take more time that scripting for what you found earlyer so it's probably obsolete now anyways...
 
     
Report this post
Post: 40251259_10 created on Mon May 12, 2008 12:24 pmPosted: Mon May 12, 2008 12:24 pm
 
     
 
 
 
 
Help me by donating
IDM

Any donation will be aprreciated
 
 
 
 
     
 
My Pc Spec's
Antec 900 Case
Asus Motherboard P5N32-E SLI
4Gb Of Ram
Nvidia 8800GT, 512Mb Video Card
750W Power Supply
Dual Gigabit Ethernet
Windows Vista Ultimate x64
Jonwi5
Get Your License
Report this post
Post: 40251259_11 created on Mon May 12, 2008 12:32 pmPosted: Mon May 12, 2008 12:32 pm
FDM works well
 
     
Report this post
Post: 40251259_12 created on Mon May 12, 2008 1:17 pmPosted: Mon May 12, 2008 1:17 pm
 
http://www.httrack.com/
     


O' Hermes Trismegistos, Orpheus, Zarathustra, Pythagoras and Plato
Mediate the wisdom!
Report this post
Post: 40251259_13 created on Mon May 12, 2008 2:13 pmPosted: Mon May 12, 2008 2:13 pm
wget + regex
 
     
Report this post
Post: 40251259_14 created on Mon May 12, 2008 2:19 pmPosted: Mon May 12, 2008 2:19 pm
 
"Minion4Hire"
That's a really specialized need. I don't think there are many (if any) downloaders that are designed for something quite like that. The few I know of can copy files from a given directory, but not multiple directories. That might be why no one has responded though - no one has anything to offer.
I have actually seen at least two that are designed for (more or less) that specific purpose.

Brute-force downloading of web-pages is useful for a certain type of security audit... you spend enough time in the CompSec field and you'll see just about anything and everything.

Unfortunately, I don't remember either of the two downloaders and even if I did they'd be complicated and difficult to use. You'll better off with a bash script and wget forming a list of URLs and downloading them.
     
Report this post
Post: 40251259_15 created on Mon May 12, 2008 2:46 pmPosted: Mon May 12, 2008 2:46 pm
Try using free download manager, i think the webspider and/or HTML crawler it has should be able to do what you're seeking
 
     
1 2 >
 Post Reply
Gaia Forums » Entertainment » Computers & Technology
 Subscribe to Topic  Report this Topic