FTP Server?

"Please leave a message at the beep, we will get back to you when your support contract expires."

Moderators: phlip, Moderators General, Prelates

Korrente
Posts: 164
Joined: Thu Dec 17, 2009 5:44 am UTC

FTP Server?

Postby Korrente » Thu Dec 16, 2010 12:18 am UTC

I'm trying to download an obscene amount of RADAR data from the NCDC. They make you select the files you want from Day X to Day Y and enter your email address. They then 'unarchive' the files you selected and email you an FTP link when they're ready.
What is the reasoning behind this? I understand that my 15 days at roughly 6 images an hour amounts to about 2500 images (in 350 files), but what are their servers doing that would take hours to 'unarchive' these? They deliver as .tar's I believe...is all that time spent de- and recompressing? Though it is a vital government service, their servers are probably just crap.

archeleus
Posts: 240
Joined: Wed Sep 29, 2010 1:49 pm UTC
Location: Valenvaryon
Contact:

Re: FTP Server?

Postby archeleus » Thu Dec 16, 2010 2:15 am UTC

Or maybe the files are stored elsewhere and then just moved to the public FTP server when requested?
I write a blog rant here.

Korrente
Posts: 164
Joined: Thu Dec 17, 2009 5:44 am UTC

Re: FTP Server?

Postby Korrente » Thu Dec 16, 2010 3:19 am UTC

Seems to be. I thought they were saying I could download them from their server or FTP them. But it's either log in via an FTP client and get everything or use your browser and get one at a time. Also I take back what I said about their server, the download is extremely fast.

keeperofdakeys
Posts: 658
Joined: Wed Oct 01, 2008 6:04 am UTC

Re: FTP Server?

Postby keeperofdakeys » Thu Dec 16, 2010 9:02 am UTC

I would put my money on decompressing and de-tarring, then tarring them up specially for you. Are they .tar.something, or just .tar? (tar puts all the files in one file, then compression is applied on that file, so you get a .tar.something).

User avatar
phillipsjk
Posts: 1213
Joined: Wed Nov 05, 2008 4:09 pm UTC
Location: Edmonton AB Canada
Contact:

Re: FTP Server?

Postby phillipsjk » Thu Dec 16, 2010 1:15 pm UTC

Maybe you are requesting the data from a tape robot. The access times for those are not great.
Did you get the number on that truck?

Korrente
Posts: 164
Joined: Thu Dec 17, 2009 5:44 am UTC

Re: FTP Server?

Postby Korrente » Thu Dec 16, 2010 5:44 pm UTC

It did mention something about it being robotic, but just in passing. I couldn't find any other good information on their equipment though.

Yes they are .tar.gz, or several .gz's in a .tar, if that's not the same thing.

Carnildo
Posts: 2023
Joined: Fri Jul 18, 2008 8:43 am UTC

Re: FTP Server?

Postby Carnildo » Sat Dec 18, 2010 9:07 am UTC

"Robotic" probably means the data is stored on a tape silo, and in order for you to download it, they need to pull it off the tape and package it up for you to download.


Return to “The Help Desk”

Who is online

Users browsing this forum: No registered users and 22 guests