1

TUTORIAL:BatchDownloadingofLiDARDEMDataviatheOpenTopographyPortal

ChristopherCrosby(ccrosby@sdsc.edu)

Lastupdated:July16,2009

INTRODUCTION:ThistutorialisdesignedtoaidOpenTopographyPortaluserswhowishtomakebulkdownloadsofLiDARDEMtilesviatheinterfacelocatedat:http://opentopo.sdsc.edu/gridsphere/gridsphere?cid=bulkdlThebatchdownloadcapabilityoftheOpenTopographyPortalisdesignedforadvancedusersseekingtodownloadlargeamountsofdataquicklyandwhohavethebandwidth,expertise,andsoftwarenecessarytomanagethegigabytesofdatathismethoddelivers.FormostuserswerecommendtheGoogleMap‐basedtiledownloadinterfacefoundinthePortalat:http://opentopography.org/demsMETHOD:TheOpenTopographyPortalprovidesbatchdownloadcapabilityfordatasetswherestandardDigitalElevationModelsareavailable.Undereachdatasetisalistofspecific"targets"oracquisitionareas.ClickingonthetargetlinkloadsapagethatlistsreferringURLsforallavailable1km2DEMtilesforadatasetonasinglepagetofacilitateeasybatchdownloading.Itisuptouserstochooseamethodtoretrievethesedatainbulk.Thistutorialprovidesabasicintroductiontofreesoftwareoptionstoretrievethesedata.AGooglesearchfor“downloadmanager”willquicklyrevealalternativeoptionsiftheonesbelowdonotworkforyou.Werecommendandwillhighlightthefollowingdownloadmanagersinthistutorial:

I. DownThemAll!extensionfortheFirefoxbrowser(http://www.downthemall.net/)

II. FreeDownloadManager(http://www.freedownloadmanager.org/)Foruserswhoarecomfortablewithcommandlineutilities,batchdownloadingofthetileDEMscanalsobeeasilyaccomplishedviathefreeGNUWgetprogram(http://www.gnu.org/software/wget/).PleaseaddressquestionsorcommentsontheOpenTopographyPortalbatchdownloadcapabilitytoinfo@opentopography.org

2

I.DownThemAll!

DownThemAll!(http://www.downthemall.net/)isapopularandfreedownloadmanagerfortheFirefoxwebbrowser.Onceyou’veinstalledDownThemAll!(dTa),usethefollowingstepstodownloaddatafromOpenTopography:

Fromthebatchdownloadspage(http://opentopo.sdsc.edu/gridsphere/gridsphere?cid=bulkdl)

Clickinga“target”(i.e.adatasetsubset)linksuchasthe“OldFaithful”portionoftheIntermountainSeismicBeltGeoEarthScopedatasetloadsapagethatlistsreferringURLsforallavailable1km2DigitalElevationModel(DEM)tilesforadatasetonasinglepagetofacilitateeasybatchdownloading.

3

EachURLonthepagebelowreturnsa.zipfilethatcontainsbareearth(ground)andunfiltered(allreturns)DEMs.Youcoulddownloadthetilesone‐by‐one,butadownloadmanagerprogramcanpullthemallwithminimalworkonyourpart.Notethateachpageliststhe#offilesandthetotaldatavolumeyouaregoingtogetback.

Aboveisthepageyou’llwanttorunthedownloadmanagertosequentiallydownloadalloftheURLslistedonthepage.

4

IMPORTANT:BeforeyourundTatodownloadthefiles,werecommendyoumakeachangetothedefaultsettingsoftheprogram.Navigateto“Tools”menuinFirefox>“DownThemAll!Tools”>“Preferences”andselectthe“Advanced”tab.Onthe“Advanced”tabyou’llwanttodisable“MultipartDownload”bymovingthesliderbarallthewaytotheleftuntilthedialogreads“1/Disabled”.Thischangeseemstoleadtofewerfailuresduringdownload.

5

Onceyou’vemodifiedthepreferencesofdTaasnotedabove,navigatebackto“Tools”>“DownThemAll!Tools”>butthistimeselect“DownThemAll!...”DoingsoloadsathedTadialogthatlookslikethis:

dTawillrecognizeeverylinkonthepagebutsinceallwewantareURLsthatpointto“.zip”files,weselectonlythe“Archives(zip,rar,…)”filteroptionandleaveallotherboxesunchecked.Inthe“Savefilesin:”box,definewhereyouwanttoputthedownloads.Onceyou’vegoteverythingset,click“Start!”

6

OncedTastartsrunningyougetadialogboxthatshowsthestatusofthedownloads.Dependinguponthenumberoffiles,serverload,andyourbandwidth,itmaytakeawhileforthedownloadstocomplete.

Insomecases,adownloadwillhangup‐forexampletheyellowbarintheexampleabovelistedasa“Servererror”.Inthiscaseyoucanselectthatfileandclickonthe“resume”buttonatthetoptorestartthedownload.Ifyouhaveafewfaileddownloads,youcanrightclickinthedTaboxandselect“Removecompletedtasks”toremovethefilesthatsucceeded.Then,selectalloftheremainingfilesandrestartthem.

7

II. FreeDownloadManagerTheFreeDownloadManager(http://www.freedownloadmanager.org/)isafree‐standingapplicationthatisagoodoptionforuserswhodonotusetheFirefoxbrowserandthereforecan’tuseDownThemAll!.FDMisawindowsonlyapplication.Onceyou’vedownloadedandinstalledFDM,followthefirsttwostepsoftheDownThemAll!tutorialabovetogettothewebpagethatliststhereferringURLsfortheDEMtilesforagivendataset:

8

TheFreeDownloadManagerinterfacelookssimilartothedTainterfacediscussedabovebutfunctionsslightlydifferentlysinceitisnotembeddedinthebrowser.

FDMhaslotsofoptions,butforthesakeofthissimpletutorial,wewillstickwiththebasics.First,itwillplacedownloadedfilesinC:\Downloads\bydefault‐tochangethedefaultdownloaddirectoryclickon“Options”>“Settings”>thenselect“NewDownload”fromthe“Downloads”menuandclickthe“Setdefaultdownloadfolder”.

9

SinceFDMisnotpartofthebrowser,itdoesn’tdiscovertheURLsonthepagethesamewaythatdTadoes.ThesimplestwaytoprovideFDMalistofURLstodownloadistoclickanddragtoselectalloftheURLsonthedownloadpagelikethis:

OncetheURLshavebeenselected,copythem(control+C)thengototheFDMinterfaceandselect“File”>“Import”>“ImportlistofURLsfromclipboard”.Youwillbepromptedto“SelectaGroup”,leavethedefaultandclickOK.

10

LikedTa,FDMwilldisplaystatusofeachdownloadasitruns:

Ifdownloadsgetstuck,theycanbestoppedandrestartedinasimilarmannertodTaasdiscussedabove.