ArcGIS Limitation >> Re: [OKGIS] 10 Meter DEM's are available at USGS for most of the state.


Chronological Thread 
  • From: "David Wheelock" <david@oda.state.ok.us>
  • To: "Oklahoma GIS Community" <okgis@gis.gis.ou.edu>
  • Subject: ArcGIS Limitation >> Re: [OKGIS] 10 Meter DEM's are available at USGS for most of the state.
  • Date: Wed, 20 Oct 2004 08:54:59 -0500
  • List-archive: <http://gis.ou.edu/pipermail/okgis>
  • List-id: Oklahoma GIS Community <okgis.gis.ou.edu>

I have also now discovered a limitation in the data size that can be handled by ArcGIS.  I downloaded and mosaiced probably about 1/8 of the state.  When I ran the Fill, Flow Direction and Flow Accumulation functions on the data in preparation for delineating watersheds I got some strange results which I reported to ESRI tech support.  They analyzed the problem and determined that it's a bug related to the large size of the dataset.  ArcGIS just can't handle that much data.
 
Currently, the only work-around is to split the data into smaller areas.
 
See the attachment for more details.

David Wheelock     Application Specialist     david@oda.state.ok.us
 
Oklahoma Department of Agriculture, Food and Forestry
2800 North Lincoln Boulevard
Oklahoma City, OK  73105-4298
Phone:  (405) 522-5477     FAX:  (405) 522-3590
http://www.oda.state.ok.us/
 
----- Original Message -----
Sent: Friday, September 24, 2004 3:36 PM
Subject: [OKGIS] 10 Meter DEM's are available at USGS for most of the state.

I have discovered that the 10 meter DEM's are available for most of the state of Oklahoma.  It looks like about 70-80% of the state.  The people I've told about this have been surprised and didn't know, so I thought I spread the word through the list.
 
Here's where you can go to get it:
  1. Click here: http://seamless.usgs.gov
  2. Then click on the link that says:  View and Order Data Sets - United States Viewer 
  3. Then you'll see a map of the United states.
  4. In the column to the right of the map, select the check box for 1/3" NED.  That's the 10 meter DEM.  Deselect the stuff that's auto-selected, if you don't want it, anything else that you do want.
  5. Zoom in on Oklahoma where you want to get the data.
  6. In the column to the left of the map, under Download select the tool on the left,  Select by Rectangle.
  7. Select an area that you want and you'll either get a download form or a message that you'll have to order it on CD's.  For one county you will have to download 4 chunks or so.  And if you're area is too large you can always break it up into separate data requests.
Regards,

David Wheelock     Application Specialist     david@oda.state.ok.us
 
Oklahoma Department of Agriculture, Food and Forestry
2800 North Lincoln Boulevard
Oklahoma City, OK  73105-4298
Phone:  (405) 522-5477     FAX:  (405) 522-3590
http://www.oda.state.ok.us/



*****
NOTE: An attachment named David Wheelock.vcf was removed from this message and deleted because it contained a windows executable or other potentially dangerous file type. Contact the system administrator if you need more information.------=_NextPart_000_00A2_01C4A24C.521A8360--
--- Begin Message ---
  • From: "ESRI Support" <nimbus@esri.com>
  • To: "David Wheelock" <david@oda.state.ok.us>
  • Subject: Re: #388730 (Matt F) winxp ai9.0 spat trying to run a field process in hydrology modeling tools in spat ana.
  • Date: Tue, 19 Oct 2004 12:11:33 -0700
*** Do not write below this line ***
David,

Our developers for Spatial Analyst have sent some info that point
this problem out to be a limitation of the data size. At ArcGIS 9.0 we are
now able to create an ESRI Grid larger than 2.14 GB. It does this by
internally blocking the pixels into tiles. But because of performance
considerations a VAT is not built for these Grids. With the size of the grid
you have it is probably creating the blocks but not building the VAT. This
VAT is a requirement for FILL, SINK and FLOWDIRECTION. So this is considered
a bug.

CQ00252902 - Sink, Fill, FlowDirection produce incorrect results for large
datasets

Unfortunately the only workaround is to physically split the raster
into smaller, overlapping pieces and run the analysis. I've gone ahead and
added you to the bug. Hopefully our programmers will figure a way around this
problem.

Regards,

Matt F.

To contact ESRI Technical Support (USA only):
Tel: (909)793-3774 Fax: (909)792-0960 E-mail: support@esri.com

==========================================================================
For additional support use our 24 Hour Support Services:

ESRI's WWW homepage http://www.esri.com
Online Support Center http://support.esri.com
Knowledge Base http://support.esri.com/search/
Discussion Forums http://support.esri.com/forums/
Scripts & Tools http://arcscripts.esri.com/
E-mail Discussion Lists http://support.esri.com/listserve/
ArcObjects Online http://arcobjectsonline.esri.com/
ArcGIS Developer Online http://arcgisdeveloperonline.esri.com
==========================================================================

-----Original Message-----
From: "David Wheelock" <david@oda.state.ok.us>
Sent: Wed, 6 Oct 2004 16:00:21 -0500
To: "ESRI Support" <nimbus@esri.com>
Subject: Re: #388730 (Matt F) winxp ai9.0 spat trying to run a field
process in hydrology modeling tools in spat ana.

Hi, Matt,

Thanks for responding. The grid is 1.4 gig, compressed would be about 850
meg. Is that too big to upload?

I, too, did a small scale test. I cancelled a running fill and then re-ran
it successfully.

Also, I have since done a successful fill on the large dataset that only
took a few hours. Since the hung fill ran for more than half a day, after
I freed up disk space and before I cancelled it, it's clear that there was
something strange going on with that.

I think there are several possibilities for this problem:
1 Bad data.
2 Some artifact of the fact that the fill process hung for lack of disk
space before I cancelled it.
3 Cancelling the fill at a certain very specific point that left some trace
of the original file name behind somewhere. This would be very difficult to
replicate since the fill process has so many steps.

I'm about prepared to treat this as an anomally and move on to the next,
more vexing issue that I haven't been able to get around.

=========================================================
Major Vexing Problem

With the large dataset, I have run a fill, flow direction and flow
accumulation. However, the flow accumulation doesn't have major streams
delineated, just a bunch of disconnected minor stream segments. I've been
through this twice, with exactly the same results.

Additionally, when running the fill and some other processes on this data,
the resulting dataset is the same size in its dimensions and cell size, but
all of the data is compressed into the upper left corner of the dataset to
just a fraction of its original size. Changing the Analysis Cell Size from
"Maximum of Inputs" to "Minimum of Inputs" fixed this, but why should it
make any difference when there's only ONE input raster.

Could the grid be to large? Its dimensions are 9,575 x 37,335.

Interestingly, the grid is a mosaic of 16 pieces downloaded from the USGS
web site. I requested and downloaded the data in two batches of 8 grids on
two different days about a week apart. In both downloads there are areas
that contain no data. However, in the first batch the no-data cells simply
contain no data. In the second batch the no-data cells are given a 0
elevation.

Also, before getting the second batch, I did a mosaic of the first batch of
8, and did the fill, flow direction and flow accumulation and everything was
normal. There was no problem with the flow accumulation.

I've had to pull off at this point because it was taking too much time and I
had other things to do. Also, tomorrow I'm going to a conference, then back
Friday, and next week I will be off, so I won't be able to do much on this
problem till I return on the week of the 18th.

What do you think?

Thanks,
David

----- Original Message -----
From: "ESRI Support" <nimbus@esri.com>
To: "David Wheelock" <david@oda.state.ok.us>
Sent: Tuesday, October 05, 2004 5:38 PM
Subject: Re: #388730 (Matt F) winxp ai9.0 spat trying to run a field process
in hydrology modeling tools in spat ana.


--- End Message ---



Archive powered by MHonArc 2.6.16.

§