Re: Subsetting and disk I/O

Hi David,

Thanks for your prompt reply.

2009/2/26 David Brown <dbrown_at_ucar.edu>:
> This approach seems to be reasonably efficient for the most common access
> patterns, but is obviously sub-optimal for your application.
> It would require some fairly significant new code development to do
> sub-setting directly
> on the packed record data, although it might not be too difficult if it was
> limited to the case
> of getting a single point. Perhaps we could consider this.
> Are you using GRIB 1 or 2 data?

I am using GRIB 2 data.

> Currently the only alternative I can suggest would be to pre-convert the
> GRIB data to NetCDF.
> This would definitely allow for faster random access of single elements.
> I can understand that this might not be feasible however.

I just tried doing that using wgrib2. The processing time for my
largest file (which is from the GFS) was less than 1 minute (without
repacking the data*). So it is definitely possible (and easy for me to
implement). Does your NetCDF reader support access to
subsets/individual points without reading an entire record? If it does
this solution will do just fine for me.

*) It is not obvious to me whether packed data are faster or slower to
access. On one hand you need to read in much less data while on the
other hand you need to use processing power to unpack it. I guess I
will have to make some tests for that (but I suspect that accessing
packed data is faster). Do you have any experience with this?

Best regards,
Jesper
_______________________________________________
pyngl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
Received on Thu Feb 26 2009 - 01:12:50 MST