Writing speed depends on indexed dimension

From: Sourish Basu <S.Basu_at_nyahnyahspammersnyahnyah>
Date: Mon Oct 04 2010 - 04:04:55 MDT

Hi All,

     I'm trying to write a very large array to a variable
var[2922,25,180,360] in a netcdf file with the option LargeFile set. The
file size /per se/ is not a problem. I'm computing and writing it in
chunks of 25x180x360. The problem is that the write speed depends on the
length of the first index. For example, the following

opt = options()
opt.Format = 'LargeFile'
fid = open_file('test.nc', 'w', options=opt)
fid.create_dimension('days',2922)
fid.create_dimension('pres_divisions',25)
fid.create_dimension('pres_levels',26)
fid.create_dimension('latitude',180)
fid.create_dimension('longitude',360)
conc_var = fid.create_variable('mixing_ratio', 'd', ('days',
'pres_divisions', 'latitude', 'longitude'))
conc_var[0] = random((25,180,360))

takes a very long time, whereas if I just change the definition of 'days' to

fid.create_dimension('days',2)

then the writing of conc_var[0] takes less than a second. I want to
write the blocks sequentially, i.e., conc_var[0], then conc_var[1], then
conc_var[2] and so on. Is there any way to speed up the writing (in the
case where 'days' is 2922) by using that fact?

Cheers,

-Sourish

_______________________________________________
pyngl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/pyngl-talk

Received on Mon Oct 4 04:04:50 2010

This archive was generated by hypermail 2.1.8 : Mon Nov 15 2010 - 09:11:30 MST