Re: problem with PyNIO dimension sizes

HI Zach,

I have checked in a fix for the single-element dimension problem.
With this fix, the following test code works without error:

import numpy
import Nio
import os
ncols = 1024
nrows = 1024
nlev = 1

prcp = numpy.zeros(nlev*nrows*ncols,dtype='float32')
prcp = prcp.reshape(nlev,nrows,ncols)

os.system("rm -f test.nc")
ncfile = Nio.open_file("test.nc","c")

ncfile.create_dimension('x',ncols)
ncfile.create_dimension('y',nrows)
ncfile.create_dimension('levels',nlev)
ncfile.create_dimension('record',1)

d4 = ('record','levels','y','x')

prcp_NC = ncfile.create_variable('prcp','f',d4)

# a number of variations that repeatedly perform the same assignment

prcp_NC[0,0,:,:] = prcp[0,:,:]
prcp_NC[:] = prcp[:,:,:]
prcp_NC[:] = prcp[:]
prcp_NC[:] = prcp[:,:]
prcp_NC[:] = prcp[:,:,:]
prcp_NC[:,:] = prcp[:,:,:]
prcp_NC[:,:,:,:] = prcp[:,:,:]
prcp_NC[:,0,:,:] = prcp[:]

ncfile.close()

------

We can get you a new version of the code if you email Mary.
  -dave

On Dec 15, 2008, at 10:57 AM, David Brown wrote:

> Hi Zach,
>
> OK, I understand. I will be looking into this immediately and
> hopefully will have a fix soon.
> -dave
>
> On Dec 14, 2008, at 1:49 PM, Zach DuFran wrote:
>
>> David-
>> Thanks for your response.
>>
>> Due to my only providing excerpts from my code, it was unclear
>> what I was doing.
>> 1. I was creating a new NetCDF file with each iteration of the t
>> loop, so I wasn't redeclaring dimensions for the same file.
>> 2. I did not want to write multiple times to the same file. I
>> was writing one time to each of the 41 (nt) files I was generating
>> from this script.
>> 3. The levels and record dimensions are simply dummy dimensions
>> that are required so that my NetCDF file format meets a specific
>> format that is required for my purposes.
>>
>> Sorry for the confusion.
>>
>> Let me know when you have a bug fix for the dimension problem.
>> Thanks!
>> Zach
>>
>> ________________________________________
>> From: David Brown [dbrown_at_ucar.edu]
>> Sent: Sunday, December 14, 2008 2:13 PM
>> To: Zach DuFran
>> Cc: pyngl-talk_at_ucar.edu Talk
>> Subject: Re: problem with PyNIO dimension sizes
>>
>> Hi Zach,
>>
>> Yes it does appear that something is not working right when some of
>> the dimensions have single elements. However, for what I think you
>> are trying to do there is a work-around. The code you posted and as
>> edited by Mary below has some problems. First of all, you can only
>> create a dimension or a variable in a file once, so you are sure to
>> get an exception the second time through the loop. Second you are
>> creating the level dimension with 1 element, yet your use of the loop
>> counter t suggests that you really mean it to have 'nt' elements.
>>
>> Assuming this is your intention, it should work with the following
>> modifications:
>>
>> import numpy
>> import Nio
>> import os
>> ncols = 1024
>> nrows = 1024
>> nt = 41
>>
>> # Ingest data
>> #databin = array.array('f')
>> #databin.read(fileobj, nt*nrows*ncols)
>> #prcp = numpy.array(databin, dtype='float32')
>> prcp = numpy.zeros(nt*nrows*ncols,dtype='float32')
>> prcp = prcp.reshape(nt,nrows,ncols)
>>
>> os.system("rm -f test.nc")
>> ncfile = Nio.open_file("test.nc","c")
>>
>> ncfile.create_dimension('x',ncols)
>> ncfile.create_dimension('y',nrows)
>> ncfile.create_dimension('levels',nt) ###
>> ncfile.create_dimension('record',1)
>>
>> d4 = ('record','levels','y','x')
>>
>> prcp_NC = ncfile.create_variable('prcp','f',d4)
>>
>> # either of the following work for me (using PyNIO 1.3.0b1)
>>
>> for t in range(nt):
>> prcp_NC[0,t,:,:] = prcp[t,:,:]
>>
>> # or most simply
>>
>> prcp_NC[:] = prcp[:]
>>
>> ---------
>>
>> However, note that if the level dimension 'nt' is actually equal to
>> 1, then neither of the methods work. This is definitely a bug and
>> will be addressed as soon as possible.
>> -dave
>>
>>
>> On Dec 12, 2008, at 3:25 PM, Mary Haley wrote:
>>
>>> Hi Zach,
>>>
>>> This looks like a bug, but I'm not certain.
>>>
>>> I took your code snippet, and made it into code I could actually
>>> run, and I got the same results as you:
>>>
>>> import numpy, os, Nio
>>>
>>> ncols = 1024
>>> nrows = 1024
>>> nt = 41
>>>
>>> prcp = numpy.zeros([nt,nrows,ncols], dtype='float32')
>>>
>>> filename = "test.nc"
>>>
>>> for t in range(nt):
>>> if os.path.exists(filename): os.remove(filename)
>>> ncfile = Nio.open_file(filename, 'c')
>>> ncfile.create_dimension('x',ncols)
>>> ncfile.create_dimension('y',nrows)
>>> ncfile.create_dimension('levels',1)
>>> ncfile.create_dimension('record',1)
>>> d4 = ('record','levels','y','x')
>>> prcp_NC = ncfile.create_variable('prcp','f',d4)
>>> print prcp_NC.shape
>>> print prcp.shape
>>> print prcp_NC[0,0,:,:].shape
>>> print prcp[t,:,:].shape
>>> prcp_NC[0,0,:,:] = prcp[t,:,:]
>>>
>>> Hopefully Dave can take a look at this and see a work-around. I
>>> tried a few things with no luck.
>>>
>>> --Mary
>>>
>>> On Fri, 12 Dec 2008 10:48:16 -0600
>>> Zach DuFran <zdufran_at_wdtinc.com> wrote:
>>>> I am trying to write a 4D variable to a NetCDF file and I am
>>>> receiving a fatal error. The line where the error occurs is
>>>> prcp_NC[0,0,:,:] = prcp[t,:,:]. I have printed out the shapes of
>>>> each variable and they match. When I switch the order in prcp_NC
>>>> to be prcp_NC[:,:,0,0] the script will run succesfully, but the
>>>> data in the NetCDF file is all blank.
>>>> If I assign prcp_NC as a 2D variable (only x and y) and do prcp_NC
>>>> [:,:] = prcp[t,:,:] the data is written correctly.
>>>> The error message for the attached script is:
>>>> fatal:Dimension sizes of left hand side do not match
>>>> right hand side
>>>> The only way I can get the assignment to a 4D variable to work is
>>>> to add 2 loops so the code reads:
>>>> for i in xrange(nrows):
>>>> for j in xrange(ncols):
>>>> prcp_NC[0,0,i,j] = prcp[t,i,j]
>>>> Of course, this method is considerably slower.
>>>> I would appreciate any help you can provide-
>>>> Zach DuFran
>>>
>>> _______________________________________________
>>> pyngl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>
> _______________________________________________
> pyngl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk

_______________________________________________
pyngl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
Received on Fri Dec 19 2008 - 11:16:26 MST