Re: Masked arrays in PyNIO 1.3.0b1

From: David Brown <dbrown_at_nyahnyahspammersnyahnyah>
Date: Tue, 2 Sep 2008 15:19:59 -0600

Hi Daran,

It seems that you are correct that PyNIO does not return masked
arrays when
data is requested through the get_value interface. This is
definitely an
oversight that will be corrected before the final release version
goes out.

Note however that given your use case:

import Nio, Ngl
dirc = Ngl.pynglpath("data")
ufile = Nio.open_file(dirc + "/cdf/Ustorm.cdf","r")
u = ufile.variables["u"]
u_data = u.get_value()

the only line that needs to be changed to work around the problem is
the last one:

u_data = u[:]

Actually this is true unless the data variable happens to be a
scalar, in which case
it will not work.

I think it will only require changes to Nio.py to fix this problem.
Assuming this is true,
I will post the new version to the mailing list once we have a solution.
  -dave

On Aug 31, 2008, at 1:06 PM, Daran L. Rife wrote:

> Hi Mary,
>
> I ran your test script, and Nio returned a masked
> array. Here is the output from my IDLE session.
>
> ---->
>
> Python 2.5 (r25:51918, Sep 19 2006, 08:49:13)
> [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
> Type "help", "copyright", "credits" or "license" for more
> information. Python 2.5
> (r25:51918, Sep 19 2006, 08:49:13)
> Type "copyright", "credits" or "license" for more information.
>
> IPython 0.8.4 -- An enhanced Interactive Python.
> ? -> Introduction and overview of IPython's features.
> %quickref -> Quick reference.
> help -> Python's own help system.
> object? -> Details about 'object'. ?object also works, ?? prints
> more.
>
> In [1]: import Nio, Ngl
> In [2]: dirc = Ngl.pynglpath("data")
> In [3]: ufile = Nio.open_file(dirc + "/cdf/Ustorm.cdf","r")
> In [4]: u = ufile.variables["u"][:]
> In [5]: print type(u)
> <class 'numpy.ma.core.MaskedArray'>
>
> <----
>
> I re-ran my "test case" using your idiom above:
>
> var = myfile.variables['my_var'][:]
>
> And Nio returns a masked array. What differs between
> your usage and mine, is that I generally keep data
> read in as a <class 'Nio.NioVariable'>, because all
> the coordinates meta data are preserved. Thus, my
> usage for your test case would be:
>
> import Nio, Ngl
> dirc = Ngl.pynglpath("data")
> ufile = Nio.open_file(dirc + "/cdf/Ustorm.cdf","r")
> u = ufile.variables["u"]
> u_data = u.get_value()
>
> Which returns a NumPy ndarray, rather than a masked
> array. Granted, it is very straight forward to tran-
> sform this to a masked array via:
>
> u_data = ma.masked_values(u.get_value(), u._FillValue)
>
> A safer way to do this (assuming no a priori knowledge
> of a variable's attributes) is:
>
> if hasattr(u, '_FillValue'):
> u_data = ma.masked_values(u.get_value(), u._FillValue)
> else: u_data = u.get_value()
>
> Perhaps I don't understand the intent of the get_value()
> method.
>
> Thanks for the example on the MaskedArrayOptions. They
> key piece I was missing was the
>
> opt = Nio.options()
> opt.MaskedArrayMode = 'MaskedIfFillAtt'
> f = Nio.open_file(filename,options=opt)
>
> Instead, I was simply doing this:
>
> f = Nio.open_file(filename, options='MaskedIfFillAtt')
>
> Which through a pile of exceptions.
>
>
> Daran
>
> --
>
>> You should indeed be getting a MaskedArray if your data contains a
>> _FillValue or
> missing_value attribute.
>>
>> Can you try a test for us?
>>
>> Run the short script below, and let me know what type "u" is:
>>
>> import Nio, Ngl
>> dirc = Ngl.pynglpath("data")
>> ufile = Nio.open_file(dirc + "/cdf/Ustorm.cdf","r")
>> u = ufile.variables["u"][:]
>> print type(u)
>>
>> It should return "<class 'numpy.ma.core.MaskedArray'>".
>>
>> If you don't have Ngl, then you can download the file from
>> http://www.ncl.ucar.edu/Applications/Data/, and run the following
>> script:
>>
>> import Nio
>> ufile = Nio.open_file("Ustorm.cdf","r")
>> u = ufile.variables["u"][:]
>> print type(u)
>>
>> As for some examples for setting the MaskedArrayMode options, I
>> believe Dave Brown is working on some. Meanwhile, you can try the
>> attached test
> program.
>>
>> --Mary
>>
>> On Sat, 30 Aug 2008, Daran L. Rife wrote:
>>
>>> Hi,
>>> This didn't seem to get through last time round.
>>> I Recently installed the latest version of PyNIO 1.3.0b1
>>> for Mac OS X Leopard (Intel) running Python 2.5:
>>> PyNIO-1.3.0b1.macosx-10.5-i386-2.5.tar.gz
>>> I was excited to learn that when reading variables from
>>> netCDF files, Nio now returns a masked array for any var-
>>> iable having the _FillValue or missing_value attribute
>>> set within the input file.
>>> When I try this, Nio still seems to return a numpy.ndarray,
>>> instead of the desired masked array. Attached below is out-
>>> put from my IDLE session. Could someone please help me un-
>>> derstand what I am doing wrong?
>>> Also, are there examples showing how to properly set the
>>> new MaskedArrayMode options?
>>> Thanks in advance for your help,
>>> Daran
>>> --
>>> --->
>>> Python 2.5 (r25:51918, Sep 19 2006, 08:49:13)
>>> Type "copyright", "credits" or "license" for more information.
>>> IPython 0.8.4 -- An enhanced Interactive Python.
>>> ? -> Introduction and overview of IPython's features.
>>> %quickref -> Quick reference.
>>> help -> Python's own help system.
>>> object? -> Details about 'object'. ?object also works, ??
>>> prints more. In [1]:
> import Nio as nio
>>> In [2]: f = nio.open_file('composite.20080523.L3m_DAY_SST_4.nc',
>>> 'r') In [3]: sst
> = f.variables['sst_data']
>>> In [4]: print sst
>>> Variable: sst_data
>>> Type: float
>>> Total Size: 3110400 bytes
>>> 777600 values
>>> Number of Dimensions: 3
>>> Dimensions and sizes: [time | 1] x [latitude | 720] x
>>> [longitude | 1080]
> Coordinates:
>>> time: [14022..14022]
>>> latitude: [24.52084..54.47918]
>>> longitude: [-98.47916..-53.52083]
>>> Number of Attributes: 5
>>> long_name : MODIS 8-day composite SST
>>> standard_name : sea_surface_temperature
>>> units : degC
>>> valid_range : [-1.979918837547302,
>>> 31.75502777099609] _FillValue
> :
>>> 1e+20
>>> In [5]: s = sst.get_value()
>>> In [6]: type(s)
>>> Out[6]: <type 'numpy.ndarray'>
>>> In [8]: from numpy import ma
>>> In [9]: s = ma.masked_values(sst.get_value(), sst._FillValue) #
>>> Force data into
> ma
>>> In [10]: type(s)
>>> Out[10]: <class 'numpy.ma.core.MaskedArray'>
>>> <---
>>> _______________________________________________
>>> pyngl-talk mailing list
>>> pyngl-talk_at_ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>
>
>
>
>
>
> _______________________________________________
> pyngl-talk mailing list
> pyngl-talk_at_ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
Received on Tue Sep 02 2008 - 15:19:59 MDT

This archive was generated by hypermail 2.2.0 : Wed Sep 03 2008 - 11:24:47 MDT