Re: PyNIO install and libjpeg

From: David B. Reusch <dreusch_at_nyahnyahspammersnyahnyah>
Date: Mon Sep 12 2011 - 14:22:34 MDT

Hi Mary,

Unfortunately this didn't work for me. I did a full rebuild with this
set; got the same error. I added F2CLIBS_PREFIX to my environment; same
error. Based on "nm nio.so", I seem to be missing over 200 symbols,
__gfortran_compare_string being the one it fails on. They seem to be
either Fortran-related (lots of math related symbols) or python-related
(_PyArg* _PyBook*, _PyDict*, _PyExc* and the like).

I took an extra close look at the install output but didn't see anything
obvious. The fink include and lib are ahead of /usr/local, but gfortran
is not installed under fink. I don't have my notes with me, but it
looks like gfortran was installed using "gfortran-macosx-x86.dmg". It's
been there since last Nov so I'd expect that something has used it
successfully since then. My numpy install from this a.m. referenced
gfortran though I don't see where it used it. numpy is working fine.

Happy to put install output or anything else somewhere for you.

Note that this all started when the pycdf module started failing with
missing netcdf symbols on my 10.6 system. It was working just fine on
my 10.5.8 system with netcdf 3.x. I installed netcdf 4.x last fall on
my 10.6 system. I've been assuming that the netcdf version is my
original problem and that led me to PyNIO as a replacement for pycdf.
After the time I've put into PyNIO, I'd like to see it through (plus
PyNIO is supported and recently updated; pycdf hasn't had an update in 3
1/2+ years). But if there's a way to get pycdf working, that'd be just
as good right now.

Thanks,
Dave

Mary Haley said the following on 9/12/2011 1:33 PM:
> David,
>
> Try:
>
> setenv F2CLIBS_PREFIX /usr/local/gfortran/lib
>
> --Mary
>
> On Sep 12, 2011, at 10:31 AM, David Reusch wrote:
>
>> Hi,
>>
>> Making some progress. I went ahead and installed python 2.7.2 from
>> source to /usr/local (therefore ahead of the Apple and fink versions).
>> That got me through the PyNIO install, i.e., no missing libraries at
>> link time.
>>
>> Now I'm stuck with a missing symbol error when I try to import Nio,
>> specifically:
>>> python
>>> Python 2.7.2 (default, Sep 12 2011, 09:04:02)
>>> [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>> import Nio
>>> Traceback (most recent call last):
>>> File "<stdin>", line 1, in<module>
>>> File "/usr/local/lib/python2.7/site-packages/PyNIO/Nio.py", line 63,
>>> in<module>
>>> from nio import *
>>> ImportError:
>>> dlopen(/usr/local/lib/python2.7/site-packages/PyNIO/nio.so, 2): Symbol
>>> not found: __gfortran_compare_string
>>> Referenced from: /usr/local/lib/python2.7/site-packages/PyNIO/nio.so
>>> Expected in: dynamic lookup
>>>
>> I did try a "setenv DYLD_LIBRARY_PATH" to point to my gfortran install
>> (/usr/local/gfortran/lib), but that didn't help. I suspect the answer
>> is in this direction, but I don't have any other ideas at the moment.
>>
>> Also, thanks for the compile-environment info. I had a feeling it was
>> something like that but hadn't found it yet. I was still poking around
>> the docs for dist-utils.
>>
>> Thanks,
>> Dave
>>
>> David Brown said the following on 9/9/2011 9:13 PM:
>>> Hi Dave,
>>> When you run 'python setup.py' most of the compiler settings are controlled by the version of python that you are using. Since the python that comes installed on the mac is a universal binary it will insist on compiling any modules as universal binaries as well because it wouldn't be able to import them otherwise. That is why I am recommending that you install another version of Python using fink, macports, or building it from source. There is also the Enthought Python Distribution (EPD), but it looks like the free version only has 32-bit for MacOS.
>>> -dave
>>>
>>> On Sep 9, 2011, at 5:11 PM, David B. Reusch wrote:
>>>
>>>> Hi,
>>>>
>>>> I can't say I was *intentionally* trying to build a universal binary. Just trying to get it to go on my Intel-based MacPro. But I see now that the system's trying to do it without being explictly asked. That's potentially all kinds of (never before seen) problems since fink only has x86_64 versions of these (and other) libraries (whereas my system-installed python is 3-way universal).
>>>>
>>>> After much searching, I still haven't found a way to keep the compiler from adding the unwanted arch flags. Plenty of help for adding them, nothing for excluding them. Also seems as though this is coming from somewhere outside of the PyNIO setup?
>>>>
>>>> I tried installing libjpeg outside of fink. Had to add LDFLAGS. That actually worked -- but then it broke on libpng! Haven't been able to fix that yet. It's starting to look like the whole reason for having fink is being subverted.
>>>>
>>>> If there was just a way to tell g++ that all I want is x86_64 and not a universal???
>>>>
>>>> Thanks,
>>>> Dave
>>>>
>>>> David Brown said the following on 9/9/11 2:28 PM:
>>>>> Hi Dave,
>>>>>
>>>>> We have not had much luck building MAC OS universal binaries here and we don't supply them for NCL or PyNGL/PyNIO as far as I know. Note that all your libraries (NetCDF, HDF5, jpeg, etc.) need to be built as universal for it to work. Before I noticed that about your output, I tried to simulate your situation by temporarily moving the jpeg library that I usually rely on in /usr/local/lib out of the way and using the macports installed version in /opt/local/lib/. As you did, I set F2CLIBS_PREFIX, but to /opt/local/lib. It worked for me, producing the following output:
>>>>>
>>>>> c++ -bundle -undefined dynamic_lookup build/temp.macosx-10.4-x86_64-2.7/niomodule.o -Llibsrc -L/usr/local/lib -L/usr/local/lib -L/Users/dbrown/local/lib -L/opt/local/lib -Lbuild/temp.macosx-10.4-x86_64-2.7 -lnio -ljpeg -lpng -lz -lnetcdf -lmfhdf -ldf -lhdf5_hl -lhdf5 -lcurl -lssl -lcrypto -lldap -lz -lhdfeos -lGctp -lgrib2c -ljasper -lpng -lhe5_hdfeos -lGctp -lgdal -lproj -lsz -lgfortran -o build/lib.macosx-10.4-x86_64-2.7/PyNIO/nio.so
>>>>>
>>>>> and not giving any error. Prior to setting F2CLIBS_PREFIX I got the same error you did:
>>>>>
>>>>> ld: library not found for -ljpeg
>>>>> collect2: ld returned 1 exit status
>>>>> ld: library not found for -ljpeg
>>>>>
>>>>> So your F2CLIBS_PREFIX should have worked, and I am speculating that perhaps it didn't because your libjpeg.a is not "universal". Are you using the MAC OS system python? If so, you may want to consider getting another version, from fink or macports perhaps, or maybe building it yourself from source.
>>>>> I will end with the caveat that Mary Haley knows a lot more about these build and install issues than I do. She is out at the moment but will be back next week.
>>>>> -dave
>>>>>
>>>>>
>>>>> On Sep 9, 2011, at 1:15 PM, David B. Reusch wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I'm trying to install PyNIO from source for the first time on a 64-but
>>>>>> Mac 10.6.8 system. I have libjpeg installed under fink (in /sw64/lib)
>>>>>> but have not been able to get the setup.py script to find it. The key
>>>>>> error output appears to be:
>>>>>>> g++-4.2 -Wl,-F. -bundle -undefined dynamic_lookup -arch i386 -arch ppc
>>>>>>> -arch x86_64 build/temp.macosx-10.6-universal-2.6/niomodule.o
>>>>>>> -Llibsrc -Lbuild/temp.macosx-10.6-universal-2.6 -lnio -ljpeg -lpng -lz
>>>>>>> -lnetcdf -o build/lib.macosx-10.6-universal-2.6/PyNIO/nio.so
>>>>>>> ld: library not found for -ljpeg
>>>>>> The script I'm using to drive my install (to keep things simpler) is:
>>>>>>> #!/bin/tcsh
>>>>>>> setenv HAS_GRIB2 0
>>>>>>> setenv HAS_HDFEOS 1
>>>>>>> setenv HAS_NETCDF4 1
>>>>>>> setenv NETCDF4_PREFIX /usr/local/netcdf-4.1.1
>>>>>>> setenv HDFEOS_PREFIX /usr/local/hdf5
>>>>>>>
>>>>>>> setenv F2CLIBS gfortran
>>>>>>> setenv F2CLIBS_PREFIX /sw64/lib
>>>>>>>
>>>>>>> sudo python setup.py install
>>>>>> Note that the lines with F2CLIBS didn't change anything after I added
>>>>>> them (presumably because gfortran wasn't having any problems...).
>>>>>>
>>>>>> I tried using setenv LDFLAGS "-L/sw64/lib -ljpeg" but that hasn't helped
>>>>>> either.
>>>>>>
>>>>>> In short, my guess is I need another setenv so that the script finds
>>>>>> /sw64/lib but I haven't been able to find the right one.
>>>>>>
>>>>>> Thanks,
>>>>>> Dave
>>>>>>
>>>>>> --
>>>>>> Associate Research Professor of Climatology
>>>>>> Dept of Earth and Environmental Science
>>>>>> MSEC 304; 801 Leroy Place
>>>>>> New Mexico Tech
>>>>>> Socorro, NM 87801
>>>>>>
>>>>>> _______________________________________________
>>>>>> pyngl-talk mailing list
>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>>>> --
>>>> Associate Research Professor of Climatology
>>>> Dept of Earth and Environmental Science
>>>> MSEC 304; 801 Leroy Place
>>>> New Mexico Tech
>>>> Socorro, NM 87801
>>>>
>> --
>> Associate Research Professor of Climatology
>> Dept of Earth and Environmental Science
>> MSEC 304; 801 Leroy Pl
>> New Mexico Tech
>> Socorro, NM 87801
>>
>>
>> --
>> Associate Research Professor of Climatology
>> Dept of Earth and Environmental Science
>> MSEC 304; 801 Leroy Place
>> New Mexico Tech
>> Socorro, NM 87801
>>
>> _______________________________________________
>> pyngl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>

-- 
Associate Research Professor of Climatology
Dept of Earth and Environmental Science
MSEC 304; 801 Leroy Pl
New Mexico Tech
Socorro, NM 87801
Received on Mon Sep 12 14:22:41 2011

This archive was generated by hypermail 2.1.8 : Mon Oct 10 2011 - 14:31:01 MDT