undefined symbol: register_derivedmscal when importing casacore.tables.

Trying to install the whole LOFAR software on an ubuntu 20.04 workstation.

Create and activate a virtualenv using the system python 3.8 in /opt/LOFAR/python:

$ python3 -m venv --prompt lofar /opt/LOFAR/python
$ source /opt/LOFAR/python/bin/activate

Install python requirements and ipython:

$ pip install numpy wheel ipython

Download casacore data into /opt/LOFAR/casa/data
Building and install casacore:

$ curl -L https://github.com/casacore/casacore/archive/refs/tags/v3.4.0.tar.gz | tar xz
$ cmake -S . -B build -DCMAKE_INSTALL_PREFIX=/opt/LOFAR/casa/3.4.0 -DUSE_OPENMP=ON -DUSE_HDF5=ON -DBUILD_PYTHON=OFF -DBUILD_PYTHON3=ON -DUSE_THREADS=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDATA_DIR=/opt/LOFAR/casa/data
$ cmake --build build -- -j24
$ cmake --install build

Download and install dysco:

$ curl -L https://github.com/aroffringa/dysco/archive/refs/tags/v1.2.tar.gz | tar xz
$ cd dysco-1.2
$ cmake -S. -B build -DCMAKE_INSTALL_PREFIX=/opt/LOFAR/casa/3.4.0 -DCMAKE_BUILD_TYPE=RelWithDebInfo
$ cmake --build build -- -j24
$ cmake --install build

Install python-casacore in --no-binary-mode:

$ export CFLAGS="--std=c++11 -L/opt/LOFAR/casa/3.4.0/lib -I/opt/LOFAR/casa/3.4.0/include"
$ export LDFLAGS="-Wl,-rpath,/opt/LOFAR/casa/3.4.0/lib" 
$ pip install --no-binary python-casacore --no-cache python-casacore

I then get:

In [1]: from casacore import tables
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 from casacore import tables

File /opt/LOFAR/python/lib/python3.8/site-packages/casacore/tables/__init__.py:60, in <module>
      1 # __init__.py: Top level .py file for python table interface
      2 # Copyright (C) 2006
      3 # Associated Universities, Inc. Washington DC, USA.
   (...)
     25 #
     26 # $Id: __init__.py,v 1.6 2006/11/06 01:54:21 gvandiep Exp $
     28 """Python interface to the Casacore tables module.
     29 
     30 A `casacore table <../../casacore/doc/html/group__Tables__module.html>`_
   (...)     57 
     58 """
---> 60 from .msutil import *
     61 from .table import table
     62 from .table import default_ms

File /opt/LOFAR/python/lib/python3.8/site-packages/casacore/tables/msutil.py:29, in <module>     27 import numpy as np
     28 import six
---> 29 from casacore.tables.table import (table, taql,
     30                                    _required_ms_desc,
     31                                    _complete_ms_desc)
     32 from casacore.tables.tableutil import (makescacoldesc, makearrcoldesc,
     33                                        makecoldesc, maketabdesc)
     36 def required_ms_desc(table=None):

File /opt/LOFAR/python/lib/python3.8/site-packages/casacore/tables/table.py:40, in <module>
     25 """Access to Casacore tables.
     26 
     27 The :class:`table` class is the main class to access a table. Its constructor
   (...)
     36 
     37 """
     39 from six import string_types
---> 40 from ._tables import (Table,
     41                       _default_ms,
     42                       _default_ms_subtable,
     43                       _required_ms_desc,
     44                       _complete_ms_desc)
     46 from .tablehelper import (_add_prefix, _remove_prefix, _do_remove_prefix,
     47                           _format_row)
     48 import six

ImportError: /opt/LOFAR/python/lib/python3.8/site-packages/casacore/tables/_tables.cpython-38-x86_64-linux-gnu.so: undefined symbol: register_derivedmscal

Any help would be much appreciated!

This may be related

python-casacore/src/tables.cc

Lines 57 to 61 in f722ae9

    
           // Register the TaQL meas and mscal functions. 
        
           // Normally they are loaded as a shared library, but that cannot 
        
           // be done if the program is built statically. 
        
           register_meas(); 
        
           register_derivedmscal();

Ah, dam. There was a hint about adding -DBUILD_SHARED_LIBS somewhere. I guess we need to add this to the casacore cmake call.

There's a hint to use this flag for the AOFlagger install on the LOFAR software page here

Note: use the -DBUILD_SHARED_LIBS=ON flag! See http://sourceforge.net/p/aoflagger

I rebuilt it from scratch with -DBUILD_SHARED_LIBS=ON for the casacore cmake but the error still remains.

I think I figured out a fix for this.

libcasa_derivedmscal.so doesn't show up in the _tables.cpython-38-x86_64-linux-gnu.so

(lofar) root@e5b-dell-09:/opt/LOFAR# ldd /opt/LOFAR/python/lib/python3.8/site-packages/casacore/tables/_tables.cpython-38-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007fff39f8a000)
        libcasa_meas.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_meas.so.6 (0x00007f3d6cb5b000)
        libcasa_ms.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_ms.so.6 (0x00007f3d6c66a000)
        libcasa_tables.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_tables.so.6 (0x00007f3d6bef6000)
        libboost_python38.so.1.71.0 => /lib/x86_64-linux-gnu/libboost_python38.so.1.71.0 (0x00007f3d6be97000)
        libcasa_python3.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_python3.so.6 (0x00007f3d6be3c000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f3d6bc5a000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f3d6bc3d000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3d6ba4b000)
        libcasa_measures.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_measures.so.6 (0x00007f3d6b725000)
        libcasa_casa.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_casa.so.6 (0x00007f3d6b2e7000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3d6b198000)
        libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f3d6b156000)
        libcasa_scimath.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_scimath.so.6 (0x00007f3d6afa0000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3d6af7d000)
        liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3 (0x00007f3d6a8d9000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3d6cd06000)
        libpython3.8.so.1.0 => /lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f3d6a383000)
        libhdf5_serial.so.103 => /lib/x86_64-linux-gnu/libhdf5_serial.so.103 (0x00007f3d6a006000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3d69ffe000)
        libcasa_scimath_f.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_scimath_f.so.6 (0x00007f3d69fd7000)
        libfftw3f_threads.so.3 => /lib/x86_64-linux-gnu/libfftw3f_threads.so.3 (0x00007f3d69fcd000)
        libfftw3_threads.so.3 => /lib/x86_64-linux-gnu/libfftw3_threads.so.3 (0x00007f3d69fc3000)
        libfftw3f.so.3 => /lib/x86_64-linux-gnu/libfftw3f.so.3 (0x00007f3d69db3000)
        libfftw3.so.3 => /lib/x86_64-linux-gnu/libfftw3.so.3 (0x00007f3d69bab000)
        libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 (0x00007f3d69b3e000)
        libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007f3d69876000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f3d69848000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f3d6982c000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f3d69827000)
        libsz.so.2 => /lib/x86_64-linux-gnu/libsz.so.2 (0x00007f3d69820000)
        libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f3d697d6000)
        libaec.so.0 => /lib/x86_64-linux-gnu/libaec.so.0 (0x00007f3d697cd000)

It seems to be that there is a default flag added to ld called --as-needed on Ubuntu which results in libraries not beeing linked (take a look here)
Defining export LDFLAGS="-Wl,-rpath,/opt/LOFAR/casa/3.4.0/lib,--no-as-needed -lcasa_derivedmscal" as recommended here resolves the error. The library now shows up

(lofar) root@e5b-dell-09:/opt/LOFAR# ldd /opt/LOFAR/python/lib/python3.8/site-packages/casacore/tables/_tables.cpython-38-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007fffe7aca000)
    --> libcasa_derivedmscal.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_derivedmscal.so.6 (0x00007f114aba3000)
        libcasa_meas.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_meas.so.6 (0x00007f114aa98000)
        libcasa_ms.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_ms.so.6 (0x00007f114a5a7000)
        libcasa_tables.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_tables.so.6 (0x00007f1149e33000)
        libboost_python38.so.1.71.0 => /lib/x86_64-linux-gnu/libboost_python38.so.1.71.0 (0x00007f1149dd4000)
        libcasa_python3.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_python3.so.6 (0x00007f1149d79000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1149b95000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1149a46000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f1149a2b000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1149a08000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1149816000)
        libcasa_measures.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_measures.so.6 (0x00007f11494f0000)
        libcasa_casa.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_casa.so.6 (0x00007f11490b0000)
        libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f114906e000)
        libcasa_scimath.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_scimath.so.6 (0x00007f1148eba000)
        liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3 (0x00007f1148816000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f114acee000)
        libpython3.8.so.1.0 => /lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f11482c0000)
        libhdf5_serial.so.103 => /lib/x86_64-linux-gnu/libhdf5_serial.so.103 (0x00007f1147f41000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1147f3b000)
        libcasa_scimath_f.so.6 => /opt/LOFAR/casa/3.4.0/lib/libcasa_scimath_f.so.6 (0x00007f1147f14000)
        libfftw3f_threads.so.3 => /lib/x86_64-linux-gnu/libfftw3f_threads.so.3 (0x00007f1147f0a000)
        libfftw3_threads.so.3 => /lib/x86_64-linux-gnu/libfftw3_threads.so.3 (0x00007f1147f00000)
        libfftw3f.so.3 => /lib/x86_64-linux-gnu/libfftw3f.so.3 (0x00007f1147cee000)
        libfftw3.so.3 => /lib/x86_64-linux-gnu/libfftw3.so.3 (0x00007f1147ae8000)
        libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 (0x00007f1147a7b000)
        libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007f11477b3000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f1147785000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1147769000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f1147762000)
        libsz.so.2 => /lib/x86_64-linux-gnu/libsz.so.2 (0x00007f114775d000)
        libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f1147713000)
        libaec.so.0 => /lib/x86_64-linux-gnu/libaec.so.0 (0x00007f114770a000)

and the module can be imported

In [1]: from casacore import tables

In [2]: tables
Out[2]: <module 'casacore.tables' from '/opt/LOFAR/python/lib/python3.8/site-packages/casacore/tables/__init__.py'>

In [3]: tables.table('/home/stefan/Documents/PhD/RADIO/data/LOFAR/CVnI/L652360_SB001_uv.pre-cal_12BF8C5ACt_121MHz.pre-cal.ms.archive')
Successful readonly open of default-locked table /home/stefan/Documents/PhD/RADIO/data/LOFAR/CVnI/L652360_SB001_uv.pre-cal_12BF8C5ACt_121MHz.pre-cal.ms.archive: 25 columns, 6798145 rows
Out[3]: <casacore.tables.table.table at 0x7ff425d540e0>

Wow that's a nasty flag. Thanks for figuring it out! So do you think we should put this flag in the setup.py of python-casacore?

From troubleshooting "as-nneded":

This usually means that a library was not linked to another library, but was using it, and then relying on the final executable to link them together

Navigation Menu

Repository navigation

undefined symbol: register_derivedmscal when importing `casacore.tables`. #227

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	// Register the TaQL meas and mscal functions.
	// Normally they are loaded as a shared library, but that cannot
	// be done if the program is built statically.
	register_meas();
	register_derivedmscal();

undefined symbol: register_derivedmscal when importing casacore.tables. #227

Description

Activity

StFroese commented on Mar 18, 2022

maxnoe commented on Mar 18, 2022

StFroese commented on Mar 18, 2022

StFroese commented on Mar 19, 2022

StFroese commented on Mar 19, 2022

tammojan commented on Mar 19, 2022

maxnoe commented on Mar 19, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

undefined symbol: register_derivedmscal when importing `casacore.tables`. #227