Skip to content

Interop with casatasks/casatools #251

New issue
New issue
Open
Open
Interop with casatasks/casatools#251

Description

@AlecThomson
AlecThomson
opened on Nov 30, 2023

Hi there,

Apologies if this is wrong stream to report this issue. Myself and @tjgalvin have had adventures trying to install both python-casacore and casatasks/casatools on Python 3.10. I've been testing on MacOS Apple Silicon, but @tjgalvin has had similar results on linux.

First, installation with pip alone does not seem to work. I needed to resort to using a conda install. There are also issues with casatasks/casatools and how it handles matplotlib - it seems to be have been built against matplotlib<=3.5.1 and newer versions break.

Here is summary of my first stages of installation trials with minimal conda installs:

#!/usr/bin/env bash

# Create a conda environment for CASA in x86_64 architecture
CONDA_SUBDIR=osx-64 conda create -n casa_py310 python=3.10 pip ipython -y
conda activate casa_py310

# Install CASA
pip install casatasks casatools casadata python-casacore

# Test CASA
python -c "from casatasks import tclean; from casacore.tables import table; print(tclean.__doc__); print(table.__doc__)"
# Fails - Matplotlib is called incorrectly
# Traceback (most recent call last):
#   File "<string>", line 1, in <module>
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatasks/__init__.py", line 249, in <module>
#     from .plotbandpass import plotbandpass
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatasks/plotbandpass.py", line 9, in <module>
#     from .private.task_plotbandpass import plotbandpass as _plotbandpass_t
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatasks/private/task_plotbandpass.py", line 36, in <module>
#     warnings.filterwarnings("ignore",category=matplotlib.cbook.MatplotlibDeprecationWarning)
# AttributeError: module 'matplotlib.cbook' has no attribute 'MatplotlibDeprecationWarning'. Did you mean: 'VisibleDeprecationWarning'?

# Fix matplotlib
pip install "matplotlib<=3.5.1"

# Test CASA
python -c "from casatasks import tclean; from casacore.tables import table; print(tclean.__doc__); print(table.__doc__)"

# Fails - boost is not installed
# Traceback (most recent call last):
#   File "<string>", line 1, in <module>
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/__init__.py", line 60, in <module>
#     from .msutil import *
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/msutil.py", line 29, in <module>
#     from casacore.tables.table import (table, taql,
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/table.py", line 40, in <module>
#     from ._tables import (Table,
# ImportError: dlopen(/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/_tables.cpython-310-darwin.so, 0x0002): Library not loaded: @rpath/libboost_python310.dylib
#   Referenced from: <E240941C-9D5F-3C5E-BF5D-93D9C4629B0E> /Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/_tables.cpython-310-darwin.so
#   Reason: tried: '/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/Users/tho822/mambaforge/envs/casa_py310/bin/../lib/libboost_python310.dylib' (no such file), '/Users/tho822/mambaforge/envs/casa_py310/bin/../lib/libboost_python310.dylib' (no such file), '/usr/local/lib/libboost_python310.dylib' (no such file), '/usr/lib/libboost_python310.dylib' (no such file, not in dyld cache)

# Install boost
conda install boost -y

# Test CASA
python -c "from casatasks import tclean; from casacore.tables import table; print(tclean.__doc__); print(table.__doc__)"

# casacore fails
# Traceback (most recent call last):
#   File "<string>", line 1, in <module>
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/__init__.py", line 60, in <module>
#     from .msutil import *
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/msutil.py", line 29, in <module>
#     from casacore.tables.table import (table, taql,
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/table.py", line 40, in <module>
#     from ._tables import (Table,
# ImportError: dlopen(/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/_tables.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '__PyTraceMalloc_NewReference'

# But casatasks works
python -c "from casatasks import tclean; print(tclean.__doc__)"
# Works

python -c "from casacore.tables import table; print(table.__doc__)"
# Fails

Next I resorted to the conda-forge build. This can seemingly work, but the order of imports matters with casatasks/casatools:

#!/usr/bin/env bash

# Create a conda environment for CASA in x86_64 architecture
CONDA_SUBDIR=osx-64 conda create -n casa_py310 python=3.10 pip ipython python-casacore -y
conda activate casa_py310
pip install "matplotlib<=3.5.1" casatasks casatools
python -c "from casatasks import tclean; from casacore.tables import table; print(tclean.__doc__); print(table.__doc__)"
# Works
python -c "from casacore.tables import table; from casatasks import tclean; print(tclean.__doc__); print(table.__doc__)"
# Fails
# Traceback (most recent call last):
#   File "<string>", line 1, in <module>
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatasks/__init__.py", line 5, in <module>
#     from casatools import logsink as _logsink
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__init__.py", line 43, in <module>
#     from .image import image
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/image.py", line 4, in <module>
#     from .__casac__.image import image as _image
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__casac__/image.py", line 10, in <module>
#     from . import _image
# ImportError: dlopen(/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__casac__/_image.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '__ZN8casacore11ArrayColumnINSt3__17complexIdEEE8putSliceEyRKNS_6SlicerERKNS_5ArrayIS3_EE'

Activity

  1. AlecThomson commented on Nov 30, 2023

    @AlecThomson
    AlecThomson
    on Nov 30, 2023
    Author

    On linux, using that last script, I get these errors:

    ~> python -c "from casatasks import tclean; from casacore.tables import table; print(tclean.__doc__); print(table.__doc__)"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/__init__.py", line 60, in <module>
    from .msutil import *
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/msutil.py", line 29, in <module>
    from casacore.tables.table import (table, taql,
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/table.py", line 40, in <module>
    from ._tables import (Table,
ImportError: /home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/../../../../libcasa_python3.so.7: undefined symbol: _ZN8casacore6VectorIxSaIxEE6resizeERKNS_9IPositionEb

    and

    ~> python -c "from casacore.tables import table; from casatasks import tclean; print(tclean.__doc__); print(table.__doc__)"
Traceback (most recent call last):
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__casac__/image.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1176, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: /home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__casac__/lib/libcasa_fits.so.7: undefined symbol: _ZTIN8casacore5ArrayIhEE

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatasks/__init__.py", line 5, in <module>
    from casatools import logsink as _logsink
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__init__.py", line 43, in <module>
    from .image import image
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/image.py", line 4, in <module>
    from .__casac__.image import image as _image
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__casac__/image.py", line 17, in <module>
    _image = swig_import_helper()
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__casac__/image.py", line 16, in swig_import_helper
    return importlib.import_module('_image')
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_image'

  2. AlecThomson commented on Aug 27, 2024

    @AlecThomson
    AlecThomson
    on Aug 27, 2024
    Author

    Hey all - just wondering if anyone has had a chance to look at this?

    Tagging @gijzelaerr @tammojan for some visibility.

  3. tammojan commented on Aug 27, 2024

    @tammojan
    tammojan
    on Aug 27, 2024
    Contributor

    Hi, thanks for the ping. I think trying to have both the CASA python bindings and the python-casacore python bindings to casacore is near impossible. That is due to the underlying casacore. If you load two python bindings, two casacores will be loaded, but the symbols of these two will shadow each other. If the versions are binary incompatible (which is probably the case) this will in the best case lead to segfaults, in a good case you get the error on importing.

    In theory this should be solvable, but on the short term we have no plans to work on this. So if you use casatools, you'd have to access the table also through casatools - or use a trick with multiple venvs or so.

  4. AlecThomson commented on Aug 27, 2024

    @AlecThomson
    AlecThomson
    on Aug 27, 2024
    Author

    Thanks so much @tammojan. Really appreciate the feedback on this. I'll give the casatools.table functionality some testing to see if we can get things working.

  5. AlecThomson commented on Jun 19, 2026

    @AlecThomson
    AlecThomson
    on Jun 19, 2026
    Author

    Hey @tammojan - sorry to bump this very old ticket! I heard (and now tested) that this seems to be resolved for Python >= 3.11. Is this expected? Is the underlying fix expected to be stable?

  6. tammojan commented on Jun 19, 2026

    @tammojan
    tammojan
    on Jun 19, 2026
    Contributor

    I'm afraid the answers to both your questions is no. The underlying issue is not resolved, casatools and python-casacore both install a separate casacore library. These are not expected to be binary (ABI) compatible, but in some cases they may be. So if it works, it's a matter of luck.

  7. AlecThomson commented on Jun 19, 2026

    @AlecThomson
    AlecThomson
    on Jun 19, 2026
    Author

    Thanks @tammojan - just discovered that the hard way tinkering 😭 Thanks for confirming, though!

    For reference I had:

    ❯ CENV=/Users/alec.thomson/miniforge3/envs/casa_py311
echo "=== casacore install source ===" && uv pip list --python $CENV/bin/python 2>/dev/null | grep -iE "casacore|casatools|casatasks|casadata"
echo "=== tools THEN core ===" && $CENV/bin/python -c "import casatools; from casacore.tables import table; print('OK')" 2>&1 | tail -2; echo "rc=${PIPESTATUS}"
echo "=== core THEN tools ===" && $CENV/bin/python -c "from casacore.tables import table; import casatools; from casatools import table as t; print('OK')" 2>&1 | tail -3; echo "rc=${PIPESTATUS}"
=== casacore install source ===
casadata                2025.9.22
casatasks               6.7.0.31
casatools               6.7.0.31
python-casacore         3.7.1
=== tools THEN core ===
OK
rc=
=== core THEN tools ===
  File "/Users/alec.thomson/miniforge3/envs/casa_py311/lib/python3.11/site-packages/casacore/tables/table.py", line 39, in <module>
    from ._tables import (Table,
ImportError: dlopen(/Users/alec.thomson/miniforge3/envs/casa_py311/lib/python3.11/site-packages/casacore/tables/_tables.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '__ZN8casacore10TableProxy10selectRowsERKNS_6VectorIxEERKNS_6StringE'
rc=
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions