Cannot uninstall PyYAML. It is a distutils installed project

[erik78se]

I'm trying to deploy a charm I've built some time ago which now fails to deploy: https://github.com/erik78se/layer-nextcloud

It seems that the layer-basic pulls in pyYAML which then collides with some system package. I'm not sure about the mechanism, but perhaps there is a way to get around this issue since I'm only including layer-basic in my charm (layer.yaml).

$ juju debug-log 
2020-10-19 17:12:42 DEBUG install   Found existing installation: six 1.15.0
2020-10-19 17:12:42 DEBUG install     Uninstalling six-1.15.0:
2020-10-19 17:12:42 DEBUG install       Successfully uninstalled six-1.15.0
2020-10-19 17:12:42 DEBUG install   Running setup.py install for six: started
2020-10-19 17:12:42 DEBUG install     Running setup.py install for six: finished with status 'done'
2020-10-19 17:12:42 DEBUG install   Found existing installation: netaddr 0.7.19
2020-10-19 17:12:42 DEBUG install     Uninstalling netaddr-0.7.19:
2020-10-19 17:12:42 DEBUG install       Successfully uninstalled netaddr-0.7.19
2020-10-19 17:12:42 DEBUG install   Running setup.py install for netaddr: started
2020-10-19 17:12:43 DEBUG install     Running setup.py install for netaddr: finished with status 'done'
2020-10-19 17:12:43 DEBUG install   Found existing installation: PyYAML 3.12
2020-10-19 17:12:43 DEBUG install Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
2020-10-19 17:12:43 DEBUG install Traceback (most recent call last):
2020-10-19 17:12:43 DEBUG install   File "/var/lib/juju/agents/unit-nextcloud-2/charm/hooks/install", line 8, in <module>
2020-10-19 17:12:43 DEBUG install     basic.bootstrap_charm_deps()
2020-10-19 17:12:43 DEBUG install   File "lib/charms/layer/basic.py", line 208, in bootstrap_charm_deps
2020-10-19 17:12:43 DEBUG install     env=_get_subprocess_env())
2020-10-19 17:12:43 DEBUG install   File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
2020-10-19 17:12:43 DEBUG install     raise CalledProcessError(retcode, cmd)
2020-10-19 17:12:43 DEBUG install subprocess.CalledProcessError: Command '['pip3', 'install', '-U', '--force-reinstall', '--no-index', '--no-cache-dir', '-f', 'wheelhouse', 'wheel', 'pbr', 'Jinja2', 'six', 'netaddr', 'PyYAML', 'MarkupSafe', 'pyaml', 'Tempita', 'charmhelpers', 'charms.reactive']' returned non-zero exit status 1.

[erik78se]

Steps to reproduce:

juju deploy cs:~erik-lonroth/nextcloud-5 --series bionic
juju deploy postgresql
juju relate postgresql:db nextcloud:postgresql

[dshcherb]

Hi Erik,

layer-basic has two ways of dealing with dependencies:

• trying to use them alongside system python packages;
• using a venv.

It also has a mechanism which creates a wheelhouse directory in the directory of a built charm that contains all the python packages that a charm needs (and C sources of binary dependencies for those packages which are built on the target machine so that the CPU architecture matches the target).

Ubuntu ships with python3-yaml so it looks like the issue is the conflict of versions between the python3-yaml shipped with Ubuntu Server and the one coming from the charm:

https://packages.ubuntu.com/search?suite=bionic&arch=any&searchon=names&keywords=python3-yaml

/var/lib/juju/agents/unit-nextcloud-0/charm# dpkg -l | grep python3-yaml
ii  python3-yaml                     3.12-1build2                                amd64        YAML parser and emitter for Python3

ls -al wheelhouse/PyYAML-5.2.tar.gz 
-rw-r--r-- 1 root root 265687 Oct 19 20:16 wheelhouse/PyYAML-5.2.tar.gz

This portion of code that is mentioned in the output is only hit when use_venv: False in layer.yaml:
https://github.com/juju-solutions/layer-basic/blob/623e69c/lib/charms/layer/basic.py#L204-L208

So, technically, adding use_venv: True to layer.yaml of the nextcloud layer should do it.

However, the behavior that I see is strange because the resulting built charm contains the following in layer.yaml:

cat /tmp/charm-builds/nextcloud/layer.yaml 
"includes":
- "layer:basic"
- "interface:http"
"options":
  "apache-php": {}
"is": "layer:apache-php"

I noticed that during the charm build process layer.yaml contains the following before it has the above content:

cat /tmp/charm-builds/nextcloud/layer.yaml 
"includes":
- "layer:options"
- "layer:basic"
- "layer:status"
- "layer:apache-php"
- "interface:http"
- "interface:mysql"
- "interface:pgsql"
- "layer:apt"
"exclude": [".travis.yml", "tests", "tox.ini", "test-requirements.txt", "unit_tests"]
"options":
  "basic":
    "use_venv": !!bool "true"
    "include_system_packages": !!bool "false"
    "packages": []
    "python_packages": []
  "status":
    "patch-hookenv": !!bool "true"
  "apt":
    "packages": []
    "version_package": ""
    "full_version": !!bool "false"
    "keys": []
  "nextcloud": {}
"repo": ""
"is": "nextcloud"

So there must be something wrong with how charm-tools process layers without a layer.yaml (layer https://github.com/juju-solutions/layer-apache-php does not have one).

[dshcherb]

Summoning @johnsca in case it looks familiar ^

[erik78se]

@dshcherb thanx, I tried the mechanism you mentioned to address the issue by adding "use_venv" to the layer.yaml but as you also found.

[dshcherb]

It looks like the composer.yaml file from the apache-php layer interferes with the charm build process:

https://github.com/juju-solutions/layer-apache-php/blob/0b1cb60/composer.yaml
https://github.com/juju/charm-tools/blob/6724b74/charmtools/build/builder.py#L127

Renaming the composer.yaml file tolayer.yaml in the included layer-apache-php before a build seems to fix it.