Repository navigation
IpcMode is not handled correctly #310
Description
When using a deployment manifest that specifies an IpcMode in hostconfig that accesses shared memory in another container, iotedgehubdev start fails due to a nonexistent container error.
Relevant version numbers:
iotedgehubdev, version 0.14.3
docker-compose version 1.28.5, build c4eb3a1f
docker-py version: 4.4.4
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
with docker-compose versions prior to 1.27.0 a different error message is thrown.
Sample deployment manifest
{
"modulesContent": {
"$edgeAgent": {
"properties.desired": {
"schemaVersion": "1.0",
"runtime": {
"type": "docker",
"settings": {
"minDockerVersion": "v1.25",
"loggingOptions": "",
"registryCredentials": {}
}
},
"systemModules": {
"edgeAgent": {
"type": "docker",
"settings": {
"image": "mcr.microsoft.com/azureiotedge-agent:1.0",
"createOptions": "{}"
}
},
"edgeHub": {
"type": "docker",
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "mcr.microsoft.com/azureiotedge-hub:1.0",
"createOptions": "{\"HostConfig\":{\"PortBindings\":{\"5671/tcp\":[{\"HostPort\":\"5671\"}],\"8883/tcp\":[{\"HostPort\":\"8883\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}}"
}
}
},
"modules": {
"lvaEdge": {
"version": "1.0",
"type": "docker",
"status": "running",
"restartPolicy": "always",
"startupOrder": 1,
"settings": {
"image": "mcr.microsoft.com/media/live-video-analytics:2",
"createOptions": "{\"HostConfig\":{\"LogConfig\":{\"Type\":\"\",\"Config\":{\"max-size\":\"10m\",\"max-file\":\"10\"}},\"Binds\":[\"/tmp/:/var/media/\"],\"IpcMode\":\"shareable\"}}"
}
},
"rtspsim": {
"version": "1.0",
"type": "docker",
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "mcr.microsoft.com/lva-utilities/rtspsim-live555:1.2",
"createOptions": "{\"HostConfig\":{\"IpcMode\":\"container:lvaEdge\"}}"
}
}
}
}
},
"$edgeHub": {
"properties.desired": {
"schemaVersion": "1.0",
"routes": {
"LVAToHub": "FROM /messages/modules/lvaEdge/outputs/* INTO $upstream"
},
"storeAndForwardConfiguration": {
"timeToLiveSecs": 7200
}
}
}
}
}
Note the IpcMode set in lvaEdge and rtpsim. The actual containers used do not matter; any containers that use that use these ipcmode settings will trigger the issue.
Steps to reproduce
iotedgehubdev start -d config/minimal.json
Network azure-iot-edge-dev is external, skipping
ERROR: Service 'rtspsim' uses the IPC namespace of container 'lvaEdge' which does not exist.
ERROR: Error while executing command: docker-compose -f /var/lib/iotedgehubdev/data/docker-compose.yml pull edgeHubDev. Command '['docker-compose', '-f', '/var/lib/iotedgehubdev/data/docker-compose.yml', 'pull', 'edgeHubDev']' returned non-zero exit status 1.
Possible solution
The IoT Edge deployment manifest module createOptions uses docker create options. The docker create container options allow the following values:
"none": own private IPC namespace, with /dev/shm not mounted
"private": own private IPC namespace
"shareable": own private IPC namespace, with a possibility to share it with other containers
"container:<name|id>": join another (shareable) container's IPC namespace
"host": use the host system's IPC namespace
However iotedgehubdev uses docker-compose to start the deployment. docker-compose requires the use of the setting ipc: "service:[service name]" when referring to another service in the same compose file. If the IpcMode value of rtpsim in the manifest above is changed from container:lvaEdge to service:lvaEdge iotedgehubdev is able to start up the solution as expected. This however will not work when deploying to a real edge device, as the service: syntax is not supported by docker create ("Error": "Invalid IPC mode: service:lvaEdge").
network _mode may also suffer from the same issue.
Activity
marianan commentedon Mar 8, 2021
Ossi Ottka (@oottka) thanks for reporting this issue. Could you clarify why is ipc needed instead of defining routes between modules?
oottka commentedon Mar 9, 2021
In this case we are using Azure LVA on IoT Edge, and have an inferencing module that uses the gRPC interface and shared memory to share frames (using the second option, "When communicating with a gRPC server running in another IoT Edge module"). This is the recommended approach for high-performance inferencing in LVA.
The edge deployment manifest used here is just for demo/bug reporting purposes. It does not contain a real inferencing module, but still can be used to reproduce and validate the issue.
konichi3 commentedon Mar 15, 2021
Ossi Ottka (@oottka)
Thank you for reporting the issue. The issue is now in our backlog. We will circle back when we have an update.
If this is blocking you, please let us know the business impact and justification.
oottka commentedon Mar 16, 2021
Thanks! There is a workaround available for this issue in the meantime: it is possible to use an external templating tool to change
container:toservice:when using iotedgehubdev.