iptb start --wait fails with dockeripfs node

[dgrisham]

When I spin up a node with iptb auto --type dockeripfs --count 1, starting the node with the --wait flag times out:

$ iptb start --wait
node QmbNGJ8hfSaepK96iZAycm3di6so9EU6syPEJKnqgVmxgn failed to come online in given time period
node[0]
gx/ipfs/QmckeQ2zrYLAXoSHYTGn5BDdb22BqbUoHEHm8KZ9YWRxd1/iptb/commands.mapWithOutput.func1
        /home/grish/.go/src/gx/ipfs/QmckeQ2zrYLAXoSHYTGn5BDdb22BqbUoHEHm8KZ9YWRxd1/iptb/commands/utils.go:195
runtime.goexit
        /usr/lib/go/src/runtime/asm_amd64.s:1337

However, the node is up when I run iptb shell 0 or check the container with docker container ls. I tried manually setting the IP address for the node from its default of 0.0.0.0 to the IP given by docker inspect <container> in the tryAPICheck() function, which fixed the issue. I glanced back at older commits of IPTB where iptb start --wait worked, but the code looks similar to the current implementation (it is late though, and I wasn't very thorough).

Any ideas what might be happening? @travisperson, @bigs?

[travisperson]

If you can attempt this again, can you tell me what the value inside of the api file is under the nodes repo dir? (Should be in $HOME/testbed/default/#/api).

Also what docker container are you using?

I'll try to repo this myself as well.

[dgrisham]

The value in the api file is /ip4/0.0.0.0/tcp/5001, and I'm getting this error with my own custom container but also when I switch back to the default ipfs/go-ipfs.

[dgrisham]

Update: I was able to reproduce this on a fresh AWS machine (with similar tools/setup to my local computer, though). I don't see the api file this time though, and I'm not actually sure where this file is created in the iptb or iptb-plugins code @travisperson?

[travisperson]

It's created by IPFS itself when the daemon starts. I think at some point we changed the API binding address for the docker plugin to the wildcard 0.0.0.0 and I don't think this is strictly needed. The way the wait check is implement works well for the local plugin but I think for the docker one we should try something different by using RunCmd it self. I believe there are certain commands which only work if the daemon is running, which we would need as simple things like id and verson can run offline. We might be able to use version with a few flags, as I believe it explicitly will mention the daemon if it connects.

[dgrisham]

I just tried ipfs version and it doesn't seem to output anything different between a running vs. inactive daemon, but ipfs diag cmds will print an error if the daemon is running (same with ipfs stats bitswap and ipfs stats bw). Not sure if there's a better way than checking whether one of those commands is successful/outputs an error or not.