Repository navigation
Validating distances against reference implementations #245
Description
For each distance we should check that either (i) netrd is the only public implementation of the distance, or (ii) that netrd's implementation of the distance produces similar outputs given the same inputs. We've done this for a bunch of them already, typically when originally implementing the distance, but we should make this process more explicit so that we don't accidentally overlook one.
- Communicability JSD
- Degree Divergence
- Deltacon
- Distributional NBD
- DK-series distance
- Frobenius
- Hamming
- Hamming-Ipsen-Mikhailov
- Ipsen-Mikhailov
- Jaccard
- Laplacian Spectral
- NBD
- NetLSD
- NetSimile
- Onion Divergence
- Polynomial Dissimilarity
- Portrait Divergence
- Quantum JSD
- Resistance Perturbation
I'll start by checking off the ones I think are novel; I'm reasonably certain that there are more we know are validated.
Activity
sdmccabe commentedon Aug 23, 2019
There are differences in output between our distance and the reference implementation of Portrait Divergence, but the differences are consistently small (the largest I've seen is 0.005, and it's usually more like 0.001). I'll keep investigating but I'd guess it's nothing.
sdmccabe commentedon Aug 23, 2019
We should bump the PyPI version after finishing this.
sdmccabe commentedon Aug 23, 2019
@leotrs I've checked off NBD because I assume the implementations are the same.
sdmccabe commentedon Aug 25, 2019
HIM is producing different outputs from the R
NetworkDistanceimplementation for RGGs (N=200, p=0.26, using the edgelists from the graphwend repo); will need to investigate further.leotrs commentedon Aug 26, 2019
At this point I wouldn't be surprised if
netrd's implementation is more updated than mine. However, you can forget about NBD as I am the maintainer of the other one. If the outputs from the two different repos are different, then probablynetrd's are correct...leotrs commentedon Aug 26, 2019
For NetSimile, I found this and this. Haven't compared them yet tho.
sdmccabe commentedon Aug 26, 2019
NetSimile is a frustrating one since there isn't a reference implementation in the sense of author's code, so we're assuming the other independent implementations are correct. When I was debugging some NetSimile issues back in the spring I remember comparing the outputs to those from the
netcomplibrary; I don't know if anything has changed since but I believe they were producing similar or identical outputs.leotrs commentedon Aug 26, 2019
We could use it as a touchstone only then. As long as we're in their ballpark, we're good.
Use proper normalization in Hamming (#250)
leotrs commentedon Oct 9, 2019
Frobenius and Jaccard depend on row ordering, yes?
Unrelatedly, they both seem to be simple enough that we can just check them off?
sdmccabe commentedon Oct 9, 2019
They should depend on row ordering, @jkbren would be able to confirm from his experiments.
They're probably simple enough to check off, but simplicity can be deceiving; see the issues we had with Jaccard before in #180.