Add git-based detection of tags at HEAD to improve PublicRelease detection

[georg-jung]

Fixes #873.

This implements my second idea described in detail in #873.

My thoughts form there:

• It would just work without any configuration needed (one might argue it's a breaking change though; could be opt-in to mitigate, wouldn't just work then).
• It hits the notch that git is the only source of truth.
• With more complicated pipelines and changing environments (e.g. the nbgv step vs. inside a docker build container), it is still straight forward (if .git is available, it works) and we don't need to worry about is this env var for the override available inside the container that performs the build?
• It feels natural, because it does what I specified in version.json and what I specified there does work in the typical case, where GITHUB_REF is not stale.
• It makes local builds where the tag is checked out equivalent to cloud builds initiated by tag push. Currently "^refs/tags/v\\d+\\.\\d+" doesn't have any effect locally if I get it right - it does in CI environments though.

This comes with parsing support for annotated tags for managed git and LibGit2. Lightweight tags pointing HEAD are considered as well as annotated tags. Nested annotated tags that indirectly point at HEAD are not considered (intentionally, changing probably wouldn't be hard). BuildingRef value is not changed.

Currently, the output of nbgv get-version -f json is extended as follows:

  "VersionHeight": 3,
  "VersionHeightOffset": 0,
  "BuildingRef": "2e6084bfeb07d7f8e426b52aa51bd7517faf5610",
  "BuildingTags": [
    "refs/tags/possiblyASecondTag",
    "refs/tags/v0.7.3",
  ],
  "Version": "0.7.3.11872",
  "CloudBuildAllVarsEnabled": false,
  "CloudBuildAllVars": {
    // ...
    "NBGV_BuildingRef": "2e6084bfeb07d7f8e426b52aa51bd7517faf5610",
    "NBGV_BuildingTags": "System.Collections.Generic.List`1[System.String]",
    "NBGV_Version": "0.7.3.11872",
    // ...
  },
  "CloudBuildVersionVarsEnabled": true,

Obviously the NBGV_BuildingTags value isn't here to stay. I thought I'd leave this for discussion, which of these values should be generated and what their value should be. I added an [Ignore] to skip it in a3089b1.

Probably I made some opinionated decisions when putting this together, don't hesitate to change what you don't like.

[dnfadmin]

All CLA requirements met.

[georg-jung]

In 3c93a89 I added a workaround for GitPack.TryGetObject which kind of violates its implicit contract (it throws if it can not get an object of the requested type). I wasn't sure about the best way to fix this. Also this is kind of unrelated to the rest of this PR.

[georg-jung]

@AArnott anything I could do from my side to assist/simplify getting this merged?

[AArnott]

I'm looking at this now and preparing some local changes to push to your PR.

[AArnott]

Thanks for putting this together. I have several comments I look forward to discussing with you.

[AArnott]

This seems to not handle a line pattern that I see in one of my repos:

^3d2137a79ee4e3621696cfbde8d4f1c0e98bc5f6
40614e83b9b3892e05efd6155c3b61035bb5542a refs/tags/drop/a.proddiag/official.00000.00
^2f0a22666a958c15f7e7486a0a14490f7b8e210f
9aa2dc36733d3f3076702b31a3dc754c144a9473 refs/tags/drop/a.proddiag/official.26601.00

See the leading caret? I don't know what that means, but it seems your line parsing assumes no such prefix exists.

[KalleOlaviNiemitalo]

The ^ prefix indicates a "peeled line" that lists the object ID referenced by the tag object on the previous line.

[georg-jung]

git's source supports that: https://github.com/git/git/blob/d9d677b2d8cc5f70499db04e633ba7a400f64cbf/refs/packed-backend.c#L542

A "record" here is one line for the reference itself and zero or one peel lines that start with '^'.

Thus, if I understand correctly, simply skipping these lines would be functionally equivalent to if they didn't exist, because we'd peel the refs in the next step. Having them does provide us with a way to speed things up though and hits the notch @AArnott mentioned here - we could decide if we're interested in this ref in place. Right?

[KalleOlaviNiemitalo]

I tried a tag referencing another tag referencing a commit, and the peeled line after the first tag listed the object ID of the second tag, not the commit ID. So if you implement some optimisation that uses those lines, please consider this scenario.

[georg-jung]

Good to know, thanks! Actually I think this behaviour makes sense and is also in line with what the code in this PR currently does. My take on "tags at HEAD" behaves like git tag --points-at HEAD would. Means it includes any lightweight as well as annotated tags pointing at HEAD. But not any tags that point at HEAD transitively. A git show HEAD on the other hand does include transitive tags. Not including transitive tags might lead to less lookups => increased performance. In a broken repo, lookup including transitive tags could lead to an infinite loop.

I guess transitive tags are kind of an edge case so my plan was to just not support them at this point. I think specifying "We do what git tag --points-at HEAD does" makes sense too. If the need to support transitive tags would arise, a second PR would probably be easier than this first one. What do you think @AArnott?

[KalleOlaviNiemitalo]

My main concern is that the behavior should not depend on whether the peeled lines are in packed-refs or not. Whether nbgv recursively follows tags in general is a matter of policy and either choice is fine with me.

[georg-jung]

Makes sense. You might want to take a look at my two recent commits in that regard.

[AArnott]

Filtering the tags at this point is quite late. NB.GV functions in some repos with tens of thousands of tags, and the code would have parsed and allocated multiple strings for all of them. I think we'll need to optimize this by walking the tags and immediately skipping lines with non-matching object IDs before we allocate anything.

[georg-jung]

I agree that this can be solved without the candidate list and that the number of string allocations can be reduced. The current implementation without tag support does however allocate multiple strings per packed-ref line too and it doesn't seem to be a performance hit or is there some detail I'm missing?

Looking up the target of an annotated tag might lead to an IO operation - read the relevant git pack. Thus, I'm wondering if collecting the candidates first wouldn't have a minor impact compared to opening the file.

I don't have any that large repo at hand. If this turns out to have a substantial negative performance impact, it might be an option to make annotated tag support opt-in/out, as reading the packed-refs file itself (plus the files in refs/tags) is sufficient for lightweight tag lookup. Edit: with the two commits incl. 49b9dc9 we don't need additional IO operations for annotated tags in packed-refs if peel lines are available.

I'm working on a commit to remove the candidate list and reduce the string allocations though. Edit: see a84ad8b

[KalleOlaviNiemitalo]

I thought there should be something to detect if reftable is used, and fall back to LibGit2Context in that case, but…

• Nerdbank.GitVersioning.ManagedGit.GitRepository already reads the packed-refs file before this PR, so it is not a new problem.
• libgit2 does not support reftable, either. reftable support for libgit2 libgit2/libgit2#5352

[AArnott]

The build failure reproduces in main, so it's unlikely to be due to this PR. I don't know how that happened, but I'll be investigating it.