Add ability to append build number to version identifier at build time

I have searched the issues of this repo and believe that this is not a duplicate.
I have searched the documentation and believe that my question is not covered.

Feature Request

Currently, poetry assumes / forces you to define your product version number explicitly in the pyproject.toml file. This is fine for building releases but doesn't work well when building pre-releases since such builds typically share the same target version. To avoid name/version clashes between packages, poetry should have some way to append a build number to the final package version at build time.

When using setuptools this is typically handled by appending a suffix such as dev or alpha followed by some sort of unique identifier such as a timestamp or build number. I'd like to see something similar in poetry.

What I thought might work best is to add an optional suffix parameter to the poetry build command which, when provided, would be appended to the version identifier stored in the the toml file. So if the static version in the toml file were, say, "1.0.0" and you ran poetry build --version_sufix = ".dev$BUILD_ID" or something similar, then the generated package would be versioned as "1.0.0.dev1234" (where "1234" is the current build number as defined by the environment variable BUILD_ID).

NOTE: this feature enhancement is one of but a few fundamental problems that are currently blocking my team from adopting poetry as a full package replacement tool for all of our projects. In the absence of this feature - or something similar - we can not facilitate cross-developer builds where each developer is working on the same project release but on different branches. Each build needs to be uniquely labelled / addressable otherwise the packages will clash with one another when published to the package repository.

All this to say, this is a relatively high priority improvement for us.

I probably also should mention that I am willing to put some effort into implementing this feature and submitting a pull request if the project maintainers don't have enough time to implement this in the near future. I'd just like to make sure the feature makes sense to the community as a whole before putting in the effort.

We have to work entirely around development wheels right now, so this would be useful.

I'm curious about the choice of arbitrary suffix string, or if you wanted to default to a suffix with compatibility with the rest of the world: The .devN suffix agrees with pep440 and pip. What about poetry itself? Perhaps this part of the source is relevant.

I'd think this is the most wanted use-case for #693. Once this has landed it should not be long until setuptools-scm/versioneer-style plugins arrive. There have been tons of requests in that direction #5, #185, #672, #1034 and probably a lot more people waiting for this on the side-lines.

Ideally having an option to pass the version to the build command. We maintain dynamic versions that are centrally stored. I'd want to pass that version along to the build and have it embedded in the resulting sdist and wheels.

It's technically possible to do this by running poetry version $VER before poetry build, but you'd need a script that can pass the right version to the command. I actually wrote a pseudo-plugin for Poetry for this specific use case, if you'd like to try it. The implementation is a hack until #693 is available :p

For anyone interested in using poetry with some basic automatic versioning, check out this past PR that was proposed, but declined in favor of a future plugin-based component.

Looking for something quite similar to be able to update the version dynamically at build, but without it being reflected inpyproject.toml.

@TheFriendlyCoder

NOTE: this feature enhancement is one of but a few fundamental problems that are currently blocking my team from adopting poetry as a full package replacement tool for all of our projects. In the absence of this feature - or something similar - we can not facilitate cross-developer builds where each developer is working on the same project release but on different branches. Each build needs to be uniquely labelled / addressable otherwise the packages will clash with one another when published to the package repository.

All this to say, this is a relatively high priority improvement for us.

Hi @TheFriendlyCoder, I had the same feature request and came across your post. I understand why this stops you from adopting poetry. I wonder if you can share your current solution to this problem?

Thanks!

I wonder if you can share your current solution to this problem?

Personally, I use setuptools-scm, which generates a version at build-time based on git metadata.

This might need the "Plugins" label.

@WhyNotHugo

@WhyNotHugo

Out of the dozens of projects which I maintain (and the several other dozens to which I contribute), none of them have a "Product Owner" or a dedicated QA team.

This just means that you are the Product Owner in that you define what will be released and what will be postponed for later.

In my reply I was careful to choose QA and testing as separate steps, meaning that in many pipelines testing will be automated and people may not always be involved. Same logic applies nevertheless.

I get it, you work in some corporate environment

I maintain a dozen of Open Source projects, just not in this account. Some for many years. I used to follow some of what you describe, until I realized that if I'm not releasing the very package that came out of testing - automated or QA'ed, I'm not doing a good job.

What are you trying to prove here?

Nothing. Just sharing what I learned not to be the best way to release packages for those who is interested. You clearly are not, which is fine. I will leave this particular sub-thread alone.

@WhyNotHugo

Once defined by the PO, version gets into the source, like pyproject.toml and becomes a pending version that matures towards the release and build numbers reflect the maturity of each artifact package, as it goes through tests. Finally, when tests are successful, you release the tested package, without changing its name, and only then you tag what was released.

High-velocity, trunk-developed projects may work very differently. They may not even assign versions.

I for one am in "tag the commit, let CI do the rest" camp like @WhyNotHugo. The tag build is reproducible enough, I don't need versions in manifests. Tools like Maven or poetry get in my way with such a workflow, and I mostly just ignore the manifest version, similar to what @mikekuzak writes. Others may choose to have their pipelines edit the manifests to set the version, which has always seemed way too complicated for little gain to me.

(Never mind that PEP440 versioning has apparently been cooked up in hell; I guess semver wasn't good enough for Python? 👀 )

@reitzig

@reitzig

High-velocity, trunk-developed projects may work very differently. They may not even assign versions.

There's always a planned version, even for train releases and the like, where you bump the version at the end of the window automatically. This is not the same as bumping version for a build, which is what many are doing.

I for one am in "tag the commit, let CI do the rest"

One builds packages towards a release, where each package gets closer and more mature to fit the release criteria. Tagging beforehand automatically means you tag it before any testing was done.

I guess semver wasn't good enough for Python?

Because Python has many features not covered by semantic versioning, such as the local version used by package maintainers to fix up upstream bugs or ABI compatibility, and a few others. Many other package managers use similar techniques, like rpm, deb, apk, etc.

Besides, semantic versioning does define build metadata, which is supposed to be used for tracking build maturity. Using version as a substitute for build numbers isn't something SemVer promotes.

It all comes down to whether you can release the exact package that went through all of your tests for a given version. If you can, then whatever approach you use is working. If you test and then need additional steps to produce a package, like building from a tag, it's not, really.

@gh-andre

@gh-andre

A version starts with the Product Owner defining a release schedule. It could be a date or recurring releases, like train releases. Version should never be defined by a build process. Once defined by the PO, version gets into the source, like pyproject.toml and becomes a pending version that matures towards the release and build numbers reflect the maturity of each artifact package, as it goes through tests. Finally, when tests are successful, you release the tested package, without changing its name, and only then you tag what was released.

This is an incredibly narrow view of how a release cycle "should" work, and doesn't fit the workflow of the majority of companies I've worked for. There's a lot of things wrong with the workflow you've described, but as others have pointed out: It's personal preference.

Where you are absolutely wrong is to claim that build numbers "never" have a place in version numbers. That's just not true. You might not have seen cases where build numbers are needed in version numbers, but examples exist.

One example is for some heavily legislated contexts (medicine, aviation, gambling, ...) and certain legal jurisdictions, there are laws in place requiring the thing that is released to be byte for byte identical as the thing that was tested and approved independently. Re-running a build rarely results in the same byte sequence if only because the build time often get's rolled into the bytes of the file.

Would you get on a plane if they told you the firmware was not byte for byte the same as the firmware which was tested?

That is, in order to be legally compliant, some companies need to be sure that the build they release is the same build they tested, even if they re-ran the build several times for the same version of code in git.

It all comes down to whether you can release the exact package that went through all of your tests for a given version.

Ironically enough, that is the exact reason you have your build tag in the version number. This way, when a user tells you which version they are on, you can be 100% sure which build they are on, not just which version of source code.

@couling

@couling You're clearly misunderstanding pretty much all of what I described. By a mile, I might add. It's the other way around - deployments that need to know they are running the exact build that was tested when the package was released, identified by the build number and without repackaging, will benefit the most from this.

Sounds like you never debugged crash dumps or linked the source to the binaries in any way. A build number is routinely tacked into the version for a released package or binaries - take any packages with debug symbols of any kind, for an example, just not in the way you think it is. Build numbers/metadata don't come from the source, but from the build system.

@gh-andre

Sounds like you never debugged crash dumps or linked the source to the binaries in any way.

@gh-andre That's extremely rude, it's slanderous and frankly shows you are not interested in listening to others' experience.

What are you trying to prove here?

Nothing. Just sharing what I learned not to be the best way to release packages for those who is interested. You clearly are not, which is fine

You seem to be just trolling here and worse ...

The version sequence isn't supposed to contain build identifiers, which are supposed to be added in the build process and go into its own spot in the package name.

This allows one to build a package, test it and release that same package without repackaging it to remove build number from the version in the package name.

Package names are embedded in Python packages (described here) and any mismatch between package names makes them unrelated packages, so adding the build identifier to the package name would absolutely force you to repackage prior to release, not prevent it as you claim. What you described there couldn't possibly work.

@couling

@couling

All I'm saying is that one should release the very package they tested, without repackaging or rebuilding it from a tag. Not sure why this causes this reaction from a few participants in this discussion.

The rest is just examples, where tracking build numbers against released packages is very important for things like aligning it with debug symbols, etc, which cannot be rebuilt to match the binaries.

That's extremely rude

You called my point of view "incredibly narrow", so against that my observation that you probably didn't match source against crash dumps is just that - an observation. I apologize if this came out as offensive - it was not my intent.

Could we please stick to technical issues, and not attack anyone else's intelligence please?

Different people and organisations do different things, lets leave it at that.

I've had some success using https://pypi.org/project/poetry-dynamic-versioning/ to override versions at build time, I expect that could be used to inject a build number.

@gh-andre

Apologies that was unusually heated of me.

@gh-andre I think you've mixed up between two very different concepts and consequently said several things which are technically untrue. You seem to have confused between:

PEP 440 pre-release suffixes which do require you to rebuild an artefact prior to actual release
Build injected version numbers

There's absolutely no connection between these two things, and this this feature request has nothing at all to do with PEP 440 numbering. There is absolutely no reason why a build injected version number cannot go out as a the final release number as long as it doesn't contain PEP 440 pre-release or dev-release suffixes.

Build injected version numbers can typically be supplied from three sources:

git tags or other SCM features
inputs to build pipelines.

Eg: in github actions drafting a "release" can trigger the build pipeline passing the release number as a variable. While the github release is in draft, the artefact can be tested prior to publishing.
build time augmentation a version number supplied from somewhere else.

Eg adding a build number to make 1.9 into 1.9.56712. Again, it is safe to release the version number 1.9.56712 if that's what your workflow dictates.

@clintonroy I've been using poetry-dynamic-versioning and it's okay for now. Unfortunately it still has some glitches and I frequently find a poetry update will modify pyproject.toml to disable it.

At this stage I'd much prefer a slicker way to embed the version number in via the CI pipeline, rather than the git tag, so to my mind the feature request is still quite useful.

FWIW, my current approach is something like this (Jenkins-style Groovy):

def isReleaseBuild = "${env.TAG_NAME}" ==~ /^v\d+\.\d+\.\d+((a|b|rc)\d+)?$/ // cf. PEP 440

// NB: poetry normalizes the version we give it.
//     So, to avoid accidents, we let it do that upfront.
def versionProposal
if (isReleaseBuild) {
    versionProposal = env.TAG_NAME - ~/v?/
} else {
    def poetryVersion = sh(script: 'poetry version -s', returnStdout: true).trim()
    versionProposal = "${poetryVersion}.dev+${env.GIT_COMMIT[0..7]}_${env.BUILD_NUMBER}"
}
toolVersion = sh(
    script: "poetry version -s '${versionProposal}'",
    returnStdout: true
).trim()

This gives me flag isReleaseBuild and a version for this build (I don't care about re-running release builds), so later in the build I can do:

sh  label: 'publish',
      script: """
          poetry version -s '${toolVersion}'
          poetry publish --build --repository '${POETRY_REPOSITORY_NAME}'
      """

This changes pyproject.toml in the build agent, but I never commit it. There, it says:

version = "0.0.0"

While exactly using the build tag semantics specified by PEP 440, you can now use the local version label at build time see #9064.

The proper fix here really is the ability to add "build tag" component in the wheel name as well as the wheel metadata. Note that iirc, this is not supported for sdists.

Modifying dev/pre/post/rc at build time is not something that we should really support.

According to PEP 440 the local version identifier was added to allow for build tags into the versioning of binary distributions.

Added the “local version identifier” and “local version label” concepts to allow system integrators to indicate patched builds in a way that is supported by the upstream tools, as well as to allow the incorporation of build tags into the versioning of binary distributions.

And according to PyPA Binary distribution format documentation

Build numbers are not a part of the distribution version and thus are difficult to reference externally, especially so outside the Python ecosystem of tools and standards. A common case where a distribution would need to referenced externally is when resolving a security vulnerability.

Due to this limitation, new distributions which need to be referenced externally should not use build numbers when building the new distribution. Instead a new distribution version should be created for such cases.

Considering all of the above, a local version tag is the right approach for this as opposed to a build tag.

This means #9064 resolves this issue and we will not support the use of build tags as it no longer recommended.

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

Navigation Menu

Repository navigation

Add ability to append build number to version identifier at build time #1208

Feature Request

16 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add ability to append build number to version identifier at build time #1208

Description

Feature Request

Activity

TheFriendlyCoder commented on Jul 2, 2019

TheFriendlyCoder commented on Jul 2, 2019

areeh commented on Jul 3, 2019

asodeur commented on Jul 4, 2019

dsully commented on Aug 2, 2019

mtkennerly commented on Oct 27, 2019

HitLuca commented on Mar 31, 2020

SailingYYC commented on Apr 11, 2020

Yajing-Zhao commented on Nov 2, 2020

WhyNotHugo commented on Nov 3, 2020

sinoroc commented on Nov 3, 2020

16 remaining items

gh-andre commented on Jun 20, 2023

reitzig commented on Jun 20, 2023

gh-andre commented on Jun 21, 2023

couling commented on Jan 5, 2024

gh-andre commented on Jan 6, 2024

couling commented on Jan 8, 2024

gh-andre commented on Jan 8, 2024

clintonroy commented on Jan 8, 2024

couling commented on Jan 9, 2024

reitzig commented on Jan 9, 2024

abn commented on Feb 29, 2024

abn commented on Mar 23, 2024

github-actions commented on Apr 24, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions