Packaging rules should support --stamp flag

[nacl]

#285 requested a feature to preserve mtime's in tar files. @aiuto brought up the below WRT how this might be assembled:

Originally posted by @aiuto in #285 (comment)

Let's discuss solutions in an issue first. I would like to to have as many people as possible weigh in on what is useful.
There are a lot of questions

• do we change the stamp of every file?
• does that include files which came from pre-made archives that have a particular timestamp of their own?
• some people want to put the timestamp into the output filename. While I find that dubious, any stamping solution should consider it.
• Why is stamp_mtime an attribute? Why not just support the --stamp flag. That seems to be the API of least surprise.
• and.... no solutions that just to .tar. We need do do the appropriate thing for each of pkg_zip, pkg_deb and pkg_rpm. But that makes getting the API right before doing any code even more important.

This issue tracks the design of a "stamping" mechanism for packages prior to implementation, especially with regards to fixing/clamping file modifications. Whether the --stamp argument should control this is also a question to consider.

[nacl]

AFAIK, my particular use case does not have a preference WRT preserving mtimes, but some things that immediately come to mind are:

• Some rules support SOURCE_DATE_EPOCH (e.g. pkg_rpm), others do not. It also applies to more than mtimes, so that may be out of scope here.

• --stamp feels like the tool of choice here, but BUILD_TIMESTAMP may also not be the right value to use. Bazel provides it for free.

[aiuto]

The least surprising behavior is that all the top level pkg rules implement stamp as done in cc_binary. I consider that a minimal requirement.

stamp

• Integer; optional; default is -1
• Enable link stamping. Whether to encode build information into the binary. Possible values:
  • stamp = 1: Stamp the build information into the binary. Stamped binaries are only rebuilt when their dependencies change. Use this if there are tests that depend on the build information.
  • stamp = 0: Always replace build information by constant values. This gives good build result caching.
  • stamp = -1: Embedding of build information is controlled by the --[no]stamp flag.

#288 demonstrates how the default behavior could be done. That is, I can get the build time for stamping if --stamp is set on the command line. I didn't do the 0 and 1 cases, nor implement for every rule.

An orthogonal feature to that is if we change the time stamps of files expanded out of archives (if I am reading #265 right). That is it's own debate. The request I have seen in the past is along the lines of

• I have a bundle of static files that will be served by a web service
• they have a set of creation dates.
• I bundle them into a zip file and check into my source tree
• I want to include them in a .deb package, but individually not as one zip
• I want the original time to be preserved.

I think we need provision for that kind of behavior. I don't have a design in mind right now. I don't think it is a blocker for --stamp support, as long as we design with the idea in mind that individual files might need to say they are immune to stamping
@nacl is rpm's use of SOURCE_DATE_EPOCH similar to this?

Then there is the question of using the time stamp in the archive name This is difficult for many reasons and we should not block basic stamping on that.

[dennis3620]

Thanks for following this up and your comments.

My main motivation is to set a more dynamically mtime for the input files. At the moment either the default behavior with portable or a static mtime can be used, but I'm looking for a "self-adjusting" mtime based on the input files. The idea was to use a new volatile status variable which is for example set by a bash script based on the input files' mtime or git command to get a less changing variable than the BUILD_TIMESTAMP.

stamp = 1: Stamp the build information into the binary. Stamped binaries are only rebuilt when their dependencies change. Use this if there are tests that depend on the build information.

If I understand this correctly with "stamp = 1" the rebuild is only triggered if the input of the specific rule change, i.e. the BUILD_TIMESTAMP is relatively close to the recent mtime of the input files. The only drawbacks are fresh builds, e.g. if no cache is available. #285 had an other intention, but if the stamp behavior is closer to the build-in rules and less complex
I guess this would even fit my needs somehow.

I will have a closer look into #288.

[aiuto]

Maybe we should start from needs for stamping, and then work back to features to achieve that.

1. We certainly need the default to be a repeatable constant value. This gives us easily testabl repeatability. That should be the default behavior for all rules.
2. On the day we want to cut a release, we probably want the time of build to be used in the binary. Here I think the default Bazel stamp behavior works well. Builds use a constant mtime by default, so during development and testing you don't overbuild. When you use --stamp, you want the package to use time of build
3. Some targets are "extra-volatile". This is the stamp=1 case. Maybe we have a process where we want to do something like bazel run :push, where push depends on a .deb pacakge we built. Should multiple runs in a row rebuild the .deb each time, even if no other sources changed? I think the current Bazel idea is right here too. Do not rebuild the binary package, but do run the action which depends on it.
4. Is there a need for users to specify a build time which is neither the epoch constant or the current time? That is the case I think you are trying to provide for.

Let's say there is a user for case 4. A hypothetical is that I am cutting a product release on Feb. 4, and I want the stamps to be 2020-02-04 33:22:11. Maybe I used the date as is, but encode something in the hh:mm:ss. We obviously don't want to have to set that in each rule, so the value has to come from the command line in some way. We would either pass the value (which requires a blaze change) or use workspace status commant (which does not).

Question 1: Can workspace status just replace BUILD_TIMESTAMP? If so, then that seems to be the way to do this.
Question 2: Is there a need for workspace status to emit 2 or more distinct time values and have different rules pick which of those they want? I strongly doubt that.
Question 3: How ofetn do we need anything beyond stamping with the current build time?

[aiuto]

Status update: #288 is ready to go

• it implements the --stamp flag and stamp attribute behavior of Bazel's *_binary rules
• It gets the time stamp from the BUILD_TIMESTAMP from the volatile file.

If we need to stamp with arbitrary times other than build time, the technique could

• be expanded to allow a timestamp to come from the command line.
• be expanded to allow a timestamp to come any user defined file.

[tkinz27]

So most of the comments here are about using stamping to determine mtimes for the files within a package. What about support though to use values from the workspace status command as part of variable inputs to the packaging rules?

I'm mostly looking for a simple way to generate this version string for our debian package

git tag "v0.0.$(git show -s --format=%ct HEAD)-$(git rev-parse --short HEAD)"

Pretty easy to output this as part of the workspace status command. The challenge I'm having is getting that into a place the pkg_deb rule can use it. I'm fairly new to bazel but it seems like each rule has to implement support for reading the ctx.version_file and ctx.info_file directly.

I am currently using a custom rule to generate a version file and use that. While the version correctly gets injected into the control file of the debian package, it is not used as part of the package name itself. I'm not sure if this is just a bug or a specific choice.