Relax computed importstr ?

[ant31]

Do you think it will be possible to use importstr with variable?

e.g:

 {
 files: [
        { 
         file: filepath,
         content: importstr filepath
         } for filepath in ['a.txt', 'b.txt', 'c.txt']
  ],
}

some_function(file):: (
    local content = importstr file;
    #process the content
  )

[ant31]

Any technical reason for importstr to not allow import variable ?
I understand for import, but I think imporstr is a little different, it's equivalent to do an file.open(filepath) in other languages, and they mostly all accept a variable.

[sparkprime]

Widening it to a variable is just as "bad" as widening it to an arbitrary expression, so if we were going to do anything it would be that.

The property that we gain by having the restriction is that you can do a simple static analysis and determine which files are dependencies, and then determine whether or not the code needs re-executing. This is potentially useful although I'm not sure anyone has made use of it yet. However if we lift the restriction and lose the property, we won't be getting it back again. So it needs careful thought.

There is one difference between Jsonnet and other programming languages, and that's that everybody writes programs that terminate in a relatively short period of time, so one answer is that if you want to know what files affect the evaluation of a config, you should execute the config. Or, in the case of deciding whether or not a cached JSON value needs recomputing, save not just the JSON output but the list of files used to compute it.

[ant31]

Thank you for the detailed explanation .

we won't be getting it back again

What about an explicit keyword or function to express it?
like: importFrom(expression) , this way users explicitly chose to lose the static analysis property ?

[ant31]

I'm closing it,
I've been able to do it easily via a native extension.

Thank you.

[sparkprime]

I suppose native extensions broke static analysis already :) Same as they do in Java etc.

[nikolay]

@sparkprime The stdlib is so limited and so opinionated that, basically, we're forced to write native extensions, which kind of defeats the purpose of using Jsonnet.

[sbarzowski]

@nikolay
It's not about the stdlib, it's about the language. Hermiticity is a core principle of Jsonnet. The output shouldn't be affected by the environment. Jsonnet code shouldn't jump around the filesystem and read files. Import mechanism is designed for importing Jsonnet libraries etc. and not for arbitrary IO.

This guarantees that it will work the same on every machine and makes it much easier to reason about the code. But of course it restricts the structure of the solution - any input data must be prepared beforehand and passed to Jsonnet.

So it's sort of a "it's not a bug it's a feature" situation. That said, it would always be nice to make passing data to Jsonnet easier (and more obvious).

we're forced to write native extensions

Native extensions are not a proper solution here. It's a way to bypass the restrictions of the language. But there are reasons for these restrictions. Also native functions were supposed to be pure functions - for cases in which Jsonnet is not fast enough or a native library is used. Using them to perform IO is an abuse of the mechanism, IMO. It's good it worked for @ant31, but I don't think it's a recommended way.

(Related #422)

[sparkprime]

If you were forced to write native functions to do things that were 1) pure and 2) relatively simple, like not regex matching or gzip compression or whatever, then there's a case for adding it to the stdlib instead.

Things that are pure, complex, but well-specified with pure implementations available in all languages (like gzip) then we could add them as builtins.

Things that have "writing" side effects should not be added as native functions by anyone. Things that have "reading" side effects could be added, as long as you're OK with it maybe reading them more times than you expect (or not at all). Typically this is OK for things like reading from the same dir (you will want to do an import cache to make it completely robust), but less OK when reading remotely.

Not having computed imports is somewhat orthogonal to all of this. It's essentially about knowing what production systems may be impacted by config refactorings. You can see this in Bazel as well - you have to predeclare what you might read. The idea is then it can always know what to rebuild.

When people ask for computed imports, I wonder what they're trying to do. We could definitely strengthen the way Jsonnet can take "input" data from outside. There was a proposal at one point to make it easier to do pipe-like functionality, e.g.

jsonnet pipe -e 'J + {x: 3}'

could be a syntax sugar for

jsonnet -e 'let J = import "/dev/stdin"; J + {x: 3}'

That gives you a jq-like functionality but it's also not that big a difference from the status quo.

[nikolay]

@sparkprime Well, that above is nice, but can't be done with importstr if you have more than external dynamic config file you want to import.

[sparkprime]

So you have more than one dynamic config file, and you also need their filenames to be parameterized from outside of Jsonnet?

[nikolay]

Yes, @sparkprime. There are many legit use cases for it.

[sparkprime]

What if there was a way to give a list of files on the commandline, and have them all accessible internally as a dict keyed by filename? You can do this now using --code-str but it'd be a clunky bash one-liner.

[nikolay]

@sparkprime That would definitely be an improvement over the status quo if you want to preserve the immutability at any price.

What I'm trying to accomplish is something along the lines of Kapitan and ksonnet, but using as much Jsonnet as possible and as little Bash as possible, too.

This approach would also allow the Bash script to concatenate/template-expand those configs and pass them to the Jsonnet compiler.