[RNHTTR]

The Airflow CLI and UI don't show Variables or Connections that are stored in a secrets backend. This can be really confusing if you don't already know that. This PR adds a warning to CLI commands and a warning banner to the list connections and list variables views in the UI when a user has enabled a secrets backend. The warning just explains that connections and variables stored in secrets backends won't be displayed using the CLI or UI.

Close: #10867

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragement file, named {pr_number}.significant.rst, in newsfragments.

[Taragolis]

cli airflow connections get <connection> will return connection from secrets backends if it exists because it use BaseHook.get_connection(args.conn_id) if it won't return but it should that mean you didn't configure cli to use alternative secrets backend.

[Taragolis]

At that moment airflow grab credentials/variables from SomeAlternateSecretsBackend, EnvironmentVariablesBackend and MetastoreBackend.

And actually it return list only from MetastoreBackend, and ignore all other backends.

[Taragolis]

My fault, actually for listing connection and variable Airflow doesn't use MetastoreBackend rather than it make direct query to Metastore

[RNHTTR]

Yeah -- connections_list and variables_list doesn't capture secrets backend conns/vars:

@suppress_logs_and_warning
def connections_list(args):
    """Lists all connections at the command line"""
    with create_session() as session:
        query = session.query(Connection)
        if args.conn_id:
            query = query.filter(Connection.conn_id == args.conn_id)
        conns = query.all()

        AirflowConsole().print_as(
            data=conns,
            output=args.output,
            mapper=_connection_mapper,
        )

[Taragolis]

Same as connections_list

[Taragolis]

Same as connections_get

[mik-laj]

#10867 is it related?

[RNHTTR]

#10867 is it related?

Yes, I think so.

[blag]

Yep, this solves #10867. Thank you for fixing this!

[blag]

It might be helpful to include examples of what are considered "alternative" backends. Really, just listing the backends that are distributed with Airflow, but are ignored by this command.

[blag]

Since this is the same warning message as connections_get, it might be useful to store it in a variable and reference it everywhere it's used - in the connections and variables commands.

Maybe put it in a new airflow.cli.commands._warnings module?

[RNHTTR]

Good idea; I'll update

[blag]

List those alternative backends distributed with Airflow here too.

And if you want extra credit, maybe count the number of connections/variables not shown, only show the warning if there are include that number in the warning message:

WARNING:
{% trans count=num_variables %}
The Airflow UI will not display the {{ num_variables }} variable stored
in alternative secrets backends (like EnvironmentVariablesBackend
and SecretsBackend).
{% pluralize %}
The Airflow UI will not display the {{ num_variables }} variables stored
in alternative secrets backends (like EnvironmentVariablesBackend
and SecretsBackend).
{% endtrans %}

[blag]

This assumes that we use the i18n extension of Jinja, and as far as I can tell, we don't yet.

[blag]

Nit: It seems a little verbose to mention "connections or variables" when the page is only dedicated to showing connections. I might just mention connections here.

[RNHTTR]

I went back and forth on this. I figured that this way it wouldn't give a user the impression that one would display in the UI but not the other. I don't feel strongly either way.

[bbovenzi]

I'd lean towards only referring to connections in the connections page and vice versa.

[ashb]

@Kinceo1 Are you trying to help out your reviews or are you just trolling?

[potiuk]

@Kinceo1 Are you trying to help out your reviews or are you just trolling?

BTW. We are going to block your account if you do not respond and show whether you have good intentions @Kinceo1 - so far it does not look like we understand your intentions, and if you won't respond shortly, we will assume they are not good and block you.

[blag]

Looks pretty good; just about ready.

The Jinja+HTML templates could use more consistent indentation (see my inline suggestions).

It's a nitpick, but I'm not a huge fan of the cli_or_ui and connection_or_variable parameter names to warn_list_secrets_alterative_backend Perhaps context and backend_type, respectively, might be better? The "blast radius" of this is rather small, and the current X_or_Y names do a good job of indicating the mutually exclusive nature of the parameter values though, so if you're happy with those names we can go with them.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.