Skip to content

ReferenceErrors when using environment-dependent materializations #1997

Open

Description

@mkamysz

We're using different materialization strategies for our models depending on the environment we're running in. We accomplish this by using project variables:

config {
    type: dataform.projectConfig.vars.env === "dev" ? "table" : "incremental"
} 

Since upgrading Dataform from 3.0.9 to 3.0.26 this leads to issues with configs that only apply to one of the materializations, for instance an updatePartitionFilter:

config {
    type: dataform.projectConfig.vars.env === "dev" ? "table" : "incremental"
    bigquery: {
        partitionBy: "event_date",
        updatePartitionFilter: "event_date >= date_sub(current_date(), interval 3 day)", 
    }
} 

Running the action with --vars env=dev, where it would be materialized as a table, now leads to compilation errors, as the updatePartitionFilter does not apply to tables:

ReferenceError: Unexpected property "updatePartitionFilter", or property value type of "string" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfig-TableConfig for allowed properties.

This has not been an issue in previous versions. Are there any remedies? We should have the flexibility to use less complex materialization in our dev environment instead of forcing all envs to be the same.

Activity

  1. mkamysz commented on Jul 24, 2025

    @mkamysz
    ContributorAuthor

    There is a workaround using JavaScript Spread syntax:

    config {
        type: dataform.projectConfig.vars.env === "dev" ? "table" : "incremental",
        bigquery: {
            partitionBy: "date",
            ...(dataform.projectConfig.vars.env === "prod" && {
                updatePartitionFilter: "date >= date_sub(current_date(), interval 7 day)"
            })
        }
    }
    

    Of course that's not really elegant. In my eyes the preferable solution would be allowing all config options from either table, incremental or view types. Maybe raise a warning if unused config is found, but not throw an error straight away.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions