Skip to content

[GH-2419] Register Sedona functions as built-in functions to support permanent VIEW creation - #2420

Merged
jiayuasu merged 3 commits into
apache:masterfrom
zhangfengcdt:feature/support.create.views
Oct 22, 2025
Merged

jiayuasu merged 3 commits into
apache:masterfrom
zhangfengcdt:feature/support.create.views

Conversation

@zhangfengcdt

Copy link
Copy Markdown
Member

Did you read the Contributor Guide?

Is this PR related to a ticket?

  • Yes, and the PR name follows the format [GH-XXX] my subject. Closes #<issue_number>

What changes were proposed in this PR?

Sedona functions are registered as temporary functions in the session FunctionRegistry, which prevents creating permanent VIEWs that reference them:

CREATE VIEW my_view AS
SELECT ST_AsText(geom) FROM buildings;

-- Error: Cannot create view that references temporary function: ST_AsText

Spark's CreateViewCommand validates that permanent VIEWs cannot reference temporary functions. Since Sedona functions are only registered in the session registry (not the built-in registry), they are treated as temporary.

This PR registers Sedona functions into Spark's built-in function registry (FunctionRegistry.builtin) in addition to the session registry. This makes Spark treat Sedona functions as built-in functions (similar to COUNT, MAX, SUM) rather than temporary functions, bypassing the CREATE VIEW validation.

How was this patch tested?

The new test validates:

  • Permanent VIEW creation with Sedona spatial functions
  • VIEW queries return correct results
  • Proper assertions instead of console output

Did this PR include necessary documentation updates?

  • No, this PR does not affect any public API so no need to change the documentation.

@zhangfengcdt
zhangfengcdt marked this pull request as ready for review October 22, 2025 03:21
@zhangfengcdt
zhangfengcdt requested a review from Copilot October 22, 2025 03:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR enables permanent VIEW creation with Sedona spatial functions by registering them as built-in functions in Spark's function registry rather than only as temporary session functions.

  • Registers Sedona functions in both session and built-in function registries
  • Adds test coverage for permanent VIEW creation with spatial functions

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
spark/common/src/main/scala/org/apache/sedona/sql/UDF/AbstractCatalog.scala Registers Sedona functions in FunctionRegistry.builtin to enable permanent VIEW support
spark/common/src/test/scala/org/apache/sedona/sql/functionTestScala.scala Adds test validating permanent VIEW creation and querying with Sedona spatial functions

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread spark/common/src/test/scala/org/apache/sedona/sql/functionTestScala.scala Outdated
@jiayuasu
jiayuasu requested a review from Kontinuation October 22, 2025 05:30
}
aggregateExpressions.foreach { f =>
sparkSession.udf.register(f.getClass.getSimpleName, functions.udaf(f))
FunctionRegistry.builtin.registerFunction(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For folks in the industry who're using HiveMetastore, IcebergRestCatalog or other table metastore options: might be a good idea to leave an option open for them to register geospatial related udf with various metastore options if they prefer to do things that way (i.e. one draw back I could think of with having too many buildin func is it complicates the resolving path for every command, whether or not they contain geospatial udf). Don't have to do this in this PR, but might be great to have a tracking issue or TODO in some form.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@haizhou-zhao pleas feel free to create a github issue on Sedona and we can then track it

@jiayuasu jiayuasu added this to the sedona-1.8.1 milestone Oct 22, 2025
@jiayuasu jiayuasu linked an issue Oct 22, 2025 that may be closed by this pull request
@jiayuasu

Copy link
Copy Markdown
Member

Thanks. Merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support creating permanent views in Sedona and Spark

4 participants