[GH-2419] Register Sedona functions as built-in functions to support permanent VIEW creation - #2420
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR enables permanent VIEW creation with Sedona spatial functions by registering them as built-in functions in Spark's function registry rather than only as temporary session functions.
- Registers Sedona functions in both session and built-in function registries
- Adds test coverage for permanent VIEW creation with spatial functions
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| spark/common/src/main/scala/org/apache/sedona/sql/UDF/AbstractCatalog.scala | Registers Sedona functions in FunctionRegistry.builtin to enable permanent VIEW support |
| spark/common/src/test/scala/org/apache/sedona/sql/functionTestScala.scala | Adds test validating permanent VIEW creation and querying with Sedona spatial functions |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| } | ||
| aggregateExpressions.foreach { f => | ||
| sparkSession.udf.register(f.getClass.getSimpleName, functions.udaf(f)) | ||
| FunctionRegistry.builtin.registerFunction( |
There was a problem hiding this comment.
For folks in the industry who're using HiveMetastore, IcebergRestCatalog or other table metastore options: might be a good idea to leave an option open for them to register geospatial related udf with various metastore options if they prefer to do things that way (i.e. one draw back I could think of with having too many buildin func is it complicates the resolving path for every command, whether or not they contain geospatial udf). Don't have to do this in this PR, but might be great to have a tracking issue or TODO in some form.
There was a problem hiding this comment.
@haizhou-zhao pleas feel free to create a github issue on Sedona and we can then track it
|
Thanks. Merging. |
…pport permanent VIEW creation (apache#2420)
Did you read the Contributor Guide?
Is this PR related to a ticket?
[GH-XXX] my subject. Closes #<issue_number>What changes were proposed in this PR?
Sedona functions are registered as temporary functions in the session
FunctionRegistry, which prevents creating permanent VIEWs that reference them:-- Error: Cannot create view that references temporary function: ST_AsText
Spark's CreateViewCommand validates that permanent VIEWs cannot reference temporary functions. Since Sedona functions are only registered in the session registry (not the built-in registry), they are treated as temporary.
This PR registers Sedona functions into Spark's built-in function registry (FunctionRegistry.builtin) in addition to the session registry. This makes Spark treat Sedona functions as built-in functions (similar to COUNT, MAX, SUM) rather than temporary functions, bypassing the CREATE VIEW validation.
How was this patch tested?
The new test validates:
Did this PR include necessary documentation updates?