Skip to content

Add DA2 and DA3 sensor depth upscale workflow - #12

Open
rurban wants to merge 7 commits into
localai-org:masterfrom
SpexAI:depth-upscale
Open

rurban wants to merge 7 commits into
localai-org:masterfrom
SpexAI:depth-upscale

Conversation

@rurban

@rurban rurban commented Sep 28, 2026 •

Copy link
Copy Markdown

Add TIFF reader support

Add depth-upscaling: interpolating a low-res depth image into a higher-res main image, so that we get for every main pixel depth and angle information.

depth-upscale is the SpexAI-specific capture workflow: it combines a DA2 or single-file
DA3 prediction from an RGB TIFF with a paired sensor TIFF. The sensor image stores each uint16
depth sample in its first two interleaved uint8 channels (high byte, then low byte); the command
resizes and smooths the model map, polynomial-fits it to valid sensor pixels, and writes an
uncompressed single-channel uint16 TIFF in calibrated sensor units. The library API also accepts
a sparse range image already projected into the main camera by PointClouds, preserving its
millimetre scale rather than stretching each result to 0..65535.

I also added a onxx to gguf helper.

We've used to do this with pytorch, which was eating memory, and is far less stable than C++

@localai-org-maint-bot localai-org-maint-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new float calibration path discards a valid calibration point in src/depth_upscale.cpp. Normalization maps the smallest finite relative depth to zero, then the fit loop skips every predictor <= 0.0. Zero here is a valid normalized predictor, not a missing sensor sample (the latter is already represented by sensor_depth[i] == 0).

A minimal case is a 1x3 relative map {1, 2, 3}, sensor ranges {1000, 2000, 3000}, degree 2 and gaussian_sigma 0. Normalization produces {0, 0.5, 1}; only two samples enter the three-coefficient fit, so it returns insufficient overlapping samples despite three non-degenerate calibration pairs. Please retain finite zero-valued predictors in the normalized float path and add this case to test_depth_upscale. Preserve any intentionally different legacy uint8 semantics explicitly.

The unconditional find_package(TIFF REQUIRED) also changes the dependency requirements for every existing embedding of the library, including callers that do not use depth-upscale. Please make that dependency opt-in, or coordinate and validate the build/package changes in LocalAI before merge.

Reviewed the complete diff at 035147c. These are source-level findings; CMake, a C++ compiler, and model fixtures are unavailable here. Maintainer edits are disabled for this fork, so I cannot push the calibration repair directly to this branch.

@rurban

rurban commented Sep 30, 2026

Copy link
Copy Markdown
Author

Should be fixed now

Comment thread make.sh
@@ -0,0 +1,2 @@
cmake -B build -DDA_BUILD_CLI=ON -DCMAKE_CUDA_ARCHITECTURES=native -DDA_GGML_CUDA=ON

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a leftover? It seems like your specific config?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants