Repository navigation
feat: Add receipt enhancement mode with live receipt edge detection - #105773
OlimpiaZurek wants to merge 19 commits into
Conversation
|
Hey, I noticed you changed If you want to automatically generate translations for other locales, an Expensify employee will have to:
Alternatively, if you are an external contributor, you can run the translation script locally with your own OpenAI API key. To learn more, try running: npx bun ./scripts/generateTranslations.ts --helpTypically, you'd want to translate only what you changed by running |
|
|
|
🚧 AndrewGable has triggered a test Expensify/App build. You can view the workflow run here. |
This comment has been minimized.
This comment has been minimized.
# Conflicts: # src/CONST/index.ts # src/pages/iou/request/step/IOURequestStepScan/components/Camera/index.native.tsx # src/pages/iou/request/step/IOURequestStepScan/components/CameraViewport.tsx
Codecov Report❌ Looks like you've decreased code coverage for some files. Please write tests to increase, or at least maintain, the existing level of code coverage. See our documentation here for how to interpret this table.
|
|
@codex review |
|
@MelvinBot review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
🤖 Web test steps verification results: ❌ fail Static review found no actionable bugs, but web verification was blocked because the handed-off browser session was unavailable. CI must restore the session before web testing can proceed.
|
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
🚧 AndrewGable has triggered a test Expensify/App build. You can view the workflow run here. |
This comment has been minimized.
This comment has been minimized.
|
Please mark as ready for review when ready |
# Conflicts: # src/CONST/index.ts
|
Also, I fixed a bug that made the app crash on release builds. @AndrewGable could we run another ad hoc build? |
|
🚧 AndrewGable has triggered a test Expensify/App build. You can view the workflow run here. |
This comment has been minimized.
This comment has been minimized.
|
Dang this feels sick! Curious what @Expensify/design thinks. (I enabled the beta in Troubleshoot > Beta Overrides) I could see us making the stroke width of the outer border a bit thicker, but we can always follow up with those visual tweaks. |
|
This is feeling really slick! I agree about making the border thicker but that's a minor visual detail. Feeling nice and smooth to me. |
|
This is testing fantastic! I quite like how it looks on my phone. Dope work! |
|
@situchan please prioritize a review of this PR to keep it moving. |
| shouldDetectDocument && !didCapturePhoto ? ( | ||
| <> | ||
| <DocumentOutline edge={documentEdge} /> | ||
| {isReceiptEnhancementToastVisible && <ReceiptEnhancementToast onHide={hideReceiptEnhancementToast} />} |
There was a problem hiding this comment.
"Receipt enhancements enabled" toast shows again later
The toast lives inside overlay, which only renders while shouldDetectDocument && !didCapturePhoto.
If the overlay unmounts before the 2 second timer ends, the timer is cleared and onHide never runs.
isReceiptEnhancementToastVisible stays true, so the toast plays again the next time the overlay renders.
Preconditions: receiptEnhancement beta on, iOS or Android, camera permission granted, receipt enhancements off.
- Tap + > Create expense > Scan
- Tap the sparkles button right of the shutter
- Within 2 seconds, tap the shutter
- On the confirmation page, tap Back
Expected: No toast, since nothing was toggled.
Actual: "Receipt enhancements enabled" shows again at the bottom of the viewfinder.
The same happens when multi-scan is turned on within 2 seconds of step 2 and then turned off.
| const {photoOutput, cameraAspectRatio, updatePhotoResolution} = useReceiptPhotoOutput(cameraPlatformConfig.scanQualityPrioritization, isInLandscapeMode); | ||
| const outputs = [photoOutput]; | ||
| // VisionCamera restarts the session, blanking the preview, whenever an output or the photo size changes. | ||
| const {photoOutput, cameraAspectRatio, updatePhotoResolution} = useReceiptPhotoOutput(cameraPlatformConfig.scanQualityPrioritization, isInLandscapeMode, canUseReceiptEnhancement); |
There was a problem hiding this comment.
Beta users get 12 MP photos even with enhancements off
shouldUseFullSensorResolution is canUseReceiptEnhancement, which checks the beta and not the toggle.
Every scan by a beta user is captured at 4032x3024 instead of 2880x2160, about twice the pixels, and the scan flow never downscales it.
App/src/hooks/useReceiptPhotoOutput.ts
Lines 24 to 27 in 9fdbc72
Preconditions: receiptEnhancement beta on, iOS, receipt enhancements off (sparkles grey).
- Tap + > Create expense > Scan
- Take a photo of a receipt
- Submit the expense, open the receipt and download it
Expected: 2880x2160, same as with the beta off.
Actual: 4032x3024, a larger upload.
There was a problem hiding this comment.
This format is intentional for beta users. With the ~5 MP format, the preview shows a narrower view than the detection frames, so the outline doesn’t line up with the receipt. I tried tying it to the toggle, but then VisionCamera replaces the photo output and restarts the session each time you toggle, and the preview goes black for a moment. Since only beta users get the larger photos, I’d keep it as is for now.
| if (!documentDetector?.isSupported) { | ||
| return; | ||
| } | ||
| if (IS_IOS) { |
There was a problem hiding this comment.
Platform branching inside one hook
IS_IOS picks the frame resolution, MAX_BEND and a 35 line branch (Vision detect(pointer) vs model detectYuv(planes) plus frame-to-camera conversion) in a file with no platform suffix. The native spec mirrors it: detect returns null on Android and detectYuv returns nil on iOS.
Fix: move per-platform frame reading into readDocument.ios.ts and readDocument.android.ts, each exporting the resolution, bend limit and a worklet returning {corners, outline, toCameraPoint}.
| import type {DocumentEdge, QuadPoint} from './types'; | ||
|
|
||
| // Must match the native detectors. | ||
| const EDGE_POINTS_PER_SIDE = 24; |
There was a problem hiding this comment.
Points-per-side constant copied in three languages
Swift, Kotlin and JS each hard-code 24, kept in step only by comments.
If one changes, isWholeReceipt goes false and the outline silently stops bending.
Fix: return pointsPerSide in DetectedDocument, or derive it from outline.length / 4 in JS.
| }; | ||
|
|
||
| interface DocumentDetectorModule extends HybridObject<{ios: 'swift'; android: 'kotlin'}> { | ||
| readonly isSupported: boolean; |
There was a problem hiding this comment.
isSupported is always true
Both native classes hard-code true. A missing module is already handled by getDocumentDetectorNitroModule() returning null, yet processFrame reads the property across JSI on every analysed frame.
Fix: drop it and check for null, or give it a real check such as the model asset existing on Android.
| type DocumentOutlineProps = { | ||
| edge: SharedValue<DocumentEdge | null>; | ||
| }; |
There was a problem hiding this comment.
edge prop needs a comment
The points are in preview view coordinates, every 24th point is a corner (the curve drawing depends on it), and null hides the outline. None of that is visible from the type.
| const outputs = [photoOutput]; | ||
| // VisionCamera restarts the session, blanking the preview, whenever an output or the photo size changes. | ||
| const {photoOutput, cameraAspectRatio, updatePhotoResolution} = useReceiptPhotoOutput(cameraPlatformConfig.scanQualityPrioritization, isInLandscapeMode, canUseReceiptEnhancement); | ||
| const {frameOutput, edge: documentEdge} = useDocumentEdgeDetection({cameraRef: camera, isEnabled: shouldDetectDocument}); |
There was a problem hiding this comment.
Every Scan open creates a camera worklet runtime, even with the beta off
useDocumentEdgeDetection runs on every native Scan mount.
In VisionCamera 5.2.3, useFrameOutput always calls VisionCamera.createFrameOutput and createRuntimeForThread (which calls createWorkletRuntime), even when the output is never attached to the camera.
With Worklets bundleMode: true, each new runtime also evaluates the app bundle, so every native user pays the time and memory on each Scan open.
Line 101 in 9fdbc72
|
|
||
| export {ContactsNitroModule, AppStartTimeNitroModule}; | ||
| let documentDetectorNitroModule: DocumentDetectorModule | null | undefined; | ||
| function getDocumentDetectorNitroModule(): DocumentDetectorModule | null { |
There was a problem hiding this comment.
The ONNX session stays in memory after Scan closes
getDocumentDetectorNitroModule() caches one detector for the whole app session.
After the first detection, the ONNX session (about 13 MB of model) and its buffers stay loaded until the app is killed.
| // Add a dependency on NitroModules | ||
| implementation project(":react-native-nitro-modules") | ||
|
|
||
| implementation "com.microsoft.onnxruntime:onnxruntime-android:1.30.0" |
There was a problem hiding this comment.
Android app size grows for every user
The 13.4 MB docquadnet256_trained_opset17.ort asset and the onnxruntime-android native libraries ship in every Android build, HybridApp included, although only beta users can turn the feature on.
We need to get sign-off on the size increase, or download the model on demand (or ship it as a Play Feature Delivery module).
There was a problem hiding this comment.
I looked at moving the model to an on-demand download (our own, or Play Feature Delivery), but both make the feature depend on the network. Scanning is something people do offline, at a restaurant, while traveling, or with a weak signal, and the app is built to work offline. With an on-demand model, the first time someone turns enhancements on without a connection they get no outline at all. A half-finished download leaves it broken until it’s retried. Play Feature Delivery also only works for Play Store installs and changes how HybridApp is built and released. Shipping the model in the app keeps the feature working offline from the first use.
I’d keep it bundled and get sign-off on the size. Who should approve it? cc: @AndrewGable
…etection only for beta users
# Conflicts: # src/CONST/index.ts
|
@situchan All comments addressed. Heads up: I will be OOO from tomorrow until the end of the week. Someone from the Callstack team will take over this PR starting Thursday, as tomorrow is a bank holiday in Poland. |
|
@AndrewGable please generate new build (android only) |
|
running new adhocs |
|
🚧 luacmartins has triggered a test Expensify/App build. You can view the workflow run here. |
This comment has been minimized.
This comment has been minimized.
| override fun open(): Double { | ||
| val owner = lastOwner.incrementAndGet() | ||
| currentOwner.set(owner) | ||
| return owner.toDouble() | ||
| } | ||
|
|
||
| override fun release(owner: Double) { | ||
| if (currentOwner.compareAndSet(owner.toLong(), NO_OWNER)) { | ||
| releaseExecutor.execute { freeResources() } | ||
| } | ||
| } |
There was a problem hiding this comment.
Android Bug: leaving a second Scan screen stops the outline on the first one
The detector keeps a single currentOwner, but two Scan screens can be mounted at once (the start page's Scan tab stays mounted under a pushed Scan step).
The second screen's open() replaces the owner, and its release() on Back sets it to 0. The first screen's frames still carry the old token, so detectYuv returns null for every frame until the page remounts.
- Tap + > Create expense > Scan and point at a receipt (outline shows)
- Take the photo
- On the confirmation page, tap the receipt > Replace
- On the new Scan screen, tap Back without taking a photo
- Tap Back on the confirmation page
- Point at the receipt again
Expected: The outline shows, as in step 1.
Actual: No outline until Create expense is closed and opened again.
|
@OlimpiaZurek let's address this comment #105773 (comment) |
The crash came from key={transactionRequestType} on the Scan tab. Every time the tab was switched, the whole Scan screen remounted and set up a new camera frame output that never got cleaned up, so memory kept climbing until iOS killed the app. Fixed by removing the key and resetting multi-scan state explicitly when the request type changes. |
# Conflicts: # src/pages/iou/request/step/IOURequestStepScan/components/Camera/index.native.tsx # src/pages/iou/request/step/IOURequestStepScan/components/MultiScanContext.tsx
Done |
|
@AndrewGable or @luacmartins please generate new build |
|
🚧 AndrewGable has triggered a test Expensify/App build. You can view the workflow run here. |
|
🧪🧪 Use the links below to test this adhoc build on Android, iOS, and Web. Happy testing! 🧪🧪
|
|
@situchan how's review going? |
| updateOutline(corners, detectedEdge, toCameraPoint, detectedAt); | ||
| }; | ||
|
|
||
| const frameOutput = useFrameOutput({ |
There was a problem hiding this comment.
The frame output leak is still there, only the tab trigger is gone
In VisionCamera 5.2.3, useFrameOutput creates the frame output and its worklet runtime in useMemo and has no unmount cleanup.
setOnFrameCallback(undefined) is never called, and with bundleMode: true each runtime also evaluates the bundle.
Removing the key stops the remount on tab switch, but every other mount of DocumentDetection still leaks one runtime: closing and reopening Create expense, pushing a second Scan screen (Replace receipt, Add receipt), or opening Scan from another entry point.
- + > Create expense > Scan
- Close Create expense
- Repeat steps 1 and 2 about 20 times
Expected: memory stays flat, as with the beta off.
Actual: memory grows with each open. This is the same mechanism as the tab-switch crash.
I haven't measured how many opens it takes to crash. Could you run this loop the same way you measured the tab case?
| detection && shouldDetectDocument && !didCapturePhoto ? ( | ||
| <> | ||
| <DocumentOutline | ||
| edge={detection.edge} | ||
| isHidden={detection.isMoving} | ||
| /> | ||
| {isReceiptEnhancementToastVisible && <ReceiptEnhancementToast onHide={hideReceiptEnhancementToast} />} |
There was a problem hiding this comment.
The toast shows again after a tab round trip (iOS)
Scan now stays mounted across tabs.
On blur the iOS preview stops, the overlay unmounts and its 2 second timer is cleared, but the flag stays true.
- + > Create expense > Scan
- Tap the sparkles button
- Within 2 seconds, tap Manual
- Tap Scan
Expected: no toast.
Actual: "Receipt enhancements enabled" shows again.
| if (!isEnabled) { | ||
| if (shownEdge.getDirty() !== null) { | ||
| setOutline(null); | ||
| } | ||
| return; |
There was a problem hiding this comment.
The previous receipt's outline shows on a new receipt (iOS)
outlineState now survives a tab switch, and the first iOS frame after the switch reuses it.
The !isEnabled branch only calls setOutline(null), so turning enhancements off and on keeps it too, on iOS and Android.
- On Scan, hold still on a crumpled receipt until the outline follows the folds
- Tap Manual
- Put a flat receipt in the same spot
- Tap Scan
Expected: four straight sides.
Actual: the old folds show for about 0.7 seconds, then the outline morphs flat.

Explanation of Change
Adds a "Receipt enhancement" mode to the native Scan camera, behind the
receiptEnhancementbeta. When the mode is on, an outline is drawn over the receipt in the live viewfinder.How the outline behaves:
This release only shows the outline. The saved photo is not cropped or edited
Fixed Issues
$ #101916
PROPOSAL:
Tests
receiptEnhancementbeta and open the Scan camera on iOS or Android.Offline tests
QA Steps
Same as tests
PR Author Checklist
### Fixed Issuessection aboveTestssectionOffline stepssectionQA stepssectiontoggleReportand notonIconClick)src/languages/*files and using the translation methodSTYLE.md) were followedAvatar, I verified the components usingAvatarare working as expected)StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))npm run compress-svg)Avataris modified, I verified thatAvataris working as expected in all cases)Designlabel and/or tagged@Expensify/designso the design team can review the changes.ScrollViewcomponent to make it scrollable when more elements are added to the page.mainbranch was merged into this PR after a review, I tested again and verified the outcome was still expected according to theTeststeps.Screenshots/Videos
Android: Native
Screen_Recording_20261106-123533_Expensify.Dev.mp4
Screen_Recording_20261106-123605_Expensify.Dev.mp4
Android: mWeb Chrome
iOS: Native
ScreenRecording_11-06-2026.12-32-08_1.MP4
ScreenRecording_11-06-2026.12-29-09_1.mov
iOS: mWeb Safari
MacOS: Chrome / Safari