Skip to content

feat: Add receipt enhancement mode with live receipt edge detection - #105773

Open
OlimpiaZurek wants to merge 19 commits into
Expensify:mainfrom
callstack-internal:feat/ai-capture-vision-detector
Open

OlimpiaZurek wants to merge 19 commits into
Expensify:mainfrom
callstack-internal:feat/ai-capture-vision-detector

Conversation

@OlimpiaZurek

@OlimpiaZurek OlimpiaZurek commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Explanation of Change

Adds a "Receipt enhancement" mode to the native Scan camera, behind the receiptEnhancement beta. When the mode is on, an outline is drawn over the receipt in the live viewfinder.

How the outline behaves:

  • While the receipt or the phone is moving, the outline shows four straight lines that follow the receipt.
  • Once the receipt holds still, the outline settles onto the paper's own edge, so crumpled, curled, torn or folded receipts are outlined the way they look.
  • When the receipt leaves the view, the outline disappears.

This release only shows the outline. The saved photo is not cropped or edited

Fixed Issues

$ #101916
PROPOSAL:

Tests

  1. Enable the receiptEnhancement beta and open the Scan camera on iOS or Android.
  2. Tap the sparkle button and verify the "Receipt enhancements enabled" toast appears.
  3. Point the camera at a receipt on a contrasting surface and verify an outline appears over it.
  4. Move the phone and verify the outline follows the receipt as straight lines.
  5. Hold the phone still and verify the outline settles onto the receipt's edge, including curled or crumpled parts.
  6. Point the camera away from the receipt and verify the outline disappears.
  7. Take a photo and verify the saved receipt image is not cropped or changed.
  8. Tap the sparkle button again and verify the outline is no longer shown.
  • Verify that no errors appear in the JS console

Offline tests

QA Steps

Same as tests

  • Verify that no errors appear in the JS console

PR Author Checklist

  • I linked the correct issue in the ### Fixed Issues section above
  • I wrote clear testing steps that cover the changes made in this PR
    • I added steps for local testing in the Tests section
    • I added steps for the expected offline behavior in the Offline steps section
    • I added steps for Staging and/or Production testing in the QA steps section
    • I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
    • I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
    • I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).
  • I included screenshots or videos for tests on all platforms
  • I ran the tests on all platforms & verified they passed on:
    • Android: Native
    • Android: mWeb Chrome
    • iOS: Native
    • iOS: mWeb Safari
    • MacOS: Chrome / Safari
  • I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
  • I followed proper code patterns (see Reviewing the code)
    • I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
    • I verified that comments were added to code that is not self explanatory
    • I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
    • I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
      • If any non-english text was added/modified, I used JaimeGPT to get English > Spanish translation. I then posted it in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:
    • I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
    • I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
    • I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
    • I verified the JSDocs style guidelines (in STYLE.md) were followed
  • If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
  • I followed the guidelines as stated in the Review Guidelines
  • I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
  • I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
  • I verified any variables that can be defined as constants (ie. in CONST.ts or at the top of the file that uses the constant) are defined as such
  • I verified that if a function's arguments changed that all usages have also been updated correctly
  • If any new file was added I verified that:
    • The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory
  • If a new CSS style is added I verified that:
    • A similar style doesn't already exist
    • The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))
  • If new assets were added or existing ones were modified, I verified that:
    • The assets are optimized and compressed (for SVG files, run npm run compress-svg)
    • The assets load correctly across all supported platforms.
  • If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
  • If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
  • If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
  • If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
  • If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:
    • I verified that all the inputs inside a form are aligned with each other.
    • I added Design label and/or tagged @Expensify/design so the design team can review the changes.
  • If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
  • I added unit tests for any new feature or bug fix in this PR to help automatically prevent regressions in this user flow.
  • If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native
Screen_Recording_20261106-123533_Expensify.Dev.mp4
Screen_Recording_20261106-123605_Expensify.Dev.mp4
Android: mWeb Chrome
iOS: Native
ScreenRecording_11-06-2026.12-32-08_1.MP4
ScreenRecording_11-06-2026.12-29-09_1.mov
iOS: mWeb Safari
MacOS: Chrome / Safari

@melvin-bot

melvin-bot Bot commented Nov 3, 2026

Copy link
Copy Markdown

Hey, I noticed you changed src/languages/en.ts in a PR from a fork. For security reasons, translations are not generated automatically for PRs from forks.

If you want to automatically generate translations for other locales, an Expensify employee will have to:

  1. Look at the code and make sure there are no malicious changes.
  2. Run the Generate static translations GitHub workflow. If you have write access and the K2 extension, you can simply click: [this button]

Alternatively, if you are an external contributor, you can run the translation script locally with your own OpenAI API key. To learn more, try running:

npx bun ./scripts/generateTranslations.ts --help

Typically, you'd want to translate only what you changed by running npx bun ./scripts/generateTranslations.ts --compare-ref main

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

⚠️ This PR is possibly changing native code and/or updating libraries, it may cause problems with HybridApp. Please check if any patch updates are required in the HybridApp repo and run an AdHoc build to verify that HybridApp will not break. Ask Contributor Plus for help if you are not sure how to handle this. ⚠️

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🚧 AndrewGable has triggered a test Expensify/App build. You can view the workflow run here.

@github-actions

This comment has been minimized.

# Conflicts:
#	src/CONST/index.ts
#	src/pages/iou/request/step/IOURequestStepScan/components/Camera/index.native.tsx
#	src/pages/iou/request/step/IOURequestStepScan/components/CameraViewport.tsx
@codecov

codecov Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Looks like you've decreased code coverage for some files. Please write tests to increase, or at least maintain, the existing level of code coverage. See our documentation here for how to interpret this table.

Files with missing lines Coverage Δ
src/CONST/SENTRY_LABEL.ts 100.00% <ø> (ø)
src/CONST/index.ts 96.15% <ø> (ø)
src/ONYXKEYS.ts 100.00% <ø> (ø)
src/hooks/useReceiptPhotoOutput.ts 100.00% <100.00%> (ø)
src/libs/ExportOnyxState/masking.ts 93.45% <ø> (ø)
src/pages/iou/request/IOURequestStartPage.tsx 85.33% <ø> (-0.10%) ⬇️
...iou/request/step/IOURequestStepScan/ScanRouter.tsx 85.89% <100.00%> (+0.18%) ⬆️
...StepScan/components/MultiScanGate/index.native.tsx 100.00% <100.00%> (ø)
...stStepScan/documentDetection/DocumentDetection.tsx 100.00% <100.00%> (ø)
.../documentDetection/isDocumentDetectionSupported.ts 100.00% <100.00%> (ø)
... and 15 more
... and 12 files with indirect coverage changes

@OlimpiaZurek

Copy link
Copy Markdown
Contributor Author

@codex review

@OlimpiaZurek

Copy link
Copy Markdown
Contributor Author

@MelvinBot review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-06T12:08:31.028592Z 43c02d7 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@MelvinBot

Copy link
Copy Markdown
Contributor

🤖 Web test steps verification results: ❌ fail

Static review found no actionable bugs, but web verification was blocked because the handed-off browser session was unavailable. CI must restore the session before web testing can proceed.

Step Status
Web Scan controls render without receipt-enhancement toggleNot verified: the browser session was unavailable.
❌
No JS console errors while opening and using ScanNot verified: Scan could not be opened.
❌
Native receipt outlinesNot verified: native receipt detection cannot be tested on web.
❌

view run

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: ffbdd6b0ce

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🚧 AndrewGable has triggered a test Expensify/App build. You can view the workflow run here.

@github-actions

This comment has been minimized.

@situchan

situchan commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Please mark as ready for review when ready

@OlimpiaZurek

Copy link
Copy Markdown
Contributor Author

Please mark as ready for review when ready

@situchan Let’s wait for feedback on this. Once it’s confirmed that this version is good, I’ll move it to review.

@OlimpiaZurek

Copy link
Copy Markdown
Contributor Author

Also, I fixed a bug that made the app crash on release builds. @AndrewGable could we run another ad hoc build?

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

🚧 AndrewGable has triggered a test Expensify/App build. You can view the workflow run here.

@github-actions

This comment has been minimized.

@OlimpiaZurek
OlimpiaZurek marked this pull request as ready for review November 6, 2026 11:43
@shawnborton

Copy link
Copy Markdown
Contributor

Dang this feels sick! Curious what @Expensify/design thinks. (I enabled the beta in Troubleshoot > Beta Overrides)

I could see us making the stroke width of the outer border a bit thicker, but we can always follow up with those visual tweaks.

@dannymcclain

Copy link
Copy Markdown
Contributor

This is feeling really slick! I agree about making the border thicker but that's a minor visual detail. Feeling nice and smooth to me.

@dubielzyk-expensify

Copy link
Copy Markdown
Contributor

This is testing fantastic! I quite like how it looks on my phone. Dope work!

@JmillsExpensify

Copy link
Copy Markdown
Contributor

@situchan please prioritize a review of this PR to keep it moving.

Comment on lines +362 to +365
shouldDetectDocument && !didCapturePhoto ? (
<>
<DocumentOutline edge={documentEdge} />
{isReceiptEnhancementToastVisible && <ReceiptEnhancementToast onHide={hideReceiptEnhancementToast} />}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Receipt enhancements enabled" toast shows again later

The toast lives inside overlay, which only renders while shouldDetectDocument && !didCapturePhoto.
If the overlay unmounts before the 2 second timer ends, the timer is cleared and onHide never runs.
isReceiptEnhancementToastVisible stays true, so the toast plays again the next time the overlay renders.

const toggleReceiptEnhancement = () => {
setIsReceiptEnhancementEnabled(!isReceiptEnhancementEnabled);
setIsReceiptEnhancementToastVisible(!isReceiptEnhancementEnabled);
};

Preconditions: receiptEnhancement beta on, iOS or Android, camera permission granted, receipt enhancements off.

  1. Tap + > Create expense > Scan
  2. Tap the sparkles button right of the shutter
  3. Within 2 seconds, tap the shutter
  4. On the confirmation page, tap Back

Expected: No toast, since nothing was toggled.
Actual: "Receipt enhancements enabled" shows again at the bottom of the viewfinder.

The same happens when multi-scan is turned on within 2 seconds of step 2 and then turned off.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

const {photoOutput, cameraAspectRatio, updatePhotoResolution} = useReceiptPhotoOutput(cameraPlatformConfig.scanQualityPrioritization, isInLandscapeMode);
const outputs = [photoOutput];
// VisionCamera restarts the session, blanking the preview, whenever an output or the photo size changes.
const {photoOutput, cameraAspectRatio, updatePhotoResolution} = useReceiptPhotoOutput(cameraPlatformConfig.scanQualityPrioritization, isInLandscapeMode, canUseReceiptEnhancement);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beta users get 12 MP photos even with enhancements off

shouldUseFullSensorResolution is canUseReceiptEnhancement, which checks the beta and not the toggle.
Every scan by a beta user is captured at 4032x3024 instead of 2880x2160, about twice the pixels, and the scan flow never downscales it.

const FULL_SENSOR_PHOTO_TARGET_RESOLUTION = {width: 4032, height: 3024};
function useReceiptPhotoOutput(qualityPrioritization: QualityPrioritization, isInLandscapeMode: boolean, shouldUseFullSensorResolution = false) {
const targetResolution = shouldUseFullSensorResolution ? FULL_SENSOR_PHOTO_TARGET_RESOLUTION : PHOTO_TARGET_RESOLUTION;

Preconditions: receiptEnhancement beta on, iOS, receipt enhancements off (sparkles grey).

  1. Tap + > Create expense > Scan
  2. Take a photo of a receipt
  3. Submit the expense, open the receipt and download it

Expected: 2880x2160, same as with the beta off.
Actual: 4032x3024, a larger upload.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This format is intentional for beta users. With the ~5 MP format, the preview shows a narrower view than the detection frames, so the outline doesn’t line up with the receipt. I tried tying it to the toggle, but then VisionCamera replaces the photo output and restarts the session each time you toggle, and the preview goes black for a moment. Since only beta users get the larger photos, I’d keep it as is for now.

if (!documentDetector?.isSupported) {
return;
}
if (IS_IOS) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Platform branching inside one hook

IS_IOS picks the frame resolution, MAX_BEND and a 35 line branch (Vision detect(pointer) vs model detectYuv(planes) plus frame-to-camera conversion) in a file with no platform suffix. The native spec mirrors it: detect returns null on Android and detectYuv returns nil on iOS.

const IS_IOS = Platform.OS === 'ios';
const MIN_MODEL_CONFIDENCE = 0.6;
const ROTATION_TO_UPRIGHT = {up: 0, right: 270, down: 180, left: 90} as const;
// Android's model takes a 256x256 input, so a small frame keeps copying and shrinking it cheap. iOS moves Vision's edge
// onto the paper in the camera image itself, so it gets a sharper frame.
const FRAME_OUTPUT_RESOLUTION = IS_IOS ? {width: 1280, height: 960} : {width: 640, height: 480};

if (IS_IOS) {
const nativeBuffer = frame.getNativeBuffer();
try {
const document = documentDetector.detect(nativeBuffer.pointer);
// VisionCamera 5.2.3 on iOS rotates converted frame points twice, swapping x and y, so Vision's raw-buffer corners are used as they are.
if (document && document.confidence >= MIN_VISION_CONFIDENCE && document.corners.length === 4) {
corners = document.corners;
detectedEdge = document.outline;
}
} finally {
nativeBuffer.release();
}
detectedAt = getTimestamp();
} else {
const [yPlane, uPlane, vPlane] = frame.getPlanes();
if (!yPlane?.isValid || !uPlane?.isValid || !vPlane?.isValid) {
throw new Error(`Frame has no YUV planes (pixelFormat=${frame.pixelFormat})`);
}
const document = documentDetector.detectYuv({
y: yPlane.getPixelBuffer(),
u: uPlane.getPixelBuffer(),
v: vPlane.getPixelBuffer(),
width: frame.width,
height: frame.height,
yRowStride: yPlane.bytesPerRow,
uvRowStride: uPlane.bytesPerRow,
// A plane's width is rowStride / pixelStride, so this gives the chroma pixel stride back.
uvPixelStride: Math.round(uPlane.bytesPerRow / uPlane.width),
rotation: ROTATION_TO_UPRIGHT[frame.orientation],
});
detectedAt = getTimestamp();
if (document && document.confidence >= MIN_MODEL_CONFIDENCE && document.corners.length === 4) {
corners = document.corners;
detectedEdge = document.outline;
toCameraPoint = (point) => frame.convertFramePointToCameraPoint({x: point.x * frame.width, y: point.y * frame.height});
}
}

Fix: move per-platform frame reading into readDocument.ios.ts and readDocument.android.ts, each exporting the resolution, bend limit and a worklet returning {corners, outline, toCameraPoint}.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

import type {DocumentEdge, QuadPoint} from './types';

// Must match the native detectors.
const EDGE_POINTS_PER_SIDE = 24;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Points-per-side constant copied in three languages

Swift, Kotlin and JS each hard-code 24, kept in step only by comments.
If one changes, isWholeReceipt goes false and the outline silently stops bending.



Fix: return pointsPerSide in DetectedDocument, or derive it from outline.length / 4 in JS.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

};

interface DocumentDetectorModule extends HybridObject<{ios: 'swift'; android: 'kotlin'}> {
readonly isSupported: boolean;

@situchan situchan Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isSupported is always true

Both native classes hard-code true. A missing module is already handled by getDocumentDetectorNitroModule() returning null, yet processFrame reads the property across JSI on every analysed frame.

Fix: drop it and check for null, or give it a real check such as the model asset existing on Android.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

Comment on lines +34 to +36
type DocumentOutlineProps = {
edge: SharedValue<DocumentEdge | null>;
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

edge prop needs a comment

The points are in preview view coordinates, every 24th point is a corner (the curve drawing depends on it), and null hides the outline. None of that is visible from the type.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

const outputs = [photoOutput];
// VisionCamera restarts the session, blanking the preview, whenever an output or the photo size changes.
const {photoOutput, cameraAspectRatio, updatePhotoResolution} = useReceiptPhotoOutput(cameraPlatformConfig.scanQualityPrioritization, isInLandscapeMode, canUseReceiptEnhancement);
const {frameOutput, edge: documentEdge} = useDocumentEdgeDetection({cameraRef: camera, isEnabled: shouldDetectDocument});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every Scan open creates a camera worklet runtime, even with the beta off

useDocumentEdgeDetection runs on every native Scan mount.
In VisionCamera 5.2.3, useFrameOutput always calls VisionCamera.createFrameOutput and createRuntimeForThread (which calls createWorkletRuntime), even when the output is never attached to the camera.
With Worklets bundleMode: true, each new runtime also evaluates the app bundle, so every native user pays the time and memory on each Scan open.


const {frameOutput, edge: documentEdge} = useDocumentEdgeDetection({cameraRef: camera, isEnabled: shouldDetectDocument});

bundleMode: true,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed


export {ContactsNitroModule, AppStartTimeNitroModule};
let documentDetectorNitroModule: DocumentDetectorModule | null | undefined;
function getDocumentDetectorNitroModule(): DocumentDetectorModule | null {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ONNX session stays in memory after Scan closes

getDocumentDetectorNitroModule() caches one detector for the whole app session.
After the first detection, the ONNX session (about 13 MB of model) and its buffers stay loaded until the app is killed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

// Add a dependency on NitroModules
implementation project(":react-native-nitro-modules")

implementation "com.microsoft.onnxruntime:onnxruntime-android:1.30.0"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Android app size grows for every user

The 13.4 MB docquadnet256_trained_opset17.ort asset and the onnxruntime-android native libraries ship in every Android build, HybridApp included, although only beta users can turn the feature on.

We need to get sign-off on the size increase, or download the model on demand (or ship it as a Play Feature Delivery module).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at moving the model to an on-demand download (our own, or Play Feature Delivery), but both make the feature depend on the network. Scanning is something people do offline, at a restaurant, while traveling, or with a weak signal, and the app is built to work offline. With an on-demand model, the first time someone turns enhancements on without a connection they get no outline at all. A half-finished download leaves it broken until it’s retried. Play Feature Delivery also only works for Play Store installs and changes how HybridApp is built and released. Shipping the model in the app keeps the feature working offline from the first use.

I’d keep it bundled and get sign-off on the size. Who should approve it? cc: @AndrewGable

@OlimpiaZurek

Copy link
Copy Markdown
Contributor Author

@situchan All comments addressed.

Heads up: I will be OOO from tomorrow until the end of the week. Someone from the Callstack team will take over this PR starting Thursday, as tomorrow is a bank holiday in Poland.

@situchan

situchan commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Bug: iOS app crashes:

  1. + > Create expense > Scan.
  2. Tap the sparkles button (off > on) and immediately tap Manual tab
  3. Tap Scan tab
  4. Tap the sparkles button (on > off).
  5. Tap the sparkles button (off > on) again and immediately tap Manual tab.

Repeat Step 3-5 a few times if crash doesn't happen

crash.mov
log

@situchan

Copy link
Copy Markdown
Contributor

@AndrewGable please generate new build (android only)

@luacmartins

Copy link
Copy Markdown
Contributor

running new adhocs

@github-actions

Copy link
Copy Markdown
Contributor

🚧 luacmartins has triggered a test Expensify/App build. You can view the workflow run here.

@github-actions

This comment has been minimized.

Comment on lines +79 to +89
override fun open(): Double {
val owner = lastOwner.incrementAndGet()
currentOwner.set(owner)
return owner.toDouble()
}

override fun release(owner: Double) {
if (currentOwner.compareAndSet(owner.toLong(), NO_OWNER)) {
releaseExecutor.execute { freeResources() }
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Android Bug: leaving a second Scan screen stops the outline on the first one

The detector keeps a single currentOwner, but two Scan screens can be mounted at once (the start page's Scan tab stays mounted under a pushed Scan step).
The second screen's open() replaces the owner, and its release() on Back sets it to 0. The first screen's frames still carry the old token, so detectYuv returns null for every frame until the page remounts.

override fun detectYuv(frame: YuvFrame): DetectedDocument? {
if (frame.owner.toLong() != currentOwner.get()) {
return null

  1. Tap + > Create expense > Scan and point at a receipt (outline shows)
  2. Take the photo
  3. On the confirmation page, tap the receipt > Replace
  4. On the new Scan screen, tap Back without taking a photo
  5. Tap Back on the confirmation page
  6. Point at the receipt again

Expected: The outline shows, as in step 1.
Actual: No outline until Create expense is closed and opened again.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@luacmartins

Copy link
Copy Markdown
Contributor

@OlimpiaZurek let's address this comment #105773 (comment)

@OlimpiaZurek

Copy link
Copy Markdown
Contributor Author

Bug: iOS app crashes:

  1. + > Create expense > Scan.
  2. Tap the sparkles button (off > on) and immediately tap Manual tab
  3. Tap Scan tab
  4. Tap the sparkles button (on > off).
  5. Tap the sparkles button (off > on) again and immediately tap Manual tab.

Repeat Step 3-5 a few times if crash doesn't happen

The crash came from key={transactionRequestType} on the Scan tab. Every time the tab was switched, the whole Scan screen remounted and set up a new camera frame output that never got cleaned up, so memory kept climbing until iOS killed the app. Fixed by removing the key and resetting multi-scan state explicitly when the request type changes.

# Conflicts:
#	src/pages/iou/request/step/IOURequestStepScan/components/Camera/index.native.tsx
#	src/pages/iou/request/step/IOURequestStepScan/components/MultiScanContext.tsx
@OlimpiaZurek

Copy link
Copy Markdown
Contributor Author

@OlimpiaZurek let's address this comment #105773 (comment)

Done

@situchan

Copy link
Copy Markdown
Contributor

@AndrewGable or @luacmartins please generate new build

@github-actions

Copy link
Copy Markdown
Contributor

🚧 AndrewGable has triggered a test Expensify/App build. You can view the workflow run here.

@github-actions

Copy link
Copy Markdown
Contributor

@luacmartins

Copy link
Copy Markdown
Contributor

@situchan how's review going?

updateOutline(corners, detectedEdge, toCameraPoint, detectedAt);
};

const frameOutput = useFrameOutput({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The frame output leak is still there, only the tab trigger is gone

In VisionCamera 5.2.3, useFrameOutput creates the frame output and its worklet runtime in useMemo and has no unmount cleanup.
setOnFrameCallback(undefined) is never called, and with bundleMode: true each runtime also evaluates the bundle.
Removing the key stops the remount on tab switch, but every other mount of DocumentDetection still leaks one runtime: closing and reopening Create expense, pushing a second Scan screen (Replace receipt, Add receipt), or opening Scan from another entry point.

  1. + > Create expense > Scan
  2. Close Create expense
  3. Repeat steps 1 and 2 about 20 times

Expected: memory stays flat, as with the beta off.
Actual: memory grows with each open. This is the same mechanism as the tab-switch crash.

I haven't measured how many opens it takes to crash. Could you run this loop the same way you measured the tab case?

Comment on lines +375 to +381
detection && shouldDetectDocument && !didCapturePhoto ? (
<>
<DocumentOutline
edge={detection.edge}
isHidden={detection.isMoving}
/>
{isReceiptEnhancementToastVisible && <ReceiptEnhancementToast onHide={hideReceiptEnhancementToast} />}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The toast shows again after a tab round trip (iOS)

Scan now stays mounted across tabs.
On blur the iOS preview stops, the overlay unmounts and its 2 second timer is cleared, but the flag stays true.

  1. + > Create expense > Scan
  2. Tap the sparkles button
  3. Within 2 seconds, tap Manual
  4. Tap Scan

Expected: no toast.
Actual: "Receipt enhancements enabled" shows again.

Comment on lines +497 to +501
if (!isEnabled) {
if (shownEdge.getDirty() !== null) {
setOutline(null);
}
return;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous receipt's outline shows on a new receipt (iOS)

outlineState now survives a tab switch, and the first iOS frame after the switch reuses it.
The !isEnabled branch only calls setOutline(null), so turning enhancements off and on keeps it too, on iOS and Android.

  1. On Scan, hold still on a crumpled receipt until the outline follows the folds
  2. Tap Manual
  3. Put a flat receipt in the same spot
  4. Tap Scan

Expected: four straight sides.
Actual: the old folds show for about 0.7 seconds, then the outline morphs flat.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants