iOS & Android integration tests by jpnurmi · Pull Request #4559 · getsentry/sentry-dotnet

jpnurmi · 2025-09-24T14:21:48Z

Helps verify that crash events are captured as expected after Cocoa and Java SDK updates (and eventually getsentry/sentry-cocoa#6193):

CrashType.Managed
CrashType.Native
NullReferenceException

For example, see

51240026268 (iOS Device Tests -> Run Integration Tests)
51593510741 (Android Device Tests -> Run Integration Tests)

#skip-changelog

Note

Adds MAUI-based Android and iOS integration tests executed on emulators/simulators in CI, with supporting test app, utilities, and workflow updates (including retries and result artifacts).

CI/Workflows:
- Android: run new integration tests on emulator via integration-test/android.Tests.ps1 with retry; checkout submodules; set up Java 17; fetch shared github-workflows; reuse emulator config via YAML anchors; upload both test_output and integration-test/mobile-app/test_output.
- iOS: add integration tests via integration-test/ios.Tests.ps1 with retry; upload both result paths.
- Build: include scripts in sparse checkout for publishing builds.
Tests/Infrastructure:
- Add MAUI test app (integration-test/net9-maui/**) targeting net9.0-android35.0 and net9.0-ios18.0 to drive crash scenarios and verify envelopes.
- New Pester test suites: integration-test/android.Tests.ps1, integration-test/ios.Tests.ps1; shared helpers in integration-test/pester.ps1.
- Utilities: new scripts/device-test-utils.ps1 (XHarness install, emulator/simulator helpers); scripts/device-test.ps1 now uses these helpers and simplifies simulator selection.
- Refactor integration-test/common.ps1 to source shared Pester extensions.

^{Written by Cursor Bugbot for commit 8144e87. This will update automatically on new commits. Configure here.}

codecov · 2025-09-24T14:35:29Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.46%. Comparing base (b3d9b2f) to head (fed73a4).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4559      +/-   ##
==========================================
- Coverage   73.50%   73.46%   -0.04%     
==========================================
  Files         482      482              
  Lines       17678    17678              
  Branches     3493     3493              
==========================================
- Hits        12994    12988       -6     
- Misses       3796     3800       +4     
- Partials      888      890       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

To make room for Android potentially coming later.

…ation tests

jpnurmi · 2025-09-25T13:08:55Z

@sentry review

Co-authored-by: Stefan Pölz <38893694+Flash0ver@users.noreply.github.com>

jamescrosswell · 2025-10-06T21:02:19Z

+    # by Android Device Tests, or manually when testing locally. This avoids
+    # slowing down non-Device Test CI builds further.
+    Install-XHarness
+    $script:emulator = Get-AndroidEmulatorId


TIL about $script scoped variables in PWSH!

jamescrosswell

Left one non-blocking suggestion, but otherwise LGTM 🎉

Technically, less correct but doesn't fail even if port forwarding had failed. This way, we can keep it simple and don't need to track it separately.

jpnurmi · 2025-10-07T11:13:14Z

Thanks for testing and helping!

Co-authored-by: Stefan Pölz <38893694+Flash0ver@users.noreply.github.com> Co-authored-by: James Crosswell <jamescrosswell@users.noreply.github.com>

Each Android emulator job ran the device tests, then Setup Environment, then the integration tests, about 22 minutes after waiting 7.5 for the device test app builds. The integration tests build their own app from source and share only the emulator, so they now run in a second job that starts with the workflow, on the same runners. The new job drops "Select Java 21". #4559 added it as "Select Java 17", when Setup Environment ended on Java 11, and since #5060 Setup Environment installs only Java 21. The new job uploads integration-test/mobile-app/test_output, where android.Tests.ps1 points xharness, without a warning when it's empty. It was empty on every run so far. On its first run, the Android workflow took 19.8 minutes instead of 30.1. The device test jobs took 3.6 to 3.8 minutes and the integration jobs 12.0 to 19.7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run the iOS integration tests in their own job The iOS job ran the device tests and then the integration tests on one Mac, about 42 minutes in all. The two build and run different apps and share only the simulator, so the integration tests now run in a second job at the same time. ios.Tests.ps1 skips every case unless a simulator is already booted, and xharness only boots one in the ios-tests job, so the new job boots its own, with a 10-minute timeout. The boot takes about 2 minutes. The job uploads no results, because ios.Tests.ps1 writes nothing to the integration-test/mobile-app/test_output folder the old job uploaded. On its first run, the iOS workflow took 18 minutes, against 38.7 for the single job the same night, and all 8 integration cases passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run the Android integration tests in their own job Each Android emulator job ran the device tests, then Setup Environment, then the integration tests, about 22 minutes after waiting 7.5 for the device test app builds. The integration tests build their own app from source and share only the emulator, so they now run in a second job that starts with the workflow, on the same runners. The new job drops "Select Java 21". #4559 added it as "Select Java 17", when Setup Environment ended on Java 11, and since #5060 Setup Environment installs only Java 21. The new job uploads integration-test/mobile-app/test_output, where android.Tests.ps1 points xharness, without a warning when it's empty. It was empty on every run so far. On its first run, the Android workflow took 19.8 minutes instead of 30.1. The device test jobs took 3.6 to 3.8 minutes and the integration jobs 12.0 to 19.7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

WIP: iOS integration test

46bbbfe

jpnurmi added 22 commits September 24, 2025 17:16

Split common.ps1 vs. setup.ps1

43b57c7

Fix empty --device argument

d79436c

Temp speed up

a916900

Try BeforeAll

67fd410

Restore, retry, artifacts

83c3084

Add test/Sentry.Maui.Device.IntegrationTestApp

6cc7696

Update solution filter

abdc057

Speed up

7ac73d0

Partial revert

adf7b68

Managed vs. Native vs. NullRefenceException

e190195

Fixup

aff8901

Move to integration-test/ios.Tests.ps1

29bef65

Fixup

2c516c2

Use shared Install-XHarness

30b08fd

Fix Install-XHarness

6389645

Add RunTestApp helper func

44a190e

Fix Install-XHarness - attempt 2

26e6d05

Prefer booted sims

5f6eb95

Convert tabs to spaces

28a57ae

Fix PreferredStates

ba70ab2

Fix Install-XHarness

64d2827

Rename to mobile.Tests.ps1 and prepare iOS context

90cda33

To make room for Android potentially coming later.

jpnurmi mentioned this pull request Sep 25, 2025

fix: AOT interop with managed .NET runtimes getsentry/sentry-cocoa#6193

Merged

7 tasks

jpnurmi added 4 commits September 25, 2025 14:56

Leave scripts/device-test-utils.ps1 in the sparse checkout for integr…

c58bd4f

…ation tests

Add missing timeout

5e01b48

Fix Android tfm

8121c4c

Prepare predictable Activity name on Android

d508359

jamescrosswell and others added 2 commits October 6, 2025 15:48

Update android.Tests.ps1

7089ea5

Get-AndroidEmulatorId: fix env:ANDROID_SERIAL check

18c7fc8

jpnurmi marked this pull request as ready for review October 6, 2025 08:46

This comment was marked as outdated.

Sign in to view

jpnurmi requested review from Flash0ver and jamescrosswell October 6, 2025 09:29

Flash0ver reviewed Oct 6, 2025

View reviewed changes

Comment thread .github/workflows/device-tests-android.yml Resolved

Apply suggestion from @Flash0ver

4e2cf5d

Co-authored-by: Stefan Pölz <38893694+Flash0ver@users.noreply.github.com>

This comment was marked as outdated.

Sign in to view

Merge remote-tracking branch 'upstream/main' into test/ios-integration

c26d952

jamescrosswell reviewed Oct 6, 2025

View reviewed changes

Comment thread integration-test/android.Tests.ps1 Resolved

jamescrosswell approved these changes Oct 6, 2025

View reviewed changes

Try-finally for adb reverse

a8c25a7

This comment was marked as outdated.

Sign in to view

Use --remove-all for port-forwarding cleanup

fed73a4

Technically, less correct but doesn't fail even if port forwarding had failed. This way, we can keep it simple and don't need to track it separately.

jpnurmi merged commit e58a8f1 into main Oct 7, 2025
32 checks passed

jpnurmi deleted the test/ios-integration branch October 7, 2025 11:13

alexsohn1126 pushed a commit that referenced this pull request Oct 11, 2025

iOS & Android integration tests (#4559)

61b76bf

Co-authored-by: Stefan Pölz <38893694+Flash0ver@users.noreply.github.com> Co-authored-by: James Crosswell <jamescrosswell@users.noreply.github.com>

This was referenced Oct 20, 2025

test: add missing native crash integration test for Android #4659

Merged

fix: AOT interop with managed .NET runtimes getsentry/sentry-native#1392

Merged

test: run mobile integration tests in Release and Debug modes #4666

Merged

Flash0ver mentioned this pull request Nov 11, 2025

bump to .NET 10.0.100 (GA) #4723

Merged

jamescrosswell mentioned this pull request Nov 17, 2025

MAUI/Mobile integration tests for .NET 10 #4744

Closed

This was referenced Oct 24, 2026

ci: run the Android integration tests in their own job #5800

Closed

ci: run the iOS and Android integration tests in their own jobs #5810

Merged

Navigation Menu

Repository navigation

iOS & Android integration tests - #4559

iOS & Android integration tests#4559
jpnurmi merged 58 commits into
mainfrom
test/ios-integration

jpnurmi commented Sep 24, 2025 •

edited

Loading

Uh oh!

codecov Bot commented Sep 24, 2025 •

edited

Loading

Uh oh!

jpnurmi commented Sep 25, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

jamescrosswell Oct 6, 2025

Uh oh!

Uh oh!

jamescrosswell left a comment

Uh oh!

This comment was marked as outdated.

Uh oh!

jpnurmi commented Oct 7, 2025

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

jpnurmi commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jpnurmi commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

jamescrosswell Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jamescrosswell left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

jpnurmi commented Oct 7, 2025

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jpnurmi commented Sep 24, 2025 •

edited

Loading

codecov Bot commented Sep 24, 2025 •

edited

Loading

jpnurmi commented Sep 25, 2025 •

edited

Loading