Performance challenges with large tenants #1236

Creating this issue as initial placeholder to address several known challenges with very large tenants.
The issues are sourced from real world large tenant experiences. This work item must not be addressed directly, but rather used as base for creating actual work items to address problematic areas.

The analysis showed several areas worth applying new approaches for:

$skiptoken expiration

Microsoft Graph documentation explicitly makes a reference to dedicated error when paging through vast amount of data: https://learn.microsoft.com/en-us/graph/paging?tabs=http#error-handling
The DirectoryPageTokenNotFoundException must be accounted for, handled and the whole process of grabbing data restarted.
Additionally, MS Graph can, without specifying any concrete reason, expire $skipToken - especially if there was too long time between calling the individual pages.
Any retries that happen, must account for the $skipToken change, and as per documentation, use the value from last successful response, instead of the original one, if that failed.

retry back-off

The analysis shows an exponential retry back off of 3s, 6s, 12s, 24s, 48s - come from the default start setting of 3 seconds x 5 retries.
We must keep track of whether this is a repeatable pattern with large tenants, or an isolated case. If repeatable, it is worth considering making the default larger - e.g. 5 or 6 seconds. Because if there are some real intermittent issues, better to wait 2 seconds more for the first retry, then risking entering throttling loop because of too short base wait.

Keeping this for deeper analysis.

Here is a comprehensive analysis of one particular failure:

Support Package Analysis

Time Range: 2026-05-12 23:12:24 → 2026-05-22 00:24:49 (9 days, 1 hour, 12 minutes)
Environment: PS 7.5.5 on Microsoft Windows 10.0.26200
Module Version: ZeroTrustAssessment 2.2.50

Summary

• Total log messages: 1,831
• Errors: 2 | Warnings: 37 | Critical: 0

Issues Found

Issue 1: Authentication Failures (Initial)

• Severity: Error + Warning (Issues 1–4)
• Time: Session start
• Function: Connect-ZtAssessment
• Source: Connect-ZtAssessment.ps1
• Message: InteractiveBrowserCredential authentication failed: User canceled authentication
• Root Cause: The user closed the browser window on the first authentication attempt. They subsequently retried and connected successfully (the assessment proceeded).
• Impact: None — transient user action, assessment continued.

Issue 2: Sign-In Log Size Limit Hit

• Severity: Warning (Issue 5)
• Time: During export phase
• Function: Export-ZtGraphEntity
• Source: Export-ZtGraphEntity.ps1 (~line 226)
• Message: Sign-in log export reached size limit of 1024 MB (current: 1025.71 MB)
• Root Cause: Large tenant with a very high volume of sign-in logs. The 1 GB cap was reached.
• Impact: Low — sign-in log export was truncated but assessment continues. This is expected behavior.
• Fix: If more sign-in data is needed, increase ZeroTrustAssessment.Export.SignInLog.MaxSizeBytes.

Issue 3: Skip Token Expiration — ServicePrincipalSignIn Export Failed

• Severity: Warning (Issues 9–12)
• Time: ~52 minutes into export
• Function: Invoke-ZtRetry → Export-ZtGraphEntity → Invoke-ZtTenantDataExport
• Source: Export-ZtGraphEntity.ps1 (~line 190), Invoke-ZtRetry.ps1
• Message: Skip token is null. It may have a typo or it has expired. (HTTP 400 on page 53)
• Root Cause: Graph API skip tokens are time-bound. Repeated stream errors and timeouts during pagination slowed the export enough for the token to expire server-side. The 400 was correctly marked as non-retryable by Test-ZtRetryableError — retrying the same expired token would fail again.
• Impact: ServicePrincipalSignIn export failed (52m57s wasted).

Issue 4: ServicePrincipal Null Reference

• Severity: Warning (Issues 14–15)
• Time: After ~1h25m of export
• Function: Invoke-ZtTenantDataExport → Wait-ZtTenantDataExport
• Source: Invoke-ZtTenantDataExport.ps1, Wait-ZtTenantDataExport.ps1
• Message: You cannot call a method on a null-valued expression
• Root Cause: Likely a cascading failure — a variable was null due to a prior failed network operation. The sustained network instability (stream errors + DNS failures) caused an upstream operation to return null, and the ServicePrincipal export called a method on it.
• Impact: ServicePrincipal export failed (1h25m42s wasted).

Issue 5: Sustained HTTP Timeouts During Application Export

• Severity: Warning (Issues 16–26)
• Time: ~03:00–04:00 UTC on May 13
• Function: Invoke-ZtRetry
• Source: Invoke-ZtRetry.ps1
• Message: The request was canceled due to the configured HttpClient.Timeout of 300 seconds elapsing
• Root Cause: Graph API responses for Application pages (520s–530s of 999-item pages) consistently exceeded the 300s timeout. This is a very large tenant. Each timeout added 5+ minutes of delay, compounding the pagination problem.
• Impact: Contributed to the Application export eventually failing via DNS failure.

Issue 6: DNS Resolution Failure — Application Export Failed

• Severity: Warning (Issues 27–34)
• Time: 04:12–04:23 UTC on May 13
• Function: Invoke-ZtRetry → Export-ZtGraphEntity → Wait-ZtTenantDataExport
• Source: Invoke-ZtRetry.ps1, Export-ZtGraphEntity.ps1
• Message: The requested name is valid, but no data of the requested type was found. (graph.microsoft.com:443) — 5 consecutive retries exhausted
• Root Cause: DNS could not resolve graph.microsoft.com. This was a sustained outage lasting ~11 minutes. All 5 retry attempts (with exponential backoff: 3s, 6s, 12s, 24s, 48s) failed.
• Impact: Application export failed on page 532 after 5h7m of export time. All progress lost.

Issue 7: DNS Failure During Test Phase

• Severity: Warning (Issues 35–39)
• Time: 04:32 UTC on May 13 (during Stage 2: Running Tests)
• Function: Invoke-ZtRetry (called from Invoke-ZtGraphRequestCache for /v1.0/organization)
• Source: Invoke-ZtRetry.ps1
• Message: Same DNS resolution failure, 5 consecutive attempts failed
• Root Cause: DNS outage persisted into the test phase. The assessment continued running tests but any test needing live Graph calls would have failed.
• Impact: Tests dependent on live Graph API calls during this window likely produced incorrect results.

Failed Exports

Successful Exports (13 of 16)

Device, SignIn, ConfigurationPolicy, RoleAssignmentScheduleInstance, RoleAssignmentScheduleInstanceGroup, RoleEligibilityScheduleInstance, RoleManagementPolicyAssignment, RoleDefinition, UserRegistrationDetails, RoleAssignment, User, RoleEligibilityScheduleInstanceGroup, RoleAssignmentGroup

Root Cause Analysis

The dominant issue is network instability on the customer's machine. Three distinct network failure modes were observed during the assessment:

1. Stream copy errors (5 occurrences) — connections dropped mid-transfer, all recovered on retry
2. HTTP timeouts (10 occurrences) — 300s timeout exceeded, indicating either slow Graph API responses for this large tenant or network latency
3. DNS resolution failures (13 occurrences) — sustained inability to resolve graph.microsoft.com, lasting ~11+ minutes, which was the fatal blow that killed the Application export after 5 hours of work

The cascade was: stream errors + timeouts slowed pagination → skip token expired for ServicePrincipalSignIn → DNS outage killed Application and affected the test phase. The ServicePrincipal null-reference failure is likely also network-related.

This is a very large tenant (~530K+ applications, 1 GB+ of sign-in logs), which makes exports slow and amplifies the impact of any network disruption.

Recommendations

1. Re-run the assessment during a stable network window — the DNS outage at ~04:12 UTC was the primary cause of failure. Ensure reliable DNS resolution and network connectivity before starting.
2. Check DNS and proxy configuration — verify the customer's DNS can reliably resolve graph.microsoft.com. Check for any corporate proxy, VPN, or firewall that might cause intermittent DNS failures.
3. Run during off-peak hours — the HTTP timeouts on Application pages suggest Graph API is responding slowly for this tenant. Off-peak hours may yield faster responses.
4. Consider wired network — if running on WiFi, switch to wired to reduce stream copy errors.
5. The initial auth failures are benign — the user simply cancelled the browser prompt once and retried successfully.
6. Sign-in log 1 GB limit is expected — if the full sign-in dataset is needed, set ZeroTrustAssessment.Export.SignInLog.MaxSizeBytes to a higher value, but note this will increase export time significantly.