Skip to content

fix(outlook): dispatch Contacts by item type and harden folder/field assumptions - #4147

Merged
Jan-Kazlouski-elastic merged 10 commits into
mainfrom
jan-kazlouski/outlook-fix
Jul 10, 2026
Merged

Jan-Kazlouski-elastic merged 10 commits into
mainfrom
jan-kazlouski/outlook-fix

Conversation

@Jan-Kazlouski-elastic

@Jan-Kazlouski-elastic Jan-Kazlouski-elastic commented Jul 9, 2026 •

Copy link
Copy Markdown
Contributor

Related to https://github.com/elastic/sdh-search/issues/1898

Follow-up to #4123. The last month has been a series of "the next malformed item aborts the whole sync" reports, most recently 'DistributionList' object has no attribute 'email_addresses'. This PR fixes that class of failure with explicit, visible handling, and audits the connector for the remaining assumptions of the same shape.

Root cause

The Contacts folder returns two item models — Contact and DistributionList — but the formatter assumed every item was a Contact and read per-contact fields off it, so a single contact group aborted the entire sync. The same "unhandled shape aborts everything" gap existed elsewhere: Calendar/Tasks folders that shared and resource mailboxes can legitimately lack, birthday events without a start, and an optional LDAP type key.

Changes

Contacts formatted by item type. _fetch_contacts now dispatches on type: a Contact goes through the strict contact_doc_formatter and a contact group through a dedicated distribution_list_doc_formatter that indexes it by name and its members' email addresses (typed "Distribution List"). The folder query fetches the union of both models' fields (incl. members). The Contacts folder contractually returns only these two models, so any other type raises a TypeError rather than being silently skipped — an unknown type is a broken assumption we want surfaced, not a whole category of items quietly dropped. (Thanks @artem-shelkovnikov for the review — explicit type handling over defensive getattr, and failing on unexpected types.)

Note on approach. An earlier revision of this PR wrapped all formatting in a broad _format_item guard that caught item-shape errors (AttributeError/KeyError/ValueError/TypeError) and skipped the item. That was dropped: swallowing those errors around every item can turn a systematic bug into a "successful" sync that returns 0 items and deletes previously indexed documents. Instead, known shapes are handled explicitly (type dispatch + targeted field guards) and genuine errors still fail the sync loudly — consistent with the principle established in #4085/#4123.

Remaining assumptions found in the audit, now fixed:

  • Calendar & Tasks folders are now skipped on ErrorFolderNotFound, mirroring the existing Contacts/mail-folder handling. Shared and resource mailboxes can legitimately lack these folders; previously that aborted the sync (same class as Fix Outlook connector crash on localized Exchange servers (contacts folder) #4065).
  • Birthdays calendar branch called .split("T") on the formatted datetime, which is None when calendar.start is missing → AttributeError. Now guarded.
  • Optional LDAP type key is read with .get(...) instead of user["type"].

Test plan

  • pytest tests/sources/test_outlook.py (65 tests), incl. coverage for:
    • contact / DistributionList type dispatch and the group formatter
    • unexpected Contacts item type → raises TypeError
    • ErrorFolderNotFound skips for Calendar / child calendars / Tasks
    • birthday without a start
    • missing LDAP type key
  • ruff check / ruff format --check

Relates to elastic/sdh-search#1898

The Exchange Contacts folder can also contain DistributionList (contact
group) items, which do not carry the per-contact fields the formatter
reads (email_addresses, phone_numbers, company_name, birthday). Accessing
those attributes raised 'DistributionList' object has no attribute
'email_addresses' and aborted the entire sync.

Read the per-contact fields defensively so a single contact group is
indexed by name instead of crashing the sync.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment on lines +148 to +150
# The Contacts folder can also return contact groups (DistributionList),
# which lack the per-contact fields below. Read them defensively so a
# single group no longer aborts the whole sync with an AttributeError.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically this change says different:

If an object returned by contacts does not have some attributes, skip them.

I'm wondering if there are other types of objects that can come here, and will these objects also have display_name?

Also I'm wondering if we should instead check the object type and return formatted documents based on type of incoming object - it will make it easier to comprehend why some objects do not have all fields and which fields are supported per object.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call, agreed on both points — I've reworked this to dispatch on item type instead.

To your questions: a Contacts-folder query in exchangelib returns exactly two item models — Contacts.supported_item_models == (Contact, DistributionList) — and both inherit display_name (plus id/last_modified_time) from the base Item, so the shared fields are always safe. DistributionList just doesn't have the per-contact fields (email_addresses, phone_numbers, company_name, birthday); instead it has members.

Changes in a9fb212:

  • Reverted contact_doc_formatter to strict Contact access (no more getattr), so it documents the real Contact schema.
  • Added a dedicated distribution_list_doc_formatter that indexes the group by name and its members' email addresses, typed as "Distribution List".
  • Dispatch by isinstance in _fetch_contacts, and the query now fetches the union of both models' fields (incl. members).

The general per-item _format_item guard stays as defense-in-depth for genuinely unexpected shapes, but the contact path is now explicit about which fields belong to which type.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update:

Removed _format_item entirely rather than keeping it as defense-in-depth.

…ield assumptions

Relates to elastic/sdh-search#1898

Instead of hardening one field at a time, isolate item-shape failures so a
single malformed Exchange object no longer aborts the whole sync:

- Add a per-item guard (_format_item) around mail, contact, task, calendar,
  and attachment formatting that skips and logs an item on shape errors
  (AttributeError/KeyError/ValueError/TypeError) while letting connection-wide
  errors (transport/SSL/rate-limit/missing mailbox) still fail the sync loudly.
- Skip Calendar and Tasks folders on ErrorFolderNotFound, mirroring the
  existing Contacts and mail-folder handling (shared/resource mailboxes may
  lack these folders).
- Guard the Birthdays calendar branch so a missing start no longer raises
  on the .split() of a None datetime.
- Read the optional LDAP "type" key defensively.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Jan-Kazlouski-elastic Jan-Kazlouski-elastic changed the title fix(outlook): handle contact groups (DistributionList) in the Contacts folder fix(outlook): isolate per-item failures so one malformed Exchange item can't abort the sync Jul 9, 2026
Jan-Kazlouski-elastic and others added 3 commits July 9, 2026 12:33
…getattr

Addresses review feedback on #4147: rather than reading contact fields
defensively (which silently tolerated any missing attribute), dispatch on the
item type returned by the Contacts folder, which exchangelib guarantees is one
of Contact or DistributionList.

- Restore the strict Contact formatter (documents the real Contact schema).
- Add a dedicated distribution_list_doc_formatter that indexes a contact group
  by name and its members' email addresses, typed as "Distribution List".
- Dispatch by isinstance in _fetch_contacts; the general _format_item guard
  remains as defense-in-depth for genuinely unexpected shapes.
- Fetch the union of Contact and DistributionList fields for the folder query.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
):
# Skip instead of aborting if the mailbox has no Calendar folder.
try:
folder = account.calendar

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sanity check - this does not produce network call, folder.all() does, right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, actually it's the other way around:
account.calendar makes the network call. It is the same as self.root.get_default_folder(Calendar), which calls GetFolder and raises ErrorFolderNotFound.

folder.all() on the other hand is a lazy QuerySet. No network call until it gets iterated, and no ErrorFolderNotFound. So we should be good here.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, then there is a problem here. exchangelib is not async, and the call to account.calendar is gonna block the whole process until it resolves. Then we'd need to wrap it into asyncio.to_thread or something similar

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This then will have to be done to the methods that do network calls below unfortunately

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. You're right. Done.

distribution_list=contact,
timezone=timezone,
)
else:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally like doing explicit type checks and warn/raise on the case when the type is unexpected.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

"""
yield content

def _format_item(self, formatter, item, account):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This actually has tendency to actually produce syncs that are successful but return 0 items. I'm not really sure we should do this.

We've tried an approach with this class:

Error Monitor swallows errors up to a certain threshold and then raises an exception if too many errors happen. It also has its downsides.

@artem-shelkovnikov artem-shelkovnikov Jul 9, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Continuing the train of thought from above:

I feel this method attempt to help not raise when formatting objects of unexpected types. E.g. you have an API returning types contact and distribution_list and you format them well, but then for some niche case the customer gets an error because an object of type broadcast_group (imaginary name) is returned and the method breaks.

So instead each formatter method can do type checks and warn or error if type is unexpected.

Warn or error? Not sure. It's all about expectation setting to me.

Warns are good because the sync does not crash
Errors are good because they tell what went wrong immediately, so you will know that something is not being synced.

Because of that Error Monitor will not actually help, but will just hide the problem.

All in all I think it's best to do type checks in formatter methods and warn on unexpected types. I am open to a discussion here, though

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Fixed. Now we're raising instead of silently skipping.

…e checks

Per review feedback on #4147, the broad _format_item guard swallowed
item-shape errors (AttributeError/KeyError/ValueError/TypeError) around every
formatter call. That can turn a systematic formatting bug into a "successful"
sync that returns 0 items and deletes previously indexed documents. Remove it
in favor of explicit, visible handling:

- Dispatch _fetch_contacts on item type: Contact -> contact_doc_formatter,
  DistributionList -> distribution_list_doc_formatter, and warn + skip any
  unexpected type instead of forcing it through a formatter.
- Call the mail/task/calendar/attachment formatters directly again, so a
  genuine formatting error fails loudly rather than silently emptying the index.

The folder guards (ErrorFolderNotFound on Calendar/Tasks), field-level null
guards (sender/organizer/attendees/birthday), the optional LDAP type key, and
the distribution list formatter remain.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Jan-Kazlouski-elastic Jan-Kazlouski-elastic changed the title fix(outlook): isolate per-item failures so one malformed Exchange item can't abort the sync fix(outlook): dispatch Contacts by item type and harden folder/field assumptions Jul 9, 2026
Jan-Kazlouski-elastic and others added 2 commits July 9, 2026 15:54
…skipping

The Contacts folder contractually returns only Contact or DistributionList, so
an unknown type is a systematic broken-assumption condition, not per-item data
variance. Warning and skipping would silently drop a whole category of items;
raise a TypeError instead so the sync fails loudly and the gap is fixed in code.

Co-authored-by: Cursor <cursoragent@cursor.com>
exchangelib is synchronous, so resolving a distinguished folder (e.g.
account.calendar) issues a blocking GetFolder network call. Running it
directly in the async get_* methods blocked the event loop until it returned.

Wrap the folder resolution in asyncio.to_thread in get_mails, get_calendars,
get_child_calendars, get_tasks and get_contacts so it runs off the event loop.
ErrorFolderNotFound raised in the thread still propagates out of the await, so
the existing skip-and-continue handling is unchanged.

Also add tests asserting resolution is offloaded and that the Contacts query
requests DistributionList fields (members).

Co-authored-by: Cursor <cursoragent@cursor.com>

@artem-shelkovnikov artem-shelkovnikov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@Jan-Kazlouski-elastic
Jan-Kazlouski-elastic enabled auto-merge (squash) July 10, 2026 14:03
@Jan-Kazlouski-elastic
Jan-Kazlouski-elastic merged commit 2285291 into main Jul 10, 2026
4 checks passed
@Jan-Kazlouski-elastic
Jan-Kazlouski-elastic deleted the jan-kazlouski/outlook-fix branch July 10, 2026 14:40
@github-actions

Copy link
Copy Markdown

💔 Failed to create backport PR(s)

Status Branch Result
✅ 9.4 #4151
✅ 9.3 #4152
❌ 9.6 The branch "9.6" is invalid or doesn't exist

Successful backport PRs will be merged automatically after passing CI.

To backport manually run:
backport --pr 4147 --autoMerge --autoMergeMethod squash

Jan-Kazlouski-elastic added a commit that referenced this pull request Jul 10, 2026
…field assumptions (#4147) (#4151)

Backports the following commits to 9.4:
- fix(outlook): dispatch Contacts by item type and harden folder/field
assumptions (#4147)

---------

Co-authored-by: Jan-Kazlouski-elastic <jan.kazlouski@elastic.co>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Jan-Kazlouski-elastic added a commit that referenced this pull request Jul 10, 2026
…field assumptions (#4147) (#4152)

Backports the following commits to 9.3:
- fix(outlook): dispatch Contacts by item type and harden folder/field
assumptions (#4147)

---------

Co-authored-by: Jan-Kazlouski-elastic <jan.kazlouski@elastic.co>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Jan-Kazlouski-elastic added a commit that referenced this pull request Jul 13, 2026
…/field assumptions (#4147) (#4153)

Backports #4147 to `8.19`.

Related to elastic/sdh-search#1898

## Summary
- Dispatch the Contacts folder by item type: a `Contact` goes through
the strict `contact_doc_formatter`, a contact group through a new
`distribution_list_doc_formatter` (indexed by name + members' emails,
typed `"Distribution List"`). The folder query now fetches the union of
both models' fields (incl. `members`). Any other type raises a
`TypeError` rather than aborting the sync with an opaque
`'DistributionList' object has no attribute 'email_addresses'`.
- Skip Calendar and Tasks folders on `ErrorFolderNotFound`, mirroring
the existing Contacts/mail-folder handling (shared/resource mailboxes
may lack these folders).
- Resolve distinguished folders off the event loop via
`asyncio.to_thread` in
`get_mails`/`get_calendars`/`get_child_calendars`/`get_tasks`/`get_contacts`
(exchangelib is synchronous).
- Guard the Birthdays branch so a missing `calendar.start` no longer
raises on `.split()` of a `None`.
- Read the optional LDAP `type` key with `.get(...)`.

## Backport notes
- The 8.19 branch still uses the monolithic
`connectors/sources/outlook.py` layout (rather than the `outlook/`
package split into `client.py`/`datasource.py` on `main`), so the
changes were manually adapted to that single module. Behaviour is
identical to the original PR.
- The incidental `NOTICE.txt` dependency bump from the original PR
(unrelated `stone`/`tzdata` version bumps that came from a main merge)
is intentionally excluded.

## Test plan
- [x] `pytest tests/sources/test_outlook.py` (71 passed)
- [x] `ruff check` / `ruff format --check`

Made with [Cursor](https://cursor.com)

Co-authored-by: Cursor <cursoragent@cursor.com>
Jan-Kazlouski-elastic added a commit that referenced this pull request Jul 13, 2026
…field assumptions (#4147) (#4154)

Backports #4147 to `9.5`.

Related to elastic/sdh-search#1898

## Summary
- Dispatch the Contacts folder by item type: a `Contact` goes through
the strict `contact_doc_formatter`, a contact group through a new
`distribution_list_doc_formatter` (indexed by name + members' emails,
typed `"Distribution List"`). The folder query fetches the union of both
models' fields (incl. `members`). Any other type raises a `TypeError`
rather than aborting the sync with an opaque `'DistributionList' object
has no attribute 'email_addresses'`.
- Skip Calendar and Tasks folders on `ErrorFolderNotFound`, mirroring
the existing Contacts/mail-folder handling.
- Resolve distinguished folders off the event loop via
`asyncio.to_thread` (exchangelib is synchronous).
- Guard the Birthdays branch so a missing `calendar.start` no longer
raises on `.split()` of a `None`.
- Read the optional LDAP `type` key with `.get(...)`.

## Backport notes
- Clean cherry-pick of `2285291`; `9.5` shares main's split `outlook/`
package layout.
- The incidental `NOTICE.txt` dependency bump from the original PR is
intentionally excluded.
- Auto-backport skipped 9.5 because `.backportrc.json` still maps
`^v9.5.0$` → `main`, which is stale now that `main` is `9.6.0` and `9.5`
was cut into its own branch. This is a bug fix, still appropriate during
the 9.5.0 feature freeze.

## Test plan
- [x] `pytest tests/sources/test_outlook.py` (71 passed)
- [x] `ruff check` / `ruff format --check`

Made with [Cursor](https://cursor.com)

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Jan-Kazlouski-elastic added a commit that referenced this pull request Jul 16, 2026
…ror handling (#4158)

Related to elastic/sdh-search#1898

Follow-up to #4147. That PR fixed the Contacts folder's
single-item-model assumption; this one applies the same "one malformed
item/folder must not abort the whole sync" principle to every remaining
folder-iteration and error-handling site the audit surfaced, most
recently the `'CalendarItem' object has no attribute 'sender'` /
localized-`Archive` class of reports.

## Root cause

The connector iterated Exchange folders assuming each folder returns
exactly one item model, that every distinguished folder resolves, and
that the only recoverable per-folder fault is `ErrorFolderNotFound`.
Real mailboxes violate all three: a Calendar/Tasks/mail folder can
contain stray item types, a child folder under Calendar may not be a
calendar folder (so the `CALENDAR_FIELDS` projection raises
`ValueError`), folder resolution can fail with
`ErrorManagedFolderNotFound`, an inaccessible mailbox raises
`ErrorAccessDenied`, an attachment can be an embedded `ItemAttachment`
(no `content`) or lack a `size`, and the timezone fallback was a bare
string that the datetime formatter can never use. Each of these aborted
the entire sync (or mis-typed an item) instead of being handled visibly.

## Changes

**Type allowlists on every folder iteration.** `_fetch_mails`,
`_fetch_tasks`, `_enqueue_calendars`, and `_fetch_contacts` now
route/skip by type: mail folders accept
`Message`/`MeetingRequest`/`MeetingResponse`/`MeetingCancellation`,
Tasks accept `Task`, Calendars accept `CalendarItem`, and Contacts route
`Contact` vs `DistributionList`. A stray type is logged and skipped
rather than crashing on a missing field. Contacts now skip-and-warn on
an unexpected type (rather than raising `TypeError` as in #4147) for
consistency with the other folders — the warning keeps it visible, and a
single odd item no longer aborts the sync.

**Broadened per-folder skips.**
`get_mails`/`get_calendars`/`get_child_calendars`/`get_tasks`/`get_contacts`
skip on `FOLDER_SKIP_ERRORS` (`ErrorFolderNotFound` +
`ErrorManagedFolderNotFound`) instead of only `ErrorFolderNotFound`.

**Child-calendar projection guard.** A non-calendar child folder under
Calendar can't be projected with `CALENDAR_FIELDS` and raises
`ValueError`; that child is now skipped (`CHILD_CALENDAR_SKIP_ERRORS`)
while healthy sibling calendars still sync.

**Account-level access errors.** `get_docs` now skips an account on
`ErrorAccessDenied` in addition to `ErrorNonExistentMailbox`.
Connection-wide `TransportError`s (e.g. TLS failures) still propagate —
silently swallowing them across all accounts would yield a doc-less
"successful" sync that deletes previously indexed documents (the
principle from #4085/#4123).

**Attachment guards.** `get_content` only extracts `FileAttachment`
content (embedded `ItemAttachment` has no `content`) and guards the
optional `size`, which may be `None`.

**Timezone fallback.** Replaced the dead `DEFAULT_TIMEZONE = "UTC"`
string fallback with the real `exchangelib.UTC` (`EWSTimeZone`), so the
`account.default_timezone or UTC` path actually formats datetimes if a
mailbox ever lacks a default timezone.

## Test plan

* `pytest tests/sources/test_outlook.py` (92 tests), incl. coverage for:
* `FOLDER_SKIP_ERRORS` (`ErrorFolderNotFound` +
`ErrorManagedFolderNotFound`) skips across mails / calendars / child
calendars / tasks / contacts
* child-calendar `ValueError` / managed-folder skip, with a healthy
sibling still syncing
* account skip on `ErrorAccessDenied`, and connection-wide errors still
re-raised
* every type in `MAIL_ITEM_TYPES` accepted, and stray non-mail /
non-task / non-calendar / unexpected-contact items skipped-and-warned
  * `get_content` skips `ItemAttachment` and `None` / `0` size
  * UTC timezone fallback when `default_timezone` is missing
* `ruff check` / `ruff format --check`
* 97% line coverage on `client.py` and `datasource.py`; every changed
line exercised

## Release Note

Harden the Outlook connector so unexpected Exchange item types,
unresolvable or inaccessible folders, embedded/size-less attachments,
and missing mailbox timezones are skipped with a warning instead of
aborting the sync.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Jan-Kazlouski-elastic added a commit that referenced this pull request Jul 17, 2026
…ount error handling (#4158) (#4178)

Backports #4158 to `8.19`.

Related to elastic/sdh-search#1898

Follow-up to #4147 / #4153. Auto-backport to 8.19 failed due to
conflicts from the `outlook/` package split on `main`.

## Summary
- Type allowlists on every folder iteration (`Message`/`Meeting*`,
`Task`, `CalendarItem`, `Contact`/`DistributionList`); stray types are
logged and skipped instead of aborting the sync.
- Broaden per-folder skips to `FOLDER_SKIP_ERRORS`
(`ErrorFolderNotFound` + `ErrorManagedFolderNotFound`).
- Skip non-calendar child folders under Calendar and non-mail
`"Archive"` folders before projecting fields.
- Skip accounts on `ErrorAccessDenied` (in addition to
`ErrorNonExistentMailbox`); connection-wide errors still propagate.
- Attachment guards: only extract `FileAttachment` content; skip missing
`name` / `None` size.
- Replace the dead `DEFAULT_TIMEZONE = "UTC"` string fallback with
`exchangelib.UTC`.
- Materialize folder querysets with `list(...)` inside
`asyncio.to_thread` so the blocking EWS fetch stays off the event loop.

## Backport notes
- The 8.19 branch still uses the monolithic
`connectors/sources/outlook.py` layout (rather than the `outlook/`
package on `main`), so the changes were manually adapted to that single
module. Behaviour matches #4158.
- Contacts unexpected-type handling now skip-and-warn (consistent with
other folders), matching main rather than the `TypeError` from #4153.

## Test plan
- [x] `pytest tests/sources/test_outlook.py` (94 passed)
- [x] `ruff check` / `ruff format --check`

Made with [Cursor](https://cursor.com)

Co-authored-by: Cursor <cursoragent@cursor.com>
Jan-Kazlouski-elastic added a commit that referenced this pull request Jul 28, 2026
…sync (#4287)

## Part of elastic/sdh-search#1898

Follow-up to #4158. The reporter's Exchange server aborts every sync
with:

```
ValueError: Item type {http://schemas.microsoft.com/exchange/services/2006/types}EndTimeZone was unexpected in a BaseFolder folder
```

### Root cause

`CALENDAR_FIELDS` asks for `start` and `end`, and exchangelib silently
appends two more fields to that projection — `calendar:StartTimeZone`
and `calendar:EndTimeZone` (`BaseFolder.normalize_fields`). The server
then returns the `EndTimeZone` element as a sibling of the
`CalendarItem` it belongs to, inside the response's item container,
rather than nested within it.

exchangelib treats every element of that container as an item, looks its
tag up in `BaseFolder.ITEM_MODEL_MAP`, and raises `ValueError` when it
is absent. This is the same failure shape as
ecederstrand/exchangelib#877 (`Booking` items on servers with Microsoft
Bookings installed).

Two things make it fatal rather than merely noisy:

* The exception is raised while
`list(folder.all().only(*CALENDAR_FIELDS))` is materialised in
`get_calendars`, so the per-folder type allowlists added in #4158 never
see any items — the entire folder is lost, not one element.
* `get_docs` skips an account only on `ErrorNonExistentMailbox` /
`ErrorAccessDenied`, so a bare `ValueError` propagates and aborts the
whole sync.

### Change

`BaseFolder.ITEM_MODEL_MAP` is replaced with a `dict` subclass whose
`__missing__` returns the generic `Item` and logs a warning naming the
exact tag — once per tag, since a single folder can hold many copies of
the same stray element. Extending that map is the extension point
exchangelib's maintainer recommends for this class of problem.

The degraded `Item` then reaches the existing allowlists from #4158,
which skip it with a warning while healthy siblings keep syncing. All
three exchangelib call sites (`GetItem`, `FindItem`, `SyncFolderItems`)
resolve tags through this one map, so mails, contacts, tasks and
calendars are covered by the single change.

Genuine errors still fail loudly. Only the unmappable element is
dropped, so this cannot turn a systematic failure into a doc-less
"successful" sync that deletes previously indexed documents — the
principle established in #4085 / #4123 / #4147.

### Testing

* `pytest tests/sources/test_outlook.py` — 103 tests, including the four
added here: known tags still resolve to their own models, an unknown tag
degrades to `Item` and warns once, the reporter's exact response shape
parses into `[CalendarItem, Item]` instead of raising, and the degraded
item is skipped by the calendar allowlist.
* Full service suite: 2302 passed, total coverage 92.19% (gate 90%).
`client.py` and `datasource.py` at 97%; every line added here is
exercised.
* `ruff check` / `ruff format --check` clean on `connectors` and
`tests`; `pyright tests` clean, `pyright connectors` unchanged at the 13
pre-existing errors present on `main`.
* Before/after check against the reporter's response shape through
exchangelib's real parsing path: `ValueError` as reported without the
change, `['CalendarItem', 'Item']` plus the new warning with it.
* No E2E coverage: Outlook has no ftest fixture (no emulated backend),
and the failure only reproduces on the reporter's server. Verification
is the parsing check above against the exact XML shape from the SDH.

## Checklists

#### Pre-Review Checklist
- [x] this PR does NOT contain credentials of any kind, such as API keys
or username/passwords (double check `config.yml.example`)
- [x] this PR has a meaningful title
- [x] this PR links to all relevant github issues that it fixes or
partially addresses
- [x] if there is no GH issue, please create it. Each PR should have a
link to an issue
- [x] this PR has a thorough description
- [x] Covered the changes with automated tests
- [x] Tested the changes locally
- [x] Added a label for each target release version (example: `v7.13.2`,
`v7.14.0`, `v8.0.0`)
- [x] Considered corresponding documentation changes
- [ ] Contributed any configuration settings changes to the
configuration reference
- [ ] if you added or changed Rich Configurable Fields for a Native
Connector, you made a corresponding PR in
[Kibana](https://github.com/elastic/kibana/blob/main/packages/kbn-search-connectors/types/native_connectors.ts)

#### Changes Requiring Extra Attention

- [ ] Security-related changes (encryption, TLS, SSRF, etc)
- [ ] New external service dependencies added.
- [x] `BaseFolder.ITEM_MODEL_MAP` is mutated process-wide at import
time. Outlook is the only consumer of exchangelib, and this mirrors the
existing process-global `BaseProtocol.HTTP_ADAPTER_CLS` assignment from
#4094. A test asserts that every model exchangelib already knows still
resolves to itself, so the map only ever adds a fallback.

## Related Pull Requests

* #4158 — per-folder item type
allowlists this change feeds into
* #4147 — Contacts item type
dispatch
* #4123 — nullable field
hardening
* #4085 — per-account skips
vs. connection-wide failures
* #4065 — localized Exchange
folder names

The 8.19 backport will need to be done by hand: that branch still has
the single-file `connectors/sources/outlook.py` layout, so the
cherry-pick conflicts (as it did for #4158).

## Release Note

Fixed an issue where the Outlook connector aborted an entire sync when
an Exchange server returned an element it did not recognise as an item,
such as a stray `EndTimeZone` alongside a calendar item. Such elements
are now skipped with a warning and the rest of the mailbox continues to
sync.

Made with [Cursor](https://cursor.com)

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants