Repository navigation
Evtx.BinaryParser.OverrunBufferException raised unexpectedly #46
Description
Activity
hey @john-corcoran
Thanks for raising this issue. The easiest way to triage it will be to share the .evtx file. I'd be happy to take a peek if you send to my email: willi.ballenthin@gmail.com
file structure looks ok:
λ C:\Users\user\env2\Scripts\python ..\scripts\evtx_info.py data\issue_46.evtx Information from file header: Format version : 3.1 Flags : 0x00000000 File is : clean Log is full : no Current chunk : 3 of 16 Oldest chunk : 5 Next record# : 521 Check sum : pass Information from chunks: Chunk file (first/last) log (first/last) Header Data - ----- --------------------- --------------------- ------ ------ 1 417 441 417 441 pass pass 2 442 466 442 466 pass pass 3 467 499 467 499 pass pass * 4 500 520 500 520 pass pass > 5 104 129 104 129 pass pass 6 130 150 130 150 pass pass 7 151 180 151 180 pass pass 8 181 205 181 205 pass pass 9 206 226 206 226 pass pass 10 227 252 227 252 pass pass 11 253 283 253 283 pass pass 12 284 304 284 304 pass pass 13 305 335 305 335 pass pass 14 336 360 336 360 pass pass 15 361 387 361 387 pass pass 16 388 416 388 416 pass passSeems like the very first record in the evtx file is problematic:
magic: 0x2a2a size: 0x17a0 num: 417 ts: 2017-11-14T11:32:41.189631 substitutions count: 20 substitutions count: 1124080761Clearly 1124080761 is not a reasonable number of substitutions.
there's a root note being parsed at 0x1836, with substitution count being parsed at 0x1844:
19:44:21,0731 [root :1046][INFO ] root node: 0x1836 19:44:21,0732 [root :1047][INFO ] ofs: 0xe (abs: 0x1844) 19:44:21,0732 [root :1048][INFO ] sub count: 1124080761but this doesn't look quite right:
1830h: 00 05 12 00 00 00 01 5C 11 00 00 3F 08 00 00 00 .......\...?.... 1840h: 00 00 00 B4 79 1C 00 43 00 72 00 79 00 70 00 74 ...´y..C.r.y.p.t 1850h: 00 52 00 65 00 74 00 72 00 69 00 65 00 76 00 65 .R.e.t.r.i.e.v.e 1860h: 00 4F 00 62 00 6A 00 65 00 63 00 74 00 42 00 79 .O.b.j.e.c.t.B.y 1870h: 00 55 00 72 00 6C 00 57 00 69 00 72 00 65 00 00 .U.r.l.W.i.r.e.. 1880h: 00 02 41 CB 00 00 00 8B 08 00 00 00 00 00 00 4F ..AË...‹.......Othe dword at 0x1844 is 0x43001C79, which is crazy. a better interpretation of these bytes is the byte at 0x1845 (value: 0x1C) is number of wchar in the utf-16le string (number of chars: 0x1C) that begins at 0x1847.
need to trace down where the parsing is faulty that leads to these incorrect fields.
file structure:
λ C:\Users\user\env2\Scripts\python ..\scripts\evtx_structure.py data\issue_46.evtx | head -n 400 File header magic: ElfFile oldest_chunk: 0x4 current_chunk_number: 0x3 next_record_number: 0x209 header_size: 0x80 minor_version: 0x1 major_version: 0x3 header_chunk_size: 0x1000 chunk_count: 0x10 flags: 0x0 checksum: 0x1d883bed verify: True dirty: False full: False Chunk offset: 0x1000 magic: ElfChnk file_first_record_number: 0x1a1 file_last_record_number: 0x1b9 log_first_record_number: 0x1a1 log_last_record_number: 0x1b9 header_size: 0x80 last_record_offset: 0xdf80 next_record_offset: 0xfe90 data_checksum: 0x20c08819 header_checksum: 0xf0aadf4cL verify: True templates: 1 Record offset: 0x1200 magic: 0x2a2a size: 0x17a0 number: 0x1a1 timestamp: 2017-11-14 11:32:41.189631 verify: True RootNode(offset=0x18) StreamStartNode(offset=0x18) TemplateInstanceNode(offset=0x1c, resident=True, length=0x571) TemplateNode(offset=0x26) StreamStartNode(offset=0x3e) OpenStartElementNode(offset=0x42) --> Event AttributeNode(offset=0x65) --> xmlns ValueNode(offset=0x7e) WstringTypeNode(offset=0x80) --> http://schemas.microsoft.com/win/2004/08/events/event CloseStartElementNode(offset=0xec) OpenStartElementNode(offset=0xed) --> System CloseStartElementNode(offset=0x10e) OpenStartElementNode(offset=0x10f) --> Provider AttributeNode(offset=0x138) --> Name ValueNode(offset=0x14f) WstringTypeNode(offset=0x151) --> Microsoft-Windows-CAPI2 AttributeNode(offset=0x181) --> Guid ValueNode(offset=0x198) WstringTypeNode(offset=0x19a) --> {5bbca4a8-b209-48dc-a8c7-b23d3e5216fb} CloseEmptyElementNode(offset=0x1e8) OpenStartElementNode(offset=0x1e9) --> EventID AttributeNode(offset=0x210) --> Qualifiers ConditionalSubstitutionNode(offset=0x233) CloseStartElementNode(offset=0x237) ConditionalSubstitutionNode(offset=0x238) CloseElementNode(offset=0x23c) OpenStartElementNode(offset=0x23d) --> Version CloseStartElementNode(offset=0x260) ConditionalSubstitutionNode(offset=0x261) CloseElementNode(offset=0x265) OpenStartElementNode(offset=0x266) --> Level CloseStartElementNode(offset=0x285) ConditionalSubstitutionNode(offset=0x286) CloseElementNode(offset=0x28a) OpenStartElementNode(offset=0x28b) --> Task CloseStartElementNode(offset=0x2a8) ConditionalSubstitutionNode(offset=0x2a9) CloseElementNode(offset=0x2ad) OpenStartElementNode(offset=0x2ae) --> Opcode CloseStartElementNode(offset=0x2cf) ConditionalSubstitutionNode(offset=0x2d0) CloseElementNode(offset=0x2d4) OpenStartElementNode(offset=0x2d5) --> Keywords CloseStartElementNode(offset=0x2fa) ConditionalSubstitutionNode(offset=0x2fb) CloseElementNode(offset=0x2ff) OpenStartElementNode(offset=0x300) --> TimeCreated AttributeNode(offset=0x32f) --> SystemTime ConditionalSubstitutionNode(offset=0x352) CloseEmptyElementNode(offset=0x356) OpenStartElementNode(offset=0x357) --> EventRecordID CloseStartElementNode(offset=0x386) ConditionalSubstitutionNode(offset=0x387) CloseElementNode(offset=0x38b) OpenStartElementNode(offset=0x38c) --> Correlation AttributeNode(offset=0x3bb) --> ActivityID ConditionalSubstitutionNode(offset=0x3de) AttributeNode(offset=0x3e2) --> RelatedActivityID ConditionalSubstitutionNode(offset=0x413) CloseEmptyElementNode(offset=0x417) OpenStartElementNode(offset=0x418) --> Execution AttributeNode(offset=0x443) --> ProcessID ConditionalSubstitutionNode(offset=0x464) AttributeNode(offset=0x468) --> ThreadID ConditionalSubstitutionNode(offset=0x487) CloseEmptyElementNode(offset=0x48b) OpenStartElementNode(offset=0x48c) --> Channel CloseStartElementNode(offset=0x4af) ValueNode(offset=0x4b0) WstringTypeNode(offset=0x4b2) --> Microsoft-Windows-CAPI2/Operational CloseElementNode(offset=0x4fa) OpenStartElementNode(offset=0x4fb) --> Computer CloseStartElementNode(offset=0x520) ValueNode(offset=0x521) WstringTypeNode(offset=0x523) --> exploited CloseElementNode(offset=0x537) OpenStartElementNode(offset=0x538) --> Security AttributeNode(offset=0x561) --> UserID ConditionalSubstitutionNode(offset=0x57c) CloseEmptyElementNode(offset=0x580) CloseElementNode(offset=0x581) OpenStartElementNode(offset=0x582) --> UserData CloseStartElementNode(offset=0x5a7) ConditionalSubstitutionNode(offset=0x5a8) CloseElementNode(offset=0x5ac) CloseElementNode(offset=0x5ad) EndOfStreamNode(offset=0x5ae) Substitutions(offset=0x5af) UnsignedByteTypeNode(offset=0x603) --> 2 UnsignedByteTypeNode(offset=0x604) --> 2 UnsignedWordTypeNode(offset=0x605) --> 53 UnsignedWordTypeNode(offset=0x607) --> 53 NullTypeNode(offset=0x609) Hex64TypeNode(offset=0x609) --> 0x4000000000000036 FiletimeTypeNode(offset=0x611) --> 2017-11-14 11:32:41.189631 NullTypeNode(offset=0x619) UnsignedDwordTypeNode(offset=0x619) --> 2856 UnsignedDwordTypeNode(offset=0x61d) --> 608 UnsignedQwordTypeNode(offset=0x621) --> 417 UnsignedByteTypeNode(offset=0x629) --> 0 SIDTypeNode(offset=0x62a) --> S-1-5-18 NullTypeNode(offset=0x636) NullTypeNode(offset=0x636) NullTypeNode(offset=0x636) NullTypeNode(offset=0x636) NullTypeNode(offset=0x636) NullTypeNode(offset=0x636) BXmlTypeNode(offset=0x636) --> RootNode(offset=0x636) ERROR: 0 Record offset: 0x29a0 magic: 0x2a2a size: 0x790 number: 0x1a2 timestamp: 2017-11-14 11:32:41.189631 verify: True RootNode(offset=0x18) StreamStartNode(offset=0x18) TemplateInstanceNode(offset=0x1c, resident=False) Substitutions(offset=0x26) UnsignedByteTypeNode(offset=0x7a) --> 4 UnsignedByteTypeNode(offset=0x7b) --> 2 UnsignedWordTypeNode(offset=0x7c) --> 41 UnsignedWordTypeNode(offset=0x7e) --> 41 NullTypeNode(offset=0x80) Hex64TypeNode(offset=0x80) --> 0x4000000000000005 FiletimeTypeNode(offset=0x88) --> 2017-11-14 11:32:41.189631 NullTypeNode(offset=0x90) UnsignedDwordTypeNode(offset=0x90) --> 2856 UnsignedDwordTypeNode(offset=0x94) --> 608 UnsignedQwordTypeNode(offset=0x98) --> 418 UnsignedByteTypeNode(offset=0xa0) --> 0 SIDTypeNode(offset=0xa1) --> S-1-5-18 NullTypeNode(offset=0xad) NullTypeNode(offset=0xad) NullTypeNode(offset=0xad) NullTypeNode(offset=0xad) NullTypeNode(offset=0xad) NullTypeNode(offset=0xad) BXmlTypeNode(offset=0xad) --> RootNode(offset=0xad) ERROR: Tried to parse beyond the end of the file (read: 0x160c1006, buffer length: 0x101000) Record offset: 0x3130 magic: 0x2a2a size: 0x1738 number: 0x1a3 timestamp: 2017-11-14 11:32:41.189631 verify: True RootNode(offset=0x18) StreamStartNode(offset=0x18) TemplateInstanceNode(offset=0x1c, resident=False) Substitutions(offset=0x26) UnsignedByteTypeNode(offset=0x7a) --> 4 UnsignedByteTypeNode(offset=0x7b) --> 2 UnsignedWordTypeNode(offset=0x7c) --> 11 UnsignedWordTypeNode(offset=0x7e) --> 11 NullTypeNode(offset=0x80) Hex64TypeNode(offset=0x80) --> 0x4000000000000003 FiletimeTypeNode(offset=0x88) --> 2017-11-14 11:32:41.189631 NullTypeNode(offset=0x90) UnsignedDwordTypeNode(offset=0x90) --> 2856 UnsignedDwordTypeNode(offset=0x94) --> 608 UnsignedQwordTypeNode(offset=0x98) --> 419 UnsignedByteTypeNode(offset=0xa0) --> 0 SIDTypeNode(offset=0xa1) --> S-1-5-18 NullTypeNode(offset=0xad) NullTypeNode(offset=0xad) NullTypeNode(offset=0xad) NullTypeNode(offset=0xad) NullTypeNode(offset=0xad) NullTypeNode(offset=0xad) BXmlTypeNode(offset=0xad) --> RootNode(offset=0xad) ERROR: Tried to parse beyond the end of the file (read: 0x5341006, buffer length: 0x101000)I have the same problem, but in this instance, I can't share the EVTX files for confidentiality reasons. I have the same parsing problem. I tried to follow some of your steps above for leads, and I will paste the output below. I'm running python 3.7.1.
python D:\tools\Parse-Evtx\scripts\evtx_info.py "Microsoft-Windows-TerminalServices-LocalSessionManager%4Operational.evtx" Information from file header: Format version : 3.1 Flags : 0x00000001 File is : dirty Log is full : no Current chunk : 8 of 9 Oldest chunk : 1 Next record# : 1018 Check sum : pass Traceback (most recent call last): File "D:\tools\Parse-Evtx\scripts\evtx_info.py", line 114, in <module> main() File "D:\tools\Parse-Evtx\scripts\evtx_info.py", line 57, in main chunk_count = sum([1 for c in fh.chunks() if c.verify()]) File "D:\tools\Parse-Evtx\scripts\evtx_info.py", line 57, in <listcomp> chunk_count = sum([1 for c in fh.chunks() if c.verify()]) File "C:\Python3\lib\site-packages\Evtx\Evtx.py", line 333, in verify return self.check_magic() and \ File "C:\Python3\lib\site-packages\Evtx\Evtx.py", line 308, in check_magic return self.magic() == "ElfChnk\x00" File "C:\Python3\lib\site-packages\Evtx\BinaryParser.py", line 211, in explicit_length_handler return f(offset, length) File "C:\Python3\lib\site-packages\Evtx\BinaryParser.py", line 475, in unpack_string return self.unpack_binary(offset, length).decode('ascii') UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 4: ordinal not in range(128)Here is the excerpt from evtx_structure.py where it failed.
Chunk offset: 0x91000 magic: ☻ ☻ file_first_record_number: 0x20048 file_last_record_number: 0x12c65ab log_first_record_number: 0x2 log_last_record_number: 0x0 header_size: 0x20800 last_record_offset: 0x3 next_record_offset: 0x20000 data_checksum: 0x0 header_checksum: 0x0 verify: False Unexpected token encountered Unexpected token encountered Traceback (most recent call last): File "C:\Python3\lib\site-packages\Evtx\BinaryParser.py", line 273, in unpack_byte return struct.unpack_from("<B", self._buf, o)[0] struct.error: unpack_from requires a buffer of at least 1 bytes During handling of the above exception, another exception occurred: Traceback (most recent call last): File "D:\tools\Parse-Evtx\scripts\evtx_structure.py", line 185, in <module> main() File "D:\tools\Parse-Evtx\scripts\evtx_structure.py", line 180, in main for line in formatter.format_header(log.get_file_header()): File "D:\tools\Parse-Evtx\scripts\evtx_structure.py", line 66, in format_header for line in self.format_chunk(chunk): File "D:\tools\Parse-Evtx\scripts\evtx_structure.py", line 89, in format_chunk yield self._l("templates: %d" % (len(chunk.templates()))) File "C:\Python3\lib\site-packages\Evtx\Evtx.py", line 411, in templates self._load_templates() File "C:\Python3\lib\site-packages\Evtx\Evtx.py", line 380, in _load_templates token = self.unpack_byte(ofs - 10) File "C:\Python3\lib\site-packages\Evtx\BinaryParser.py", line 275, in unpack_byte raise OverrunBufferException(o, len(self._buf)) Evtx.BinaryParser.OverrunBufferException: Tried to parse beyond the end of the file (read: 0x43dbfc, buffer length: 0x101000)Experiencing the same exact issues. Any recommendations?
@williballenthin I believe this evt(x) file causes the same exception: https://send.firefox.com/download/1804ced0a1ad523c/#A_E9qDRE3rU1poCUk_zq4g
I hope this helps.
(it's produced by wusa.exe)
Hi,
I'm seeing exception Evtx.BinaryParser.OverrunBufferException raised unexpectedly when parsing event logs "Microsoft-Windows-Windows Firewall With Advanced Security%4Firewall.evtx", "Microsoft-Windows-WindowsUpdateClient%4Operational.evtx", and "Microsoft-Windows-CAPI2%4Operational.evtx" (all extracted from a Windows 7 testing instance) using evtx_dump.py.
For Microsoft-Windows-Windows Firewall With Advanced Security%4Firewall.evtx, the same behaviour is seen across Windows with Python 2.7 and Ubuntu with Python 2.7 / 3.5. Traceback from Ubuntu with Python 3.5 as follows:
For Microsoft-Windows-WindowsUpdateClient%4Operational.evtx, Python 3.5 on Ubuntu gives Evtx.BinaryParser.OverrunBufferException, while Python 2.7 on Windows / Ubuntu gives a MemoryError. Python 3.5 traceback as follows:
Python 2.7 traceback (from Ubuntu) as follows:
And similarly for Microsoft-Windows-CAPI2%4Operational.evtx, Python 3.5 on Ubuntu gives Evtx.BinaryParser.OverrunBufferException, while Python 2.7 on Windows / Ubuntu BSODs / has process 'Killed'. Traceback for Python 3.5 on Ubuntu as follows:
I can forward the EVTX files in question if that assists?