Abstract download file extraction and conversion - #1685
Merged
navarone-feekery merged 2 commits intoSep 27, 2023
Merged
Conversation
artem-shelkovnikov
approved these changes
Sep 27, 2023
artem-shelkovnikov
left a comment
Member
There was a problem hiding this comment.
Good stuff! Will you follow-up with using it in other sources, or we create an issue and address later?
Contributor
Author
|
@artem-shelkovnikov Thanks! I'm intending to use this in all sources with file downloads, I'll be making some batch PRs for that now :) |
navarone-feekery
enabled auto-merge (squash)
September 27, 2023 09:48
navarone-feekery
deleted the
navarone/5857-generalise-extraction-service-calls
branch
September 27, 2023 09:59
💔 Failed to create backport PR(s)The backport operation could not be completed due to the following error: The backport PRs will be merged automatically after passing CI. To backport manually run: |
3 tasks done
This was referenced Sep 27, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to https://github.com/elastic/enterprise-search-team/issues/5857
This PR moves the logic for downloading files to temp files, and their extraction/conversion to b64 to BaseDataSource.
The new tool is very generic, so it can be easily added to other connectors in future PRs.
If this is approved, I will also generalise/abstract the file size and file extension check.
This PR uses two connectors as examples for how this tool would be used:
This is a "standard" download and extraction case. The download func uses
response.content.iter_chunked, which is the same pattern that most connectors use. This download func is now abstracted onto theBaseDataSourceclass asgeneric_chunked_download_func.ABS has an irregular download method (it uses a library and has its own chunking method). As a result, its download func can't be abstracted. Instead the download func is defined in the ABS connector and is passed as a partial arg.
Checklists
Pre-Review Checklist
v7.13.2,v7.14.0,v8.0.0)