text "slicer" based on time interval #280

Activity

1. keighrim

   added

   ✨NNew feature or requestNew feature or request

   on Jun 5, 2024
2. clams-bot

   added this to infraon Jun 5, 2024
3. github-project-automation

   moved this to Todo in infraon Jun 5, 2024
4. bohJiang12 commented on Jun 7, 2024

   

   bohJiang12

   on Jun 7, 2024

   Contributor

   More actions

   In terms of implementing get_annotations_between_time function, some questions about:

   • Return format: my idea is to return a dict whose key is the string ("timeFrame", "timePoint") and the value is a list of TF/TP annotations
   • Judging whether a TimeFrame is in a time interval: should the condition be only true if the entire TimeFrame is in the interval? Or we can say it's also true for when they overlap, e.g. [2, 8] overlaps with [3, 10]?
   • Other annotations: Since our goal is to "slice" text from transcribed texts, I'm wondering if there's any other annotations other than Token annotations.
5. clams-bot

   assigned

   bohJiang12

   on Jun 7, 2024
6. keighrim commented on Jun 8, 2024

   

   keighrim

   on Jun 8, 2024

   MemberAuthor

   More actions

   • Return format: my idea is to return a dict whose key is the string ("timeFrame", "timePoint") and the value is a list of TF/TP annotations

   I was naively thinking that they can be just in a single list, when starting this feature requests. For the downstream use of this method, for example mmif.utils.text_document_helper.slice_text, if the return type is just one iterable;

   def slice_text(mmif, start_time, end_time):
       time_annotations = mmif.get_annotations_between_time(start_time, end_time)
       slice = []
       for ann in time_annotations: 
           if ann._attype == AnnotationTypes.Token: # this has to be actually lapps.discriminators.Uri.TOKEN, I use AnnotationTypes just for demonstration 
               slice.append(ann.get_propery('text')
       return ' '.join(slice)

   Then internally, the implementation can be something like

   class Mmif(...): 
       ...
       def get_annotations_between_time(self, s, e):
           time_anns = []
           for v in self.views:
               for a in v.annotations: 
                   if a.attype in (AnnotationTypes.TimePoint, AnnotationTypes.TimePoint): 
                       if is_in_time(a, s, e):
                           time_anns.append(a)
           time_anns = sorted(time_anns, key=sort_by_start_time_or_timepoint)
           for ann in time_anns:
               for aligned in self.find_aligned_annotations(ann):
                   yield aligned
       ...

   This is just a suggestion, so feel free to come up with any working implementation that works best for the downstream usage.

   • Judging whether a TimeFrame is in a time interval: should the condition be only true if the entire TimeFrame is in the interval? Or we can say it's also true for when they overlap, e.g. [2, 8] overlaps with [3, 10]?

   I think we definitely don't want to include full overlaps (e.g., [2, 13] overlaps with [3, 10]). For partial overlaps, maybe we can add a parameter to select the behavior?

       def get_annotations_between_time(self, s, e, allow_overlap=False):
           time_anns = []
           for v in self.views:
               for a in v.annotations: 
                   if a.attype in (AnnotationTypes.TimePoint, AnnotationTypes.TimePoint): 
                       if self.is_in_time(a, s, e, allow_overlap):
                           time_anns.append(a)
           ...

   • Other annotations: Since our goal is to "slice" text from transcribed texts, I'm wondering if there's any other annotations other than Token annotations.

   Technically, all types under Region in this tree https://vocab.lappsgrid.org/ are slice-able text annotations. Usually, when there's multiple levels of text annotations in one MMIF (look at the spacy output for example), only the lowest level of the character-based annotation (Token for most case) is directly aligned to time-based annotations.

   Another case we need to be careful here is OCR . OCR is done on images, and images (still frames) are extracted from and aligned to TimePoint annotations. OCR results are stored in TextDocument annotation type (along with lower-level tokenizations, if available). Here's an example MMIF from SWT-docTR pipeline (search for v_2 to go to OCR view) https://github.com/clamsproject/aapb-evaluations/blob/main/ocr_eval/preds%40doctr-wrapper1.1%40aapb-annotation-44/cpb-aacip-0515ac167c0.swt.stitch.doctr.mmif.credits.mmif
7. keighrim

   closed this as completedin #284on Jun 14, 2024
8. github-project-automation

   moved this from Todo to Done in infraon Jun 14, 2024
9. clams-bot

   unassigned

   bohJiang12

   on Jun 14, 2024