Resizing Images and Boxes for Vision Scenarios #6629

Activity

1. ghost added

   untriagedNew issue has not been triagedNew issue has not been triaged

   on Apr 24, 2023
2. JakeRadMSFT commented on Apr 24, 2023

   

   JakeRadMSFT

   on Apr 24, 2023

   Last edited by JakeRadMSFT

   MemberAuthor

   More actions

   @luisquintanilla - @michaelgsharp and I were chatting and it probably makes sense to have a ObjectDetectionResize transform in ML.NET. Something that resizes images and bounding boxes the same way.

   We would need to figure out what type of resizes we want to support.

   An added benefit to this ... is if we make sure all images are the same size ... we can batch. This requires re-sizing larger and re-sizing smaller to make it happen.

   👍React with 👍1Reacted by Luis Quintanilla
3. justinormont commented on Apr 24, 2023

   

   justinormont

   on Apr 24, 2023

   Contributor

   More actions

   @JakeRadMSFT : You could create a generic image+keypoints resize transform. This could then in future handle segmentation, key point detection (e.g. human pose detection), and various other ML tasks which include points which scale with the image.

   Related work for co-handling the image+keypoints, is an image augmentation transform. To handle image rotation, keystoning, crop-and-zoom, skew, etc (list), the labeled points need to move with the image pixels.

   One hard part is understanding the various label formats; another is properly handling box/segment/instance removal if a box/etc is partially or fully cropped out of out image.

   👍React with 👍2Reacted by Luis Quintanilla and yyl9510
4. JakeRadMSFT commented on Apr 25, 2023

   

   JakeRadMSFT

   on Apr 25, 2023

   Last edited by JakeRadMSFT

   MemberAuthor

   More actions

   Copying some notes from meeting.

   Yangyu recommends to resize the image resolution to 800x600.

   There are several points need to be careful:

   • Remember to resize image resolution and bounding box location/size at the same time in training dataset.
   • It's better to keep the image resolution same or similar in both training and testing stage for one model instance (reduce resolution gap).
   • It's better to keep the width-height ratio not change during resizing.

   
5. JakeRadMSFT commented on Apr 25, 2023

   

   JakeRadMSFT

   on Apr 25, 2023

   Last edited by JakeRadMSFT

   MemberAuthor

   More actions

   Seems like we can do what we want to with the ResizeTransformer using IsoPad resize

   machinelearning/src/Microsoft.ML.ImageAnalytics/MLImage.cs

   Line 284 in 2ede226

   private SKBitmap ResizeWithPadding(int width, int height)

   and this code (based on the ISO Pad Resize code).

   void CalculateAspectAndOffset(float sourceWidth, float sourceHeight, float destinationWidth, float destinationHeight, out float xOffset, out float yOffset, out float aspect)
   {
       float widthAspect = destinationWidth / sourceWidth;
       float heightAspect = destinationHeight / sourceHeight;
       xOffset = 0;
       yOffset = 0;
       if (heightAspect < widthAspect)
       {
           aspect = heightAspect;
           xOffset = (destinationWidth - (sourceWidth * aspect)) / 2;
       }
       else
       {
           aspect = widthAspect;
           yOffset = (destinationHeight - (sourceHeight * aspect)) / 2;
       }
   }
   
   void ResizeKeyPoint(float sourceWidth, float sourceHeight, float destinationWidth, float destinationHeight, float sourceX, float sourceY, ref float destinationX, ref float destinationY)
   {
       float xOffset, yOffset, aspect;
       CalculateAspectAndOffset(sourceWidth, sourceHeight, destinationWidth, destinationHeight, out xOffset, out yOffset, out aspect);
   
       destinationX = xOffset + (sourceX * aspect);
       destinationY = yOffset + (sourceY * aspect);
   }
   
   void ReverseResizeKeyPoint(float sourceWidth, float sourceHeight, float destinationWidth, float destinationHeight, float destinationX, float destinationY, ref float sourceX, ref float sourceY)
   {
       float xOffset, yOffset, aspect;
       CalculateAspectAndOffset(sourceWidth, sourceHeight, destinationWidth, destinationHeight, out xOffset, out yOffset, out aspect);
   
       sourceX = (destinationX - xOffset) / aspect;
       sourceY = (destinationY - yOffset) / aspect;
   }

   xOffset, yOffset, and aspect can be re-used for all key points for the same image.

   👍React with 👍1Reacted by Justin Ormont
6. luisquintanilla commented on Apr 25, 2023

   

   luisquintanilla

   on Apr 25, 2023

   Contributor

   More actions

   Thanks for the feedback @justinormont and @JakeRadMSFT for looking into it.

   For resizing I'm okay to proceed with the recommended solution from Yangyu and Jake.

   @michaelgsharp thoughts?

   👍React with 👍1Reacted by Justin Ormont
7. luisquintanilla commented on Apr 27, 2023

   

   luisquintanilla

   on Apr 27, 2023

   Contributor

   More actions

   One additional recommendation here - if we know that 800x600 is the preferred size, let's make those the default values in the methods. Users can provide a different size if needed.
8. JakeRadMSFT

   changed the title [-]Object Detection Stalls with Large Images[/-] [+]Resizing Images and Boxes for Vision Scenarios[/+] on Apr 27, 2023
9. yyl9510 commented on Apr 27, 2023

   

   yyl9510

   on Apr 27, 2023

   Last edited by yyl9510

   More actions

   Copying some notes from meeting.

   Yangyu recommends to resize the image resolution to 800x600.

   There are several points need to be careful:

   • Remember to resize image resolution and bounding box location/size at the same time in training dataset.
   • It's better to keep the image resolution same or similar in both training and testing stage for one model instance (reduce resolution gap).
   • It's better to keep the width-height ratio not change during resizing.

   

   What @JakeRadMSFT said is very correct. For large images, resizing is necessary. The white padding part can be set to (r,g,b) = (0,0,0).
10. luisquintanilla

    added this to the ML.NET 3.0 milestone on May 10, 2023
11. ghost removed

    untriagedNew issue has not been triagedNew issue has not been triaged

    on May 10, 2023
12. rosebyte

    added

    area-VisionImage processing and computer visionImage processing and computer vision

    on Sep 8, 2026
13. rosebyte

    added

    yellowmaintenance onlymaintenance only

    on Sep 21, 2026