Skip to content

Prediction on still images has worse performance on iOS than on Lobe desktop #22

Open
Open
Prediction on still images has worse performance on iOS than on Lobe desktop#22

Description

@technoplato
technoplato
opened on Feb 8, 2021

Hello, love the app, first of all. This is how training should be!

I did run into a snag: I've successfully trained the model to recognize screenshots from audible, youtube, and apple podcasts and that model works very well on desktop using the "Play" section.

However, when I export the model and use it in the iOS Bootstrap, it does very bad at recognizing any screenshot accurately.

All I've done with code is replaced LobeModel with SavedModel directly from Lobe.

Any suggestions?

Video demo link: https://www.youtube.com/watch?v=_rtKnlmsxzM

Activity

  1. technoplato commented on Feb 8, 2021

    @technoplato
    technoplato
    on Feb 8, 2021
    Author

    Disregard.

    There is a popup when you go to export your model in the desktop application that asks if you'd like to optimize your model.

    I clicked optimize the second go round and everything works flawlessly now. Thanks!

  2. technoplato
    closed this as completedon Feb 8, 2021
  3. technoplato
    changed the title [-]Lobe - Model works perfectly with identical images on Desktop, but not on iOS bootstrap app[/-] [+][Solution - Optimize Model] Lobe - Model works perfectly with identical images on Desktop, but not on iOS bootstrap app[/+] on Feb 8, 2021
  4. technoplato
    changed the title [-][Solution - Optimize Model] Lobe - Model works perfectly with identical images on Desktop, but not on iOS bootstrap app[/-] [+]Lobe - Model works perfectly with identical images on Desktop, but not on iOS bootstrap app[/+] on Feb 9, 2021
  5. technoplato
    reopened this on Feb 9, 2021

  6. technoplato commented on Feb 9, 2021

    @technoplato
    technoplato
    on Feb 9, 2021
    Author

    Sorry Lobe people for the back and forth, I’m going to reopen this as the performance on device, even with the optimized model is much less accurate than that on the desktop application.

    Any tips would be appreciated.

  7. ellbosch commented on Feb 9, 2021

    @ellbosch
    ellbosch
    on Feb 9, 2021
    Contributor

    Thanks @technoplato—we will be deploying changes as soon as this week to fix these issues. You are correct to note that optimizing your model will not resolve this issue :)

  8. technoplato commented on Feb 9, 2021

    @technoplato
    technoplato
    on Feb 9, 2021
    Author

    Well that's great to hear! Glad I'm not crazy.

    As a side note, is Object Detection getting close or is that still pretty far out in the timeline?

    Thanks!

  9. ellbosch commented on Feb 10, 2021

    @ellbosch
    ellbosch
    on Feb 10, 2021
    Contributor

    Object detection is a rather large release and a top priority, but we can't say more than that. We are starting an insiders ring to test early features with object detection, if you are interested in joining email lobeai@microsoft.com.

  10. technoplato commented on Feb 10, 2021

    @technoplato
    technoplato
    on Feb 10, 2021 via email
    Author

  11. ellbosch commented on Feb 10, 2021

    @ellbosch
    ellbosch
    on Feb 10, 2021
    Contributor

    For the particular bug you posted, it's a bug specific to image processing. There's actually nothing wrong with the Core ML model—it should behave as you would expect. We're just plugging in a bad input to the model which creates the weird results you're seeing.

  12. technoplato commented on Feb 10, 2021

    @technoplato
    technoplato
    on Feb 10, 2021 via email
    Author

  13. ellbosch commented on Feb 10, 2021

    @ellbosch
    ellbosch
    on Feb 10, 2021
    Contributor

    You are correct that it's a bug with the starter code. Here's a brief explanation on how iOS-bootstrap works:

    1. Video capture: first, the iOS device's camera transmits video feed.
    2. Image processing: a frame of the video feed needs formatting (with cropping and orientation adjustments) before we can predict.
    3. Prediction: the processed image is assigned a prediction from the Core ML model's classifier.

    Step 2 above is what's broken, and should be fixed this week.

    I highly recommend our docs if you'd like to learn more! Please let me know if you have any other questions.

  14. technoplato commented on Feb 11, 2021

    @technoplato
    technoplato
    on Feb 11, 2021 via email
    Author

  15. ellbosch commented on Feb 11, 2021

    @ellbosch
    ellbosch
    on Feb 11, 2021
    Contributor

    Good question! Yes, we also format images selected from device library before sending to prediction (i.e. we need images formatted into a square aspect ratio).

  16. 6 remaining items

  17. ellbosch commented on Mar 25, 2021

    @ellbosch
    ellbosch
    on Mar 25, 2021
    Contributor

    Hello @technoplato! I believe PR #25 might have resolved this bug. Would you mind validating this by testing the u/elbosc/63/model-preprocessing branch with your project?

  18. technoplato commented on Mar 25, 2021

    @technoplato
    technoplato
    on Mar 25, 2021 via email
    Author

  19. technoplato commented on Mar 26, 2021

    @technoplato
    technoplato
    on Mar 26, 2021
    Author

    Hey @ellbosch I'm running into 'Network Disconnected' errors when trying to download the application either through https://lobeprod.azureedge.net/downloads/macos/Lobe.zip or through the home page (which I assume uses the same endpoint)

    Can you send me the zipped application directly at halfjew22@gmail.com? Really looking forward to playing with the new tool and demos

    EDIT: I'm trying to download it through my phone and it's working so far. Strange.

  20. mbeissinger commented on Mar 26, 2021

    @mbeissinger
    mbeissinger
    on Mar 26, 2021

    @technoplato sent you an email with a link as well

  21. technoplato commented on Mar 26, 2021

    @technoplato
    technoplato
    on Mar 26, 2021
    Author

    Unfortunately, no dice here either. I'm going to upload my training set and see if I'm just doing something wrong: https://drive.google.com/drive/folders/1nGTCSE9dsgz1ZJDVgc4rMGmBoyNJ-wmk?usp=sharing

    I'd upload the model as well but I'm suspicious that's what I'm messing up.

    Received the email @mbeissinger. Going to try exporting a new model, but after trying a v.8 model with the new iOS bootstrap that @ellbosch linked above, I'm still getting poor results

  22. mbeissinger commented on Mar 26, 2021

    @mbeissinger
    mbeissinger
    on Mar 26, 2021

    @technoplato is the model in iOS giving different results than from Lobe's Use tab?

    One thing from quickly looking at the dataset -- it is probably overfitting since most of the images in each class look very similar to each other. I would try to prune out images that pretty much look the same and try to capture more varied examples. Here is our docs section about improving the model/dataset: https://docs.lobe.ai/docs/improving/improving#why-is-lobe-not-predicting-well-on-new-images-in-use

  23. technoplato commented on Mar 26, 2021

    @technoplato
    technoplato
    on Mar 26, 2021
    Author

    @mbeissinger Thank you for the suggestion.

    I've got 99% accuracy within the Lobe Desktop application. The issue arises when I attempt to use the model in the ios-bootstrap. I believe Lobe Desktop's almost perfect accuracy implies that, despite a relatively repetitive dataset, the model "functions", even if just in the Desktop application

  24. mbeissinger commented on Mar 26, 2021

    @mbeissinger
    mbeissinger
    on Mar 26, 2021

    @technoplato ah ok! So is the CoreML model not loading/giving any results or are they different results than in Lobe?

  25. technoplato commented on Mar 26, 2021

    @technoplato
    technoplato
    on Mar 26, 2021 via email
    Author

  26. mbeissinger commented on Mar 26, 2021

    @mbeissinger
    mbeissinger
    on Mar 26, 2021

    @technoplato ok great I'll see if we can replicate

  27. technoplato commented on Mar 26, 2021

    @technoplato
    technoplato
    on Mar 26, 2021 via email
    Author

  28. ellbosch commented on Mar 30, 2021

    @ellbosch
    ellbosch
    on Mar 30, 2021
    Contributor

    @technoplato the only thing driving me nuts is that this bug still exists. Sorry we still haven't resolved this!

  29. technoplato commented on Mar 30, 2021

    @technoplato
    technoplato
    on Mar 30, 2021 via email
    Author

  30. mbeissinger commented on Mar 30, 2021

    @mbeissinger
    mbeissinger
    on Mar 30, 2021

    @technoplato @ellbosch I verified we have automatic tests for the model outputs going through the converters, so likely a bug in the ios processing. There will be some slight differences in outputs due to the different frameworks but they should not be major. I'm working on adding a file picker to the web-bootstrap so it isn't just the webcam -- you should be able to test the same image within Lobe and in the starter project soon. Can you also try it with the Lobe Connect local API using the same image as something you drag into the Use tab?

  31. technoplato commented on Apr 17, 2021

    @technoplato
    technoplato
    on Apr 17, 2021
    Author

    Sorry for my delay @mbeissinger finally got around to this and the Lobe Connect Run mode works perfectly!

    python3 lobeconnect.py
correct label: youtube.png
predicted label:	youtube
confidence:		1
correct label: audible.png
predicted label:	audible
confidence:		1

    Confirming the React App works as well. Thanks for the updates to allow static image selection!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions