Skip to content

[PAID] [$500] Country search does not consider Latin or Spanish letters in search #29826

Closed

Description

@m-natarajan

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!


Version Number: 1.3.85-1
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: @dhanashree-sawant
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1697544511147309

Action Performed:

  1. Open the app
  2. Open settings->profile->personal details->address->country
  3. Search with letter 'Å' and observe that even though we have country with that letter in list, app still doesn't show that on n top of search eg: 'Åland Islands'
  4. Similarily search with letter 'é', we have multiple countries with that letter but search doesn't show them on top eg: 'São Tomé & Príncipe', 'Saint Barthélemy'

Expected Result:

App should consider latin or spanish letter in country search and display their results on top as we generally do on user search page and send message page

Actual Result:

App does not consider latin or spanish letter in country search and does not display their results on top

Workaround:

unknown

Platforms:

Which of our officially supported platforms is this issue occurring on?

  • Android: Native
  • Android: mWeb Chrome
  • iOS: Native
  • iOS: mWeb Safari
  • MacOS: Chrome / Safari
  • MacOS: Desktop

Screenshots/Videos

Android: Native
Android.native.country.search.not.specific.variation.letters.mp4
Android: mWeb Chrome
Android.chrome.search.not.change.by.variation.letters.mp4
iOS: Native
ios.native.country.search.not.specific.variation.letters.mov
iOS: mWeb Safari
ios.safari.country.search.not.specific.variation.letters.mov
MacOS: Chrome / Safari
mac.chrome.country.search.not.specific.variation.letters.mov
MacOS: Desktop
mac.desktop.country.search.not.specific.variation.letters.mov

View all open jobs on GitHub

Upwork Automation - Do Not Edit
  • Upwork Job URL: https://www.upwork.com/jobs/~01c62c8f44bacecbe2
  • Upwork Job ID: 1714381091711156224
  • Last Price Increase: 2023-10-31
  • Automatic offers:
    • situchan | Reviewer | 27762279
    • graylewis | Contributor | 27762281
    • dhanashree-sawant | Reporter | 27762282

Activity

  1. added
    ExternalAdded to denote the issue can be worked on by a contributor
    BugSomething is broken. Auto assigns a BugZero manager.
    on Oct 17, 2023
  2. changed the title [-]Country search does not consider Latin or Spanish letters in search[/-] [+][$500] Country search does not consider Latin or Spanish letters in search[/+] on Oct 17, 2023
  3. melvin-bot commented on Oct 17, 2023

    @melvin-bot
  4. melvin-bot commented on Oct 17, 2023

    @melvin-bot

    Triggered auto assignment to @strepanier03 (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

  5. added
    Help WantedApply this label when an issue is open to proposals by contributors
    on Oct 17, 2023
  6. melvin-bot commented on Oct 17, 2023

    @melvin-bot

    Bug0 Triage Checklist (Main S/O)

    • This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
    • This bug is not a duplicate report (check E/App issues and #expensify-bugs)
      • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
    • This bug is reproducible using the reproduction steps in the OP. S/O
      • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
      • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
    • This issue is filled out as thoroughly and clearly as possible
      • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
    • I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync
  7. melvin-bot commented on Oct 17, 2023

    @melvin-bot

    Triggered auto assignment to Contributor-plus team member for initial proposal review - @situchan (External)

  8. graylewis commented on Oct 17, 2023

    @graylewis
    Contributor

    Proposal

    Please re-state the problem that we are trying to solve in this issue.

    The country search page doesn't prioritize search results that contain the same diacritical marks as the search term over normalized matches

    What is the root of the issue

    Items in the array of countries being searched get sanitized with the StringUtils.sanitizeString() function which:
    Removes diacritical marks and non-alphabetic and non-latin characters from a string.

    By running this on the array of countries to be searched and the search term, both searches with diacritical marks and searches without are treated the same. This is almost ideal behavior, but prioritizing exact matches with diacritical marks would be even better.

    What changes do you think we should make in order to solve the problem?

    In the searchCountryOptions file, where the results are generated by matching the search term against items of the countriesData array, we can add a sorting mechanism to prioritize exact matches including diacritical marks. In addition to this, we'll have to maintain the diacritical marks for the array of countries to be searched and then normalize the values as needed, rather than normalizing them all at once up front.

    In code (this code is obviously sloppy, it's just for illustration purposes):

        const filteredData = countriesData.filter((country) => StringUtils.sanitizeString(country.searchValue).includes(trimmedSearchValue));
    
        let halfSorted = filteredData.sort((a, b) => {
            if (a.searchValue.toLowerCase().substring(2, trimmedSearchValue.length + 2) === trimmedSearchValue.toLowerCase()) {
                return -1;
            }
            if (b.searchValue.toLowerCase().substring(2, trimmedSearchValue.length + 2) === trimmedSearchValue.toLowerCase()) {
                return 1;
            }
            return 0;
        })
    
        let fullSorted;
        if (trimmedSearchValue !== searchValue) {
            // Diacritic detected, prioritize diacritic matches
            fullSorted = halfSorted.sort((a,b) => {
                if (    
                    a.text.toLowerCase().includes(searchValue.toLowerCase())
                ) {
                    return -1
                } 
                if (
                    b.text.toLowerCase().includes(searchValue.toLowerCase())
                ) {
                    return 1;
                }
                return 0;
            });
        } else {
            fullSorted = halfSorted.sort((a, b) => {
                if (a.value.toLowerCase() === trimmedSearchValue) {
                    return -1;
                }
                if (b.value.toLowerCase() === trimmedSearchValue) {
                    return 1;
                }
                return 0;
            });
        }

    Updated logic to reflect updated requirements:

    1. Data is filtered based on the original search term comparison. This includes all possible matches.
    2. A sort is run to prioritize exact matches from the beginning of the country name, as requested in the updated requirements. E.g. "Bar" returns Barbados first since it's the beginning of the country name.
    3. The sanitized search term is compared to the raw search term. If they're different, then diacritics/non-latin chars are present in the search term. In this case, exact matches including diacritics are pushed to the top.
    4. If diacritics aren't present, then as is done in the existing code, we prioritize exact country code matches. E.g. "US" returns the united states before anything else. (Since country codes never contain diacritics, there's no need to run this step if diacritics are in the search term)

    The conditional sort based on the presence of diacritics ensures we only run two sorts regardless of the search term.

    As a side note, the call to the searchCountryOptions function should be memoized, as it currently runs every frame unnecessarily. This will more than offset any potential performance costs of running an additional sort on the search results array.

    My solution in action:
    Screenshot 2023-10-24 at 3 09 19 PM

    Screenshot 2023-10-24 at 3 23 21 PM Screenshot 2023-10-24 at 3 23 36 PM image

    What alternative solutions did you explore? (Optional)
    NA

  9. Victor-Nyagudi commented on Oct 18, 2023

    @Victor-Nyagudi
    Contributor

    It seems this method removes letters with diacritics i.e. accented letters like é, so that could be affecting country search results.

    /**
    * Removes diacritical marks and non-alphabetic and non-latin characters from a string.
    * @param str - The input string to be sanitized.
    * @returns The sanitized string
    */
    function sanitizeString(str: string): string {
    return _.deburr(str).toLowerCase().replaceAll(CONST.REGEX.NON_ALPHABETIC_AND_NON_LATIN_CHARS, '');
    }

    I'm not sure if this is the intended behavior or a regression, so I asked about it in the pull request that introduced this method.

  10. strepanier03 commented on Oct 20, 2023

    @strepanier03
    Contributor

    @situchan - Friendly bump on the proposal above.

  11. 82 remaining items

  12. changed the title [-][$500] Country search does not consider Latin or Spanish letters in search[/-] [+][HOLD for payment 2024-02-26] [$500] Country search does not consider Latin or Spanish letters in search[/+] on Feb 19, 2024
  13. melvin-bot commented on Feb 19, 2024

    @melvin-bot

    Reviewing label has been removed, please complete the "BugZero Checklist".

  14. melvin-bot commented on Feb 19, 2024

    @melvin-bot

    The solution for this issue has been 🚀 deployed to production 🚀 in version 1.4.42-5 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

    If no regressions arise, payment will be issued on 2024-02-26. 🎊

    For reference, here are some details about the assignees on this issue:

  15. melvin-bot commented on Feb 19, 2024

    @melvin-bot

    BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

    • [@situchan] The PR that introduced the bug has been identified. Link to the PR:
    • [@situchan] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:
    • [@situchan] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion:
    • [@situchan] Determine if we should create a regression test for this bug.
    • [@situchan] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.
    • [@strepanier03] Link the GH issue for creating/updating the regression test once above steps have been agreed upon:
  16. strepanier03 commented on Feb 26, 2024

    @strepanier03
    Contributor

    @graylewis and @dhanashree-sawant - I've paid you both via Upwork and closed the contracts.

    @situchan - I'll check in later today to see if the checklist is done and if so move forward then. If not, I'll check tomorrow morning.

  17. situchan commented on Feb 26, 2024

    @situchan
    Contributor

    This is not really bug but improvement. We added enough automated tests so no need regression test.

  18. strepanier03 commented on Feb 26, 2024

    @strepanier03
    Contributor

    Got it thanks for the context @situchan - I'll handle now.

  19. strepanier03 commented on Feb 26, 2024

    @strepanier03
    Contributor

    Okay, payment sent and contract closed.

    Thanks again everyone 👏

  20. changed the title [-][HOLD for payment 2024-02-26] [$500] Country search does not consider Latin or Spanish letters in search[/-] [+][PAID] [$500] Country search does not consider Latin or Spanish letters in search[/+] on Feb 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Awaiting PaymentAuto-added when associated PR is deployed to productionBugSomething is broken. Auto assigns a BugZero manager.DailyKSv2EngineeringExternalAdded to denote the issue can be worked on by a contributor

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions