You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[HOLD for payment 2023-10-02] [$1000] Chat - Mention is not displayed in message with full 15000 letters #21817
Write message with 15000 letters (max limit) and include mention in it in middle or in end
Observe that app does not recognize mention in the message
Expected Result:
App should recognize mention in message with 15000 letters (max limit)
Actual Result:
App does not recognize mention in message with 15000 letters (max limit)
Workaround:
Unknown
Platforms:
Which of our officially supported platforms is this issue occurring on?
Android / native
Android / Chrome
iOS / native
iOS / Safari
MacOS / Chrome / Safari
MacOS / Desktop
Version Number: 1.3.33-3 Reproducible in staging?: y Reproducible in production?: y If this was caught during regression testing, add the test name, ID and link from TestRail: Email or phone of affected tester (no customers): Logs:https://stackoverflow.com/c/expensify/questions/4856 Notes/Photos/Videos: Any additional supporting documentation
What changes do you think we should make in order to solve the problem?
To correctly display markdown in all messages: MAX_MARKUP_LENGTH should be equal to MAX_COMMENT_LENGTH.
However, this leaves an edge case in the composer, where it is possible to type messages that exceed MAX_COMMENT_LENGTH. The message in the composer is parsed prior to determining its length, as this is how the BE counts messages also.
* For comments shorter than 10k chars, convert the comment from MD into HTML because that's how it is stored in the database
* For longer comments, skip parsing, but still escape the text, and display plaintext for performance reasons. It takes over 40s to parse a 100k long string!!
As you remove characters from a long message, the character count will decrease but then suddenly jump as markdown parsing will be applied. Please see the video below for a demonstration.
Video
chrome_RO2IrooXyk.mp4
In conclusion, we can consider these solutions:
Option 1. Keep the current counting algorithm and increase MAX_MARKUP_LENGTH to a value higher than MAX_COMMENT_LENGTH.
This ensures the entire message is parsed, as well as prevents the message length jumping behavior in the composer near MAX_COMMENT_LENGTH.
Option 2. Make MAX_MARKUP_LENGTH equal to MAX_COMMENT_LENGTH, then:
getCommentLength must be changed to count the unprocessed message (i.e. how it's typed by the user). This counting change should also be applied to the BE, which may make it more complex as the BE does not currently see the plaintext message and secondly due to storage limit considerations and the need to validate this. After all, post-markdown processing is how the message is actually stored in the BE.
text.length < CONST.MAX_MARKUP_LENGTH should be made inclusive, i.e. <= instead of <
Option 3. Optimize markdown processing speed and remove MAX_MARKUP_LENGTH altogether (or drastically increase)
Slack does not have any responsiveness issues when typing long messages (tried 375k+) in their composer and only limit the length of pasting long texts. I think there is plenty room for optimization and the only reason of existence (to my knowledge) for MAX_MARKUP_LENGTH is to prevent performance degradation.
Marked.js handles complex 100k markdown texts in the ~50ms range. CommonMark requires ~20ms. Expensify's implementation however requires tens of seconds (freezing the tab) due to its heavy reliance on regular expressions. It's a simple, but poorly performant implementation. Arguably this is the true root cause of the problem.
What alternative solutions did you explore? (Optional)
📣 @samh-nl! 📣
Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork.
Please follow these steps:
Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details.
Format:
Contributor details
Your Expensify account email: <REPLACE EMAIL HERE>
Upwork Profile Link: <REPLACE LINK HERE>
changed the title [-]Chat - Mention is not displayed in message with full 15000 letters[/-][+][$1000] Chat - Mention is not displayed in message with full 15000 letters[/+]on Jun 29, 2023
BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:
This issue is resulted by multiple PRs and was kind of expected result back then.
Activity
melvin-bot commentedon Jun 28, 2023
Triggered auto assignment to @sophiepintoraetz (
Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.melvin-bot commentedon Jun 28, 2023
Bug0 Triage Checklist (Main S/O)
Platformsin OP are ✅)samh-nl commentedon Jun 28, 2023
Proposal
Please re-state the problem that we are trying to solve in this issue.
Mentions are not displayed in comments that approach the 15000 character limit.
What is the root cause of that problem?
MAX_COMMENT_LENGTHis higher thanMAX_MARKUP_LENGTH, leading to long messages not being displayed with markdown.App/src/CONST.js
Line 1233 in ecfcd83
What changes do you think we should make in order to solve the problem?
To correctly display markdown in all messages:
MAX_MARKUP_LENGTHshould be equal toMAX_COMMENT_LENGTH.However, this leaves an edge case in the composer, where it is possible to type messages that exceed
MAX_COMMENT_LENGTH. The message in the composer is parsed prior to determining its length, as this is how the BE counts messages also.App/src/libs/ReportUtils.js
Lines 2277 to 2281 in 67bac9d
App/src/libs/ReportUtils.js
Lines 1289 to 1299 in 67bac9d
As you remove characters from a long message, the character count will decrease but then suddenly jump as markdown parsing will be applied. Please see the video below for a demonstration.
Video
chrome_RO2IrooXyk.mp4
In conclusion, we can consider these solutions:
Option 1. Keep the current counting algorithm and increase
MAX_MARKUP_LENGTHto a value higher thanMAX_COMMENT_LENGTH.This ensures the entire message is parsed, as well as prevents the message length jumping behavior in the composer near
MAX_COMMENT_LENGTH.Option 2. Make
MAX_MARKUP_LENGTHequal toMAX_COMMENT_LENGTH, then:getCommentLengthmust be changed to count the unprocessed message (i.e. how it's typed by the user). This counting change should also be applied to the BE, which may make it more complex as the BE does not currently see the plaintext message and secondly due to storage limit considerations and the need to validate this. After all, post-markdown processing is how the message is actually stored in the BE.text.length < CONST.MAX_MARKUP_LENGTHshould be made inclusive, i.e.<=instead of<Option 3. Optimize markdown processing speed and remove MAX_MARKUP_LENGTH altogether (or drastically increase)
Slack does not have any responsiveness issues when typing long messages (tried 375k+) in their composer and only limit the length of pasting long texts. I think there is plenty room for optimization and the only reason of existence (to my knowledge) for
MAX_MARKUP_LENGTHis to prevent performance degradation.Marked.js handles complex 100k markdown texts in the ~50ms range. CommonMark requires ~20ms. Expensify's implementation however requires tens of seconds (freezing the tab) due to its heavy reliance on regular expressions. It's a simple, but poorly performant implementation. Arguably this is the true root cause of the problem.
What alternative solutions did you explore? (Optional)
N/A
EDIT: Revised proposal.
melvin-bot commentedon Jun 28, 2023
📣 @samh-nl! 📣
Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork.
Please follow these steps:
Format:
yh-0218 commentedon Jun 28, 2023
Proposal
Please re-state the problem that we are trying to solve in this issue.
App should recognize mention in message with 15000 letters (max limit)
What is the root cause of that problem?
MAX_MARKUP_LENGTHandMAX_COMMENT_LENGTHare different.What changes do you think we should make in order to solve the problem?
MAX_COMMENT_LENGTHto 10,000 orMAX_MARKUP_LENGTHto 15,000What alternative solutions did you explore? (Optional)
sophiepintoraetz commentedon Jun 29, 2023
[-]Chat - Mention is not displayed in message with full 15000 letters[/-][+][$1000] Chat - Mention is not displayed in message with full 15000 letters[/+]melvin-bot commentedon Jun 29, 2023
Job added to Upwork: https://www.upwork.com/jobs/~015b3ce1ef6e882fd1
melvin-bot commentedon Jun 29, 2023
Current assignee @sophiepintoraetz is eligible for the External assigner, not assigning anyone new.
119 remaining items
sobitneupane commentedon Oct 9, 2023
This issue is resulted by multiple PRs and was kind of expected result back then.
sobitneupane commentedon Oct 9, 2023
Regression Test Proposal
Do we agree 👍 or 👎
sobitneupane commentedon Oct 9, 2023
#21817 (comment)
Requested payment on newDot.
JmillsExpensify commentedon Oct 9, 2023
$1,000 payment approved for @sobitneupane based on BZ summary.
JmillsExpensify commentedon Oct 9, 2023
Re-opening this issue since we still need the regression tested added to TestRail.
neil-marcellini commentedon Oct 12, 2023
Not over due
melvin-bot commentedon Oct 16, 2023
@sobitneupane, @neil-marcellini, @NickTooker, @sophiepintoraetz, @samh-nl Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!
sophiepintoraetz commentedon Oct 18, 2023
done https://github.com/Expensify/Expensify/issues/327101