Skip to content

Performance issues by many reports in the database #77

Open

Description

@filipwoz

Unfortunately the new KLOV has still the same problem as the old one: When database is growing the performance of loading the reports is continuously falling until it exceeds user acceptance boarder.

I observed it for example with the following number of records in the Mongo collections:

Table | Record Count
project | 18
report | 340
test | 15.230
log | 198.085
media | 25.388
exception | 502

After deleting projects and getting below the log < 110.000, media < 15.000 it was acceptable again, but sorry it is quite unprofessional that here no strategy is offered to help that situation. Have you through an optimizing the queries?

This ways actually my main reason why I have not bought the KLOV professional licence for the old version and why I evaluated the new one. In such a shape I am also no willing to spend money on the new KLOV.

Activity

  1. filipwoz commented on Oct 1, 2021

    @filipwoz
    Author

    I have helped myself temporarily by writing a utility working on MongoDB level which is deleting reports older than X working days and X is configurable per KLOV project name. I am scheduling it every day, keeping my KLOV usable this way.

    But at the end the performance is an issue here and you should work on it. I have an impression that you are not using the advantages of No-SQL DB properly. Why to use No-SQL when at the end you are keeping the test reports decomposed in different collections and you have to join it before rendering. With such data model a normal SQL DB would be quicker in my opinion.
    In a No-SQL DB like Mongo you could have been saving the complete test reports as JSON documents in one collection (report/test/log/exception all in one collection) without a need to join anything after that. Then the real benefit of No-SQL would be utilized. I am not sure about it because I am also new to No-SQL but this is how I understand it. No-SQL at the end should boost performance and not the other way round - it is its main benefit.

  2. iamakshayshar commented on Feb 10, 2022

    @iamakshayshar

    @filipwoz : if possible, can you please share the utility and its documentation with me.
    email : akshay.sharma979@gmail.com

    Thanks in advance.

  3. parkerqian commented on Feb 18, 2022

    @parkerqian

    I have helped myself temporarily by writing a utility working on MongoDB level which is deleting reports older than X working days and X is configurable per KLOV project name. I am scheduling it every day, keeping my KLOV usable this way.

    But at the end the performance is an issue here and you should work on it. I have an impression that you are not using the advantages of No-SQL DB properly. Why to use No-SQL when at the end you are keeping the test reports decomposed in different collections and you have to join it before rendering. With such data model a normal SQL DB would be quicker in my opinion. In a No-SQL DB like Mongo you could have been saving the complete test reports as JSON documents in one collection (report/test/log/exception all in one collection) without a need to join anything after that. Then the real benefit of No-SQL would be utilized. I am not sure about it because I am also new to No-SQL but this is how I understand it. No-SQL at the end should boost performance and not the other way round - it is its main benefit.

    I have helped myself temporarily by writing a utility working on MongoDB level which is deleting reports older than X working days and X is configurable per KLOV project name. I am scheduling it every day, keeping my KLOV usable this way.

    But at the end the performance is an issue here and you should work on it. I have an impression that you are not using the advantages of No-SQL DB properly. Why to use No-SQL when at the end you are keeping the test reports decomposed in different collections and you have to join it before rendering. With such data model a normal SQL DB would be quicker in my opinion. In a No-SQL DB like Mongo you could have been saving the complete test reports as JSON documents in one collection (report/test/log/exception all in one collection) without a need to join anything after that. Then the real benefit of No-SQL would be utilized. I am not sure about it because I am also new to No-SQL but this is how I understand it. No-SQL at the end should boost performance and not the other way round - it is its main benefit.

    agree with you .
    My other problem is that my testng.xml includes so many case (perhaps more then 800) .Every case will output log info .
    At the end the total document is more then 16MB , mongodb can not hander that @iamakshayshar

  4. iamakshayshar commented on Feb 19, 2022

    @iamakshayshar

    I have helped myself temporarily by writing a utility working on MongoDB level which is deleting reports older than X working days and X is configurable per KLOV project name. I am scheduling it every day, keeping my KLOV usable this way.

    But at the end the performance is an issue here and you should work on it. I have an impression that you are not using the advantages of No-SQL DB properly. Why to use No-SQL when at the end you are keeping the test reports decomposed in different collections and you have to join it before rendering. With such data model a normal SQL DB would be quicker in my opinion. In a No-SQL DB like Mongo you could have been saving the complete test reports as JSON documents in one collection (report/test/log/exception all in one collection) without a need to join anything after that. Then the real benefit of No-SQL would be utilized. I am not sure about it because I am also new to No-SQL but this is how I understand it. No-SQL at the end should boost performance and not the other way round - it is its main benefit.

    I have helped myself temporarily by writing a utility working on MongoDB level which is deleting reports older than X working days and X is configurable per KLOV project name. I am scheduling it every day, keeping my KLOV usable this way.

    But at the end the performance is an issue here and you should work on it. I have an impression that you are not using the advantages of No-SQL DB properly. Why to use No-SQL when at the end you are keeping the test reports decomposed in different collections and you have to join it before rendering. With such data model a normal SQL DB would be quicker in my opinion. In a No-SQL DB like Mongo you could have been saving the complete test reports as JSON documents in one collection (report/test/log/exception all in one collection) without a need to join anything after that. Then the real benefit of No-SQL would be utilized. I am not sure about it because I am also new to No-SQL but this is how I understand it. No-SQL at the end should boost performance and not the other way round - it is its main benefit.

    agree with you .
    My other problem is that my testng.xml includes so many case (perhaps more then 800) .Every case will output log info .
    At the end the total document is more then 16MB , mongodb can not hander that @iamakshayshar

    Yes, this is also the issue with Klov. One other problem is lack of support. 😟

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions