Skip to content

feat: improve performance of count_data - #1091

Merged
empiredan merged 13 commits into
apache:masterfrom
GehaFearless:lgh_count_data
Aug 30, 2022
Merged

empiredan merged 13 commits into
apache:masterfrom
GehaFearless:lgh_count_data

Conversation

@GehaFearless

@GehaFearless GehaFearless commented Aug 1, 2022 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

issue: #1090
same job as before: #728

When we precisely count data for a large table, it will cost minutes or hours.
However, it's unnecessarily return key-values from server to client.

What is changed and how does it work?

Actually, we just need the count of data. So we just need transfer the count of data from server to client, but not the detailed data.

In my test, it will 2x on onebox faster than before.
for my bench on onebox๏ผš
1> before
wecom-temp-d7e2a1e011357a0ca4c3ad44d910e69e

2> this commit
image

Tests
  • Unit test
  • Manual test (add detailed scripts or steps below)
Related changes
  • Need to update the documentation
  • Need to be included in the release note

@empiredan empiredan changed the title improve performance of count_data feat: improve performance of count_data Aug 1, 2022
Comment thread src/shell/commands/data_operations.cpp Resolved
Comment thread src/shell/commands/data_operations.cpp Resolved
Comment thread src/shell/commands/data_operations.cpp Resolved
Comment thread src/server/pegasus_server_impl.cpp Resolved
Comment thread src/server/pegasus_server_impl.cpp Resolved
Comment thread src/server/pegasus_server_impl.cpp Resolved
Comment thread src/server/pegasus_server_impl.cpp Resolved
Comment thread src/shell/command_helper.h Resolved
Comment thread src/shell/command_helper.h Resolved
@GehaFearless
GehaFearless force-pushed the lgh_count_data branch 2 times, most recently from f4c5af1 to 1be846c Compare August 3, 2022 11:44
Comment thread idl/rrdb.thrift Resolved
Comment thread src/client_lib/pegasus_scanner_impl.cpp Resolved
Comment thread src/client_lib/pegasus_scanner_impl.cpp Resolved
Comment thread src/client_lib/pegasus_scanner_impl.cpp Resolved
Comment thread src/client_lib/pegasus_scanner_impl.cpp Resolved
Comment thread idl/rrdb.thrift Resolved
Comment thread src/server/pegasus_server_impl.cpp Resolved
Comment thread src/server/pegasus_server_impl.cpp Resolved
Comment thread src/server/pegasus_server_impl.cpp Resolved
Comment thread src/server/pegasus_server_impl.cpp Resolved
Comment thread src/server/pegasus_server_impl.cpp Resolved
Comment thread src/client_lib/pegasus_scanner_impl.cpp Resolved
Comment thread src/server/pegasus_server_impl.cpp Resolved
Comment thread src/server/pegasus_server_impl.cpp Resolved
Comment thread src/server/pegasus_server_impl.cpp Resolved
Comment thread src/server/pegasus_server_impl.cpp Resolved
@GehaFearless
GehaFearless force-pushed the lgh_count_data branch 3 times, most recently from 8d2fe21 to bb901e0 Compare August 11, 2022 03:40
@acelyc111

Copy link
Copy Markdown
Member

Could you give some performance comparation with the old version ?

Comment thread src/server/pegasus_server_impl.cpp Resolved
Comment thread src/client_lib/pegasus_scanner_impl.cpp Resolved
Comment thread src/client_lib/pegasus_scanner_impl.cpp Resolved
Comment thread src/server/pegasus_server_impl.cpp Resolved
@GehaFearless
GehaFearless force-pushed the lgh_count_data branch 3 times, most recently from 2e0e6fd to dad6442 Compare August 15, 2022 14:06
Comment thread src/server/pegasus_server_impl.cpp Resolved
Comment thread src/test/function_test/test_scan.cpp Resolved
Comment thread src/shell/commands/data_operations.cpp Resolved
Comment thread idl/rrdb.thrift Resolved
Comment thread src/client_lib/pegasus_scanner_impl.cpp Resolved
Comment thread src/client_lib/pegasus_scanner_impl.cpp Resolved
Comment thread src/client_lib/pegasus_scanner_impl.cpp Resolved
Comment thread src/shell/commands/data_operations.cpp Resolved
Comment thread src/client_lib/pegasus_client_impl.h Resolved
Comment thread src/test/function_test/test_scan.cpp Resolved
Comment thread src/client_lib/pegasus_scanner_impl.cpp Resolved
Comment thread src/client_lib/pegasus_client_impl.h Resolved
Comment thread src/client_lib/pegasus_client_impl.h Resolved
acelyc111
acelyc111 previously approved these changes Aug 24, 2022
Comment thread src/client_lib/pegasus_scanner_impl.cpp Resolved
Comment thread src/client_lib/pegasus_scanner_impl.cpp Resolved
Comment thread src/shell/commands/data_operations.cpp Resolved
acelyc111 and others added 3 commits August 29, 2022 11:52
Co-authored-by: Dan Wang <wangdan@apache.org>
Co-authored-by: Dan Wang <wangdan@apache.org>
Co-authored-by: Dan Wang <wangdan@apache.org>
@empiredan
empiredan merged commit 7e26c9f into apache:master Aug 30, 2022
@empiredan empiredan mentioned this pull request Aug 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants