Skip to content

fix(ddl_client): sleep for a while before retry once meta server is busy - #1453

Merged
empiredan merged 23 commits into
apache:masterfrom
empiredan:ddl-client-busy-retry
Apr 25, 2023
Merged

empiredan merged 23 commits into
apache:masterfrom
empiredan:ddl-client-busy-retry

Conversation

@empiredan

@empiredan empiredan commented Apr 18, 2023 •

Copy link
Copy Markdown
Contributor

#1449

Under some circumstances, creating/dropping table might spend more time,
which could even be greater than RPC timeout. Then, once DDL client launched
another requests to meta server, it would receive ERR_BUSY_CREATING or
ERR_BUSY_DROPPING. Between these two attempts there is no interval, thus
these attempts are ineffective.

Therefore, a retry mechanism is introduced to resolve this problem:

  • the max number of attempts is configurable;
  • the retry mechanism for failed sending request to meta server due to
    ERR_NETWORK_FAILURE or ERR_TIMEOUT is unchanged, and the max
    number of attempts is restrict by the configuration;
  • once response is ERR_BUSY_CREATING or ERR_BUSY_DROPPING, DDL
    client would also retry, and between two attempts there is an configurable
    interval.

Some new configurations are added:

[ddl_client]
+ ddl_client_max_attempt_count = 3
+ ddl_client_retry_interval_ms = 10000

@github-actions github-actions Bot added the cpp label Apr 18, 2023
Comment thread src/client/replication_ddl_client.cpp Resolved
Comment thread src/client/test/CMakeLists.txt Resolved
Comment thread src/client/replication_ddl_client.h Resolved
Comment thread src/client/replication_ddl_client.cpp Resolved
@empiredan
empiredan marked this pull request as draft April 20, 2023 03:11
@empiredan empiredan changed the title fix(ddl_client): after receiving ERR_BUSY_CREATING or ERR_BUSY_DROPPING sleep for a while before retry fix(ddl_client): sleep for a while before retry once meta server is busy Apr 22, 2023
@empiredan
empiredan marked this pull request as ready for review April 23, 2023 14:02
Comment thread src/client/replication_ddl_client.cpp Resolved
@empiredan
empiredan merged commit b11b45d into apache:master Apr 25, 2023
@empiredan empiredan mentioned this pull request Aug 21, 2023
@empiredan empiredan added the type/config-change Added or modified configuration that should be noted on release note of new version. label Nov 29, 2023
GehaFearless pushed a commit to GehaFearless/incubator-pegasus that referenced this pull request Feb 28, 2024
…erver is busy

https://jira.sensorsdata.cn/browse/SKV-618

社区issue:apache#1449

社区pr:apache#1453

Under some circumstances, creating/dropping table might spend more time,
which could even be greater than RPC timeout. Then, once DDL client launched
another requests to meta server, it would receive `ERR_BUSY_CREATING` or
`ERR_BUSY_DROPPING`. Between these two attempts there is no interval, thus
these attempts are ineffective.

Therefore, a retry mechanism is introduced to resolve this problem:

- the max number of attempts is configurable; 
- the retry mechanism for failed sending request to meta server due to
`ERR_NETWORK_FAILURE` or `ERR_TIMEOUT` is unchanged, and the max
number of attempts is restrict by the configuration;
- once response is `ERR_BUSY_CREATING` or `ERR_BUSY_DROPPING`, DDL
client would also retry, and between two attempts there is an configurable
interval.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build cpp github scripts type/config-change Added or modified configuration that should be noted on release note of new version.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants