[Task]: 增加worker进程与scheduler进程断链后重试次数

### Checklist - [x] 1. 我已经搜索过相关问题，但没有得到预期的帮助. (https://gitee.com/mindspore/mindspore/issues) - [x] 2. 最新版本中该错误尚未修复. - [x] 3. 请注意，如果您提交的Bug描述缺少相应的环境信息和最小可复现的demo，我们将很难复现和解决该问题，从而降低收到反馈的可能性，甚至该问题将被关闭. ### 🐞 问题详细描述 worker进程与scheduler进程之间用心跳保活，当心跳消息没有被worker进程正常接收到时，worker会向scheduler重连接一次（通过Reconnect函数）： ```cpp bool ComputeGraphNode::Heartbeat() { ··· while (enable_hb_) { HeartbeatMessage hb_msg; hb_msg.set_node_id(node_id_); const auto &server_url = meta_server_addr_.GetUrl(); std::string content = hb_msg.SerializeAsString(); auto message = CreateMessage(server_url, MessageName::kHeartbeat, content); MS_EXCEPTION_IF_NULL(message); MS_VLOG(VL_DISTRIBUTED_TRACE) << "Start heart beat."; MessageBase *response = hb_client_->ReceiveSync(std::move(message)); if (response == nullptr) { MS_LOG(ERROR) << "Failed to send heartbeat message to meta server node and try to reconnect to the meta server."; if (!Reconnect()) { if (!enable_recovery_ && topo_state_ != TopoState::kInitializing) { topo_state_ = TopoState::kFailed; if (abnormal_callback_ != nullptr) { (*abnormal_callback_)(); } MS_LOG(EXCEPTION) << "Failed to connect to the meta server. Maybe it has exited. Please check scheduler's log."; } else { MS_LOG(ERROR) << "Failed to connect to the meta server. Maybe it has exited. Please check scheduler's log."; } } } ··· ``` 实际场景中，host侧的网络波动导致的心跳断链或者路由阻塞导致心跳消息丢失，往往会持续一段时间；在host异常时间内重连接也会失败，导致集群被标记为异常，然后退出；建议增加重连次数，而不是仅尝试重连一次。 ### 接口变更 新增环境变量`MS_HEARTBEAT_RETRY_TIMEOUT`来控制重连的超时时间，默认为20(秒)。