[ghstack-poisoned]

1. We want to take option 3 as discussed in #135712, so every time when we retry, we create a new TCPStore so that we don't need to append attemp prefix and avoid eventually TCPStore sync failure.
2. Upon checking the code, we creating a new TCPStore in c10d_rendezvous_backend.py before we even hitting the the logic inside dynamic_rendezvous.py. Since we use `multi_tenant=True`, so we don't create new server in TCPStore.cpp if this flag is set:
```cpp
if (opts.multiTenant) {
    std::lock_guard<std::mutex> guard{cache_mutex_};

    // If the caller is okay with a multi-tenant store, first check if we
    // already have a TCPServer running on the specified port.
    if (opts.port > 0) {
      auto pos = cachedServers_.find(opts.port);
      if (pos != cachedServers_.end()) {
        server = pos->second.lock();
        if (server != nullptr) {
          return server;
        }

        // Looks like the TCPStore has been disposed, make sure that we release
        // the control block.
        cachedServers_.erase(pos);
      }
    }

    server = startCore();

    cachedServers_.emplace(server->port(), server);
  }
```
so if we are sure the TCPStore server is destroyed everytime we retry, we are recreating a TCPStore server already; otherwise, we need more changes to the elastic code.
3. This changes will force broadcasting a new mast address and port everytime when dynamic_rendezvous's `next_rendezvous` is called.

Send a PR first to collect feedback.

cc XilunWu H-Huang awgu kwen2501 wanchaol fegin wz337 wconstab d4l3k c-p-i-o 

[ghstack-poisoned]

…ous"


1. We want to take option 3 as discussed in #135712, so every time when we retry, we create a new TCPStore so that we don't need to append attemp prefix and avoid eventually TCPStore sync failure.
2. Upon checking the code, we creating a new TCPStore in c10d_rendezvous_backend.py before we even hitting the the logic inside dynamic_rendezvous.py. Since we use `multi_tenant=True`, so we don't create new server in TCPStore.cpp if this flag is set:
```cpp
if (opts.multiTenant) {
    std::lock_guard<std::mutex> guard{cache_mutex_};

    // If the caller is okay with a multi-tenant store, first check if we
    // already have a TCPServer running on the specified port.
    if (opts.port > 0) {
      auto pos = cachedServers_.find(opts.port);
      if (pos != cachedServers_.end()) {
        server = pos->second.lock();
        if (server != nullptr) {
          return server;
        }

        // Looks like the TCPStore has been disposed, make sure that we release
        // the control block.
        cachedServers_.erase(pos);
      }
    }

    server = startCore();

    cachedServers_.emplace(server->port(), server);
  }
```
so if we are sure the TCPStore server is destroyed everytime we retry, we are recreating a TCPStore server already; otherwise, we need more changes to the elastic code.
3. This changes will force broadcasting a new mast address and port everytime when dynamic_rendezvous's `next_rendezvous` is called.

Send a PR first to collect feedback.

cc XilunWu H-Huang awgu kwen2501 wanchaol fegin wz337 wconstab d4l3k c-p-i-o 

[ghstack-poisoned]

…ous"


1. We want to take option 3 as discussed in #135712, so every time when we retry, we create a new TCPStore so that we don't need to append attemp prefix and avoid eventually TCPStore sync failure.
2. Upon checking the code, we creating a new TCPStore in c10d_rendezvous_backend.py before we even hitting the the logic inside dynamic_rendezvous.py. Since we use `multi_tenant=True`, so we don't create new server in TCPStore.cpp if this flag is set:
```cpp
if (opts.multiTenant) {
    std::lock_guard<std::mutex> guard{cache_mutex_};

    // If the caller is okay with a multi-tenant store, first check if we
    // already have a TCPServer running on the specified port.
    if (opts.port > 0) {
      auto pos = cachedServers_.find(opts.port);
      if (pos != cachedServers_.end()) {
        server = pos->second.lock();
        if (server != nullptr) {
          return server;
        }

        // Looks like the TCPStore has been disposed, make sure that we release
        // the control block.
        cachedServers_.erase(pos);
      }
    }

    server = startCore();

    cachedServers_.emplace(server->port(), server);
  }
```
so if we are sure the TCPStore server is destroyed everytime we retry, we are recreating a TCPStore server already; otherwise, we need more changes to the elastic code.
3. This changes will force broadcasting a new mast address and port everytime when dynamic_rendezvous's `next_rendezvous` is called.

Send a PR first to collect feedback.

cc XilunWu H-Huang awgu kwen2501 wanchaol fegin wz337 wconstab d4l3k c-p-i-o 

[ghstack-poisoned]

…ous"


1. We want to take option 3 as discussed in #135712, so every time when we retry, we create a new TCPStore so that we don't need to append attemp prefix and avoid eventually TCPStore sync failure.
2. Upon checking the code, we creating a new TCPStore in c10d_rendezvous_backend.py before we even hitting the the logic inside dynamic_rendezvous.py. Since we use `multi_tenant=True`, so we don't create new server in TCPStore.cpp if this flag is set:
```cpp
if (opts.multiTenant) {
    std::lock_guard<std::mutex> guard{cache_mutex_};

    // If the caller is okay with a multi-tenant store, first check if we
    // already have a TCPServer running on the specified port.
    if (opts.port > 0) {
      auto pos = cachedServers_.find(opts.port);
      if (pos != cachedServers_.end()) {
        server = pos->second.lock();
        if (server != nullptr) {
          return server;
        }

        // Looks like the TCPStore has been disposed, make sure that we release
        // the control block.
        cachedServers_.erase(pos);
      }
    }

    server = startCore();

    cachedServers_.emplace(server->port(), server);
  }
```
so if we are sure the TCPStore server is destroyed everytime we retry, we are recreating a TCPStore server already; otherwise, we need more changes to the elastic code.
3. This changes will force broadcasting a new mast address and port everytime when dynamic_rendezvous's `next_rendezvous` is called.

Send a PR first to collect feedback.

cc XilunWu H-Huang awgu kwen2501 wanchaol fegin wz337 wconstab d4l3k c-p-i-o 

[ghstack-poisoned]

…ous"


1. We want to take option 3 as discussed in #135712, so every time when we retry, we create a new TCPStore server first so that we don't need to append attempt count as prefix and avoid eventually TCPStore sync failure. (This is only for the TCPStore sharing enabled case) 
2. We reuse the port from the newly created TCPStore server and first bind it to port 0 so that the system will allocate a new available one to us.
3. Then the port be broadcasted for dynamic_rendezvous.

Only one more question, what do we do about the store created from (_create_tcp_store) torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py, are we ok with creating a duplicate TCPStore server?

cc XilunWu H-Huang awgu kwen2501 wanchaol fegin wz337 wconstab d4l3k c-p-i-o 

[ghstack-poisoned]

1. We want to take option 3 as discussed in #135712, so every time when we retry, we create a new TCPStore server first so that we don't need to append attempt count as prefix and avoid eventually TCPStore sync failure. (This is only for the TCPStore sharing enabled case) 
2. We start a new server bound to an ephemeral port (i.e. 0) so it gets assigned to a free port. We then pass that downstream (trainer or c10d). By doing so, TCPStore is managed by the elastic agent rather than having a race condition on binding to a specific port in the trainer.
3. Then the port be broadcasted for dynamic_rendezvous.

Only one more question, what do we do about the store created from (_create_tcp_store) torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py, are we ok with creating a duplicate TCPStore server?

cc XilunWu H-Huang awgu kwen2501 wanchaol fegin wz337 wconstab d4l3k c-p-i-o 

[ghstack-poisoned]

1. We want to take option 3 as discussed in #135712, so every time when we retry, we create a new TCPStore server first so that we don't need to append attempt count as prefix and avoid eventually TCPStore sync failure. (This is only for the TCPStore sharing enabled case) 
2. We start a new server bound to an ephemeral port (i.e. 0) so it gets assigned to a free port. We then pass that downstream (trainer or c10d). By doing so, TCPStore is managed by the elastic agent rather than having a race condition on binding to a specific port in the trainer.
3. Then the port be broadcasted for dynamic_rendezvous.

Only one more question, what do we do about the store created from (_create_tcp_store) torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py, are we ok with creating a duplicate TCPStore server?

cc XilunWu H-Huang awgu kwen2501 wanchaol fegin wz337 wconstab d4l3k c-p-i-o 

[ghstack-poisoned]