🐛 Describe the bug

In the example below, we're wrapping a simple model with FSDP, then saving a checkpoint using torch.distributed.checkpoint API. The checkpoint saves successfully, no errors. But in forward, an error occurs:

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 74, in _wrap
    fn(i, *args)
  File "/home/adrian/repositories/lightning/examples/pytorch/bug_report/bug_report_model.py", line 47, in main
    _ = model(x1, x2)
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 844, in forward
    args, kwargs = _root_pre_forward(self, self, args, kwargs)
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 540, in _root_pre_forward
    _lazy_init(state, module)
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 171, in _lazy_init
    _share_state_and_init_handle_attrs(state, root_module)
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 239, in _share_state_and_init_handle_attrs
    _p_assert(
  File "/home/adrian/.conda/envs/lightning/lib/python3.10/site-packages/torch/distributed/utils.py", line 147, in _p_assert
    raise AssertionError(s)
AssertionError: Non-root FSDP instance's `_is_root` should not have been set yet or should have been set to `False`

Reproducing code:

import os
import torch
import torch.multiprocessing as mp
import torch.distributed as dist
from torch.distributed.checkpoint import FileSystemWriter, save_state_dict
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp.api import StateDictType
from torch.distributed.fsdp.wrap import ModuleWrapPolicy


class BoringModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.model = torch.nn.Sequential(torch.nn.Linear(5, 5), torch.nn.ReLU(), torch.nn.Linear(5, 1))
        self.pretrained_model = torch.nn.Sequential(
            torch.nn.Linear(5, 5),
            torch.nn.ReLU(),
        )

    def forward(self, x1, x2):
        x2_hidden = self.pretrained_model(x2)
        hiddens = torch.cat([x1, x2_hidden], dim=1)
        return self.model(hiddens)


def main(rank):
    os.environ["MASTER_ADDR"] = "localhost"
    os.environ["MASTER_PORT"] = "12355 "
    dist.init_process_group("nccl", rank=rank, world_size=2)
    torch.cuda.set_device(rank)
    
    model = BoringModel()
    model = FSDP(model, auto_wrap_policy=ModuleWrapPolicy({torch.nn.Linear}), device_id=rank)

    state_dict_type_context = FSDP.state_dict_type(
        module=model, state_dict_type=StateDictType.SHARDED_STATE_DICT,
    )

    with state_dict_type_context:
        state_dict = model.pretrained_model.state_dict()
    
    writer = FileSystemWriter(path="model.ckpt")
    save_state_dict(state_dict, writer)

    x1 = torch.ones((2, 6, 5))
    x2 = torch.ones((2, 6, 5)) * 2
    _ = model(x1, x2)


if __name__ == "__main__":
    mp.spawn(main, nprocs=2)

PyTorch nightly 2.2.0.dev20230920+cu121
Reported originally in Lightning: Lightning-AI/pytorch-lightning#18944

Versions

Collecting environment information...
PyTorch version: 2.2.0.dev20230920+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.26.0
Libc version: glibc-2.35

Python version: 3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-75-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB
GPU 4: NVIDIA A100-SXM4-40GB
GPU 5: NVIDIA A100-SXM4-40GB
GPU 6: NVIDIA A100-SXM4-40GB
GPU 7: NVIDIA A100-SXM4-40GB

Nvidia driver version: 525.125.06
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 43 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 256
On-line CPU(s) list: 0-254
Off-line CPU(s) list: 255
Vendor ID: AuthenticAMD
Model name: AMD EPYC 7742 64-Core Processor
CPU family: 23
Model: 49
Thread(s) per core: 2
Core(s) per socket: 64
Socket(s): 2
Stepping: 0
Frequency boost: enabled
CPU max MHz: 2250.0000
CPU min MHz: 0.0000
BogoMIPS: 4499.83
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es
Virtualization: AMD-V
L1d cache: 4 MiB (128 instances)
L1i cache: 4 MiB (128 instances)
L2 cache: 64 MiB (128 instances)
L3 cache: 512 MiB (32 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-63,128-191
NUMA node1 CPU(s): 64-127,192-254
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] mypy==1.4.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.2
[pip3] onnx==1.12.0
[pip3] onnxruntime==1.14.1
[pip3] pytorch-lightning==2.0.6
[pip3] pytorch-triton==2.1.0+6e4932cda8
[pip3] torch==2.2.0.dev20230920+cu121
[pip3] torchmetrics==1.0.3
[pip3] torchvision==0.17.0.dev20230920+cu121
[pip3] triton==2.0.0
[conda] numpy 1.24.2 pypi_0 pypi
[conda] pytorch-lightning 2.0.6 pypi_0 pypi
[conda] pytorch-triton 2.1.0+6e4932cda8 pypi_0 pypi
[conda] torch 2.2.0.dev20230920+cu121 pypi_0 pypi
[conda] torchmetrics 1.0.3 pypi_0 pypi
[conda] torchvision 0.17.0.dev20230920+cu121 pypi_0 pypi
[conda] triton 2.0.0 pypi_0 pypi

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @kiukchung @LucasLLC @d4l3k