I didnt understand this comment which took me sometime to solve for, adding what I found so others can use it:

We have downloaded models locally, you can also load from huggingface control_sd15_seg is converted from control_sd15_seg.safetensors using instructions above and

To use any control model already present in control net models , then the way to do it is :

1. Download the models and annotators from the controlnet huggingface repo and place it under models folder: https://huggingface.co/lllyasviel/ControlNet

2. Use the takuma repo and download the diffusers

pip install "git+https://github.com/takuma104/diffusers.git@controlnet"
git clone https://github.com/takuma104/diffusers.git
cd diffusers
git checkout controlnet

3. Run this script to create the model which can be used with the pipeline, for which you will need to install transformers:
   conda install -c huggingface transformers
python ./scripts/convert_controlnet_to_diffusers.py --checkpoint_path [./models/control_sd15_canny.pth] --dump_path [./models/control_sd15_canny]--device cpu

4. check path of installed diffusers :
   assume you already know the absolute path of installed diffusers
   cp pipeline_stable_diffusion_controlnet_inpaint.py PATH/pipelines/stable_diffusion
   The PATH of the installed diffusers is not the same as the diffusers folder present in the repo. This installed diffusers will be present in the anaconda3/envs/control/lib/python3.8/site-packages/diffusers

5. Importing in files:

In addition to importing it in the following

PATH/pipelines/__init__.py
PATH/__init__.py

add the following line to the pipelines/stable_diffusion/init.py as well
from .pipeline_stable_diffusion_controlnet_inpaint import StableDiffusionControlNetInpaintPipeline

After this the code should work.