Skip to content

[Tracker] modularize inferencing during and after training in the example scripts #6545

Closed

Description

@sayakpaul

We provide support for running validation inference during and after training in our officially maintained training examples. This is very helpful to keep track of the training progress.

We could modularize some bits in the example to reduce the LoC.

The train_lcm_distill_lora_sdxl.py script already does this:

def log_validation(vae, args, accelerator, weight_dtype, step, unet=None, is_final_validation=False):

It would be nice to follow something similar for the rest of the scripts too. Here's a handy list of the scripts where we'd like to incorporate this change:

Feel free to comment here if you're interested.

Also, when opening PRs, please target one example at a time and please tag me in the PRs.

Activity

  1. charchit7 commented on Jan 12, 2024

    @charchit7
    Contributor

    Hey @sayakpaul for text-to-image, SD
    log-validation is present :

    def log_validation(vae, text_encoder, tokenizer, unet, args, accelerator, weight_dtype, epoch):

    haven't gone through the code, will check more if we need to modify it.

  2. sayakpaul commented on Jan 12, 2024

    @sayakpaul
    MemberAuthor

    Ideally, this should be delegated to log_validation() too:

    # Run a final round of inference.

    As done in train_lcm_distill_lora_sdxl.py.

  3. charchit7 commented on Jan 12, 2024

    @charchit7
    Contributor

    Got it. @sayakpaul I missed that part. Thanks!

  4. kim-sangyeon commented on Jan 14, 2024

    @kim-sangyeon
    Contributor

    Hey @sayakpaul, @charchit7
    I'm interested in incorporating the change to InstructPix2Pix SD

  5. coolyashas commented on Jan 14, 2024

    @coolyashas

    Hey @sayakpaul, @charchit7 I'm interested in incorporating the change to InstructPix2Pix SD

    Hey @sang-k, I have been looking into InstructPix2Pix SD too. Would be happy to collaborate if necessary.

  6. kim-sangyeon commented on Jan 14, 2024

    @kim-sangyeon
    Contributor

    @coolyashas
    Oh, I see.
    I will look into InstructPix2Pix SDXL then!

  7. coolyashas commented on Jan 21, 2024

    @coolyashas

    Working on InstructPix2Pix SD

  8. changed the title [-][Examples] modularize inferencing during and after training[/-] [+][Tracker] modularize inferencing during and after training in the example scripts[/+] on Jan 29, 2024
  9. ParagEkbote commented on Mar 12, 2025

    @ParagEkbote
    Contributor

    Has this issue been resolved or are there any pending scripts I could contribute to modularize?

    cc: @sayakpaul

  10. sayakpaul commented on Mar 12, 2025

    @sayakpaul
    MemberAuthor

    Yeah I think this can be closed now :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions