refactor DPMSolverMultistepScheduler using sigmas #4986

yiyixuxu · 2023-09-12T02:19:41Z

I think I overcomplicated things with this #4690 by trying to follow k-diffusion implementation.

trying it again with a simpler approach here: we simply calculate lambda_t, sigma_t and alpha_t from sigma

to-do

dpmsolver_multistep
dpmsolver_singlestep
DEIS
UniPC

testing

Notes:
I compared the results against the current implementation (for dpm multistep only). There is some slight numerical difference when using k-sigmas; for example, in the last testing examples below on the 4th row (sde-dpmsolver++, use_karras_sigma=True), you can visually see the outputs are slightly different.
Also, I had to loosen a test here #4986 (comment).
However, the new implementation is more accurate by using the sigma directly.

import torch
from diffusers import StableDiffusionKDiffusionPipeline, DPMSolverMultistepScheduler, StableDiffusionPipeline
import gc

# make sure to update your branch name 
branch = 'dpm-mstep-sigma-2'
#branch = "main"


# test 1: dpmsolver_+++, use_karras_sigma=False 
seed = 33

pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")
pipe = pipe.to("cuda")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config, use_karras_sigmas=False
)

prompt = "an astronaut riding a horse on mars"

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(prompt, generator=generator, num_inference_steps=20).images[0]

image.save(f"[{branch}]_test_dpmsolver++.png")


# test 2: dpmsolver_+++, use_karras_sigma=True 
seed = 33

pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")
pipe = pipe.to("cuda")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config, use_karras_sigmas=True
)

prompt = "an astronaut riding a horse on mars"

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(prompt, generator=generator, num_inference_steps=20).images[0]

image.save(f"[{branch}]_test_dpmsolver++_k_sigma.png")

# # test 3: sde-dpmsolver++, use_karras_sigma=False 

seed = 33

pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")
pipe = pipe.to("cuda")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config, use_karras_sigmas=False, algorithm_type="sde-dpmsolver++"
)

prompt = "an astronaut riding a horse on mars"

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(prompt, generator=generator, num_inference_steps=20).images[0]

image.save(f"[{branch}]_test_sde-dpmsolver++.png")

## test 4: sde-dpmsolver++, use_karras_sigma=True
seed = 33

pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")
pipe = pipe.to("cuda")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config, use_karras_sigmas=True, algorithm_type="sde-dpmsolver++"
)

prompt = "an astronaut riding a horse on mars"

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(prompt, generator=generator, num_inference_steps=20).images[0]
image.save(f"[{branch}]_test_sde-dpmsolver++_k_sigma.png")

main	this PR

add print lines add print lines and change

yiyixuxu · 2023-09-13T06:27:28Z

tests/schedulers/test_scheduler_dpm_multi_inverse.py

@@ -54,6 +54,7 @@ def check_over_configs(self, time_step=0, **config):

            output, new_output = sample, sample
            for t in range(time_step, time_step + scheduler.config.solver_order + 1):
+                t = scheduler.timesteps[t]


We used 0,1,2 directly as timestep index here, but they are not even in the self.timestepes. In the previous implementation, it will default to the last timesteps when it is outside of the timesteps range, but I don't think it is intended. I changed it here, I think it makes more sense this way. Let me know if it's not the case

src/diffusers/schedulers/scheduling_dpmsolver_multistep.py

yiyixuxu · 2023-09-19T02:15:32Z

tests/schedulers/test_scheduler_unipc.py

@@ -241,11 +242,3 @@ def test_fp16_support(self):
            sample = scheduler.step(residual, t, sample).prev_sample

        assert sample.dtype == torch.float16
-
-    def test_unique_timesteps(self, **config):


don't need this test anymore because we allow duplicated timesteps now

yiyixuxu · 2023-09-19T03:34:11Z

@patrickvonplaten another review please :)

patrickvonplaten · 2023-09-19T09:48:34Z

tests/schedulers/test_scheduler_dpm_multi.py

        for scheduler_class in self.scheduler_classes:
            scheduler_config = self.get_scheduler_config(**config)
            scheduler = scheduler_class(**scheduler_config)

            scheduler.set_timesteps(scheduler.config.num_train_timesteps)
-            assert len(scheduler.timesteps.unique()) == scheduler.num_inference_steps
+            assert len(scheduler.timesteps) == scheduler.num_inference_steps


patrickvonplaten

Very nice!

rvorias · 2023-09-22T12:04:43Z

@yiyixuxu This refactor has broken stable diffusion (1.5) img2img for unipc

both use denoising strength = 0.6

Using unipc code before this commit:

Using unipc after this commit:

ljk1291 · 2023-09-22T13:19:16Z

@yiyixuxu This refactor has broken stable diffusion (1.5) img2img for unipc

both use denoising strength = 0.6

Using unipc code before this commit:

Using unipc after this commit:

Same with DPMSolverMultistepScheduler using Karras sigmas. Just returns noise with anything under 1.0 denoising strength. Havent tested it without Karras sigmas.

yiyixuxu · 2023-09-22T18:11:27Z

@ljk1291
oh, thanks for reporting! I'm looking into it now!
if you have a reproducible example and can open a new issue - that would be great!

burgalon · 2023-10-17T12:01:22Z

it seems like this bug still resurfaces somehow when using the legacy inpainting pipeline for some reason

yiyixuxu · 2023-10-17T23:20:31Z

Hi @burgalon

can you be more specific? could you maybe open a bug report and provide an example?

thanks!

YiYi

burgalon · 2023-11-26T08:14:16Z

@yiyixuxu
oops... happened to reply here #4631 (comment)

--------- Co-authored-by: yiyixuxu <yixu310@gmail,com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

yiyixuxu and others added 29 commits July 21, 2023 03:52

add index_counter

8f78025

update test

3b886af

update

a05a13a

add print lines add print lines and change

fix

670c782

style

c95b545

Merge branch 'main' into dpm-mstep-sigma

9eeb5e9

sde-dpmsolver+++

515c105

v_prediction

a85a18c

remove round

f238e0d

try it differently

7198998

add print lines

67ef0e3

add

86601bc

add

0445412

style

f687c7d

add

48a9b1e

fix

b3bf644

fix calculation

c5a6cdb

update add_noise

3932182

remove the unused timestep args

2f785fb

refactor dpm inverse

f39cc96

fix last sigma + unique timesteps

b28320e

add sigma_max

8431b9a

fix test t -> timesteps[t]

3e0826b

inverse

3d31065

add

dd8ec06

Merge remote-tracking branch 'origin/main' into dpm-mstep-sigma-2

b7f910c

doc string for step index

709095a

fix

af5fcd6

2e-3

9128336

yiyixuxu commented Sep 13, 2023

View reviewed changes

yiyixuxu added 5 commits September 17, 2023 06:08

deprecate

25c0432

update dpm singlestep

911ca6a

fix

65fe7c3

style

d53056c

deis

13729a9

patrickvonplaten mentioned this pull request Sep 18, 2023

DPMSolverSinglestepScheduler with use_karras_sigmas=True triggers RuntimeError: a Tensor with 2 elements cannot be converted to Scalar #4631

Closed

yiyixuxu added 3 commits September 18, 2023 21:20

add unipc

b7223a8

fix

02d07d4

flip sigmas

466cb53

yiyixuxu commented Sep 19, 2023

View reviewed changes

src/diffusers/schedulers/scheduling_dpmsolver_multistep.py Show resolved Hide resolved

yiyixuxu commented Sep 19, 2023

View reviewed changes

patrickvonplaten reviewed Sep 19, 2023

View reviewed changes

patrickvonplaten approved these changes Sep 19, 2023

View reviewed changes

yiyixuxu merged commit 8263cf0 into main Sep 19, 2023
11 of 12 checks passed

yiyixuxu deleted the dpm-mstep-sigma-2 branch September 19, 2023 21:21

yiyixuxu mentioned this pull request Sep 19, 2023

DPMSolverMultistepScheduler has wrong number of timesteps given num_inference_steps with StableDiffusionXL pipelines #4122

Closed

yiyixuxu mentioned this pull request Sep 23, 2023

fix the add_noise function for dpm-multi et al #5158

Merged

yiyixuxu mentioned this pull request Oct 18, 2023

Fix an error when DPM Solver Single Step contains duplicate timesteps #4231

Closed

yiyixuxu mentioned this pull request Jan 8, 2024

add Sa-Solver #5975

Merged

yiyixuxu mentioned this pull request Jan 15, 2024

fix DPM Scheduler with use_karras_sigmas option #6477

Merged

yiyixuxu mentioned this pull request Sep 24, 2024

Profile and speedup pipelines #9475

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor DPMSolverMultistepScheduler using sigmas #4986

refactor DPMSolverMultistepScheduler using sigmas #4986

yiyixuxu commented Sep 12, 2023 •

edited

Loading

yiyixuxu Sep 13, 2023 •

edited

Loading

yiyixuxu Sep 19, 2023

yiyixuxu commented Sep 19, 2023

patrickvonplaten Sep 19, 2023

patrickvonplaten left a comment

rvorias commented Sep 22, 2023 •

edited

Loading

ljk1291 commented Sep 22, 2023 •

edited

Loading

yiyixuxu commented Sep 22, 2023

burgalon commented Oct 17, 2023

yiyixuxu commented Oct 17, 2023

burgalon commented Nov 26, 2023 •

edited

Loading

refactor DPMSolverMultistepScheduler using sigmas #4986

refactor DPMSolverMultistepScheduler using sigmas #4986

Conversation

yiyixuxu commented Sep 12, 2023 • edited Loading

to-do

testing

yiyixuxu Sep 13, 2023 • edited Loading

Choose a reason for hiding this comment

yiyixuxu Sep 19, 2023

Choose a reason for hiding this comment

yiyixuxu commented Sep 19, 2023

patrickvonplaten Sep 19, 2023

Choose a reason for hiding this comment

patrickvonplaten left a comment

Choose a reason for hiding this comment

rvorias commented Sep 22, 2023 • edited Loading

ljk1291 commented Sep 22, 2023 • edited Loading

yiyixuxu commented Sep 22, 2023

burgalon commented Oct 17, 2023

yiyixuxu commented Oct 17, 2023

burgalon commented Nov 26, 2023 • edited Loading

yiyixuxu commented Sep 12, 2023 •

edited

Loading

yiyixuxu Sep 13, 2023 •

edited

Loading

rvorias commented Sep 22, 2023 •

edited

Loading

ljk1291 commented Sep 22, 2023 •

edited

Loading

burgalon commented Nov 26, 2023 •

edited

Loading