27 Pull requests merged by 15 people

• [chore] fix: retain memory utility.

  #9543 merged Sep 28, 2024

• [train_instruct_pix2pix.py]Fix the LR schedulers when num_train_epochs is passed in a distributed training env

  #9316 merged Sep 28, 2024

• [refactor] remove conv_cache from CogVideoX VAE

  #9524 merged Sep 28, 2024

• [Core] fix variant-identification.

  #9253 merged Sep 28, 2024

• [Tests] [LoRA] clean up the serialization stuff.

  #9512 merged Sep 27, 2024

• [examples] add train flux-controlnet scripts in example.

  #9324 merged Sep 27, 2024

• [LoRA] make set_adapters() method more robust.

  #9535 merged Sep 27, 2024

• [bug] Precedence of operations in VAE should be slicing -> tiling

  #9342 merged Sep 26, 2024

• [Tests] Fix ChatGLMTokenizer

  #9536 merged Sep 26, 2024

• flux controlnet fix (control_modes batch & others)

  #9507 merged Sep 26, 2024

• [Schedulers] Add beta sigmas / beta noise schedule

  #9509 merged Sep 25, 2024

• [Doc] Improved level of clarity for latents_to_rgb.

  #9529 merged Sep 25, 2024

• update get_parameter_dtype

  #9526 merged Sep 25, 2024

• [docs] Model sharding

  #9521 merged Sep 25, 2024

• Update community_projects.md

  #9266 merged Sep 25, 2024

• [Community Pipeline] Batched implementation of Flux with CFG

  #9513 merged Sep 25, 2024

• Add exponential sigmas to other schedulers and update docs

  #9518 merged Sep 25, 2024

• a few fix for SingleFile tests

  #9522 merged Sep 24, 2024

• [CI] allow faster downloads from the Hub in CI.

  #9478 merged Sep 24, 2024

• [Doc] Fix path and and also import imageio

  #9506 merged Sep 23, 2024

• fix bugs for sd3 controlnet training

  #9489 merged Sep 23, 2024

• Add Noise Schedule/Schedule Type to Schedulers Overview documentation

  #9504 merged Sep 23, 2024

• Allow DDPMPipeline half precision

  #9222 merged Sep 23, 2024

• [Schedulers] Add exponential sigmas / exponential noise schedule

  #9499 merged Sep 23, 2024

• Fix the bug of sd3 controlnet training when using gradient checkpointing.

  #9498 merged Sep 23, 2024

• [Tests] Reduce the model size in the lumina test

  #8985 merged Sep 23, 2024

• [Cog] some minor fixes and nits

  #9466 merged Sep 23, 2024

12 Pull requests opened by 8 people

• [Tests] clean up and refactor gradient checkpointing tests

  #9494 opened Sep 23, 2024

• [WIP][LoRA] allow loras to be loaded with low_cpu_mem_usage.

  #9510 opened Sep 24, 2024

• Fix the bug that `joint_attention_kwargs` is not passed to the FLUX's transformer attention processors

  #9517 opened Sep 24, 2024

• SingleFile mapping dict copies

  #9533 opened Sep 26, 2024

• Fixed noise_pred_text referenced before assignment.

  #9537 opened Sep 26, 2024

• Add beta sigmas to other schedulers and update docs

  #9538 opened Sep 26, 2024

• [LoRA] support Kohya Flux LoRAs that have text encoders as well

  #9542 opened Sep 27, 2024

• [LoRA] Handle DoRA better

  #9547 opened Sep 28, 2024

• Fixed the issue on flux dreambooth lora training

  #9549 opened Sep 28, 2024

• Image dimension checking for ControlNet FLUX

  #9550 opened Sep 28, 2024

• Update distributed_inference.md to include `transformer.device_map`

  #9553 opened Sep 29, 2024

• [experiment] one shot decoding instead of framewise in Cog VAE

  #9554 opened Sep 29, 2024

21 Issues closed by 12 people

• TypeError: couldn't find storage object Float8_e4m3fnStorage

  #9298 closed Sep 29, 2024

• How to use x-labs flux controlnet models in diffusers?

  #9551 closed Sep 29, 2024

• ValueError: Multiple file extensions found at ./models/coreml-stable-diffusion-v1-4_original_packages.Cannot infer resource type from contents.

  #9471 closed Sep 28, 2024

• operator torchvision::nms does not exist

  #9109 closed Sep 28, 2024

• Inference speed half on version 0.26.0 upwards than 0.25.1

  #8880 closed Sep 27, 2024

• Training results do not make sense for sdxl+controlnet

  #9179 closed Sep 27, 2024

• [Flux ControlNet] ControlNet initialization from transformer seems to be broken

  #9540 closed Sep 27, 2024

• `lora_scale` has no effect when loading with Flux

  #9525 closed Sep 27, 2024

• Support loading quantized models from quanto directly

  #9058 closed Sep 26, 2024

• callback / cannot yield intermediate images on the fly during inference

  #9407 closed Sep 25, 2024

• Error Mapping on sd3, sdxl and upcoming flux controlnet training scripts

  #9523 closed Sep 25, 2024

• self.scheduler.add_noise did not add any noise for the given latent

  #9519 closed Sep 24, 2024

• AttributeError in `diffusers.models.transformers.transformer_flux` - Missing `rope` Attribute

  #9267 closed Sep 24, 2024

• Flux-dev - Unable to load LoRA weights after fp8 quantisation

  #9514 closed Sep 24, 2024

• [Schedulers] Add SGMUniform

  #9416 closed Sep 23, 2024

• SD3ControlNetModel forward function error

  #9496 closed Sep 23, 2024

• SDXL PAG with IPAdapter is not working.

  #9495 closed Sep 23, 2024

• SD3 + SDXL refine fix lying on grass. How to do in diffusers colab workflow?

  #8693 closed Sep 23, 2024

• More thorough guidance for multiple IP adapter images/masks and a single IP Adapter

  #8626 closed Sep 23, 2024

• Pixart-sigma is incompatible with mixed precision inference

  #8612 closed Sep 23, 2024

• Lumina pipeline fails to generate any image

  #9488 closed Sep 23, 2024

22 Issues opened by 20 people

• [Flux Controlnet] Add control_guidance_start and control_guidance_end

  #9555 opened Sep 29, 2024

• Using a custom pipeline with from_single_file

  #9552 opened Sep 29, 2024

• Still Issue on flux dreambooth lora training #9237

  #9548 opened Sep 28, 2024

• Flux Controlnet Train Example, will run out of memory on validation step

  #9546 opened Sep 28, 2024

• Training a specific Flux Controlnet Model from Controlnet-Union

  #9541 opened Sep 27, 2024

• "index_select_cuda" not implemented for 'Float8_e4m3fn' error from CogVideoXImageToVideoPipeline

  #9539 opened Sep 26, 2024

• [FluxMultiControlNetModel] object has no attribute 'config

  #9534 opened Sep 26, 2024

• Overflow error when using a pretrained repo locally

  #9532 opened Sep 25, 2024

• SDXL max sigma value should be doubled for 1024px generations

  #9531 opened Sep 25, 2024

• Freeing GPU memory after `torch.compile` StableDiffusionXLPipeline UNet

  #9530 opened Sep 25, 2024

• load_ip_adapter for distilled sd models

  #9528 opened Sep 25, 2024

• dtype error when using controlnet fp32 and mainpipe bf16

  #9527 opened Sep 25, 2024

• UNetMotionModel.dtype is really expensive to call, is it possible to cache it during inference?

  #9520 opened Sep 24, 2024

• parameters `joint_attention_kwargs` doesn't be passed to FLUX's transformers model

  #9516 opened Sep 24, 2024

• The `JointAttnProcessor2_0` in the `SD3Transformer2DModel` does not include `RMSNorm`.

  #9515 opened Sep 24, 2024

• Multi-controlnet batching for `StableDiffusionXLControlNetInpaintPipeline`

  #9511 opened Sep 24, 2024

• AnimateDiff SparseCtrl RGB does not work as expected

  #9508 opened Sep 23, 2024

• Can't upscale anything. Keep getting "KeyError: 'middle_block_out.0.weight'" error/

  #9505 opened Sep 23, 2024

• Confusion about `FrozenDict` in `configuration_utils.py`

  #9503 opened Sep 23, 2024

• Dreambooth Flux training does not save a model for around 10-15 minutes

  #9501 opened Sep 23, 2024

• Dreambooth Flux training failed on saving a checkpoint

  #9500 opened Sep 23, 2024

• Dreambooth Flux training error: RuntimeError: mat2 must be a matrix, got 1-D tensor

  #9497 opened Sep 23, 2024

58 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [LoRA] fix: lora loading when using with a device_mapped model.

  #9449 commented on Sep 28, 2024 • 3 new comments

• Support FLUX nf4 & pf8 for GPUs with 6GB/8GB VRAM (method and checkpoints by lllyasviel)

  #9149 commented on Sep 28, 2024 • 0 new comments

• Finetuning on dataset

  #9317 commented on Sep 28, 2024 • 0 new comments

• FLUX dreambooth train on multigpu with deepspeed

  #9484 commented on Sep 29, 2024 • 0 new comments

• Unload lora specific lora weight

  #9325 commented on Sep 29, 2024 • 0 new comments

• RunwayML obsoletion causes issues

  #9322 commented on Sep 29, 2024 • 0 new comments

• Finetuned models aren't saved or loaded properly in train_custom_diffusion.py

  #9320 commented on Sep 29, 2024 • 0 new comments

• support kolors with controlnet and ipadapter-Faceid

  #9315 commented on Sep 29, 2024 • 0 new comments

• [Scheduler] Add Variational Diffusion Models (VDM) scheduler

  #7737 commented on Sep 29, 2024 • 0 new comments

• Fix `latents.dtype` before `vae.decode()` at ROCm devices in `StableDiffusionPipeline`s

  #7886 commented on Sep 27, 2024 • 0 new comments

• add PAG support for SD Controlnet Img2Img

  #8864 commented on Sep 29, 2024 • 0 new comments

• Add PAG support to StableDiffusionControlNetPAGInpaintPipeline

  #8875 commented on Sep 29, 2024 • 0 new comments

• [`Research Project`] Add AnyText: Multilingual Visual Text Generation And Editing

  #8998 commented on Sep 28, 2024 • 0 new comments

• [`Community Pipeline`] Add 🪆Matryoshka Diffusion Models

  #9157 commented on Sep 29, 2024 • 0 new comments

• refactor SD3, speedup tests

  #9169 commented on Sep 27, 2024 • 0 new comments

• [Quantization] Add quantization support for `bitsandbytes`

  #9213 commented on Sep 29, 2024 • 0 new comments

• Fix bug in Textual Inversion Unloading

  #9304 commented on Sep 29, 2024 • 0 new comments

• [Official callbacks] SDXL Controlnet CFG Cutoff

  #9311 commented on Sep 28, 2024 • 0 new comments

• [do not merge] testing rotary embedding + torch.compile

  #9321 commented on Sep 29, 2024 • 0 new comments

• make `pipelines` tests device-agnostic (part1)

  #9399 commented on Sep 26, 2024 • 0 new comments

• Fix flux controlnet mode to take into account batch size

  #9406 commented on Sep 25, 2024 • 0 new comments

• [Flux] Add advanced training script + support textual inversion inference

  #9434 commented on Sep 23, 2024 • 0 new comments

• [Tests] Fix CI for deprecated attention block when used with `device_map`

  #9442 commented on Sep 26, 2024 • 0 new comments

• [WIP][LoRA] Implement hot-swapping of LoRA

  #9453 commented on Sep 28, 2024 • 0 new comments

• add PAG support for SD Img2Img

  #9463 commented on Sep 25, 2024 • 0 new comments

• [Fix] remove setting lr for T5 text encoder when using prodigy in flux dreambooth lora script

  #9473 commented on Sep 23, 2024 • 0 new comments

• Profile and speedup pipelines

  #9475 commented on Sep 27, 2024 • 0 new comments

• Support bfloat16 for Upsample2D

  #9480 commented on Sep 27, 2024 • 0 new comments

• [training] CogVideoX-I2V LoRA

  #9482 commented on Sep 27, 2024 • 0 new comments

• FluxPipeline - Multi-GPU Issue - When you define transformer= you get "Expected all tensors to be on the same device"

  #9450 commented on Sep 23, 2024 • 0 new comments

• FLUX error when loading with low_cpu_mem_usage=False and ignore_mismatched_sizes=True

  #9343 commented on Sep 23, 2024 • 0 new comments

• Redundant reinitialization of text encoders in train_dreambooth_lora_flux

  #9358 commented on Sep 23, 2024 • 0 new comments

• ValueError: --image_column' value 'image' needs to be one of: text

  #9210 commented on Sep 23, 2024 • 0 new comments

• Remove redundant comparison inside the diffusion loop of stable video diffusion pipeline

  #9425 commented on Sep 23, 2024 • 0 new comments

• [Schedulers] Analysis of `simple`, `exponential`, `polyexponential` and `beta`

  #9490 commented on Sep 23, 2024 • 0 new comments

• train_dreambooth_lora_flux with prodigy and train_text_encoder causes IndexError: list index out of range

  #9464 commented on Sep 24, 2024 • 0 new comments

• StableDiffusionPipeline.from_single_file() with CKPT fails: "UnpicklingError: Weights only load failed"

  #8874 commented on Sep 24, 2024 • 0 new comments

• [Quantization] bring quantization to diffusers core

  #9174 commented on Sep 24, 2024 • 0 new comments

• Storage-Efficient Bulk Conversion of Single-File Model Layout to Diffusers-Multifolder

  #9260 commented on Sep 24, 2024 • 0 new comments

• [BUG] SNR gamma in v_prediction

  #9451 commented on Sep 25, 2024 • 0 new comments

• torch.compile + cudagraph failed with StableDiffusionXL

  #8879 commented on Sep 25, 2024 • 0 new comments

• How can I manually update some of their checkpoints of UNet2/3DConditionModel objects?

  #9276 commented on Sep 25, 2024 • 0 new comments

• The different quality between ComfyUI and Diffusers ?

  #9271 commented on Sep 25, 2024 • 0 new comments

• Incomplete Lora support for Flux Dev from SimpleTuner

  #9270 commented on Sep 25, 2024 • 0 new comments

• Can we allow making everything on gpu/cuda for scheduler?

  #9485 commented on Sep 25, 2024 • 0 new comments

• Implementing training-free RB-Modulation pipeline for most used models

  #9283 commented on Sep 26, 2024 • 0 new comments

• Issue running stable diffusion dreambooth on mac m3 max apple silicon.

  #7498 commented on Sep 26, 2024 • 0 new comments

• Issues of training dreambooth using DoRA with text encoder

  #7431 commented on Sep 26, 2024 • 0 new comments

• train_dreambooth_lora_flux validation RuntimeError: Input type (float) and bias type (c10::BFloat16) should be the same

  #9476 commented on Sep 26, 2024 • 0 new comments

• Loading PT Files for VAE with the AutoEncoder is still broken!!!

  #9154 commented on Sep 27, 2024 • 0 new comments

• Cannot import StableDiffusion3ControlNetInpaintingPipeline

  #9410 commented on Sep 27, 2024 • 0 new comments

• [Add] VEnhancer - the interpolation and upscaler for CogVideoX-5b

  #9303 commented on Sep 27, 2024 • 0 new comments

• Convert From XLab controlnet to diffusers format script

  #9301 commented on Sep 27, 2024 • 0 new comments

• please add flux img2img with controlnet pipeline

  #9151 commented on Sep 27, 2024 • 0 new comments

• DPMSolverSinglestepScheduler Step ValueError in SDXL Pipeline

  #9359 commented on Sep 28, 2024 • 0 new comments

• FLUX.1-dev dreambooth train on multigpu

  #9278 commented on Sep 28, 2024 • 0 new comments

• [Flux IPadapter] Support Xlabs IPadapter in diffusers

  #9403 commented on Sep 28, 2024 • 0 new comments

• Consistency training fails to converge

  #8277 commented on Sep 28, 2024 • 0 new comments