Model reference · open weights
instruct-pix2pix is an open-weight image model from timbrooks. instruct-pix2pix (FP32) weighs 1.4 GB; the smallest configuration that runs it is RTX 3060 12 GB.
What it is
| Released by | timbrooks |
|---|---|
| Type | Image models |
| Task | Image edit |
| Parameters (lead) | 860M |
| Runs with | diffusers |
| Released | 2023-01-20 |
| Popularity | 27k downloads / month |
| Weights | 1.4 GB (instruct-pix2pix (FP32), file size) |
| Licence | Open weights |
What it runs on
Weights 1.4 GB (file size) · its biggest part 860 MB · working memory for one 1024×1024 image about 5.0 GB · overhead about 537 MB.
| Card | One 1024×1024 image | Counted memory |
|---|---|---|
| RTX 3060 12 GB | fits | 11.6 GB |
| RTX 4060 Ti 16 GB | fits | 15.4 GB |
| RTX 3090 24 GB | fits | 23.4 GB |
| RTX 4090 24 GB | fits | 23.4 GB |
| RTX 5090 32 GB | fits | 31.0 GB |
| L40S 48 GB | fits | 44.0 GB |
| A100 80 GB | fits | 78.2 GB |
| H100 80 GB | fits | 78.1 GB |
| RTX PRO 6000 Blackwell 96 GB | fits | 93.8 GB |
| DGX Spark (GB10) 128 GB unified | fits | 107 GB |
| H200 141 GB | fits | 138 GB |
| B200 180 GB | fits | 176 GB |
Estimates, not measurements: the weights are the build's file size; one 1024×1024 image needs about 5 GB of working memory (larger images more); "encoders offloaded" means only the biggest part is on the card at once — diffusers' model offload, or ComfyUI unloading the text encoder. diffusers can also place a pipeline's parts on separate cards (device_map) — not estimated here. Counted memory is 92 % of what CUDA reports for the card.
From the model card
GitHub: https://github.com/timothybrooks/instruct-pix2pix
To use InstructPix2Pix, install diffusers using main for now. The pipeline will be available in the next release
pip install diffusers accelerate safetensors transformers
import PIL
import requests
import torch
from diffusers import StableDiffusionInstructPix2PixPipeline, EulerAncestralDiscreteScheduler
model_id = "timbrooks/instruct-pix2pix"
pipe = StableDiffusionInstructPix2PixPipeline.from_pretrained(model_id, torch_dtype=torch.float16, safety_checker=None)
pipe.to("cuda")
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)
url = "https://raw.githubusercontent.com/timothybrooks/instruct-pix2pix/main/imgs/example.jpg"
def download_image(url):
image = PIL.Image.open(requests.get(url, stream=True).raw)
image = PIL.ImageOps.exif_transpose(image)
image = image.convert("RGB")
return image
image = download_image(url)
prompt = "turn him into cyborg"
images = pipe(prompt, image=image, num_inference_steps=10, image_guidance_scale=1).images
images[0]
Quoted from the model card on Hugging Face — the full card is behind the Hugging Face link above.
Running it yourself
Rent a machine by the hour — ComfyUI is installed on it. Open ComfyUI through the tunnel: its default workflow loads a checkpoint — choose this model's file in Load Checkpoint, with the settings its model card gives.
# on your rented machine: pip install diffusers transformers accelerate
import torch
from diffusers import DiffusionPipeline
from diffusers.utils import load_image
pipe = DiffusionPipeline.from_pretrained("timbrooks/instruct-pix2pix", torch_dtype=torch.bfloat16).to("cuda")
start = load_image("/workspace/in.png")
image = pipe(prompt="the same scene at golden hour", image=start).images[0]
image.save("/workspace/out.png")# on your rented machine (the ssh line is on its page in the console)
# get REPO FILE FOLDER: one file into /workspace/models/FOLDER, where ComfyUI loads it from
get() { hf download "$1" "$2" --local-dir /workspace/hf-files && mkdir -p "/workspace/models/$3" && mv "/workspace/hf-files/$2" "/workspace/models/$3/$4"; }
# the model (7.2 GB)
get timbrooks/instruct-pix2pix instruct-pix2pix-00-22000.safetensors checkpoints
start-comfyui# on your computer, in a second terminal: ComfyUI in your browser at http://localhost:8188
# HOST and PORT are your machine's, from its page in the console
ssh -L 8188:localhost:8188 dev@HOST -p PORT