Model reference · open weights
FuseCap_Image_Captioning is an open-weight language model from noamrot. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.
Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.
What it is
| Released by | noamrot |
|---|---|
| Type | Language models |
| Task | Image→text |
| Context | 512 tokens |
| Runs with | transformers |
| Released | 2023-05-31 |
| Popularity | 3k downloads / month |
| Licence | Open weights |
About
A framework designed to generate semantically rich image captions.
💻 Project Page: For more details, visit the official project page.
📝 Read the Paper: You can find the paper here.
🚀 Demo: Try out our BLIP-based model demo trained using FuseCap.
📂 Code Repository: The code for FuseCap can be found in the GitHub repository.
🗃️ Datasets: The fused captions datasets can be accessed from here.
Our BLIP-based model can be run using the following code,
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
processor = BlipProcessor.from_pretrained("noamrot/FuseCap")
model = BlipForConditionalGeneration.from_pretrained("noamrot/FuseCap").to(device)
img_url = 'https://huggingface.co/spaces/noamrot/FuseCap/resolve/main/bike.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
text = "a picture of "
inputs = processor(raw_image, text, return_tensors="pt").to(device)
out = model.generate(**inputs, num_beams = 3)
print(processor.decode(out[0], skip_special_tokens=True))
The official codebase, datasets and trained models for this project will be released soon.
@inproceedings{rotstein2024fusecap,
title={Fusecap: Leveraging large language models for enriched fused image captions},
author={Rotstein, Noam and Bensa{\"\i}d, David and Brody, Shaked and Ganz, Roy and Kimmel, Ron},
booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
pages={5689--5700},
year={2024}
}
From the published model card. Full card on the HuggingFace links in the sidebar.
Using it via the API
Once AxForge deploys fusecap-image-captioning for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (fusecap-image-captioning below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/chat/completions \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"fusecap-image-captioning","messages":[{"role":"user","content":"Hello"}]}'
Create an account — your API key is available in the console. 3M free tokens every 30 days with every new account.