Model reference · open weights
g-large-textcaps is an open-weight language model from microsoft. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.
Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.
What it is
| Released by | microsoft |
|---|---|
| Type | Language models |
| Task | Image→text |
| Context | 1k tokens |
| Runs with | transformers |
| Released | 2023-01-02 |
| Popularity | 559 downloads / month |
| Licence | Open weights |
About
GIT (short for GenerativeImage2Text) model, large-sized version, fine-tuned on TextCaps. It was introduced in the paper GIT: A Generative Image-to-text Transformer for Vision and Language by Wang et al. and first released in this repository.
Disclaimer: The team releasing GIT did not write a model card for this model so this model card has been written by the Hugging Face team.
GIT is a Transformer decoder conditioned on both CLIP image tokens and text tokens. The model is trained using "teacher forcing" on a lot of (image, text) pairs.
The goal for the model is simply to predict the next text token, giving the image tokens and previous text tokens.
The model has full access to (i.e. a bidirectional attention mask is used for) the image patch tokens, but only has access to the previous text tokens (i.e. a causal attention mask is used for the text tokens) when predicting the next text token.
This allows the model to be used for tasks like:
You can use the raw model for image captioning. See the model hub to look for fine-tuned versions on a task that interests you.
For code examples, we refer to the documentation.
From the paper:
We collect 0.8B image-text pairs for pre-training, which include COCO (Lin et al., 2014), Conceptual Captions (CC3M) (Sharma et al., 2018), SBU (Ordonez et al., 2011), Visual Genome (VG) (Krishna et al., 2016), Conceptual Captions (CC12M) (Changpinyo et al., 2021), ALT200M (Hu et al., 2021a), and an extra 0.6B data following a similar collection procedure in Hu et al. (2021a).
=> however this is for the model referred to as "GIT" in the paper, which is not open-sourced.
This checkpoint is "GIT-large", which is a smaller variant of GIT trained on 20 million image-text pairs.
Next, the model was fine-tuned on TextCaps.
See table 11 in the paper for more details.
We refer to the original repo regarding details for preprocessing during training.
During validation, one resizes the shorter edge of each image, after which center cropping is performed to a fixed-size resolution. Next, frames are normalized across the RGB channels with the ImageNet mean and standard deviation.
For evaluation results, we refer readers to the paper.
From the published model card. Full card on the HuggingFace links in the sidebar.
Using it via the API
Once AxForge deploys g-large-textcaps for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (g-large-textcaps below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/chat/completions \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"g-large-textcaps","messages":[{"role":"user","content":"Hello"}]}'
Create an account — your API key is available in the console. 3M free tokens every 30 days with every new account.