Update README.md
Browse files
README.md
CHANGED
|
@@ -11,7 +11,8 @@ license: apache-2.0
|
|
| 11 |
---
|
| 12 |
# Model card for DePlot
|
| 13 |
|
| 14 |
-
|
|
|
|
| 15 |
|
| 16 |
|
| 17 |
# Table of Contents
|
|
@@ -30,7 +31,25 @@ The abstract of the paper states that:
|
|
| 30 |
|
| 31 |
# Using the model
|
| 32 |
|
| 33 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
|
| 35 |
You can use the [`convert_pix2struct_checkpoint_to_pytorch.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/pix2struct/convert_pix2struct_original_pytorch_to_hf.py) script as follows:
|
| 36 |
```bash
|
|
@@ -51,24 +70,6 @@ model.push_to_hub("USERNAME/MODEL_NAME")
|
|
| 51 |
processor.push_to_hub("USERNAME/MODEL_NAME")
|
| 52 |
```
|
| 53 |
|
| 54 |
-
## Run a prediction
|
| 55 |
-
|
| 56 |
-
You can run a prediction by querying an input image together with a question as follows:
|
| 57 |
-
```python
|
| 58 |
-
from transformers import Pix2StructForConditionalGeneration, Pix2StructProcessor
|
| 59 |
-
import requests
|
| 60 |
-
from PIL import Image
|
| 61 |
-
|
| 62 |
-
model = Pix2StructForConditionalGeneration.from_pretrained('google/deplot')
|
| 63 |
-
processor = Pix2StructProcessor.from_pretrained('google/deplot')
|
| 64 |
-
url = "https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/5090.png"
|
| 65 |
-
image = Image.open(requests.get(url, stream=True).raw)
|
| 66 |
-
|
| 67 |
-
inputs = processor(images=image, text="Generate underlying data table of the figure below:", return_tensors="pt")
|
| 68 |
-
predictions = model.generate(**inputs, max_new_tokens=512)
|
| 69 |
-
print(processor.decode(predictions[0], skip_special_tokens=True))
|
| 70 |
-
```
|
| 71 |
-
|
| 72 |
# Contribution
|
| 73 |
|
| 74 |
This model was originally contributed by Fangyu Liu, Julian Martin Eisenschlos et al. and added to the Hugging Face ecosystem by [Younes Belkada](https://huggingface.co/ybelkada).
|
|
|
|
| 11 |
---
|
| 12 |
# Model card for DePlot
|
| 13 |
|
| 14 |
+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/deplot_architecture.png"
|
| 15 |
+
alt="drawing" width="600"/>
|
| 16 |
|
| 17 |
|
| 18 |
# Table of Contents
|
|
|
|
| 31 |
|
| 32 |
# Using the model
|
| 33 |
|
| 34 |
+
You can run a prediction by querying an input image together with a question as follows:
|
| 35 |
+
|
| 36 |
+
```python
|
| 37 |
+
from transformers import Pix2StructProcessor, Pix2StructForConditionalGeneration
|
| 38 |
+
import requests
|
| 39 |
+
from PIL import Image
|
| 40 |
+
|
| 41 |
+
processor = Pix2StructProcessor.from_pretrained('google/deplot')
|
| 42 |
+
model = Pix2StructForConditionalGeneration.from_pretrained('google/deplot')
|
| 43 |
+
|
| 44 |
+
url = "https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/5090.png"
|
| 45 |
+
image = Image.open(requests.get(url, stream=True).raw)
|
| 46 |
+
|
| 47 |
+
inputs = processor(images=image, text="Generate underlying data table of the figure below:", return_tensors="pt")
|
| 48 |
+
predictions = model.generate(**inputs, max_new_tokens=512)
|
| 49 |
+
print(processor.decode(predictions[0], skip_special_tokens=True))
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
# Converting from T5x to huggingface
|
| 53 |
|
| 54 |
You can use the [`convert_pix2struct_checkpoint_to_pytorch.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/pix2struct/convert_pix2struct_original_pytorch_to_hf.py) script as follows:
|
| 55 |
```bash
|
|
|
|
| 70 |
processor.push_to_hub("USERNAME/MODEL_NAME")
|
| 71 |
```
|
| 72 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
# Contribution
|
| 74 |
|
| 75 |
This model was originally contributed by Fangyu Liu, Julian Martin Eisenschlos et al. and added to the Hugging Face ecosystem by [Younes Belkada](https://huggingface.co/ybelkada).
|