Spaces:

marianvd-01
/

transformer-visualizer

Sleeping

marianvd-01 commited on Apr 11

Commit

38d431d

verified ·

1 Parent(s): 9d0184c

Update visualize.py

Files changed (1) hide show

visualize.py CHANGED Viewed

@@ -66,6 +66,9 @@ def show_embeddings(tokenizer, model, text):
     return fig
 def compare_model_sizes():
     from model_utils import MODEL_OPTIONS
     from transformers import AutoModel

     return fig
+def get_token_list(tokenizer, text):
+    return tokenizer.tokenize(text)
 def compare_model_sizes():
     from model_utils import MODEL_OPTIONS
     from transformers import AutoModel