Set `clean_up_tokenization_spaces` to false (#13) dac9c9e verified patrickvonplaten Xenova HF Staff commited on Jul 19, 2024
Do not add eos token by default (#8) 4f81d78 verified patrickvonplaten Xenova HF Staff commited on Jul 18, 2024