Upload KL3M multi-word tokenizer (32K) - Update README

README.md

@@ -1,199 +1,298 @@
 ---
+language:
+- en
+license: mit
+tags:
+- tokenizer
+- legal
+- bpe
+- byte-pair-encoding
+- multi-word
+- kl3m
+- legal-domain
+- hierarchical
+pipeline_tag: fill-mask
 library_name: transformers
-tags: []
 ---
 
-# Model Card for Model ID
+# KL3M Multi-Word Tokenizer - 32K
 
-<!-- Provide a quick summary of what the model is/does. -->
+This is the **32,768 token** variant of the KL3M (Kelvin Legal Large Language Model) multi-word tokenizer family, optimized for legal domain text with hierarchical vocabulary nesting.
 
+## Overview
 
+The KL3M multi-word tokenizers are a family of byte-pair encoding (BPE) tokenizers trained on ~44GB of legal domain text from the [KL3M dataset](https://aleainstitute.ai/work/kl3m/) (copyright-clean legal corpus from the ALEA Institute). Unlike general-purpose tokenizers, these tokenizers:
 
-## Model Details
+- **Capture multi-word legal phrases** as single tokens (e.g., "United States", "with respect to", "Supreme Court")
+- **Use hierarchical vocabulary nesting** where smaller vocabularies are proper subsets of larger ones
+- **Enable vocabulary expansion experiments** and transfer learning across vocabulary sizes
+- **Optimize for legal domain text** while maintaining general-purpose capability
 
-### Model Description
+## Tokenizer Family
 
-<!-- Provide a longer summary of what this model is. -->
+This tokenizer is part of a hierarchically nested family. Token IDs in smaller vocabularies are **identical** across all larger vocabularies, enabling seamless vocabulary expansion:
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+| Vocabulary Size | HuggingFace Repository | File Size |
+|----------------|------------------------|-----------|
+| 4,096 (4K) | [alea-institute/kl3m-multi-word-001-4k](https://huggingface.co/alea-institute/kl3m-multi-word-001-4k) | 118 KB |
+| 8,192 (8K) | [alea-institute/kl3m-multi-word-001-8k](https://huggingface.co/alea-institute/kl3m-multi-word-001-8k) | 249 KB |
+| 16,384 (16K) | [alea-institute/kl3m-multi-word-001-16k](https://huggingface.co/alea-institute/kl3m-multi-word-001-16k) | 529 KB |
+| 32,768 (32K) | [alea-institute/kl3m-multi-word-001-32k](https://huggingface.co/alea-institute/kl3m-multi-word-001-32k) | 1.2 MB |
+| 65,536 (65K) | [alea-institute/kl3m-multi-word-001-65k](https://huggingface.co/alea-institute/kl3m-multi-word-001-65k) | 2.4 MB |
+| 131,072 (131K) | [alea-institute/kl3m-multi-word-001-131k](https://huggingface.co/alea-institute/kl3m-multi-word-001-131k) | 5.2 MB |
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+**→ You are viewing: 32,768 (32K)**
 
-### Model Sources [optional]
+## Key Features
 
-<!-- Provide the basic links for the model. -->
+### 1. Multi-Word Tokenization
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+Legal text contains frequent multi-word phrases that benefit from being treated as single tokens. The larger vocabularies capture increasingly sophisticated legal terminology:
 
-## Uses
+**Example: "with respect to" (common legal phrase)**
+```python
+# 4K tokenizer: 3 tokens
+['with respec', 't ', 'to'] → [2286, 282, 393]
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+# 131K tokenizer: 1 token
+['with respect to'] → [15878]
+```
 
-### Direct Use
+**Example: "Supreme Court"**
+```python
+# 4K tokenizer: 5 tokens
+['Sup', 'rem', 'e ', 'Cour', 't'] → [4062, 1847, 265, 3032, 123]
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+# 131K tokenizer: 1 token
+['Supreme Court'] → [81439]
+```
 
-[More Information Needed]
+**Other multi-word tokens in this vocabulary:**
+- Common legal phrases: "United States" (→1 token), "in accordance with" (→1 token), "on behalf of" (→1 token)
+- Frequent constructions: "of the " (→1 token), "in the " (→1 token), ", the " (→1 token)
+- Legal terminology: "the defendant", "the Court", "Therefore,", "However,"
 
-### Downstream Use [optional]
+### 2. Hierarchical Token ID Nesting
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+Token IDs are **preserved across vocabulary sizes** — a token with ID 1846 in the 4K vocabulary has the **same ID** in all larger vocabularies:
 
-[More Information Needed]
+```python
+# "of the" tokenizes to ID 1846 in ALL vocabulary sizes
+4K:     [1846]
+8K:     [1846]
+16K:    [1846]
+32K:    [1846]
+65K:    [1846]
+131K:   [1846]
 
-### Out-of-Scope Use
+# Special tokens are identical across all sizes
+<|start|>:  [0]
+<|end|>:    [1]
+<|pad|>:    [2]
+```
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+This enables:
+- **Vocabulary expansion during training**: Start with 4K vocab, expand to 8K → 16K → 32K
+- **Embedding transfer**: Initialize larger vocabulary models from smaller ones
+- **Controlled ablation studies**: Isolate the effect of vocabulary size
 
-[More Information Needed]
+### 3. Special Tokens
 
-## Bias, Risks, and Limitations
+All tokenizers include 7 special tokens with consistent IDs:
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+| Token | ID | Purpose |
+|-------|---:|---------|
+| `<\|start\|>` | 0 | Start of sequence (GPT-style) |
+| `<\|end\|>` | 1 | End of sequence |
+| `<\|pad\|>` | 2 | Padding token |
+| `<\|unk\|>` | 3 | Unknown token |
+| `<\|cls\|>` | 4 | Classification token (BERT-style) |
+| `<\|sep\|>` | 5 | Separator token (BERT-style) |
+| `<\|mask\|>` | 6 | Mask token (MLM training) |
 
-[More Information Needed]
+## Usage
 
-### Recommendations
+### Quick Start
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+```python
+from transformers import PreTrainedTokenizerFast
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+# Load tokenizer
+tokenizer = PreTrainedTokenizerFast.from_pretrained("alea-institute/kl3m-multi-word-001-32k")
 
-## How to Get Started with the Model
+# Tokenize text
+text = "The Supreme Court held that the defendant violated due process."
+tokens = tokenizer.tokenize(text)
+ids = tokenizer.encode(text)
 
-Use the code below to get started with the model.
+print(f"Tokens: {tokens}")
+print(f"Token IDs: {ids}")
+```
 
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
+### Using with 🤗 Tokenizers Library
 
+```python
+from tokenizers import Tokenizer
 
-#### Training Hyperparameters
+# Load tokenizer
+tokenizer = Tokenizer.from_pretrained("alea-institute/kl3m-multi-word-001-32k")
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+# Encode text
+encoding = tokenizer.encode("in accordance with the United States Code")
+print(f"Tokens: {encoding.tokens}")
+print(f"IDs: {encoding.ids}")
+```
 
-#### Speeds, Sizes, Times [optional]
+### Configuration for Training
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+```python
+from transformers import PreTrainedTokenizerFast
 
-[More Information Needed]
+tokenizer = PreTrainedTokenizerFast.from_pretrained("alea-institute/kl3m-multi-word-001-32k")
 
-## Evaluation
+# Configure special tokens for your model
+tokenizer.pad_token = "<|pad|>"
+tokenizer.eos_token = "<|end|>"
+tokenizer.bos_token = "<|start|>"
+tokenizer.unk_token = "<|unk|>"
+tokenizer.cls_token = "<|cls|>"  # For BERT-style models
+tokenizer.sep_token = "<|sep|>"  # For BERT-style models
+tokenizer.mask_token = "<|mask|>"  # For masked language modeling
+```
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
+## Training Details
 
-#### Metrics
+### Training Corpus
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+- **Source**: KL3M (Kelvin Legal Large Language Model) dataset
+- **Size**: ~44.2 GB (44,168,540,153 bytes)
+- **Content**: 1,018,355,750 lines, 5,997,814,602 words
+- **Domain**: Legal text (court opinions, contracts, statutes, legal documents)
+- **License**: Copyright-clean corpus from the ALEA Institute
 
-[More Information Needed]
+### Training Method
 
-### Results
+Trained using the `bbpe` (Binary Byte Pair Encoding) Rust crate with multi-word optimization:
 
-[More Information Needed]
+```bash
+zcat /nas4/data/kl3m/kl3m-bbpe-sample.txt.gz | \
+bbpe train -v - \
+     --max-entropy 7.0 \
+     --preprocessor unicode-whitespace \
+     --preprocessor-probability 0.1 \
+     --vocab-size 131072 \
+     --family-size 65536 --family-size 32768 --family-size 16384 \
+     --family-size 8192 --family-size 4096 \
+     --family-template tokenizer-{size}.json \
+     --output tokenizer-131072.json
+```
 
-#### Summary
+Parameters:
+- `max-entropy 7.0`: Entropy threshold balancing multi-word phrases with common tokens
+- `family-size`: Creates nested vocabulary families ensuring ID consistency
+- `preprocessor unicode-whitespace`: Whitespace normalization
 
+## Use Cases
 
+### 1. Legal Language Models
 
-## Model Examination [optional]
+Train domain-specific language models optimized for legal text:
 
-<!-- Relevant interpretability work for the model goes here -->
+```python
+from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast
 
-[More Information Needed]
+tokenizer = PreTrainedTokenizerFast.from_pretrained("alea-institute/kl3m-multi-word-001-32k")
+model = AutoModelForCausalLM.from_pretrained("your-legal-model")
 
-## Environmental Impact
+# The model will efficiently process legal terminology
+text = "The Court held that the statute of limitations had expired."
+inputs = tokenizer(text, return_tensors="pt")
+outputs = model(**inputs)
+```
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+### 2. Vocabulary Ablation Studies
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+Compare model performance across vocabulary sizes:
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+```python
+# Train models with different vocabulary sizes
+for vocab_size in ["4k", "8k", "16k", "32k", "65k", "131k"]:
+    tokenizer = PreTrainedTokenizerFast.from_pretrained(
+        f"alea-institute/kl3m-multi-word-001-{vocab_size}"
+    )
+    # Train model and compare convergence, perplexity, downstream performance
+```
 
-## Technical Specifications [optional]
+### 3. Curriculum Learning with Vocabulary Expansion
 
-### Model Architecture and Objective
+Leverage hierarchical nesting for progressive vocabulary growth:
 
-[More Information Needed]
+```python
+# Stage 1: Train with 4K vocabulary
+tokenizer_4k = PreTrainedTokenizerFast.from_pretrained("alea-institute/kl3m-multi-word-001-4k")
+# ... train model ...
 
-### Compute Infrastructure
+# Stage 2: Expand to 16K vocabulary (embeddings for IDs 0-4095 are identical!)
+tokenizer_16k = PreTrainedTokenizerFast.from_pretrained("alea-institute/kl3m-multi-word-001-16k")
+# ... expand model embeddings and continue training ...
+```
 
-[More Information Needed]
+## Research Applications
 
-#### Hardware
+These tokenizers enable research into:
 
-[More Information Needed]
+1. **Vocabulary Size Effects**: How does vocabulary size affect convergence speed, final perplexity, and downstream task performance?
 
-#### Software
+2. **Domain-Specific Tokenization**: Do legal domain tokenizers outperform general-purpose tokenizers (GPT-4, LLaMA) on legal tasks?
 
-[More Information Needed]
+3. **Multi-Word Phrase Modeling**: Does capturing legal phrases as single tokens improve legal reasoning and understanding?
 
-## Citation [optional]
+4. **Hierarchical Curriculum Learning**: Can progressive vocabulary expansion improve training efficiency or final performance?
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+5. **Transfer Learning**: Can models trained on smaller vocabularies transfer knowledge to larger vocabularies?
 
-**BibTeX:**
+## Citation
 
-[More Information Needed]
+If you use these tokenizers in your research, please cite:
 
-**APA:**
+```bibtex
+@misc{kl3m-multi-word-tokenizers-2025,
+  title={KL3M Multi-Word Tokenizers: Hierarchically Nested BPE for Legal Domain Language Modeling},
+  author={ALEA Institute},
+  year={2025},
+  url={https://huggingface.co/alea-institute/kl3m-multi-word-001-32k}
+}
+```
 
-[More Information Needed]
+Also consider citing the KL3M dataset:
 
-## Glossary [optional]
+```bibtex
+@article{kl3m-data-2025,
+  title={The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models},
+  author={Bommarito, Michael and others},
+  journal={arXiv preprint arXiv:2504.07854},
+  year={2025}
+}
+```
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+## License
 
-[More Information Needed]
+These tokenizers are released under the MIT License. The training corpus (KL3M dataset) is copyright-clean and permissively licensed.
 
-## More Information [optional]
+## Links
 
-[More Information Needed]
+- **ALEA Institute**: https://aleainstitute.ai/
+- **KL3M Project**: https://aleainstitute.ai/work/kl3m/
+- **KL3M Dataset Paper**: https://arxiv.org/html/2504.07854
+- **Research Repository**: https://github.com/alea-institute/multi-word-tokenization
 
-## Model Card Authors [optional]
+## Acknowledgments
 
-[More Information Needed]
+These tokenizers were created as part of research into vocabulary size effects on legal language model performance. The KL3M dataset and tokenizers are stewarded by the ALEA Institute for public benefit.
 
-## Model Card Contact
+---
 
-[More Information Needed]
+**Version**: 001
+**Created**: November 2025
+**Vocabulary Size**: 32,768 tokens
+**Domain**: Legal text (with general-purpose capability)