Upload KazEmbed-V5: Best BASE-size Kazakh embedding model

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+    "word_embedding_dimension": 768,
+    "pooling_mode_cls_token": false,
+    "pooling_mode_mean_tokens": true,
+    "pooling_mode_max_tokens": false,
+    "pooling_mode_mean_sqrt_len_tokens": false,
+    "pooling_mode_weightedmean_tokens": false,
+    "pooling_mode_lasttoken": false,
+    "include_prompt": true
+}

README.md

@@ -0,0 +1,155 @@
+---
+language:
+- kk
+- ru
+- en
+license: apache-2.0
+tags:
+- sentence-transformers
+- feature-extraction
+- sentence-similarity
+- embedding
+- retrieval
+- kazakh
+- rag
+datasets:
+- issai/kazqad
+- issai/kazqad-retrieval
+- DarkyMan/powerful-kazakh-dialogue
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
+base_model: intfloat/multilingual-e5-base
+---
+
+# KazEmbed-V5: Kazakh Embedding Model for RAG
+
+🏆 **Best BASE-size embedding model for Kazakh language retrieval tasks**
+
+## Model Description
+
+KazEmbed-V5 is a fine-tuned version of [multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) optimized for Kazakh language retrieval and RAG (Retrieval-Augmented Generation) applications.
+
+### Key Features
+
+- 🇰🇿 **Specialized for Kazakh**: Fine-tuned on 61,255 Kazakh text pairs
+- 📈 **+2.1% MRR improvement** over multilingual-e5-base
+- ⚡ **Efficient**: 278M parameters (base-size model)
+- 🔍 **RAG-optimized**: Trained specifically for retrieval tasks
+
+## Benchmark Results
+
+| Model | Hits@1 | Hits@5 | MRR | Params |
+|-------|--------|--------|-----|--------|
+| **KazEmbed-V5 (Ours)** | **72%** | **96%** | **0.835** | 278M |
+| multilingual-e5-base | 72% | 96% | 0.818 | 278M |
+| multilingual-e5-large | 85% | 99% | 0.909 | 560M |
+| paraphrase-mpnet-v2 | 53% | 80% | 0.648 | 278M |
+| LaBSE | 48% | 73% | 0.601 | 471M |
+
+*Evaluated on KazQAD test set with TF-IDF hard negatives (100 candidates per query)*
+
+## Usage
+
+### Installation
+
+```bash
+pip install sentence-transformers
+```
+
+### Basic Usage
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer('YOUR_USERNAME/kazembed-v5')
+
+# For queries (questions)
+query = "query: Қазақстанның астанасы қай қала?"
+query_embedding = model.encode(query)
+
+# For passages (documents)
+passage = "passage: Астана — Қазақстан Республикасының астанасы."
+passage_embedding = model.encode(passage)
+
+# Calculate similarity
+from sklearn.metrics.pairwise import cosine_similarity
+similarity = cosine_similarity([query_embedding], [passage_embedding])[0][0]
+print(f"Similarity: {similarity:.4f}")
+```
+
+### For RAG Applications
+
+```python
+from sentence_transformers import SentenceTransformer
+import numpy as np
+
+model = SentenceTransformer('YOUR_USERNAME/kazembed-v5')
+
+# Your document corpus
+documents = [
+    "Астана — Қазақстан Республикасының астанасы.",
+    "Алматы — Қазақстанның ең үлкен қаласы.",
+    "Қазақстан — Орталық Азиядағы мемлекет.",
+]
+
+# Encode documents (do once, store in vector DB)
+doc_embeddings = model.encode(["passage: " + doc for doc in documents])
+
+# Query
+query = "Қазақстанның астанасы қай қала?"
+query_embedding = model.encode("query: " + query)
+
+# Find most similar
+similarities = np.dot(doc_embeddings, query_embedding)
+best_idx = np.argmax(similarities)
+print(f"Best match: {documents[best_idx]}")
+```
+
+## Training Details
+
+### Training Data
+
+| Dataset | Pairs | Description |
+|---------|-------|-------------|
+| KazQAD | 6,640 | Question-Context pairs |
+| KazQAD-Retrieval | 44,615 | Title-Text pairs |
+| Powerful-Kazakh-Dialogue | 10,000 | User-Assistant pairs |
+| **Total** | **61,255** | Retrieval-focused pairs |
+
+### Training Configuration
+
+- **Base Model**: intfloat/multilingual-e5-base
+- **Epochs**: 2
+- **Batch Size**: 16
+- **Learning Rate**: 1e-5
+- **Loss**: MultipleNegativesRankingLoss
+- **Hardware**: NVIDIA GPU
+
+### Training Strategy
+
+We found that:
+1. **Retrieval-only data** works best (no NLI/STS data)
+2. **2 epochs** is optimal (1 = underfit, 3 = overfit)
+3. **Larger batch size** (16) provides more in-batch negatives
+
+## Limitations
+
+- Optimized for Kazakh; performance on other languages may vary
+- Best for retrieval tasks; may not be optimal for semantic similarity
+- Requires `query:` and `passage:` prefixes for best results
+
+## Citation
+
+```bibtex
+@misc{kazembed2024,
+  title={KazEmbed-V5: A Fine-tuned Embedding Model for Kazakh Language Retrieval},
+  author={Your Name},
+  year={2024},
+  howpublished={HuggingFace Hub}
+}
+```
+
+## Acknowledgments
+
+- Base model: [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base)
+- Training data: [ISSAI](https://issai.nu.edu.kz/) for KazQAD dataset

config.json

@@ -0,0 +1,27 @@
+{
+  "architectures": [
+    "XLMRobertaModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "xlm-roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "output_past": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.57.3",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 250002
+}

config_sentence_transformers.json

@@ -0,0 +1,14 @@
+{
+  "model_type": "SentenceTransformer",
+  "__version__": {
+    "sentence_transformers": "5.1.2",
+    "transformers": "4.57.3",
+    "pytorch": "2.9.1+cu128"
+  },
+  "prompts": {
+    "query": "",
+    "document": ""
+  },
+  "default_prompt_name": null,
+  "similarity_fn_name": "cosine"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0579b07073d7da806177eaa5abe6a3be9fdb94d1c1a620313bc78ddeab247e34
+size 1112197096

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.models.Normalize"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+    "max_seq_length": 512,
+    "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:883b037111086fd4dfebbbc9b7cee11e1517b5e0c0514879478661440f137085
+size 17082987

tokenizer_config.json

@@ -0,0 +1,62 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "250001": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "max_length": 512,
+  "model_max_length": 512,
+  "pad_to_multiple_of": null,
+  "pad_token": "<pad>",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "</s>",
+  "stride": 0,
+  "tokenizer_class": "XLMRobertaTokenizerFast",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "<unk>"
+}