sh7vashrestha/Bert_Small_13M_SentimentalAnalysis

KDLLM Student Model — Distilled Lightweight BERT

This repository hosts the Student Model distilled under the KDLLM framework proposed in:

KDLLM: Knowledge Distillation for Compressed and Copyright-Safe Large Language Model Sharing
Shiva Shrestha et al.
Tsinghua Science and Technology, 2025
Manuscript ID: TST-2025-0253

---

📄 Overview

The KDLLM Student model is a compact distilled version of the fine tuned teacher bert-base-uncased model - sh7vashrestha/BertBaseUncased-SenetimentAnalysis. It preserves most of the teacher's performance while significantly reducing model size and computational requirements. The student model was trained using behavioral knowledge distillation:

• Soft-label distillation using:
  • ✅ Kullback-Leibler Divergence (KLD)
• Ground-truth supervised learning (cross-entropy loss).

---

🔧 Model Architecture

Component	Configuration
Layers	4
Hidden Size	312
Attention Heads	6
Feedforward Size	1024
Dropout	0.2
Parameters	~13.9M
File Size	~54MB

---

📊 Performance Summary

Distillation Loss	Accuracy	F1 Score
KLD	86.30%	86.23%

• Dataset: IMDb sentiment classification (binary, 50k samples)
• Compression ratio: ~7.74× smaller than teacher.

---

🚀 Inference Example

from transformers import BertForSequenceClassification, BertTokenizer

tokenizer = BertTokenizer.from_pretrained("sh7vashrestha/Bert_Small_13M_SentimentalAnalysis")
model = BertForSequenceClassification.from_pretrained("sh7vashrestha/Bert_Small_13M_SentimentalAnalysis")

inputs = tokenizer("I absolutely loved the movie!", return_tensors="pt")
outputs = model(**inputs)
prediction = outputs.logits.argmax(dim=1)
print(prediction)
label_mapping = {0: "negative", 1: "positive"}
prediction_label = label_mapping[prediction.item()]
print(prediction_label)