Introducing the Palmyra-mini family: Powerful, lightweight, and ready to reason!
•
58
AI & ML interests
AGI, LLMs, Knowledge Graph, Palmyra, Domain Specific LLM
Recent Activity
View all activity
Articles
zhenyu-writer
updated
a
dataset
1 day ago
zhenyu-writer
published
a
dataset
1 day ago
zhenyu-writer
updated
a
dataset
2 days ago
zhenyu-writer
published
a
dataset
2 days ago
zhenyu-writer
updated
a
dataset
2 days ago
zhenyu-writer
published
a
dataset
2 days ago
zhenyu-writer
updated
a
dataset
3 days ago
zhenyu-writer
published
a
dataset
3 days ago
zhenyu-writer
updated
a
dataset
3 days ago
zhenyu-writer
updated
a
dataset
4 days ago
sanderland
authored
a
paper
about 1 month ago
dmytro-writer
authored
a
paper
6 months ago
sanderland
authored
a
paper
6 months ago
wassemgtk
authored
a
paper
6 months ago
sanderland
authored
a
paper
7 months ago
Post
3238
I’ve been diving into the iRoPE architecture from Llama 4—a game-changer for long-context models! It interleaves local attention (with RoPE) for short contexts and global attention (with inference-time temp scaling) for long-range reasoning, aiming for infinite context. I’m going to try writing iRoPE—who wants to help?
Code: https://github.com/wassemgtk/iRoPE-try/blob/main/iRoPE.ipynb
Code: https://github.com/wassemgtk/iRoPE-try/blob/main/iRoPE.ipynb
-
- 1 reply
sanderland
authored
a
paper
8 months ago
Post
2133
For fun, a new project: SuperTokenizer! A BPE tokenizer trained on C4 to beat GPT-4. Byte-level, A100-powered, and open-source. Messing around with tokens!
https://github.com/wassemgtk/SuperTokenizer
https://github.com/wassemgtk/SuperTokenizer
- 1 reply
Post
1918
# GESAL: Real-Time Adaptation for LLMs
We’re excited to unveil **Graph-Enhanced Singular Adaptive Learning (GESAL)**, a framework that lets LLMs like
🔗 **Code**: [https://github.com/writer/AI-Adaptive-Learning-GESAL](https://github.com/writer/AI-Adaptive-Learning-GESAL)
---
## Why GESAL?
Static LLMs struggle to adapt without heavy retraining. GESAL solves this with:
- **SVF**: Adapts weights via \( W' = U (\Sigma \cdot z) V^T \), using few parameters.
- **Graph Memory**: Stores adaptations in nodes for scalability.
- **RL**: Updates via \( J(z) = \mathbb{E}[\log \pi_z(y|x) r] \) based on feedback.
---
## How It Works
Ask "How many R’s in ‘strawberry’?" If it says "2" and you say "no," GESAL learns to say "3" next time, avoiding repeats.
---
## Try It
Built with Hugging Face’s
Needs a Hugging Face token for Llama-3.2-1B.
---
## Results
GESAL hits 95% accuracy after 5 feedbacks vs. LoRA’s 70%. It’s efficient (~0.5M params) and scalable.
We’re excited to unveil **Graph-Enhanced Singular Adaptive Learning (GESAL)**, a framework that lets LLMs like
meta-llama/Llama-3.2-1B adapt in real time using user feedback. Check out the code and white paper on GitHub!🔗 **Code**: [https://github.com/writer/AI-Adaptive-Learning-GESAL](https://github.com/writer/AI-Adaptive-Learning-GESAL)
---
## Why GESAL?
Static LLMs struggle to adapt without heavy retraining. GESAL solves this with:
- **SVF**: Adapts weights via \( W' = U (\Sigma \cdot z) V^T \), using few parameters.
- **Graph Memory**: Stores adaptations in nodes for scalability.
- **RL**: Updates via \( J(z) = \mathbb{E}[\log \pi_z(y|x) r] \) based on feedback.
---
## How It Works
Ask "How many R’s in ‘strawberry’?" If it says "2" and you say "no," GESAL learns to say "3" next time, avoiding repeats.
---
## Try It
Built with Hugging Face’s
transformers:pip install transformers torch numpy
python Adaptive_Learning_(GESAL).pyNeeds a Hugging Face token for Llama-3.2-1B.
---
## Results
GESAL hits 95% accuracy after 5 feedbacks vs. LoRA’s 70%. It’s efficient (~0.5M params) and scalable.
-
- 15 replies
wassemgtk
authored
a
paper
10 months ago