Update README.md
Browse files
README.md
CHANGED
|
@@ -19,6 +19,47 @@ This is a layer-pruned pre-trained language model sliced with [mergekit](https:/
|
|
| 19 |
|
| 20 |
|
| 21 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
## Merge Details
|
| 23 |
### Merge Method
|
| 24 |
|
|
|
|
| 19 |
|
| 20 |
|
| 21 |
|
| 22 |
+
|
| 23 |
+
## Quick eval
|
| 24 |
+
|
| 25 |
+
Quick eval for: pszemraj/Mistral-7B-v0.3-prune6
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
hf (pretrained=pszemraj/Mistral-7B-v0.3-prune6,trust_remote_code=True,dtype=bfloat16), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 2
|
| 29 |
+
|
| 30 |
+
| Tasks |Version|Filter|n-shot| Metric | Value | |Stderr|
|
| 31 |
+
|--------------|------:|------|-----:|----------|------:|---|-----:|
|
| 32 |
+
|arc_easy | 1|none | 0|acc | 0.6393|± |0.0099|
|
| 33 |
+
| | |none | 0|acc_norm | 0.6309|± |0.0099|
|
| 34 |
+
|boolq | 2|none | 0|acc | 0.7599|± |0.0075|
|
| 35 |
+
|lambada_openai| 1|none | 0|perplexity|10.1184|± |0.2771|
|
| 36 |
+
| | |none | 0|acc | 0.5507|± |0.0069|
|
| 37 |
+
|openbookqa | 1|none | 0|acc | 0.2200|± |0.0185|
|
| 38 |
+
| | |none | 0|acc_norm | 0.3580|± |0.0215|
|
| 39 |
+
|piqa | 1|none | 0|acc | 0.7203|± |0.0105|
|
| 40 |
+
| | |none | 0|acc_norm | 0.7350|± |0.0103|
|
| 41 |
+
|winogrande | 1|none | 0|acc | 0.6906|± |0.0130|
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
### original
|
| 45 |
+
|
| 46 |
+
bootstrapping for stddev: perplexity
|
| 47 |
+
hf (pretrained=mistralai/Mistral-7B-v0.3,trust_remote_code=True,dtype=bfloat16), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 2
|
| 48 |
+
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|
| 49 |
+
|--------------|------:|------|-----:|----------|-----:|---|-----:|
|
| 50 |
+
|arc_easy | 1|none | 0|acc |0.7959|± |0.0083|
|
| 51 |
+
| | |none | 0|acc_norm |0.7832|± |0.0085|
|
| 52 |
+
|boolq | 2|none | 0|acc |0.8202|± |0.0067|
|
| 53 |
+
|lambada_openai| 1|none | 0|perplexity|3.2578|± |0.0601|
|
| 54 |
+
| | |none | 0|acc |0.7518|± |0.0060|
|
| 55 |
+
|openbookqa | 1|none | 0|acc |0.3340|± |0.0211|
|
| 56 |
+
| | |none | 0|acc_norm |0.4420|± |0.0222|
|
| 57 |
+
|piqa | 1|none | 0|acc |0.8009|± |0.0093|
|
| 58 |
+
| | |none | 0|acc_norm |0.8215|± |0.0089|
|
| 59 |
+
|winogrande | 1|none | 0|acc |0.7380|± |0.0124|
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
|
| 63 |
## Merge Details
|
| 64 |
### Merge Method
|
| 65 |
|