Skip to content

Instantly share code, notes, and snippets.

@darkacorn

darkacorn/gist:71658f280ea0fc0ad4b97d2a616f4ce8 Secret

Created September 16, 2023 22:34
Show Gist options
  • Save darkacorn/71658f280ea0fc0ad4b97d2a616f4ce8 to your computer and use it in GitHub Desktop.
Save darkacorn/71658f280ea0fc0ad4b97d2a616f4ce8 to your computer and use it in GitHub Desktop.
Download ZIP
100k test . exllama2(testbranch) + fa 1 - 100k in 128t steps
Raw
gistfile1.txt
(venv) (extras) ➜ exllamav2 git:(test) ✗ python test_inference.py -m ./models/CodeLlama-34B-instruct-4.0bpw-h6-exl2 -p "once upon a time" -s -l 98304
-- Model: ./models/CodeLlama-34B-instruct-4.0bpw-h6-exl2
-- Options: ['length: 98304', 'rope_scale 1.0', 'rope_alpha 1.0']
-- Loading model...
-- Loading tokenizer...
-- Warmup...
-- Generating (greedy sampling)...
once upon a time, I was a young man, and I was in love with a young woman. She was a beautiful woman, and I loved her very much. I was very happy with her, and I thought that I would be happy with her forever. But, as it often happens, our love was not meant to be.
One day, I was walking down the street, and I saw her walking towards me. She was walking with another man, and I could see that they were very much in love. I felt a pang of jealousy, and I felt very sad. I knew that I could never be with her again
Prompt processed in 0.06 seconds, 4 tokens, 67.14 tokens/second
Response generated in 4.08 seconds, 128 tokens, 31.35 tokens/second
-- Measuring token speed...
** Position 1 + 127 tokens: 31.5753 t/s
** Position 128 + 128 tokens: 31.6137 t/s
** Position 256 + 128 tokens: 31.4325 t/s
** Position 384 + 128 tokens: 31.2780 t/s
** Position 512 + 128 tokens: 30.7072 t/s
** Position 640 + 128 tokens: 30.4688 t/s
** Position 768 + 128 tokens: 30.7787 t/s
** Position 896 + 128 tokens: 30.6061 t/s
** Position 1024 + 128 tokens: 30.1270 t/s
** Position 1152 + 128 tokens: 30.0503 t/s
** Position 1280 + 128 tokens: 29.7615 t/s
** Position 1408 + 128 tokens: 29.6711 t/s
** Position 1536 + 128 tokens: 29.5386 t/s
** Position 1664 + 128 tokens: 29.4738 t/s
** Position 1792 + 128 tokens: 29.4152 t/s
** Position 1920 + 128 tokens: 29.0489 t/s
** Position 2048 + 128 tokens: 28.9033 t/s
** Position 2176 + 128 tokens: 28.8572 t/s
** Position 2304 + 128 tokens: 28.8322 t/s
** Position 2432 + 128 tokens: 28.7923 t/s
** Position 2560 + 128 tokens: 28.3350 t/s
** Position 2688 + 128 tokens: 28.3396 t/s
** Position 2816 + 128 tokens: 28.3114 t/s
** Position 2944 + 128 tokens: 28.2951 t/s
** Position 3072 + 128 tokens: 28.2587 t/s
** Position 3200 + 128 tokens: 27.8467 t/s
** Position 3328 + 128 tokens: 27.8160 t/s
** Position 3456 + 128 tokens: 27.7989 t/s
** Position 3584 + 128 tokens: 27.8029 t/s
** Position 3712 + 128 tokens: 27.7755 t/s
** Position 3840 + 128 tokens: 27.3631 t/s
** Position 3968 + 128 tokens: 27.3385 t/s
** Position 4096 + 128 tokens: 27.3367 t/s
** Position 4224 + 128 tokens: 27.3336 t/s
** Position 4352 + 128 tokens: 27.3106 t/s
** Position 4480 + 128 tokens: 26.9106 t/s
** Position 4608 + 128 tokens: 26.9104 t/s
** Position 4736 + 128 tokens: 26.9030 t/s
** Position 4864 + 128 tokens: 26.8936 t/s
** Position 4992 + 128 tokens: 26.8675 t/s
** Position 5120 + 128 tokens: 26.4791 t/s
** Position 5248 + 128 tokens: 26.4643 t/s
** Position 5376 + 128 tokens: 26.4681 t/s
** Position 5504 + 128 tokens: 26.4699 t/s
** Position 5632 + 128 tokens: 26.4290 t/s
** Position 5760 + 128 tokens: 26.0589 t/s
** Position 5888 + 128 tokens: 26.0545 t/s
** Position 6016 + 128 tokens: 26.0492 t/s
** Position 6144 + 128 tokens: 26.0418 t/s
** Position 6272 + 128 tokens: 26.0226 t/s
** Position 6400 + 128 tokens: 25.6594 t/s
** Position 6528 + 128 tokens: 25.6496 t/s
** Position 6656 + 128 tokens: 25.6482 t/s
** Position 6784 + 128 tokens: 25.6418 t/s
** Position 6912 + 128 tokens: 25.6231 t/s
** Position 7040 + 128 tokens: 25.2637 t/s
** Position 7168 + 128 tokens: 25.2629 t/s
** Position 7296 + 128 tokens: 25.2511 t/s
** Position 7424 + 128 tokens: 25.2524 t/s
** Position 7552 + 128 tokens: 25.2336 t/s
** Position 7680 + 128 tokens: 24.8848 t/s
** Position 7808 + 128 tokens: 24.8758 t/s
** Position 7936 + 128 tokens: 24.8785 t/s
** Position 8064 + 128 tokens: 24.8749 t/s
** Position 8192 + 128 tokens: 24.8503 t/s
** Position 8320 + 128 tokens: 24.5163 t/s
** Position 8448 + 128 tokens: 24.5081 t/s
** Position 8576 + 128 tokens: 24.5087 t/s
** Position 8704 + 128 tokens: 24.5046 t/s
** Position 8832 + 128 tokens: 24.4818 t/s
** Position 8960 + 128 tokens: 24.1547 t/s
** Position 9088 + 128 tokens: 24.1565 t/s
** Position 9216 + 128 tokens: 24.1504 t/s
** Position 9344 + 128 tokens: 24.1495 t/s
** Position 9472 + 128 tokens: 24.1181 t/s
** Position 9600 + 128 tokens: 23.8081 t/s
** Position 9728 + 128 tokens: 23.8030 t/s
** Position 9856 + 128 tokens: 23.7986 t/s
** Position 9984 + 128 tokens: 23.7926 t/s
** Position 10112 + 128 tokens: 23.7667 t/s
** Position 10240 + 128 tokens: 23.4616 t/s
** Position 10368 + 128 tokens: 23.4596 t/s
** Position 10496 + 128 tokens: 23.4495 t/s
** Position 10624 + 128 tokens: 23.4440 t/s
** Position 10752 + 128 tokens: 23.4241 t/s
** Position 10880 + 128 tokens: 23.1272 t/s
** Position 11008 + 128 tokens: 23.1173 t/s
** Position 11136 + 128 tokens: 23.1104 t/s
** Position 11264 + 128 tokens: 23.1080 t/s
** Position 11392 + 128 tokens: 23.0686 t/s
** Position 11520 + 128 tokens: 22.7892 t/s
** Position 11648 + 128 tokens: 22.7846 t/s
** Position 11776 + 128 tokens: 22.7795 t/s
** Position 11904 + 128 tokens: 22.7744 t/s
** Position 12032 + 128 tokens: 22.7525 t/s
** Position 12160 + 128 tokens: 22.4659 t/s
** Position 12288 + 128 tokens: 22.4674 t/s
** Position 12416 + 128 tokens: 22.4668 t/s
** Position 12544 + 128 tokens: 22.4491 t/s
** Position 12672 + 128 tokens: 22.4255 t/s
** Position 12800 + 128 tokens: 22.1597 t/s
** Position 12928 + 128 tokens: 22.1588 t/s
** Position 13056 + 128 tokens: 22.1400 t/s
** Position 13184 + 128 tokens: 22.1343 t/s
** Position 13312 + 128 tokens: 22.1004 t/s
** Position 13440 + 128 tokens: 21.8426 t/s
** Position 13568 + 128 tokens: 21.8428 t/s
** Position 13696 + 128 tokens: 21.8379 t/s
** Position 13824 + 128 tokens: 21.8274 t/s
** Position 13952 + 128 tokens: 21.8012 t/s
** Position 14080 + 128 tokens: 21.5445 t/s
** Position 14208 + 128 tokens: 21.5482 t/s
** Position 14336 + 128 tokens: 21.5287 t/s
** Position 14464 + 128 tokens: 21.5117 t/s
** Position 14592 + 128 tokens: 21.4929 t/s
** Position 14720 + 128 tokens: 21.2440 t/s
** Position 14848 + 128 tokens: 21.2461 t/s
** Position 14976 + 128 tokens: 21.2341 t/s
** Position 15104 + 128 tokens: 21.2280 t/s
** Position 15232 + 128 tokens: 21.1930 t/s
** Position 15360 + 128 tokens: 20.9563 t/s
** Position 15488 + 128 tokens: 20.9607 t/s
** Position 15616 + 128 tokens: 20.9536 t/s
** Position 15744 + 128 tokens: 20.9447 t/s
** Position 15872 + 128 tokens: 20.9196 t/s
** Position 16000 + 128 tokens: 20.6799 t/s
** Position 16128 + 128 tokens: 20.6832 t/s
** Position 16256 + 128 tokens: 20.6715 t/s
** Position 16384 + 128 tokens: 20.6570 t/s
** Position 16512 + 128 tokens: 20.6358 t/s
** Position 16640 + 128 tokens: 20.4054 t/s
** Position 16768 + 128 tokens: 20.4034 t/s
** Position 16896 + 128 tokens: 20.4059 t/s
** Position 17024 + 128 tokens: 20.3895 t/s
** Position 17152 + 128 tokens: 20.3707 t/s
** Position 17280 + 128 tokens: 20.1369 t/s
** Position 17408 + 128 tokens: 20.1434 t/s
** Position 17536 + 128 tokens: 20.1304 t/s
** Position 17664 + 128 tokens: 20.1283 t/s
** Position 17792 + 128 tokens: 20.1176 t/s
** Position 17920 + 128 tokens: 19.8874 t/s
** Position 18048 + 128 tokens: 19.8845 t/s
** Position 18176 + 128 tokens: 19.8754 t/s
** Position 18304 + 128 tokens: 19.8638 t/s
** Position 18432 + 128 tokens: 19.8465 t/s
** Position 18560 + 128 tokens: 19.6324 t/s
** Position 18688 + 128 tokens: 19.6340 t/s
** Position 18816 + 128 tokens: 19.6322 t/s
** Position 18944 + 128 tokens: 19.6181 t/s
** Position 19072 + 128 tokens: 19.5942 t/s
** Position 19200 + 128 tokens: 19.3912 t/s
** Position 19328 + 128 tokens: 19.3898 t/s
** Position 19456 + 128 tokens: 19.3870 t/s
** Position 19584 + 128 tokens: 19.3749 t/s
** Position 19712 + 128 tokens: 19.3538 t/s
** Position 19840 + 128 tokens: 19.1492 t/s
** Position 19968 + 128 tokens: 19.1578 t/s
** Position 20096 + 128 tokens: 19.1474 t/s
** Position 20224 + 128 tokens: 19.1366 t/s
** Position 20352 + 128 tokens: 19.1127 t/s
** Position 20480 + 128 tokens: 18.9146 t/s
** Position 20608 + 128 tokens: 18.9104 t/s
** Position 20736 + 128 tokens: 18.9059 t/s
** Position 20864 + 128 tokens: 18.9014 t/s
** Position 20992 + 128 tokens: 18.8787 t/s
** Position 21120 + 128 tokens: 18.6797 t/s
** Position 21248 + 128 tokens: 18.6847 t/s
** Position 21376 + 128 tokens: 18.6827 t/s
** Position 21504 + 128 tokens: 18.6702 t/s
** Position 21632 + 128 tokens: 18.6580 t/s
** Position 21760 + 128 tokens: 18.4694 t/s
** Position 21888 + 128 tokens: 18.4765 t/s
** Position 22016 + 128 tokens: 18.4687 t/s
** Position 22144 + 128 tokens: 18.4585 t/s
** Position 22272 + 128 tokens: 18.4408 t/s
** Position 22400 + 128 tokens: 18.2526 t/s
** Position 22528 + 128 tokens: 18.2567 t/s
** Position 22656 + 128 tokens: 18.2510 t/s
** Position 22784 + 128 tokens: 18.2440 t/s
** Position 22912 + 128 tokens: 18.2185 t/s
** Position 23040 + 128 tokens: 18.0393 t/s
** Position 23168 + 128 tokens: 18.0449 t/s
** Position 23296 + 128 tokens: 18.0337 t/s
** Position 23424 + 128 tokens: 18.0327 t/s
** Position 23552 + 128 tokens: 18.0052 t/s
** Position 23680 + 128 tokens: 17.8285 t/s
** Position 23808 + 128 tokens: 17.8295 t/s
** Position 23936 + 128 tokens: 17.8247 t/s
** Position 24064 + 128 tokens: 17.8175 t/s
** Position 24192 + 128 tokens: 17.7962 t/s
** Position 24320 + 128 tokens: 17.6166 t/s
** Position 24448 + 128 tokens: 17.6180 t/s
** Position 24576 + 128 tokens: 17.6193 t/s
** Position 24704 + 128 tokens: 17.6156 t/s
** Position 24832 + 128 tokens: 17.5961 t/s
** Position 24960 + 128 tokens: 17.4273 t/s
** Position 25088 + 128 tokens: 17.4334 t/s
** Position 25216 + 128 tokens: 17.4203 t/s
** Position 25344 + 128 tokens: 17.4208 t/s
** Position 25472 + 128 tokens: 17.4000 t/s
** Position 25600 + 128 tokens: 17.2340 t/s
** Position 25728 + 128 tokens: 17.2391 t/s
** Position 25856 + 128 tokens: 17.2397 t/s
** Position 25984 + 128 tokens: 17.2358 t/s
** Position 26112 + 128 tokens: 17.2139 t/s
** Position 26240 + 128 tokens: 17.0507 t/s
** Position 26368 + 128 tokens: 17.0513 t/s
** Position 26496 + 128 tokens: 17.0512 t/s
** Position 26624 + 128 tokens: 17.0494 t/s
** Position 26752 + 128 tokens: 17.0275 t/s
** Position 26880 + 128 tokens: 16.8585 t/s
** Position 27008 + 128 tokens: 16.8693 t/s
** Position 27136 + 128 tokens: 16.8563 t/s
** Position 27264 + 128 tokens: 16.8471 t/s
** Position 27392 + 128 tokens: 16.8315 t/s
** Position 27520 + 128 tokens: 16.6759 t/s
** Position 27648 + 128 tokens: 16.6823 t/s
** Position 27776 + 128 tokens: 16.6758 t/s
** Position 27904 + 128 tokens: 16.6618 t/s
** Position 28032 + 128 tokens: 16.6495 t/s
** Position 28160 + 128 tokens: 16.4907 t/s
** Position 28288 + 128 tokens: 16.4981 t/s
** Position 28416 + 128 tokens: 16.4948 t/s
** Position 28544 + 128 tokens: 16.4906 t/s
** Position 28672 + 128 tokens: 16.4704 t/s
** Position 28800 + 128 tokens: 16.3111 t/s
** Position 28928 + 128 tokens: 16.3189 t/s
** Position 29056 + 128 tokens: 16.3124 t/s
** Position 29184 + 128 tokens: 16.3061 t/s
** Position 29312 + 128 tokens: 16.2886 t/s
** Position 29440 + 128 tokens: 16.1357 t/s
** Position 29568 + 128 tokens: 16.1505 t/s
** Position 29696 + 128 tokens: 16.1366 t/s
** Position 29824 + 128 tokens: 16.1323 t/s
** Position 29952 + 128 tokens: 16.1130 t/s
** Position 30080 + 128 tokens: 15.9660 t/s
** Position 30208 + 128 tokens: 15.9751 t/s
** Position 30336 + 128 tokens: 15.9761 t/s
** Position 30464 + 128 tokens: 15.9638 t/s
** Position 30592 + 128 tokens: 15.9537 t/s
** Position 30720 + 128 tokens: 15.7994 t/s
** Position 30848 + 128 tokens: 15.8105 t/s
** Position 30976 + 128 tokens: 15.8005 t/s
** Position 31104 + 128 tokens: 15.7899 t/s
** Position 31232 + 128 tokens: 15.7755 t/s
** Position 31360 + 128 tokens: 15.6306 t/s
** Position 31488 + 128 tokens: 15.6477 t/s
** Position 31616 + 128 tokens: 15.6393 t/s
** Position 31744 + 128 tokens: 15.6379 t/s
** Position 31872 + 128 tokens: 15.6197 t/s
** Position 32000 + 128 tokens: 15.4740 t/s
** Position 32128 + 128 tokens: 15.4845 t/s
** Position 32256 + 128 tokens: 15.4768 t/s
** Position 32384 + 128 tokens: 15.4770 t/s
** Position 32512 + 128 tokens: 15.4603 t/s
** Position 32640 + 128 tokens: 15.3124 t/s
** Position 32768 + 128 tokens: 15.3096 t/s
** Position 32896 + 128 tokens: 15.2976 t/s
** Position 33024 + 128 tokens: 15.2937 t/s
** Position 33152 + 128 tokens: 15.2767 t/s
** Position 33280 + 128 tokens: 15.1337 t/s
** Position 33408 + 128 tokens: 15.1481 t/s
** Position 33536 + 128 tokens: 15.1377 t/s
** Position 33664 + 128 tokens: 15.1355 t/s
** Position 33792 + 128 tokens: 15.1220 t/s
** Position 33920 + 128 tokens: 14.9886 t/s
** Position 34048 + 128 tokens: 14.9991 t/s
** Position 34176 + 128 tokens: 14.9939 t/s
** Position 34304 + 128 tokens: 14.9908 t/s
** Position 34432 + 128 tokens: 14.9706 t/s
** Position 34560 + 128 tokens: 14.8346 t/s
** Position 34688 + 128 tokens: 14.8536 t/s
** Position 34816 + 128 tokens: 14.8533 t/s
** Position 34944 + 128 tokens: 14.8432 t/s
** Position 35072 + 128 tokens: 14.8321 t/s
** Position 35200 + 128 tokens: 14.6969 t/s
** Position 35328 + 128 tokens: 14.7088 t/s
** Position 35456 + 128 tokens: 14.7062 t/s
** Position 35584 + 128 tokens: 14.6928 t/s
** Position 35712 + 128 tokens: 14.6842 t/s
** Position 35840 + 128 tokens: 14.5506 t/s
** Position 35968 + 128 tokens: 14.5657 t/s
** Position 36096 + 128 tokens: 14.5635 t/s
** Position 36224 + 128 tokens: 14.5574 t/s
** Position 36352 + 128 tokens: 14.5506 t/s
** Position 36480 + 128 tokens: 14.4242 t/s
** Position 36608 + 128 tokens: 14.4383 t/s
** Position 36736 + 128 tokens: 14.4378 t/s
** Position 36864 + 128 tokens: 14.4335 t/s
** Position 36992 + 128 tokens: 14.4157 t/s
** Position 37120 + 128 tokens: 14.2919 t/s
** Position 37248 + 128 tokens: 14.3118 t/s
** Position 37376 + 128 tokens: 14.3067 t/s
** Position 37504 + 128 tokens: 14.2981 t/s
** Position 37632 + 128 tokens: 14.2788 t/s
** Position 37760 + 128 tokens: 14.1544 t/s
** Position 37888 + 128 tokens: 14.1729 t/s
** Position 38016 + 128 tokens: 14.1678 t/s
** Position 38144 + 128 tokens: 14.1595 t/s
** Position 38272 + 128 tokens: 14.1428 t/s
** Position 38400 + 128 tokens: 14.0208 t/s
** Position 38528 + 128 tokens: 14.0338 t/s
** Position 38656 + 128 tokens: 14.0348 t/s
** Position 38784 + 128 tokens: 14.0309 t/s
** Position 38912 + 128 tokens: 14.0165 t/s
** Position 39040 + 128 tokens: 13.8909 t/s
** Position 39168 + 128 tokens: 13.9045 t/s
** Position 39296 + 128 tokens: 13.9009 t/s
** Position 39424 + 128 tokens: 13.8933 t/s
** Position 39552 + 128 tokens: 13.8873 t/s
** Position 39680 + 128 tokens: 13.7725 t/s
** Position 39808 + 128 tokens: 13.7743 t/s
** Position 39936 + 128 tokens: 13.7713 t/s
** Position 40064 + 128 tokens: 13.7696 t/s
** Position 40192 + 128 tokens: 13.7549 t/s
** Position 40320 + 128 tokens: 13.6461 t/s
** Position 40448 + 128 tokens: 13.6566 t/s
** Position 40576 + 128 tokens: 13.6548 t/s
** Position 40704 + 128 tokens: 13.6536 t/s
** Position 40832 + 128 tokens: 13.6391 t/s
** Position 40960 + 128 tokens: 13.5260 t/s
** Position 41088 + 128 tokens: 13.5409 t/s
** Position 41216 + 128 tokens: 13.5389 t/s
** Position 41344 + 128 tokens: 13.5355 t/s
** Position 41472 + 128 tokens: 13.5236 t/s
** Position 41600 + 128 tokens: 13.4011 t/s
** Position 41728 + 128 tokens: 13.4125 t/s
** Position 41856 + 128 tokens: 13.4182 t/s
** Position 41984 + 128 tokens: 13.4124 t/s
** Position 42112 + 128 tokens: 13.4024 t/s
** Position 42240 + 128 tokens: 13.2782 t/s
** Position 42368 + 128 tokens: 13.3024 t/s
** Position 42496 + 128 tokens: 13.3043 t/s
** Position 42624 + 128 tokens: 13.2985 t/s
** Position 42752 + 128 tokens: 13.2896 t/s
** Position 42880 + 128 tokens: 13.1652 t/s
** Position 43008 + 128 tokens: 13.1838 t/s
** Position 43136 + 128 tokens: 13.1787 t/s
** Position 43264 + 128 tokens: 13.1772 t/s
** Position 43392 + 128 tokens: 13.1672 t/s
** Position 43520 + 128 tokens: 13.0544 t/s
** Position 43648 + 128 tokens: 13.0666 t/s
** Position 43776 + 128 tokens: 13.0642 t/s
** Position 43904 + 128 tokens: 13.0684 t/s
** Position 44032 + 128 tokens: 13.0519 t/s
** Position 44160 + 128 tokens: 12.9422 t/s
** Position 44288 + 128 tokens: 12.9567 t/s
** Position 44416 + 128 tokens: 12.9552 t/s
** Position 44544 + 128 tokens: 12.9579 t/s
** Position 44672 + 128 tokens: 12.9459 t/s
** Position 44800 + 128 tokens: 12.8417 t/s
** Position 44928 + 128 tokens: 12.8536 t/s
** Position 45056 + 128 tokens: 12.8504 t/s
** Position 45184 + 128 tokens: 12.8483 t/s
** Position 45312 + 128 tokens: 12.8432 t/s
** Position 45440 + 128 tokens: 12.7357 t/s
** Position 45568 + 128 tokens: 12.7498 t/s
** Position 45696 + 128 tokens: 12.7455 t/s
** Position 45824 + 128 tokens: 12.7355 t/s
** Position 45952 + 128 tokens: 12.7238 t/s
** Position 46080 + 128 tokens: 12.6227 t/s
** Position 46208 + 128 tokens: 12.6422 t/s
** Position 46336 + 128 tokens: 12.6307 t/s
** Position 46464 + 128 tokens: 12.6315 t/s
** Position 46592 + 128 tokens: 12.6235 t/s
** Position 46720 + 128 tokens: 12.5195 t/s
** Position 46848 + 128 tokens: 12.5296 t/s
** Position 46976 + 128 tokens: 12.5252 t/s
** Position 47104 + 128 tokens: 12.5198 t/s
** Position 47232 + 128 tokens: 12.5181 t/s
** Position 47360 + 128 tokens: 12.4111 t/s
** Position 47488 + 128 tokens: 12.4292 t/s
** Position 47616 + 128 tokens: 12.4268 t/s
** Position 47744 + 128 tokens: 12.4145 t/s
** Position 47872 + 128 tokens: 12.4118 t/s
** Position 48000 + 128 tokens: 12.3044 t/s
** Position 48128 + 128 tokens: 12.3165 t/s
** Position 48256 + 128 tokens: 12.3178 t/s
** Position 48384 + 128 tokens: 12.3139 t/s
** Position 48512 + 128 tokens: 12.3062 t/s
** Position 48640 + 128 tokens: 12.2042 t/s
** Position 48768 + 128 tokens: 12.2153 t/s
** Position 48896 + 128 tokens: 12.2151 t/s
** Position 49024 + 128 tokens: 12.2090 t/s
** Position 49152 + 128 tokens: 12.2025 t/s
** Position 49280 + 128 tokens: 12.0977 t/s
** Position 49408 + 128 tokens: 12.1157 t/s
** Position 49536 + 128 tokens: 12.1117 t/s
** Position 49664 + 128 tokens: 12.1067 t/s
** Position 49792 + 128 tokens: 12.1104 t/s
** Position 49920 + 128 tokens: 12.0096 t/s
** Position 50048 + 128 tokens: 12.0191 t/s
** Position 50176 + 128 tokens: 12.0234 t/s
** Position 50304 + 128 tokens: 12.0184 t/s
** Position 50432 + 128 tokens: 12.0061 t/s
** Position 50560 + 128 tokens: 11.9108 t/s
** Position 50688 + 128 tokens: 11.9254 t/s
** Position 50816 + 128 tokens: 11.9283 t/s
** Position 50944 + 128 tokens: 11.9244 t/s
** Position 51072 + 128 tokens: 11.9235 t/s
** Position 51200 + 128 tokens: 11.8233 t/s
** Position 51328 + 128 tokens: 11.8368 t/s
** Position 51456 + 128 tokens: 11.8389 t/s
** Position 51584 + 128 tokens: 11.8357 t/s
** Position 51712 + 128 tokens: 11.8285 t/s
** Position 51840 + 128 tokens: 11.7321 t/s
** Position 51968 + 128 tokens: 11.7425 t/s
** Position 52096 + 128 tokens: 11.7421 t/s
** Position 52224 + 128 tokens: 11.7428 t/s
** Position 52352 + 128 tokens: 11.7384 t/s
** Position 52480 + 128 tokens: 11.6409 t/s
** Position 52608 + 128 tokens: 11.6557 t/s
** Position 52736 + 128 tokens: 11.6539 t/s
** Position 52864 + 128 tokens: 11.6508 t/s
** Position 52992 + 128 tokens: 11.6508 t/s
** Position 53120 + 128 tokens: 11.5554 t/s
** Position 53248 + 128 tokens: 11.5709 t/s
** Position 53376 + 128 tokens: 11.5705 t/s
** Position 53504 + 128 tokens: 11.5690 t/s
** Position 53632 + 128 tokens: 11.5628 t/s
** Position 53760 + 128 tokens: 11.4682 t/s
** Position 53888 + 128 tokens: 11.4785 t/s
** Position 54016 + 128 tokens: 11.4803 t/s
** Position 54144 + 128 tokens: 11.4718 t/s
** Position 54272 + 128 tokens: 11.4663 t/s
** Position 54400 + 128 tokens: 11.3785 t/s
** Position 54528 + 128 tokens: 11.3879 t/s
** Position 54656 + 128 tokens: 11.3910 t/s
** Position 54784 + 128 tokens: 11.3832 t/s
** Position 54912 + 128 tokens: 11.3787 t/s
** Position 55040 + 128 tokens: 11.2940 t/s
** Position 55168 + 128 tokens: 11.3083 t/s
** Position 55296 + 128 tokens: 11.3029 t/s
** Position 55424 + 128 tokens: 11.3009 t/s
** Position 55552 + 128 tokens: 11.2990 t/s
** Position 55680 + 128 tokens: 11.2116 t/s
** Position 55808 + 128 tokens: 11.2223 t/s
** Position 55936 + 128 tokens: 11.2202 t/s
** Position 56064 + 128 tokens: 11.2204 t/s
** Position 56192 + 128 tokens: 11.2138 t/s
** Position 56320 + 128 tokens: 11.1235 t/s
** Position 56448 + 128 tokens: 11.1363 t/s
** Position 56576 + 128 tokens: 11.1378 t/s
** Position 56704 + 128 tokens: 11.1348 t/s
** Position 56832 + 128 tokens: 11.1338 t/s
** Position 56960 + 128 tokens: 11.0475 t/s
** Position 57088 + 128 tokens: 11.0617 t/s
** Position 57216 + 128 tokens: 11.0598 t/s
** Position 57344 + 128 tokens: 11.0567 t/s
** Position 57472 + 128 tokens: 11.0518 t/s
** Position 57600 + 128 tokens: 10.9667 t/s
** Position 57728 + 128 tokens: 10.9781 t/s
** Position 57856 + 128 tokens: 10.9785 t/s
** Position 57984 + 128 tokens: 10.9809 t/s
** Position 58112 + 128 tokens: 10.9747 t/s
** Position 58240 + 128 tokens: 10.8874 t/s
** Position 58368 + 128 tokens: 10.8998 t/s
** Position 58496 + 128 tokens: 10.8998 t/s
** Position 58624 + 128 tokens: 10.8988 t/s
** Position 58752 + 128 tokens: 10.8977 t/s
** Position 58880 + 128 tokens: 10.8106 t/s
** Position 59008 + 128 tokens: 10.8229 t/s
** Position 59136 + 128 tokens: 10.8214 t/s
** Position 59264 + 128 tokens: 10.8133 t/s
** Position 59392 + 128 tokens: 10.8117 t/s
** Position 59520 + 128 tokens: 10.7252 t/s
** Position 59648 + 128 tokens: 10.7398 t/s
** Position 59776 + 128 tokens: 10.7427 t/s
** Position 59904 + 128 tokens: 10.7410 t/s
** Position 60032 + 128 tokens: 10.7406 t/s
** Position 60160 + 128 tokens: 10.6602 t/s
** Position 60288 + 128 tokens: 10.6691 t/s
** Position 60416 + 128 tokens: 10.6683 t/s
** Position 60544 + 128 tokens: 10.6647 t/s
** Position 60672 + 128 tokens: 10.6604 t/s
** Position 60800 + 128 tokens: 10.5789 t/s
** Position 60928 + 128 tokens: 10.5925 t/s
** Position 61056 + 128 tokens: 10.5909 t/s
** Position 61184 + 128 tokens: 10.5902 t/s
** Position 61312 + 128 tokens: 10.5901 t/s
** Position 61440 + 128 tokens: 10.5063 t/s
** Position 61568 + 128 tokens: 10.5183 t/s
** Position 61696 + 128 tokens: 10.5172 t/s
** Position 61824 + 128 tokens: 10.5177 t/s
** Position 61952 + 128 tokens: 10.5186 t/s
** Position 62080 + 128 tokens: 10.4338 t/s
** Position 62208 + 128 tokens: 10.4491 t/s
** Position 62336 + 128 tokens: 10.4498 t/s
** Position 62464 + 128 tokens: 10.4432 t/s
** Position 62592 + 128 tokens: 10.4422 t/s
** Position 62720 + 128 tokens: 10.3660 t/s
** Position 62848 + 128 tokens: 10.3802 t/s
** Position 62976 + 128 tokens: 10.3791 t/s
** Position 63104 + 128 tokens: 10.3777 t/s
** Position 63232 + 128 tokens: 10.3766 t/s
** Position 63360 + 128 tokens: 10.2966 t/s
** Position 63488 + 128 tokens: 10.3049 t/s
** Position 63616 + 128 tokens: 10.3056 t/s
** Position 63744 + 128 tokens: 10.3059 t/s
** Position 63872 + 128 tokens: 10.3031 t/s
** Position 64000 + 128 tokens: 10.2239 t/s
** Position 64128 + 128 tokens: 10.2364 t/s
** Position 64256 + 128 tokens: 10.2369 t/s
** Position 64384 + 128 tokens: 10.2378 t/s
** Position 64512 + 128 tokens: 10.2319 t/s
** Position 64640 + 128 tokens: 10.1510 t/s
** Position 64768 + 128 tokens: 10.1655 t/s
** Position 64896 + 128 tokens: 10.1650 t/s
** Position 65024 + 128 tokens: 10.1623 t/s
** Position 65152 + 128 tokens: 10.1670 t/s
** Position 65280 + 128 tokens: 10.0860 t/s
** Position 65408 + 128 tokens: 10.0987 t/s
** Position 65536 + 128 tokens: 10.0917 t/s
** Position 65664 + 128 tokens: 10.0929 t/s
** Position 65792 + 128 tokens: 10.0905 t/s
** Position 65920 + 128 tokens: 10.0159 t/s
** Position 66048 + 128 tokens: 10.0270 t/s
** Position 66176 + 128 tokens: 10.0276 t/s
** Position 66304 + 128 tokens: 10.0238 t/s
** Position 66432 + 128 tokens: 10.0213 t/s
** Position 66560 + 128 tokens: 9.9451 t/s
** Position 66688 + 128 tokens: 9.9580 t/s
** Position 66816 + 128 tokens: 9.9599 t/s
** Position 66944 + 128 tokens: 9.9588 t/s
** Position 67072 + 128 tokens: 9.9528 t/s
** Position 67200 + 128 tokens: 9.8819 t/s
** Position 67328 + 128 tokens: 9.8933 t/s
** Position 67456 + 128 tokens: 9.8922 t/s
** Position 67584 + 128 tokens: 9.8931 t/s
** Position 67712 + 128 tokens: 9.8901 t/s
** Position 67840 + 128 tokens: 9.8196 t/s
** Position 67968 + 128 tokens: 9.8320 t/s
** Position 68096 + 128 tokens: 9.8280 t/s
** Position 68224 + 128 tokens: 9.8316 t/s
** Position 68352 + 128 tokens: 9.8270 t/s
** Position 68480 + 128 tokens: 9.7566 t/s
** Position 68608 + 128 tokens: 9.7690 t/s
** Position 68736 + 128 tokens: 9.7716 t/s
** Position 68864 + 128 tokens: 9.7679 t/s
** Position 68992 + 128 tokens: 9.7621 t/s
** Position 69120 + 128 tokens: 9.6945 t/s
** Position 69248 + 128 tokens: 9.7053 t/s
** Position 69376 + 128 tokens: 9.7073 t/s
** Position 69504 + 128 tokens: 9.7070 t/s
** Position 69632 + 128 tokens: 9.7019 t/s
** Position 69760 + 128 tokens: 9.6328 t/s
** Position 69888 + 128 tokens: 9.6361 t/s
** Position 70016 + 128 tokens: 9.6442 t/s
** Position 70144 + 128 tokens: 9.6384 t/s
** Position 70272 + 128 tokens: 9.6394 t/s
** Position 70400 + 128 tokens: 9.5683 t/s
** Position 70528 + 128 tokens: 9.5834 t/s
** Position 70656 + 128 tokens: 9.5810 t/s
** Position 70784 + 128 tokens: 9.5794 t/s
** Position 70912 + 128 tokens: 9.5767 t/s
** Position 71040 + 128 tokens: 9.5048 t/s
** Position 71168 + 128 tokens: 9.5178 t/s
** Position 71296 + 128 tokens: 9.5161 t/s
** Position 71424 + 128 tokens: 9.5156 t/s
** Position 71552 + 128 tokens: 9.5160 t/s
** Position 71680 + 128 tokens: 9.4482 t/s
** Position 71808 + 128 tokens: 9.4575 t/s
** Position 71936 + 128 tokens: 9.4612 t/s
** Position 72064 + 128 tokens: 9.4572 t/s
** Position 72192 + 128 tokens: 9.4555 t/s
** Position 72320 + 128 tokens: 9.3844 t/s
** Position 72448 + 128 tokens: 9.4011 t/s
** Position 72576 + 128 tokens: 9.3976 t/s
** Position 72704 + 128 tokens: 9.3963 t/s
** Position 72832 + 128 tokens: 9.3965 t/s
** Position 72960 + 128 tokens: 9.3273 t/s
** Position 73088 + 128 tokens: 9.3406 t/s
** Position 73216 + 128 tokens: 9.3400 t/s
** Position 73344 + 128 tokens: 9.3406 t/s
** Position 73472 + 128 tokens: 9.3388 t/s
** Position 73600 + 128 tokens: 9.2740 t/s
** Position 73728 + 128 tokens: 9.2876 t/s
** Position 73856 + 128 tokens: 9.2891 t/s
** Position 73984 + 128 tokens: 9.2854 t/s
** Position 74112 + 128 tokens: 9.2841 t/s
** Position 74240 + 128 tokens: 9.2185 t/s
** Position 74368 + 128 tokens: 9.2325 t/s
** Position 74496 + 128 tokens: 9.2306 t/s
** Position 74624 + 128 tokens: 9.2292 t/s
** Position 74752 + 128 tokens: 9.2202 t/s
** Position 74880 + 128 tokens: 9.1596 t/s
** Position 75008 + 128 tokens: 9.1755 t/s
** Position 75136 + 128 tokens: 9.1762 t/s
** Position 75264 + 128 tokens: 9.1732 t/s
** Position 75392 + 128 tokens: 9.1738 t/s
** Position 75520 + 128 tokens: 9.1046 t/s
** Position 75648 + 128 tokens: 9.1204 t/s
** Position 75776 + 128 tokens: 9.1209 t/s
** Position 75904 + 128 tokens: 9.1185 t/s
** Position 76032 + 128 tokens: 9.1170 t/s
** Position 76160 + 128 tokens: 9.0448 t/s
** Position 76288 + 128 tokens: 9.0643 t/s
** Position 76416 + 128 tokens: 9.0645 t/s
** Position 76544 + 128 tokens: 9.0639 t/s
** Position 76672 + 128 tokens: 9.0601 t/s
** Position 76800 + 128 tokens: 8.9861 t/s
** Position 76928 + 128 tokens: 9.0078 t/s
** Position 77056 + 128 tokens: 9.0128 t/s
** Position 77184 + 128 tokens: 9.0131 t/s
** Position 77312 + 128 tokens: 9.0091 t/s
** Position 77440 + 128 tokens: 8.9336 t/s
** Position 77568 + 128 tokens: 8.9553 t/s
** Position 77696 + 128 tokens: 8.9580 t/s
** Position 77824 + 128 tokens: 8.9577 t/s
** Position 77952 + 128 tokens: 8.9563 t/s
** Position 78080 + 128 tokens: 8.8848 t/s
** Position 78208 + 128 tokens: 8.9051 t/s
** Position 78336 + 128 tokens: 8.9083 t/s
** Position 78464 + 128 tokens: 8.9069 t/s
** Position 78592 + 128 tokens: 8.9039 t/s
** Position 78720 + 128 tokens: 8.8343 t/s
** Position 78848 + 128 tokens: 8.8552 t/s
** Position 78976 + 128 tokens: 8.8561 t/s
** Position 79104 + 128 tokens: 8.8567 t/s
** Position 79232 + 128 tokens: 8.8570 t/s
** Position 79360 + 128 tokens: 8.7921 t/s
** Position 79488 + 128 tokens: 8.8082 t/s
** Position 79616 + 128 tokens: 8.8075 t/s
** Position 79744 + 128 tokens: 8.8063 t/s
** Position 79872 + 128 tokens: 8.8038 t/s
** Position 80000 + 128 tokens: 8.7332 t/s
** Position 80128 + 128 tokens: 8.7532 t/s
** Position 80256 + 128 tokens: 8.7550 t/s
** Position 80384 + 128 tokens: 8.7544 t/s
** Position 80512 + 128 tokens: 8.7506 t/s
** Position 80640 + 128 tokens: 8.6876 t/s
** Position 80768 + 128 tokens: 8.7069 t/s
** Position 80896 + 128 tokens: 8.7074 t/s
** Position 81024 + 128 tokens: 8.7034 t/s
** Position 81152 + 128 tokens: 8.7058 t/s
** Position 81280 + 128 tokens: 8.6300 t/s
** Position 81408 + 128 tokens: 8.6507 t/s
** Position 81536 + 128 tokens: 8.6537 t/s
** Position 81664 + 128 tokens: 8.6494 t/s
** Position 81792 + 128 tokens: 8.6523 t/s
** Position 81920 + 128 tokens: 8.5730 t/s
** Position 82048 + 128 tokens: 8.5908 t/s
** Position 82176 + 128 tokens: 8.5948 t/s
** Position 82304 + 128 tokens: 8.5886 t/s
** Position 82432 + 128 tokens: 8.5977 t/s
** Position 82560 + 128 tokens: 8.5230 t/s
** Position 82688 + 128 tokens: 8.5428 t/s
** Position 82816 + 128 tokens: 8.5413 t/s
** Position 82944 + 128 tokens: 8.5365 t/s
** Position 83072 + 128 tokens: 8.5387 t/s
** Position 83200 + 128 tokens: 8.4746 t/s
** Position 83328 + 128 tokens: 8.4919 t/s
** Position 83456 + 128 tokens: 8.4912 t/s
** Position 83584 + 128 tokens: 8.4978 t/s
** Position 83712 + 128 tokens: 8.4981 t/s
** Position 83840 + 128 tokens: 8.4273 t/s
** Position 83968 + 128 tokens: 8.4429 t/s
** Position 84096 + 128 tokens: 8.4464 t/s
** Position 84224 + 128 tokens: 8.4431 t/s
** Position 84352 + 128 tokens: 8.4455 t/s
** Position 84480 + 128 tokens: 8.3814 t/s
** Position 84608 + 128 tokens: 8.3935 t/s
** Position 84736 + 128 tokens: 8.3964 t/s
** Position 84864 + 128 tokens: 8.4027 t/s
** Position 84992 + 128 tokens: 8.4047 t/s
** Position 85120 + 128 tokens: 8.3360 t/s
** Position 85248 + 128 tokens: 8.3569 t/s
** Position 85376 + 128 tokens: 8.3562 t/s
** Position 85504 + 128 tokens: 8.3567 t/s
** Position 85632 + 128 tokens: 8.3617 t/s
** Position 85760 + 128 tokens: 8.2886 t/s
** Position 85888 + 128 tokens: 8.3050 t/s
** Position 86016 + 128 tokens: 8.3074 t/s
** Position 86144 + 128 tokens: 8.3043 t/s
** Position 86272 + 128 tokens: 8.3093 t/s
** Position 86400 + 128 tokens: 8.2427 t/s
** Position 86528 + 128 tokens: 8.2632 t/s
** Position 86656 + 128 tokens: 8.2623 t/s
** Position 86784 + 128 tokens: 8.2645 t/s
** Position 86912 + 128 tokens: 8.2703 t/s
** Position 87040 + 128 tokens: 8.1931 t/s
** Position 87168 + 128 tokens: 8.2145 t/s
** Position 87296 + 128 tokens: 8.2148 t/s
** Position 87424 + 128 tokens: 8.2130 t/s
** Position 87552 + 128 tokens: 8.2147 t/s
** Position 87680 + 128 tokens: 8.1500 t/s
** Position 87808 + 128 tokens: 8.1711 t/s
** Position 87936 + 128 tokens: 8.1781 t/s
** Position 88064 + 128 tokens: 8.1783 t/s
** Position 88192 + 128 tokens: 8.1727 t/s
** Position 88320 + 128 tokens: 8.1149 t/s
** Position 88448 + 128 tokens: 8.1296 t/s
** Position 88576 + 128 tokens: 8.1292 t/s
** Position 88704 + 128 tokens: 8.1314 t/s
** Position 88832 + 128 tokens: 8.1286 t/s
** Position 88960 + 128 tokens: 8.0652 t/s
** Position 89088 + 128 tokens: 8.0808 t/s
** Position 89216 + 128 tokens: 8.0850 t/s
** Position 89344 + 128 tokens: 8.0805 t/s
** Position 89472 + 128 tokens: 8.0794 t/s
** Position 89600 + 128 tokens: 8.0201 t/s
** Position 89728 + 128 tokens: 8.0345 t/s
** Position 89856 + 128 tokens: 8.0404 t/s
** Position 89984 + 128 tokens: 8.0390 t/s
** Position 90112 + 128 tokens: 8.0446 t/s
** Position 90240 + 128 tokens: 7.9764 t/s
** Position 90368 + 128 tokens: 7.9908 t/s
** Position 90496 + 128 tokens: 7.9918 t/s
** Position 90624 + 128 tokens: 7.9859 t/s
** Position 90752 + 128 tokens: 7.9945 t/s
** Position 90880 + 128 tokens: 7.9304 t/s
** Position 91008 + 128 tokens: 7.9521 t/s
** Position 91136 + 128 tokens: 7.9525 t/s
** Position 91264 + 128 tokens: 7.9551 t/s
** Position 91392 + 128 tokens: 7.9582 t/s
** Position 91520 + 128 tokens: 7.8928 t/s
** Position 91648 + 128 tokens: 7.9120 t/s
** Position 91776 + 128 tokens: 7.9180 t/s
** Position 91904 + 128 tokens: 7.9150 t/s
** Position 92032 + 128 tokens: 7.9142 t/s
** Position 92160 + 128 tokens: 7.8561 t/s
** Position 92288 + 128 tokens: 7.8736 t/s
** Position 92416 + 128 tokens: 7.8805 t/s
** Position 92544 + 128 tokens: 7.8756 t/s
** Position 92672 + 128 tokens: 7.8807 t/s
** Position 92800 + 128 tokens: 7.8086 t/s
** Position 92928 + 128 tokens: 7.8304 t/s
** Position 93056 + 128 tokens: 7.8331 t/s
** Position 93184 + 128 tokens: 7.8321 t/s
** Position 93312 + 128 tokens: 7.8348 t/s
** Position 93440 + 128 tokens: 7.7684 t/s
** Position 93568 + 128 tokens: 7.7803 t/s
** Position 93696 + 128 tokens: 7.7820 t/s
** Position 93824 + 128 tokens: 7.7840 t/s
** Position 93952 + 128 tokens: 7.7812 t/s
** Position 94080 + 128 tokens: 7.7266 t/s
** Position 94208 + 128 tokens: 7.7482 t/s
** Position 94336 + 128 tokens: 7.7471 t/s
** Position 94464 + 128 tokens: 7.7447 t/s
** Position 94592 + 128 tokens: 7.7475 t/s
** Position 94720 + 128 tokens: 7.6862 t/s
** Position 94848 + 128 tokens: 7.7010 t/s
** Position 94976 + 128 tokens: 7.7038 t/s
** Position 95104 + 128 tokens: 7.7063 t/s
** Position 95232 + 128 tokens: 7.7021 t/s
** Position 95360 + 128 tokens: 7.6472 t/s
** Position 95488 + 128 tokens: 7.6678 t/s
** Position 95616 + 128 tokens: 7.6730 t/s
** Position 95744 + 128 tokens: 7.6733 t/s
** Position 95872 + 128 tokens: 7.6753 t/s
** Position 96000 + 128 tokens: 7.6124 t/s
** Position 96128 + 128 tokens: 7.6270 t/s
** Position 96256 + 128 tokens: 7.6286 t/s
** Position 96384 + 128 tokens: 7.6316 t/s
** Position 96512 + 128 tokens: 7.6326 t/s
** Position 96640 + 128 tokens: 7.5752 t/s
** Position 96768 + 128 tokens: 7.5930 t/s
** Position 96896 + 128 tokens: 7.5941 t/s
** Position 97024 + 128 tokens: 7.5894 t/s
** Position 97152 + 128 tokens: 7.5972 t/s
** Position 97280 + 128 tokens: 7.5350 t/s
** Position 97408 + 128 tokens: 7.5490 t/s
** Position 97536 + 128 tokens: 7.5543 t/s
** Position 97664 + 128 tokens: 7.5566 t/s
** Position 97792 + 128 tokens: 7.5556 t/s
** Position 97920 + 128 tokens: 7.4965 t/s
** Position 98048 + 128 tokens: 7.5126 t/s
** Position 98176 + 128 tokens: 7.5143 t/s
(venv) (extras) ➜ exllamav2 git:(test) ✗
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment