Skip to content

Instantly share code, notes, and snippets.

@Ralakus

Ralakus/bias.from_scratch.parallel.log Secret

Last active May 8, 2024 16:49
Show Gist options
  • Save Ralakus/5747fb7550b41b7a3f22f5de4b4414d0 to your computer and use it in GitHub Desktop.
Save Ralakus/5747fb7550b41b7a3f22f5de4b4414d0 to your computer and use it in GitHub Desktop.
Download ZIP
llama.cpp parallel inferrencing producing the unknown token (token id 0) on finetuned mixtral
Raw
bias.from_scratch.parallel.log
[1715185979] Log start
[1715185979] Cmd: ../llama.cpp/build/bin/parallel -m Mistral-Custom/ggml-model-q4_k_m.gguf -ngl 999 -c 1024 --parallel 4 --sequences 4 --seed 25519 -n 512 --logit-bias 0-999999999999
[1715185980] warming up the model with an empty run
[1715185980] main: Simulating parallel requests from clients:
[1715185980] main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
[1715185980]
[1715185980] main: Evaluating the system prompt ...
[1715185980]
[1715185980] Processing requests ...
[1715185980] main: clearing the KV cache
[1715185980] Client 0, seq 0, started decoding ...
[1715185980] Client 1, seq 1, started decoding ...
[1715185980] Client 2, seq 2, started decoding ...
[1715185980] Client 3, seq 3, started decoding ...
[1715185980] main : decoded batch of 59 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] main : decoded batch of 4 tokens
[1715185980] Client 1, seq 1/ 4, prompt 15 t, response 31 t, time 0.52 s, speed 88.90 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: The best way to cook a steak depends on your preference for doneness. Here's a simple method for a pan-seared steak:
[1715185980] main : decoded batch of 3 tokens
[1715185980] main : decoded batch of 3 tokens
[1715185980] main : decoded batch of 3 tokens
[1715185980] main : decoded batch of 3 tokens
[1715185980] main : decoded batch of 3 tokens
[1715185980] main : decoded batch of 3 tokens
[1715185980] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] main : decoded batch of 3 tokens
[1715185981] Client 3, seq 3/ 4, prompt 22 t, response 81 t, time 1.09 s, speed 94.24 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: Yes, I can certainly help explain the Special Theory of Relativity. The Special Theory of Relativity is a physical theory proposed by Albert Einstein in 1905. It is based on two main postulates: the laws of physics are the same in all inertial frames of reference, and the speed of light in a vacuum is the same for all observers, regardless of their motion.
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] main : decoded batch of 2 tokens
[1715185981] Client 0, seq 0/ 4, prompt 11 t, response 107 t, time 1.36 s, speed 86.51 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical one that has puzzled humans for centuries. There are many different perspectives on this question, and there is no definitive answer. Some people believe that the meaning of life is to seek happiness, knowledge, or personal growth. Others believe that it is to serve a higher power or to contribute to the greater good of humanity. Ultimately, the meaning of life is a personal question, and each person must find their own answer based on their beliefs, values, and experiences.
[1715185981] Client 2, seq 2/ 4, prompt 11 t, response 107 t, time 1.36 s, speed 86.51 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical one that has puzzled humans for centuries. There are many different perspectives on this question, and there is no definitive answer. Some people believe that the meaning of life is to seek happiness, knowledge, or personal growth. Others believe that it is to serve a higher power or to contribute to the greater good of humanity. Ultimately, the meaning of life is a personal question, and each person must find their own answer based on their beliefs, values, and experiences.
[1715185981] main: clearing the KV cache
[1715185981]
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
[1715185981] External prompt file: used built-in defaults
[1715185981] Model and path used: Mistral-Custom/ggml-model-q4_k_m.gguf
[1715185981] Total prompt tokens: 59, speed: 41.91 t/s
[1715185981] Total gen tokens: 326, speed: 231.58 t/s
[1715185981] Total speed (AVG): speed: 273.49 t/s
[1715185981] Cache misses: 0
[1715185981]
Raw
bias.from_scratch.parallel.out
Log start
llama_model_loader: loaded meta data with 25 key-value pairs and 291 tensors from Mistral-Custom/ggml-model-q4_k_m.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = .
llama_model_loader: - kv 2: llama.vocab_size u32 = 32001
llama_model_loader: - kv 3: llama.context_length u32 = 32768
llama_model_loader: - kv 4: llama.embedding_length u32 = 4096
llama_model_loader: - kv 5: llama.block_count u32 = 32
llama_model_loader: - kv 6: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 7: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 8: llama.attention.head_count u32 = 32
llama_model_loader: - kv 9: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 10: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 11: llama.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 12: general.file_type u32 = 15
llama_model_loader: - kv 13: tokenizer.ggml.model str = llama
llama_model_loader: - kv 14: tokenizer.ggml.tokens arr[str,32001] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv 15: tokenizer.ggml.scores arr[f32,32001] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,32001] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 17: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 18: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 19: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 20: tokenizer.ggml.padding_token_id u32 = 32000
llama_model_loader: - kv 21: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 22: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 23: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
llama_model_loader: - kv 24: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type q4_K: 193 tensors
llama_model_loader: - type q6_K: 33 tensors
llm_load_vocab: special tokens definition check successful ( 260/32001 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32001
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 8B
llm_load_print_meta: model ftype = Q4_K - Medium
llm_load_print_meta: model params = 7.24 B
llm_load_print_meta: model size = 4.07 GiB (4.83 BPW)
llm_load_print_meta: general.name = .
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: PAD token = 32000 '[PAD]'
llm_load_print_meta: LF token = 13 '<0x0A>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size = 0.30 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors: CPU buffer size = 70.31 MiB
llm_load_tensors: CUDA0 buffer size = 4095.06 MiB
................................................................................................
llama_new_context_with_model: n_ctx = 1024
llama_new_context_with_model: n_batch = 1024
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA0 KV buffer size = 128.00 MiB
llama_new_context_with_model: KV self size = 128.00 MiB, K (f16): 64.00 MiB, V (f16): 64.00 MiB
llama_new_context_with_model: CUDA_Host output buffer size = 0.61 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 98.00 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 10.01 MiB
llama_new_context_with_model: graph nodes = 1030
llama_new_context_with_model: graph splits = 2
main: Simulating parallel requests from clients:
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
main: Evaluating the system prompt ...
Processing requests ...
main: clearing the KV cache
Client 0, seq 0, started decoding ...
Client 1, seq 1, started decoding ...
Client 2, seq 2, started decoding ...
Client 3, seq 3, started decoding ...
Client 1, seq 1/ 4, prompt 15 t, response 31 t, time 0.52 s, speed 88.90 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: The best way to cook a steak depends on your preference for doneness. Here's a simple method for a pan-seared steak:
Client 3, seq 3/ 4, prompt 22 t, response 81 t, time 1.09 s, speed 94.24 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: Yes, I can certainly help explain the Special Theory of Relativity. The Special Theory of Relativity is a physical theory proposed by Albert Einstein in 1905. It is based on two main postulates: the laws of physics are the same in all inertial frames of reference, and the speed of light in a vacuum is the same for all observers, regardless of their motion.
Client 0, seq 0/ 4, prompt 11 t, response 107 t, time 1.36 s, speed 86.51 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical one that has puzzled humans for centuries. There are many different perspectives on this question, and there is no definitive answer. Some people believe that the meaning of life is to seek happiness, knowledge, or personal growth. Others believe that it is to serve a higher power or to contribute to the greater good of humanity. Ultimately, the meaning of life is a personal question, and each person must find their own answer based on their beliefs, values, and experiences.
Client 2, seq 2/ 4, prompt 11 t, response 107 t, time 1.36 s, speed 86.51 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical one that has puzzled humans for centuries. There are many different perspectives on this question, and there is no definitive answer. Some people believe that the meaning of life is to seek happiness, knowledge, or personal growth. Others believe that it is to serve a higher power or to contribute to the greater good of humanity. Ultimately, the meaning of life is a personal question, and each person must find their own answer based on their beliefs, values, and experiences.
main: clearing the KV cache
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
External prompt file: used built-in defaults
Model and path used: Mistral-Custom/ggml-model-q4_k_m.gguf
Total prompt tokens: 59, speed: 41.91 t/s
Total gen tokens: 326, speed: 231.58 t/s
Total speed (AVG): speed: 273.49 t/s
Cache misses: 0
llama_print_timings: load time = 501.22 ms
llama_print_timings: sample time = 6.19 ms / 330 runs ( 0.02 ms per token, 53303.18 tokens per second)
llama_print_timings: prompt eval time = 1354.07 ms / 684 tokens ( 1.98 ms per token, 505.14 tokens per second)
llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: total time = 1407.94 ms / 685 tokens
No new questions so proceed with build-in defaults.
run parameters as at 2024-05-08 12:33:01
Raw
cpu.parallel.log
[1714506924] Log start
[1714506924] Cmd: llama.cpp/build/bin/parallel -m Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf -c 32768 --parallel 4 --sequences 4 --seed 25519 -t 12 --numa numactl
[1714506924] warming up the model with an empty run
[1714506925] main: Simulating parallel requests from clients:
[1714506925] main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
[1714506925]
[1714506925] main: Evaluating the system prompt ...
[1714506927]
[1714506927] Processing requests ...
[1714506927] main: clearing the KV cache
[1714506927] Client 0, seq 0, started decoding ...
[1714506927] Client 1, seq 1, started decoding ...
[1714506927] Client 2, seq 2, started decoding ...
[1714506927] Client 3, seq 3, started decoding ...
[1714506929] main : decoded batch of 59 tokens
[1714506929] main : decoded batch of 4 tokens
[1714506929] main : decoded batch of 4 tokens
[1714506930] main : decoded batch of 4 tokens
[1714506930] main : decoded batch of 4 tokens
[1714506931] main : decoded batch of 4 tokens
[1714506931] main : decoded batch of 4 tokens
[1714506932] main : decoded batch of 4 tokens
[1714506932] main : decoded batch of 4 tokens
[1714506933] main : decoded batch of 4 tokens
[1714506933] main : decoded batch of 4 tokens
[1714506934] main : decoded batch of 4 tokens
[1714506934] main : decoded batch of 4 tokens
[1714506935] main : decoded batch of 4 tokens
[1714506935] main : decoded batch of 4 tokens
[1714506935] Client 1, seq 1/ 4, prompt 15 t, response 14 t, time 8.32 s, speed 3.49 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: The best way to cook a steak is to follow these steps:
[1714506936] main : decoded batch of 3 tokens
[1714506936] main : decoded batch of 3 tokens
[1714506937] main : decoded batch of 3 tokens
[1714506937] main : decoded batch of 3 tokens
[1714506937] main : decoded batch of 3 tokens
[1714506938] main : decoded batch of 3 tokens
[1714506938] main : decoded batch of 3 tokens
[1714506938] main : decoded batch of 3 tokens
[1714506939] main : decoded batch of 3 tokens
[1714506939] main : decoded batch of 3 tokens
[1714506939] main : decoded batch of 3 tokens
[1714506940] main : decoded batch of 3 tokens
[1714506940] main : decoded batch of 3 tokens
[1714506941] main : decoded batch of 3 tokens
[1714506941] main : decoded batch of 3 tokens
[1714506941] main : decoded batch of 3 tokens
[1714506942] main : decoded batch of 3 tokens
[1714506942] main : decoded batch of 3 tokens
[1714506942] main : decoded batch of 3 tokens
[1714506943] main : decoded batch of 3 tokens
[1714506943] main : decoded batch of 3 tokens
[1714506944] main : decoded batch of 3 tokens
[1714506944] main : decoded batch of 3 tokens
[1714506944] main : decoded batch of 3 tokens
[1714506945] main : decoded batch of 3 tokens
[1714506945] main : decoded batch of 3 tokens
[1714506946] main : decoded batch of 3 tokens
[1714506946] main : decoded batch of 3 tokens
[1714506946] main : decoded batch of 3 tokens
[1714506947] main : decoded batch of 3 tokens
[1714506947] main : decoded batch of 3 tokens
[1714506947] main : decoded batch of 3 tokens
[1714506948] main : decoded batch of 3 tokens
[1714506948] main : decoded batch of 3 tokens
[1714506949] main : decoded batch of 3 tokens
[1714506949] main : decoded batch of 3 tokens
[1714506949] main : decoded batch of 3 tokens
[1714506950] main : decoded batch of 3 tokens
[1714506950] main : decoded batch of 3 tokens
[1714506951] main : decoded batch of 3 tokens
[1714506951] main : decoded batch of 3 tokens
[1714506951] main : decoded batch of 3 tokens
[1714506952] main : decoded batch of 3 tokens
[1714506952] main : decoded batch of 3 tokens
[1714506952] main : decoded batch of 3 tokens
[1714506953] main : decoded batch of 3 tokens
[1714506953] main : decoded batch of 3 tokens
[1714506954] main : decoded batch of 3 tokens
[1714506954] main : decoded batch of 3 tokens
[1714506955] main : decoded batch of 3 tokens
[1714506955] main : decoded batch of 3 tokens
[1714506956] main : decoded batch of 3 tokens
[1714506956] main : decoded batch of 3 tokens
[1714506957] main : decoded batch of 3 tokens
[1714506957] main : decoded batch of 3 tokens
[1714506958] main : decoded batch of 3 tokens
[1714506958] main : decoded batch of 3 tokens
[1714506959] main : decoded batch of 3 tokens
[1714506959] main : decoded batch of 3 tokens
[1714506960] main : decoded batch of 3 tokens
[1714506960] main : decoded batch of 3 tokens
[1714506960] main : decoded batch of 3 tokens
[1714506961] main : decoded batch of 3 tokens
[1714506961] main : decoded batch of 3 tokens
[1714506962] main : decoded batch of 3 tokens
[1714506962] Client 0, seq 0/ 4, prompt 11 t, response 79 t, time 34.95 s, speed 2.58 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has been debated throughout history. Some people believe that the meaning of life is to seek happiness, fulfillment, and personal growth. Others believe that the meaning of life is to serve a higher purpose, such as a god or a moral principle. Ultimately, the meaning of life is a personal belief that each individual must determine for themselves.</s>
[1714506962] main : decoded batch of 2 tokens
[1714506963] main : decoded batch of 2 tokens
[1714506963] main : decoded batch of 2 tokens
[1714506964] main : decoded batch of 2 tokens
[1714506964] main : decoded batch of 2 tokens
[1714506964] main : decoded batch of 2 tokens
[1714506965] main : decoded batch of 2 tokens
[1714506965] main : decoded batch of 2 tokens
[1714506965] main : decoded batch of 2 tokens
[1714506966] main : decoded batch of 2 tokens
[1714506966] main : decoded batch of 2 tokens
[1714506967] main : decoded batch of 2 tokens
[1714506967] main : decoded batch of 2 tokens
[1714506967] main : decoded batch of 2 tokens
[1714506968] main : decoded batch of 2 tokens
[1714506968] main : decoded batch of 2 tokens
[1714506968] main : decoded batch of 2 tokens
[1714506969] main : decoded batch of 2 tokens
[1714506969] main : decoded batch of 2 tokens
[1714506969] main : decoded batch of 2 tokens
[1714506970] main : decoded batch of 2 tokens
[1714506970] main : decoded batch of 2 tokens
[1714506971] main : decoded batch of 2 tokens
[1714506971] main : decoded batch of 2 tokens
[1714506971] main : decoded batch of 2 tokens
[1714506972] main : decoded batch of 2 tokens
[1714506972] Client 2, seq 2/ 4, prompt 11 t, response 105 t, time 44.61 s, speed 2.60 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has been debated throughout history. Some people believe that the meaning of life is to seek happiness, fulfillment, and personal growth. Others believe that the meaning of life is to serve a higher purpose, such as a god or a moral principle. Still others believe that life has no inherent meaning, and that it is up to each individual to create their own purpose. Ultimately, the meaning of life is a subjective question that each person must answer for themselves.
[1714506972] Client 3, seq 3/ 4, prompt 22 t, response 105 t, time 44.61 s, speed 2.85 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: Yes, I can explain the Special Theory of Relativity. The Special Theory of Relativity was proposed by Albert Einstein in 1905. It is a theory of physics that describes the behavior of objects that are moving at constant speeds in a straight line. The theory is based on two postulates: (1) the laws of physics are the same in all inertial frames of reference, and (2) the speed of light in a vacuum is the same for all observers, regardless of their motion.
[1714506972] main: clearing the KV cache
[1714506972]
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
[1714506972] External prompt file: used built-in defaults
[1714506972] Model and path used: Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf
[1714506972] Total prompt tokens: 59, speed: 1.26 t/s
[1714506972] Total gen tokens: 303, speed: 6.49 t/s
[1714506972] Total speed (AVG): speed: 7.76 t/s
[1714506972] Cache misses: 0
[1714506972]
Raw
cpu.parallel.out
Log start
llama_model_loader: loaded meta data with 27 key-value pairs and 323 tensors from Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = .
llama_model_loader: - kv 2: llama.vocab_size u32 = 32001
llama_model_loader: - kv 3: llama.context_length u32 = 32768
llama_model_loader: - kv 4: llama.embedding_length u32 = 4096
llama_model_loader: - kv 5: llama.block_count u32 = 32
llama_model_loader: - kv 6: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 7: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 8: llama.attention.head_count u32 = 32
llama_model_loader: - kv 9: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 10: llama.expert_count u32 = 8
llama_model_loader: - kv 11: llama.expert_used_count u32 = 2
llama_model_loader: - kv 12: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 13: llama.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 14: general.file_type u32 = 15
llama_model_loader: - kv 15: tokenizer.ggml.model str = llama
llama_model_loader: - kv 16: tokenizer.ggml.tokens arr[str,32001] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv 17: tokenizer.ggml.scores arr[f32,32001] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,32001] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 22: tokenizer.ggml.padding_token_id u32 = 32000
llama_model_loader: - kv 23: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 24: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 25: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
llama_model_loader: - kv 26: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type f16: 32 tensors
llama_model_loader: - type q8_0: 64 tensors
llama_model_loader: - type q4_K: 113 tensors
llama_model_loader: - type q5_K: 32 tensors
llama_model_loader: - type q6_K: 17 tensors
llm_load_vocab: special tokens definition check successful ( 260/32001 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32001
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 8
llm_load_print_meta: n_expert_used = 2
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 8x7B
llm_load_print_meta: model ftype = Q4_K - Medium
llm_load_print_meta: model params = 46.70 B
llm_load_print_meta: model size = 26.49 GiB (4.87 BPW)
llm_load_print_meta: general.name = .
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: PAD token = 32000 '[PAD]'
llm_load_print_meta: LF token = 13 '<0x0A>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size = 0.16 MiB
llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/33 layers to GPU
llm_load_tensors: CPU buffer size = 27127.87 MiB
.............................................................................................
llama_new_context_with_model: n_ctx = 32768
llama_new_context_with_model: n_batch = 2048
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA_Host KV buffer size = 4096.00 MiB
llama_new_context_with_model: KV self size = 4096.00 MiB, K (f16): 2048.00 MiB, V (f16): 2048.00 MiB
llama_new_context_with_model: CUDA_Host output buffer size = 0.61 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 2265.00 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 72.01 MiB
llama_new_context_with_model: graph nodes = 1510
llama_new_context_with_model: graph splits = 388
main: Simulating parallel requests from clients:
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
main: Evaluating the system prompt ...
Processing requests ...
main: clearing the KV cache
Client 0, seq 0, started decoding ...
Client 1, seq 1, started decoding ...
Client 2, seq 2, started decoding ...
Client 3, seq 3, started decoding ...
Client 1, seq 1/ 4, prompt 15 t, response 14 t, time 8.32 s, speed 3.49 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: The best way to cook a steak is to follow these steps:
Client 0, seq 0/ 4, prompt 11 t, response 79 t, time 34.95 s, speed 2.58 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has been debated throughout history. Some people believe that the meaning of life is to seek happiness, fulfillment, and personal growth. Others believe that the meaning of life is to serve a higher purpose, such as a god or a moral principle. Ultimately, the meaning of life is a personal belief that each individual must determine for themselves.</s>
Client 2, seq 2/ 4, prompt 11 t, response 105 t, time 44.61 s, speed 2.60 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has been debated throughout history. Some people believe that the meaning of life is to seek happiness, fulfillment, and personal growth. Others believe that the meaning of life is to serve a higher purpose, such as a god or a moral principle. Still others believe that life has no inherent meaning, and that it is up to each individual to create their own purpose. Ultimately, the meaning of life is a subjective question that each person must answer for themselves.
Client 3, seq 3/ 4, prompt 22 t, response 105 t, time 44.61 s, speed 2.85 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: Yes, I can explain the Special Theory of Relativity. The Special Theory of Relativity was proposed by Albert Einstein in 1905. It is a theory of physics that describes the behavior of objects that are moving at constant speeds in a straight line. The theory is based on two postulates: (1) the laws of physics are the same in all inertial frames of reference, and (2) the speed of light in a vacuum is the same for all observers, regardless of their motion.
main: clearing the KV cache
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
External prompt file: used built-in defaults
Model and path used: Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf
Total prompt tokens: 59, speed: 1.26 t/s
Total gen tokens: 303, speed: 6.49 t/s
Total speed (AVG): speed: 7.76 t/s
Cache misses: 0
llama_print_timings: load time = 1310.86 ms
llama_print_timings: sample time = 6.23 ms / 307 runs ( 0.02 ms per token, 49246.07 tokens per second)
llama_print_timings: prompt eval time = 46617.64 ms / 661 tokens ( 70.53 ms per token, 14.18 tokens per second)
llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: total time = 46673.27 ms / 662 tokens
No new questions so proceed with build-in defaults.
run parameters as at 2024-04-30 15:56:12
Raw
cpu.tasks.txt
# TASK 0
The Kessler syndrome, also known as the Kessler effect, is a scenario in which the density of objects in low Earth orbit (LEO) is high enough that collisions between objects could cause a cascade of additional collisions, creating a debris field that could make LEO unusable for future generations. Satellite constellations, such as SpaceX's Starlink, could potentially accelerate the Kessler syndrome in a few ways:
1. Increased number of satellites: Starlink plans to launch thousands of satellites into LEO, which will significantly increase the number of objects in this region of space. This will increase the likelihood of collisions between satellites, as well as between satellites and existing debris.
2. Lower altitude: Starlink satellites operate at an altitude of around 550 kilometers, which is lower than many other satellite systems. This means that the satellites (and any resulting debris) will have a shorter orbital lifetime, increasing the likelihood that they will re-enter Earth's atmosphere and burn up. However, it also means that any collisions that do occur will happen at higher velocities, which could result in more debris being generated.
3. Greater potential for collisions: The design of the Starlink constellation, with satellites in closely spaced orbital planes, could increase the potential for collisions. While the satellites are designed to maneuver to avoid collisions, there is still a risk that unforeseen circumstances could lead to a collision.
4. Difficulty in tracking debris: With thousands of satellites in orbit, tracking and predicting the movements of debris will become increasingly challenging. This could make it more difficult to avoid collisions, further increasing the risk of a cascading debris event.
It is important to note that the Kessler syndrome is still largely theoretical, and it is not clear how likely it is to occur. However, the deployment of large satellite constellations, such as Starlink, has raised concerns about the potential for accelerating the onset of the Kessler syndrome. SpaceX and other companies deploying satellite constellations have recognized these concerns and are taking steps to minimize the risk of collisions and the generation of debris. These steps include designing satellites to deorbit at the end of their lifespan, using active debris removal technologies, and implementing responsible space traffic management practices.
# TASK 1
The Kessler syndrome, also known as the Kessler effect, is a scenario in which the density of objects in low Earth orbit (LEO) is high enough that collisions between objects could cause a cascade of additional collisions, leading to an exponential increase in the number of debris and a corresponding decrease in the usefulness of LEO for satellite operations.
Satellite constellations, such as SpaceX's Starlink, which plan to deploy large numbers of satellites in LEO, have the potential to accelerate the onset of the Kessler syndrome. This is because the deployment of a large number of satellites in a relatively small volume of space increases the likelihood of collisions between satellites, either during the deployment phase or during the operational phase. Additionally, the low altitude of LEO means that debris created by collisions will quickly re-enter the atmosphere and burn up, but not before potentially damaging other operational satellites.
Furthermore, the constellations are planned to be deployed in a short period of time, which could lead to a rapid increase in the number of satellites in LEO, and therefore the risk of collisions. Additionally, the satellites in the constellations are designed to have a relatively short lifespan, which means that they will need to be replaced frequently, further increasing the number of satellites in LEO.
Another concern is that the satellites in the constellations are designed to be relatively small and lightweight, which makes them more difficult to track and increases the risk of collisions with other objects in LEO.
If the Kessler syndrome were to occur, it could have significant consequences for the use of LEO for satellite operations, including communications, navigation, and remote sensing. It could also make it more difficult and expensive to launch and operate satellites in LEO.
Therefore, it is important to carefully consider the potential impacts of satellite constellations on the space environment and to take steps to mitigate the risk of collisions and the creation of debris. This could include measures such as designing satellites to be more robust and able to maneuver to avoid collisions, as well as implementing regulations and guidelines to govern the deployment and operation of satellite constellations.
# TASK 2
The Kessler syndrome, also known as the Kessler effect, is a scenario in which the density of objects in low Earth orbit (LEO) is high enough that collisions between objects could cause a cascade of additional collisions, creating a debris field that could make LEO unusable for future generations. Satellite constellations, such as SpaceX's Starlink, could potentially accelerate the Kessler syndrome in a few ways:
1. Increased number of satellites: Starlink plans to launch thousands of satellites into LEO, which will significantly increase the number of objects in this region of space. This will increase the likelihood of collisions between satellites, as well as between satellites and existing debris.
2. Lower altitude: Starlink satellites operate at an altitude of around 550 kilometers, which is lower than many other satellite systems. This means that the satellites (and any resulting debris) will have a shorter orbital lifetime, increasing the chances that they will re-enter Earth's atmosphere and burn up. However, it also means that collisions at this altitude will produce more debris that will remain in LEO.
3. Greater potential for collisions: Due to the sheer number of satellites in the Starlink constellation, there is a greater potential for collisions between satellites, both within the constellation and with other satellites and debris. Additionally, the constellation's design includes frequent maneuvers to avoid collisions, which could increase the risk of accidents.
4. Difficulty in tracking and removing debris: With thousands of satellites in orbit, tracking and removing debris will become increasingly challenging. This is because the debris field will be more complex and dynamic, making it harder to predict the movement of individual objects.
While Starlink and other satellite constellations have the potential to bring numerous benefits, such as improved global internet coverage, it is crucial to consider the long-term consequences of their deployment. Measures to mitigate the risk of the Kessler syndrome, such as designing satellites to disintegrate upon re-entry, tracking and removing debris, and implementing responsible space traffic management practices, will be essential to ensure the sustainable use of LEO for future generations.
# TASK 3
The Kessler syndrome, also known as the Kessler effect, is a scenario in which the density of objects in low Earth orbit (LEO) is high enough that collisions between objects could cause a cascade of additional collisions, creating a debris field that could make LEO too dangerous for further space travel. Satellite constellations, such as SpaceX's Starlink, could potentially accelerate the Kessler syndrome in a few ways:
* Increased number of satellites: Starlink plans to launch thousands of satellites into LEO, which will significantly increase the number of objects in this region of space. This will increase the likelihood of collisions between satellites, as well as between satellites and existing debris.
* Lower altitude: Starlink satellites operate at an altitude of around 550 kilometers, which is lower than many other satellite systems. This means that the satellites (and any resulting debris) will have a shorter decay time and will therefore remain in LEO for a longer period of time, increasing the chances of collisions.
* Increased visibility: The Starlink satellites are designed to be visible from the ground, which means that they will be more likely to be hit by other satellites or debris. Additionally, the brightness of the satellites could make it more difficult for astronomers to observe other objects in the sky.
* End-of-life disposal: SpaceX has stated that they will deorbit the satellites at the end of their life, but there is still a risk that the satellites could explode or fragment during the deorbiting process, creating additional debris.
All these factors combined, satellite constellations such as Starlink, with their large number of satellites, low altitude, and high visibility, could pose a significant issue in the future by accelerating the Kessler syndrome. However, it is worth noting that the Kessler syndrome is still a theoretical scenario, and it is not clear how likely it is to occur. Additionally, SpaceX and other companies are working on ways to mitigate the risk of collisions and the creation of debris.
Raw
create-model.py
"""Writes the default Mistral Instuct model with added tokens."""
from typing import Final
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL: Final[str] = "mistralai/Mistral-7B-Instruct-v0.2"
OUTPUT_DIR: Final[str] = "Mistral-Custom"
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(MODEL)
tokenizer.add_special_tokens({"pad_token": "[PAD]"})
model.resize_token_embeddings(len(tokenizer))
tokenizer.save_pretrained(OUTPUT_DIR)
model.save_pretrained(OUTPUT_DIR)
Raw
cuda-mmq.parallel.log
[1714677938] Log start
[1714677938] Cmd: llama.cpp/build/bin/parallel -m Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf -ngl 999 -c 32768 --parallel 4 --sequences 4 --seed 25519
[1714677941] warming up the model with an empty run
[1714677941] main: Simulating parallel requests from clients:
[1714677941] main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
[1714677941]
[1714677941] main: Evaluating the system prompt ...
[1714677941]
[1714677941] Processing requests ...
[1714677941] main: clearing the KV cache
[1714677941] Client 0, seq 0, started decoding ...
[1714677941] Client 1, seq 1, started decoding ...
[1714677941] Client 2, seq 2, started decoding ...
[1714677941] Client 3, seq 3, started decoding ...
[1714677942] main : decoded batch of 59 tokens
[1714677942] main : decoded batch of 4 tokens
[1714677942] main : decoded batch of 4 tokens
[1714677942] main : decoded batch of 4 tokens
[1714677942] main : decoded batch of 4 tokens
[1714677942] main : decoded batch of 4 tokens
[1714677942] Client 3, seq 3/ 4, prompt 22 t, response 5 t, time 0.45 s, speed 59.68 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: Yes, I am▅
[1714677942] main : decoded batch of 3 tokens
[1714677942] main : decoded batch of 3 tokens
[1714677942] main : decoded batch of 3 tokens
[1714677942] main : decoded batch of 3 tokens
[1714677942] main : decoded batch of 3 tokens
[1714677942] main : decoded batch of 3 tokens
[1714677942] main : decoded batch of 3 tokens
[1714677942] main : decoded batch of 3 tokens
[1714677942] main : decoded batch of 3 tokens
[1714677942] main : decoded batch of 3 tokens
[1714677942] Client 2, seq 2/ 4, prompt 11 t, response 15 t, time 0.79 s, speed 32.84 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has▅▅▅a</s>
[1714677942] main : decoded batch of 2 tokens
[1714677942] main : decoded batch of 2 tokens
[1714677942] main : decoded batch of 2 tokens
[1714677942] main : decoded batch of 2 tokens
[1714677942] main : decoded batch of 2 tokens
[1714677942] main : decoded batch of 2 tokens
[1714677942] main : decoded batch of 2 tokens
[1714677942] main : decoded batch of 2 tokens
[1714677942] main : decoded batch of 2 tokens
[1714677942] main : decoded batch of 2 tokens
[1714677942] main : decoded batch of 2 tokens
[1714677942] main : decoded batch of 2 tokens
[1714677942] main : decoded batch of 2 tokens
[1714677943] main : decoded batch of 2 tokens
[1714677943] main : decoded batch of 2 tokens
[1714677943] main : decoded batch of 2 tokens
[1714677943] Client 1, seq 1/ 4, prompt 15 t, response 31 t, time 1.28 s, speed 35.97 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: The best way to▅ to cook a steak▅assistant is to▅assistant follow these▅▅▅▅▅▅▅▅▅ass▅
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] main : decoded batch of 1 tokens
[1714677943] Client 0, seq 0/ 4, prompt 11 t, response 79 t, time 2.10 s, speed 42.89 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has been debated throughout history. Some people believe that the meaning of life is to seek happiness, fulfillment, and personal growth. Others believe that the meaning of life is to serve a higher purpose, such as a god or a moral principle. Ultimately, the meaning of life is a personal belief that each individual must determine for themselves.</s>
[1714677943] main: clearing the KV cache
[1714677943]
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
[1714677943] External prompt file: used built-in defaults
[1714677943] Model and path used: Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf
[1714677943] Total prompt tokens: 59, speed: 23.56 t/s
[1714677943] Total gen tokens: 130, speed: 51.91 t/s
[1714677943] Total speed (AVG): speed: 75.47 t/s
[1714677943] Cache misses: 0
[1714677943]
Raw
cuda-mmq.parallel.out
Log start
llama_model_loader: loaded meta data with 27 key-value pairs and 323 tensors from Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = .
llama_model_loader: - kv 2: llama.vocab_size u32 = 32001
llama_model_loader: - kv 3: llama.context_length u32 = 32768
llama_model_loader: - kv 4: llama.embedding_length u32 = 4096
llama_model_loader: - kv 5: llama.block_count u32 = 32
llama_model_loader: - kv 6: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 7: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 8: llama.attention.head_count u32 = 32
llama_model_loader: - kv 9: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 10: llama.expert_count u32 = 8
llama_model_loader: - kv 11: llama.expert_used_count u32 = 2
llama_model_loader: - kv 12: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 13: llama.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 14: general.file_type u32 = 15
llama_model_loader: - kv 15: tokenizer.ggml.model str = llama
llama_model_loader: - kv 16: tokenizer.ggml.tokens arr[str,32001] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv 17: tokenizer.ggml.scores arr[f32,32001] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,32001] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 22: tokenizer.ggml.padding_token_id u32 = 32000
llama_model_loader: - kv 23: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 24: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 25: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
llama_model_loader: - kv 26: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type f16: 32 tensors
llama_model_loader: - type q8_0: 64 tensors
llama_model_loader: - type q4_K: 113 tensors
llama_model_loader: - type q5_K: 32 tensors
llama_model_loader: - type q6_K: 17 tensors
llm_load_vocab: special tokens definition check successful ( 260/32001 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32001
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 8
llm_load_print_meta: n_expert_used = 2
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 8x7B
llm_load_print_meta: model ftype = Q4_K - Medium
llm_load_print_meta: model params = 46.70 B
llm_load_print_meta: model size = 26.49 GiB (4.87 BPW)
llm_load_print_meta: general.name = .
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: PAD token = 32000 '[PAD]'
llm_load_print_meta: LF token = 13 '<0x0A>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes
ggml_cuda_init: CUDA_USE_TENSOR_CORES: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size = 0.32 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors: CPU buffer size = 70.31 MiB
llm_load_tensors: CUDA0 buffer size = 27057.56 MiB
.............................................................................................
llama_new_context_with_model: n_ctx = 32768
llama_new_context_with_model: n_batch = 2048
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA0 KV buffer size = 4096.00 MiB
llama_new_context_with_model: KV self size = 4096.00 MiB, K (f16): 2048.00 MiB, V (f16): 2048.00 MiB
llama_new_context_with_model: CUDA_Host output buffer size = 0.61 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 2144.00 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 72.01 MiB
llama_new_context_with_model: graph nodes = 1510
llama_new_context_with_model: graph splits = 2
main: Simulating parallel requests from clients:
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
main: Evaluating the system prompt ...
Processing requests ...
main: clearing the KV cache
Client 0, seq 0, started decoding ...
Client 1, seq 1, started decoding ...
Client 2, seq 2, started decoding ...
Client 3, seq 3, started decoding ...
Client 3, seq 3/ 4, prompt 22 t, response 5 t, time 0.45 s, speed 59.68 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: Yes, I am▅
Client 2, seq 2/ 4, prompt 11 t, response 15 t, time 0.79 s, speed 32.84 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has▅▅▅a</s>
Client 1, seq 1/ 4, prompt 15 t, response 31 t, time 1.28 s, speed 35.97 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: The best way to▅ to cook a steak▅assistant is to▅assistant follow these▅▅▅▅▅▅▅▅▅ass▅
Client 0, seq 0/ 4, prompt 11 t, response 79 t, time 2.10 s, speed 42.89 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has been debated throughout history. Some people believe that the meaning of life is to seek happiness, fulfillment, and personal growth. Others believe that the meaning of life is to serve a higher purpose, such as a god or a moral principle. Ultimately, the meaning of life is a personal belief that each individual must determine for themselves.</s>
main: clearing the KV cache
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
External prompt file: used built-in defaults
Model and path used: Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf
Total prompt tokens: 59, speed: 23.56 t/s
Total gen tokens: 130, speed: 51.91 t/s
Total speed (AVG): speed: 75.47 t/s
Cache misses: 0
llama_print_timings: load time = 2638.09 ms
llama_print_timings: sample time = 2.40 ms / 134 runs ( 0.02 ms per token, 55856.61 tokens per second)
llama_print_timings: prompt eval time = 1670.06 ms / 440 tokens ( 3.80 ms per token, 263.46 tokens per second)
llama_print_timings: eval time = 811.82 ms / 48 runs ( 16.91 ms per token, 59.13 tokens per second)
llama_print_timings: total time = 2504.88 ms / 488 tokens
No new questions so proceed with build-in defaults.
run parameters as at 2024-05-02 15:25:43
Raw
f16.from_scratch.parallel.log
[1715186744] Log start
[1715186744] Cmd: ../llama.cpp/build/bin/parallel -m Mistral-Custom/ggml-model-f16.gguf -ngl 999 -c 1024 --parallel 4 --sequences 4 --seed 25519 -n 512
[1715186746] warming up the model with an empty run
[1715186746] main: Simulating parallel requests from clients:
[1715186746] main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
[1715186746]
[1715186746] main: Evaluating the system prompt ...
[1715186746]
[1715186746] Processing requests ...
[1715186746] main: clearing the KV cache
[1715186746] Client 0, seq 0, started decoding ...
[1715186746] Client 1, seq 1, started decoding ...
[1715186746] Client 2, seq 2, started decoding ...
[1715186746] Client 3, seq 3, started decoding ...
[1715186746] main : decoded batch of 59 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186746] main : decoded batch of 4 tokens
[1715186747] main : decoded batch of 4 tokens
[1715186747] main : decoded batch of 4 tokens
[1715186747] main : decoded batch of 4 tokens
[1715186747] main : decoded batch of 4 tokens
[1715186747] main : decoded batch of 4 tokens
[1715186747] Client 1, seq 1/ 4, prompt 15 t, response 36 t, time 1.01 s, speed 50.37 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: The best way to cook a steak depends on your preference for the level of doneness. Here is a simple method for cooking a steak on the stovetop:
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186747] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] main : decoded batch of 3 tokens
[1715186748] Client 0, seq 0/ 4, prompt 11 t, response 101 t, time 2.70 s, speed 41.56 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical one and has been asked for centuries. There are many different interpretations and beliefs about this question. Some people believe that the meaning of life is to seek happiness, knowledge, or personal fulfillment. Others believe that the meaning of life is to serve a higher power or to contribute to the greater good of humanity. Ultimately, the answer to this question is a personal one and may depend on an individual's beliefs, values, and experiences.
[1715186748] main : decoded batch of 2 tokens
[1715186748] main : decoded batch of 2 tokens
[1715186748] main : decoded batch of 2 tokens
[1715186748] main : decoded batch of 2 tokens
[1715186748] main : decoded batch of 2 tokens
[1715186748] main : decoded batch of 2 tokens
[1715186748] main : decoded batch of 2 tokens
[1715186748] main : decoded batch of 2 tokens
[1715186749] main : decoded batch of 2 tokens
[1715186749] main : decoded batch of 2 tokens
[1715186749] main : decoded batch of 2 tokens
[1715186749] main : decoded batch of 2 tokens
[1715186749] Client 2, seq 2/ 4, prompt 11 t, response 113 t, time 3.00 s, speed 41.33 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical and metaphysical question that has been debated throughout history. Many different perspectives have been proposed, including religious, philosophical, and scientific. Some people believe that life has no inherent meaning and that it is up to each individual to create their own purpose. Others believe that life has a specific meaning or purpose that is given by God or the universe. Ultimately, the answer to this question is a matter of personal belief and there is no definitive answer that is accepted by all people.
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] main : decoded batch of 1 tokens
[1715186749] Client 3, seq 3/ 4, prompt 22 t, response 137 t, time 3.58 s, speed 44.39 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: Yes, I can certainly help explain the Special Theory of Relativity. The Special Theory of Relativity is a physical theory proposed by Albert Einstein in 1905. It is based on two main postulates: the laws of physics are the same in all inertial frames of reference, and the speed of light in a vacuum is the same for all observers, regardless of their motion or the source of the light. The theory leads to several counterintuitive results, such as time dilation, length contraction, and the equivalence of mass and energy (E=mc^2). These effects have been experimentally confirmed and are fundamental to modern physics.
[1715186749] main: clearing the KV cache
[1715186749]
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
[1715186749] External prompt file: used built-in defaults
[1715186749] Model and path used: Mistral-Custom/ggml-model-f16.gguf
[1715186749] Total prompt tokens: 59, speed: 16.35 t/s
[1715186749] Total gen tokens: 387, speed: 107.27 t/s
[1715186749] Total speed (AVG): speed: 123.63 t/s
[1715186749] Cache misses: 0
[1715186749]
Raw
f16.from_scratch.parallel.out
Log start
llama_model_loader: loaded meta data with 24 key-value pairs and 291 tensors from Mistral-Custom/ggml-model-f16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = .
llama_model_loader: - kv 2: llama.vocab_size u32 = 32001
llama_model_loader: - kv 3: llama.context_length u32 = 32768
llama_model_loader: - kv 4: llama.embedding_length u32 = 4096
llama_model_loader: - kv 5: llama.block_count u32 = 32
llama_model_loader: - kv 6: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 7: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 8: llama.attention.head_count u32 = 32
llama_model_loader: - kv 9: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 10: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 11: llama.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 12: general.file_type u32 = 1
llama_model_loader: - kv 13: tokenizer.ggml.model str = llama
llama_model_loader: - kv 14: tokenizer.ggml.tokens arr[str,32001] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv 15: tokenizer.ggml.scores arr[f32,32001] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,32001] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 17: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 18: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 19: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 20: tokenizer.ggml.padding_token_id u32 = 32000
llama_model_loader: - kv 21: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 22: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 23: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type f16: 226 tensors
llm_load_vocab: special tokens definition check successful ( 260/32001 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32001
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 8B
llm_load_print_meta: model ftype = F16
llm_load_print_meta: model params = 7.24 B
llm_load_print_meta: model size = 13.49 GiB (16.00 BPW)
llm_load_print_meta: general.name = .
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: PAD token = 32000 '[PAD]'
llm_load_print_meta: LF token = 13 '<0x0A>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size = 0.30 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors: CPU buffer size = 250.01 MiB
llm_load_tensors: CUDA0 buffer size = 13563.02 MiB
...................................................................................................
llama_new_context_with_model: n_ctx = 1024
llama_new_context_with_model: n_batch = 1024
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA0 KV buffer size = 128.00 MiB
llama_new_context_with_model: KV self size = 128.00 MiB, K (f16): 64.00 MiB, V (f16): 64.00 MiB
llama_new_context_with_model: CUDA_Host output buffer size = 0.61 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 98.00 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 10.01 MiB
llama_new_context_with_model: graph nodes = 1030
llama_new_context_with_model: graph splits = 2
main: Simulating parallel requests from clients:
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
main: Evaluating the system prompt ...
Processing requests ...
main: clearing the KV cache
Client 0, seq 0, started decoding ...
Client 1, seq 1, started decoding ...
Client 2, seq 2, started decoding ...
Client 3, seq 3, started decoding ...
Client 1, seq 1/ 4, prompt 15 t, response 36 t, time 1.01 s, speed 50.37 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: The best way to cook a steak depends on your preference for the level of doneness. Here is a simple method for cooking a steak on the stovetop:
Client 0, seq 0/ 4, prompt 11 t, response 101 t, time 2.70 s, speed 41.56 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical one and has been asked for centuries. There are many different interpretations and beliefs about this question. Some people believe that the meaning of life is to seek happiness, knowledge, or personal fulfillment. Others believe that the meaning of life is to serve a higher power or to contribute to the greater good of humanity. Ultimately, the answer to this question is a personal one and may depend on an individual's beliefs, values, and experiences.
Client 2, seq 2/ 4, prompt 11 t, response 113 t, time 3.00 s, speed 41.33 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical and metaphysical question that has been debated throughout history. Many different perspectives have been proposed, including religious, philosophical, and scientific. Some people believe that life has no inherent meaning and that it is up to each individual to create their own purpose. Others believe that life has a specific meaning or purpose that is given by God or the universe. Ultimately, the answer to this question is a matter of personal belief and there is no definitive answer that is accepted by all people.
Client 3, seq 3/ 4, prompt 22 t, response 137 t, time 3.58 s, speed 44.39 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: Yes, I can certainly help explain the Special Theory of Relativity. The Special Theory of Relativity is a physical theory proposed by Albert Einstein in 1905. It is based on two main postulates: the laws of physics are the same in all inertial frames of reference, and the speed of light in a vacuum is the same for all observers, regardless of their motion or the source of the light. The theory leads to several counterintuitive results, such as time dilation, length contraction, and the equivalence of mass and energy (E=mc^2). These effects have been experimentally confirmed and are fundamental to modern physics.
main: clearing the KV cache
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
External prompt file: used built-in defaults
Model and path used: Mistral-Custom/ggml-model-f16.gguf
Total prompt tokens: 59, speed: 16.35 t/s
Total gen tokens: 387, speed: 107.27 t/s
Total speed (AVG): speed: 123.63 t/s
Cache misses: 0
llama_print_timings: load time = 1568.09 ms
llama_print_timings: sample time = 7.01 ms / 391 runs ( 0.02 ms per token, 55785.42 tokens per second)
llama_print_timings: prompt eval time = 2966.57 ms / 721 tokens ( 4.11 ms per token, 243.04 tokens per second)
llama_print_timings: eval time = 578.22 ms / 24 runs ( 24.09 ms per token, 41.51 tokens per second)
llama_print_timings: total time = 3607.73 ms / 745 tokens
No new questions so proceed with build-in defaults.
run parameters as at 2024-05-08 12:45:49
Raw
f32.from_scratch.parallel.log
[1715186854] Log start
[1715186854] Cmd: ../llama.cpp/build/bin/parallel -m Mistral-Custom/ggml-model-f32.gguf -ngl 999 -c 1024 --parallel 4 --sequences 4 --seed 25519 -n 512
[1715186911] warming up the model with an empty run
[1715186911] main: Simulating parallel requests from clients:
[1715186911] main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
[1715186911]
[1715186911] main: Evaluating the system prompt ...
[1715186911]
[1715186911] Processing requests ...
[1715186911] main: clearing the KV cache
[1715186911] Client 0, seq 0, started decoding ...
[1715186911] Client 1, seq 1, started decoding ...
[1715186911] Client 2, seq 2, started decoding ...
[1715186911] Client 3, seq 3, started decoding ...
[1715186911] main : decoded batch of 59 tokens
[1715186911] main : decoded batch of 4 tokens
[1715186911] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186912] main : decoded batch of 4 tokens
[1715186913] main : decoded batch of 4 tokens
[1715186913] main : decoded batch of 4 tokens
[1715186913] main : decoded batch of 4 tokens
[1715186913] main : decoded batch of 4 tokens
[1715186913] main : decoded batch of 4 tokens
[1715186913] main : decoded batch of 4 tokens
[1715186913] main : decoded batch of 4 tokens
[1715186913] main : decoded batch of 4 tokens
[1715186913] Client 1, seq 1/ 4, prompt 15 t, response 31 t, time 1.60 s, speed 28.70 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: The best way to cook a steak depends on your preference for doneness. Here's a simple method for a pan-seared steak:
[1715186913] main : decoded batch of 3 tokens
[1715186913] main : decoded batch of 3 tokens
[1715186913] main : decoded batch of 3 tokens
[1715186913] main : decoded batch of 3 tokens
[1715186913] main : decoded batch of 3 tokens
[1715186913] main : decoded batch of 3 tokens
[1715186913] main : decoded batch of 3 tokens
[1715186913] main : decoded batch of 3 tokens
[1715186913] main : decoded batch of 3 tokens
[1715186913] main : decoded batch of 3 tokens
[1715186913] main : decoded batch of 3 tokens
[1715186913] main : decoded batch of 3 tokens
[1715186913] main : decoded batch of 3 tokens
[1715186913] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186914] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186915] main : decoded batch of 3 tokens
[1715186916] main : decoded batch of 3 tokens
[1715186916] main : decoded batch of 3 tokens
[1715186916] Client 0, seq 0/ 4, prompt 11 t, response 91 t, time 4.36 s, speed 23.39 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical one and has been asked for centuries. There are many different interpretations and beliefs about this question. Some people believe that the meaning of life is to seek happiness, knowledge, or personal fulfillment. Others believe that the meaning of life is to serve a higher power or to contribute to the greater good of humanity. Ultimately, the answer to this question is a matter of personal belief and interpretation.
[1715186916] Client 2, seq 2/ 4, prompt 11 t, response 91 t, time 4.36 s, speed 23.39 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical one and has been asked for centuries. There are many different interpretations and beliefs about this question. Some people believe that the meaning of life is to seek happiness, knowledge, or personal fulfillment. Others believe that the meaning of life is to serve a higher power or to contribute to the greater good of humanity. Ultimately, the answer to this question is a matter of personal belief and interpretation.
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186916] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186917] main : decoded batch of 1 tokens
[1715186918] main : decoded batch of 1 tokens
[1715186918] main : decoded batch of 1 tokens
[1715186918] main : decoded batch of 1 tokens
[1715186918] main : decoded batch of 1 tokens
[1715186918] main : decoded batch of 1 tokens
[1715186918] main : decoded batch of 1 tokens
[1715186918] main : decoded batch of 1 tokens
[1715186918] main : decoded batch of 1 tokens
[1715186918] main : decoded batch of 1 tokens
[1715186918] main : decoded batch of 1 tokens
[1715186918] main : decoded batch of 1 tokens
[1715186918] Client 3, seq 3/ 4, prompt 22 t, response 146 t, time 6.75 s, speed 24.88 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: Yes, I can certainly help explain the Special Theory of Relativity. The Special Theory of Relativity is a physical theory proposed by Albert Einstein in 1905. It is based on two main postulates: the laws of physics are the same in all inertial frames of reference, and the speed of light in a vacuum is the same for all observers, regardless of their motion or the source of the light. The theory leads to several counterintuitive results, such as time dilation, length contraction, and the equivalence of mass and energy (as expressed in the famous equation E=mc²). It revolutionized our understanding of space and time and had a profound impact on modern physics.
[1715186918] main: clearing the KV cache
[1715186918]
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
[1715186918] External prompt file: used built-in defaults
[1715186918] Model and path used: Mistral-Custom/ggml-model-f32.gguf
[1715186918] Total prompt tokens: 59, speed: 8.67 t/s
[1715186918] Total gen tokens: 359, speed: 52.74 t/s
[1715186918] Total speed (AVG): speed: 61.40 t/s
[1715186918] Cache misses: 0
[1715186918]
Raw
f32.from_scratch.parallel.out
Log start
llama_model_loader: loaded meta data with 24 key-value pairs and 291 tensors from Mistral-Custom/ggml-model-f32.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = .
llama_model_loader: - kv 2: llama.vocab_size u32 = 32001
llama_model_loader: - kv 3: llama.context_length u32 = 32768
llama_model_loader: - kv 4: llama.embedding_length u32 = 4096
llama_model_loader: - kv 5: llama.block_count u32 = 32
llama_model_loader: - kv 6: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 7: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 8: llama.attention.head_count u32 = 32
llama_model_loader: - kv 9: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 10: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 11: llama.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 12: general.file_type u32 = 0
llama_model_loader: - kv 13: tokenizer.ggml.model str = llama
llama_model_loader: - kv 14: tokenizer.ggml.tokens arr[str,32001] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv 15: tokenizer.ggml.scores arr[f32,32001] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,32001] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 17: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 18: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 19: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 20: tokenizer.ggml.padding_token_id u32 = 32000
llama_model_loader: - kv 21: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 22: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 23: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
llama_model_loader: - type f32: 291 tensors
llm_load_vocab: special tokens definition check successful ( 260/32001 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32001
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 8B
llm_load_print_meta: model ftype = all F32
llm_load_print_meta: model params = 7.24 B
llm_load_print_meta: model size = 26.98 GiB (32.00 BPW)
llm_load_print_meta: general.name = .
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: PAD token = 32000 '[PAD]'
llm_load_print_meta: LF token = 13 '<0x0A>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size = 0.30 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors: CPU buffer size = 500.02 MiB
llm_load_tensors: CUDA0 buffer size = 27125.03 MiB
...................................................................................................
llama_new_context_with_model: n_ctx = 1024
llama_new_context_with_model: n_batch = 1024
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA0 KV buffer size = 128.00 MiB
llama_new_context_with_model: KV self size = 128.00 MiB, K (f16): 64.00 MiB, V (f16): 64.00 MiB
llama_new_context_with_model: CUDA_Host output buffer size = 0.61 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 98.00 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 10.01 MiB
llama_new_context_with_model: graph nodes = 1030
llama_new_context_with_model: graph splits = 2
main: Simulating parallel requests from clients:
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
main: Evaluating the system prompt ...
Processing requests ...
main: clearing the KV cache
Client 0, seq 0, started decoding ...
Client 1, seq 1, started decoding ...
Client 2, seq 2, started decoding ...
Client 3, seq 3, started decoding ...
Client 1, seq 1/ 4, prompt 15 t, response 31 t, time 1.60 s, speed 28.70 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: The best way to cook a steak depends on your preference for doneness. Here's a simple method for a pan-seared steak:
Client 0, seq 0/ 4, prompt 11 t, response 91 t, time 4.36 s, speed 23.39 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical one and has been asked for centuries. There are many different interpretations and beliefs about this question. Some people believe that the meaning of life is to seek happiness, knowledge, or personal fulfillment. Others believe that the meaning of life is to serve a higher power or to contribute to the greater good of humanity. Ultimately, the answer to this question is a matter of personal belief and interpretation.
Client 2, seq 2/ 4, prompt 11 t, response 91 t, time 4.36 s, speed 23.39 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical one and has been asked for centuries. There are many different interpretations and beliefs about this question. Some people believe that the meaning of life is to seek happiness, knowledge, or personal fulfillment. Others believe that the meaning of life is to serve a higher power or to contribute to the greater good of humanity. Ultimately, the answer to this question is a matter of personal belief and interpretation.
Client 3, seq 3/ 4, prompt 22 t, response 146 t, time 6.75 s, speed 24.88 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: Yes, I can certainly help explain the Special Theory of Relativity. The Special Theory of Relativity is a physical theory proposed by Albert Einstein in 1905. It is based on two main postulates: the laws of physics are the same in all inertial frames of reference, and the speed of light in a vacuum is the same for all observers, regardless of their motion or the source of the light. The theory leads to several counterintuitive results, such as time dilation, length contraction, and the equivalence of mass and energy (as expressed in the famous equation E=mc²). It revolutionized our understanding of space and time and had a profound impact on modern physics.
main: clearing the KV cache
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
External prompt file: used built-in defaults
Model and path used: Mistral-Custom/ggml-model-f32.gguf
Total prompt tokens: 59, speed: 8.67 t/s
Total gen tokens: 359, speed: 52.74 t/s
Total speed (AVG): speed: 61.40 t/s
Cache misses: 0
llama_print_timings: load time = 57567.90 ms
llama_print_timings: sample time = 7.02 ms / 363 runs ( 0.02 ms per token, 51687.31 tokens per second)
llama_print_timings: prompt eval time = 4366.27 ms / 662 tokens ( 6.60 ms per token, 151.62 tokens per second)
llama_print_timings: eval time = 2381.82 ms / 55 runs ( 43.31 ms per token, 23.09 tokens per second)
llama_print_timings: total time = 6807.98 ms / 717 tokens
No new questions so proceed with build-in defaults.
run parameters as at 2024-05-08 12:48:38
Raw
from_scratch.parallel.log
[1715185850] Log start
[1715185850] Cmd: ../llama.cpp/build/bin/parallel -m Mistral-Custom/ggml-model-q4_k_m.gguf -ngl 999 -c 1024 --parallel 4 --sequences 4 --seed 25519 -n 512
[1715185850] warming up the model with an empty run
[1715185850] main: Simulating parallel requests from clients:
[1715185850] main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
[1715185850]
[1715185850] main: Evaluating the system prompt ...
[1715185850]
[1715185850] Processing requests ...
[1715185850] main: clearing the KV cache
[1715185850] Client 0, seq 0, started decoding ...
[1715185850] Client 1, seq 1, started decoding ...
[1715185850] Client 2, seq 2, started decoding ...
[1715185850] Client 3, seq 3, started decoding ...
[1715185850] main : decoded batch of 59 tokens
[1715185850] main : decoded batch of 4 tokens
[1715185850] main : decoded batch of 4 tokens
[1715185850] main : decoded batch of 4 tokens
[1715185850] main : decoded batch of 4 tokens
[1715185850] main : decoded batch of 4 tokens
[1715185850] main : decoded batch of 4 tokens
[1715185850] main : decoded batch of 4 tokens
[1715185850] main : decoded batch of 4 tokens
[1715185850] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185851] main : decoded batch of 4 tokens
[1715185852] main : decoded batch of 4 tokens
[1715185852] main : decoded batch of 4 tokens
[1715185852] main : decoded batch of 4 tokens
[1715185852] Client 1, seq 1/ 4, prompt 15 t, response 86 t, time 1.27 s, speed 79.68 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: ▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅ Assist: To cook▅▅▅▅▅ Assis▅:▅▅▅[PAD]▅ Ass▅ Assistant: To▅▅▅ Assistant▅▅ To▅▅ Assistant▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅:▅:▅:▅▅:▅:
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] main : decoded batch of 3 tokens
[1715185852] Client 0, seq 0/ 4, prompt 11 t, response 107 t, time 1.51 s, speed 78.10 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical one that has puzzled humans for centuries. There are many different perspectives on this question, and there is no definitive answer. Some people believe that the meaning of life is to seek happiness, knowledge, or personal growth. Others believe that it is to serve a higher power or to contribute to the greater good of humanity. Ultimately, the meaning of life is a personal question, and each person must find their own answer based on their beliefs, values, and experiences.
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] main : decoded batch of 2 tokens
[1715185852] Client 3, seq 3/ 4, prompt 22 t, response 134 t, time 1.79 s, speed 86.98 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: ▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅ I'd be happy to help▅▅▅▅ Sh▅ yes,▅ certainly!▅ The▅▅ A Special Theory of Relativity is a▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅ I'▅ I▅▅▅▅▅▅▅▅ I▅▅▅▅▅▅▅▅▅▅▅▅▅▅ I▅▅▅▅▅▅▅ I'm glad▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅ I▅▅▅▅▅▅▅
[1715185852] main : decoded batch of 1 tokens
[1715185852] main : decoded batch of 1 tokens
[1715185852] main : decoded batch of 1 tokens
[1715185852] main : decoded batch of 1 tokens
[1715185852] main : decoded batch of 1 tokens
[1715185852] main : decoded batch of 1 tokens
[1715185852] main : decoded batch of 1 tokens
[1715185852] main : decoded batch of 1 tokens
[1715185852] main : decoded batch of 1 tokens
[1715185852] main : decoded batch of 1 tokens
[1715185852] main : decoded batch of 1 tokens
[1715185852] main : decoded batch of 1 tokens
[1715185852] main : decoded batch of 1 tokens
[1715185852] Client 2, seq 2/ 4, prompt 11 t, response 147 t, time 1.92 s, speed 82.08 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical one that has puzzled humans for▅ I'm not sure how long▅ I'▅ I apologize for the▅ I'll▅ I'll continue this thought later. The meaning of life is a question that has been explored by many philosophers, theologians, and scientists throughout history. Some propose that the meaning of life is to seek happiness, knowledge,▅▅▅ I▅▅ Some▅▅▅ Some propose that▅▅▅▅▅▅▅▅ Some propose that the meaning of life is to serve a higher power or to fulfill a purpose. Ultimately, the answer to this question is a personal one that may vary from person to person.
[1715185852] main: clearing the KV cache
[1715185852]
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
[1715185852] External prompt file: used built-in defaults
[1715185852] Model and path used: Mistral-Custom/ggml-model-q4_k_m.gguf
[1715185852] Total prompt tokens: 59, speed: 29.96 t/s
[1715185852] Total gen tokens: 474, speed: 240.73 t/s
[1715185852] Total speed (AVG): speed: 270.70 t/s
[1715185852] Cache misses: 0
[1715185852]
Raw
from_scratch.parallel.out
Log start
llama_model_loader: loaded meta data with 25 key-value pairs and 291 tensors from Mistral-Custom/ggml-model-q4_k_m.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = .
llama_model_loader: - kv 2: llama.vocab_size u32 = 32001
llama_model_loader: - kv 3: llama.context_length u32 = 32768
llama_model_loader: - kv 4: llama.embedding_length u32 = 4096
llama_model_loader: - kv 5: llama.block_count u32 = 32
llama_model_loader: - kv 6: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 7: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 8: llama.attention.head_count u32 = 32
llama_model_loader: - kv 9: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 10: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 11: llama.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 12: general.file_type u32 = 15
llama_model_loader: - kv 13: tokenizer.ggml.model str = llama
llama_model_loader: - kv 14: tokenizer.ggml.tokens arr[str,32001] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv 15: tokenizer.ggml.scores arr[f32,32001] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,32001] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 17: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 18: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 19: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 20: tokenizer.ggml.padding_token_id u32 = 32000
llama_model_loader: - kv 21: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 22: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 23: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
llama_model_loader: - kv 24: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type q4_K: 193 tensors
llama_model_loader: - type q6_K: 33 tensors
llm_load_vocab: special tokens definition check successful ( 260/32001 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32001
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 8B
llm_load_print_meta: model ftype = Q4_K - Medium
llm_load_print_meta: model params = 7.24 B
llm_load_print_meta: model size = 4.07 GiB (4.83 BPW)
llm_load_print_meta: general.name = .
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: PAD token = 32000 '[PAD]'
llm_load_print_meta: LF token = 13 '<0x0A>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size = 0.30 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors: CPU buffer size = 70.31 MiB
llm_load_tensors: CUDA0 buffer size = 4095.06 MiB
................................................................................................
llama_new_context_with_model: n_ctx = 1024
llama_new_context_with_model: n_batch = 1024
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA0 KV buffer size = 128.00 MiB
llama_new_context_with_model: KV self size = 128.00 MiB, K (f16): 64.00 MiB, V (f16): 64.00 MiB
llama_new_context_with_model: CUDA_Host output buffer size = 0.61 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 98.00 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 10.01 MiB
llama_new_context_with_model: graph nodes = 1030
llama_new_context_with_model: graph splits = 2
main: Simulating parallel requests from clients:
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
main: Evaluating the system prompt ...
Processing requests ...
main: clearing the KV cache
Client 0, seq 0, started decoding ...
Client 1, seq 1, started decoding ...
Client 2, seq 2, started decoding ...
Client 3, seq 3, started decoding ...
Client 1, seq 1/ 4, prompt 15 t, response 86 t, time 1.27 s, speed 79.68 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: ▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅ Assist: To cook▅▅▅▅▅ Assis▅:▅▅▅[PAD]▅ Ass▅ Assistant: To▅▅▅ Assistant▅▅ To▅▅ Assistant▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅:▅:▅:▅▅:▅:
Client 0, seq 0/ 4, prompt 11 t, response 107 t, time 1.51 s, speed 78.10 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical one that has puzzled humans for centuries. There are many different perspectives on this question, and there is no definitive answer. Some people believe that the meaning of life is to seek happiness, knowledge, or personal growth. Others believe that it is to serve a higher power or to contribute to the greater good of humanity. Ultimately, the meaning of life is a personal question, and each person must find their own answer based on their beliefs, values, and experiences.
Client 3, seq 3/ 4, prompt 22 t, response 134 t, time 1.79 s, speed 86.98 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: ▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅ I'd be happy to help▅▅▅▅ Sh▅ yes,▅ certainly!▅ The▅▅ A Special Theory of Relativity is a▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅ I'▅ I▅▅▅▅▅▅▅▅ I▅▅▅▅▅▅▅▅▅▅▅▅▅▅ I▅▅▅▅▅▅▅ I'm glad▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅ I▅▅▅▅▅▅▅
Client 2, seq 2/ 4, prompt 11 t, response 147 t, time 1.92 s, speed 82.08 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The question of the meaning of life is a philosophical one that has puzzled humans for▅ I'm not sure how long▅ I'▅ I apologize for the▅ I'll▅ I'll continue this thought later. The meaning of life is a question that has been explored by many philosophers, theologians, and scientists throughout history. Some propose that the meaning of life is to seek happiness, knowledge,▅▅▅ I▅▅ Some▅▅▅ Some propose that▅▅▅▅▅▅▅▅ Some propose that the meaning of life is to serve a higher power or to fulfill a purpose. Ultimately, the answer to this question is a personal one that may vary from person to person.
main: clearing the KV cache
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
External prompt file: used built-in defaults
Model and path used: Mistral-Custom/ggml-model-q4_k_m.gguf
Total prompt tokens: 59, speed: 29.96 t/s
Total gen tokens: 474, speed: 240.73 t/s
Total speed (AVG): speed: 270.70 t/s
Cache misses: 0
llama_print_timings: load time = 651.59 ms
llama_print_timings: sample time = 8.77 ms / 478 runs ( 0.02 ms per token, 54522.64 tokens per second)
llama_print_timings: prompt eval time = 1760.92 ms / 819 tokens ( 2.15 ms per token, 465.10 tokens per second)
llama_print_timings: eval time = 129.18 ms / 13 runs ( 9.94 ms per token, 100.63 tokens per second)
llama_print_timings: total time = 1969.87 ms / 832 tokens
No new questions so proceed with build-in defaults.
run parameters as at 2024-05-08 12:30:52
Raw
gpu.bias.parallel.log
[1714510828] Log start
[1714510828] Cmd: llama.cpp/build/bin/parallel -m Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf -ngl 999 -c 32768 --parallel 2 --sequences 4 --seed 25519 --logit-bias 0-999999999
[1714510830] warming up the model with an empty run
[1714510830] main: Simulating parallel requests from clients:
[1714510830] main: n_parallel = 2, n_sequences = 4, cont_batching = 1, system tokens = 299
[1714510830]
[1714510830] main: Evaluating the system prompt ...
[1714510831]
[1714510831] Processing requests ...
[1714510831] main: clearing the KV cache
[1714510831] Client 0, seq 0, started decoding ...
[1714510831] Client 1, seq 1, started decoding ...
[1714510831] main : decoded batch of 26 tokens
[1714510831] main : decoded batch of 2 tokens
[1714510831] main : decoded batch of 2 tokens
[1714510831] main : decoded batch of 2 tokens
[1714510831] main : decoded batch of 2 tokens
[1714510831] main : decoded batch of 2 tokens
[1714510831] main : decoded batch of 2 tokens
[1714510831] main : decoded batch of 2 tokens
[1714510831] main : decoded batch of 2 tokens
[1714510831] main : decoded batch of 2 tokens
[1714510831] main : decoded batch of 2 tokens
[1714510831] main : decoded batch of 2 tokens
[1714510831] main : decoded batch of 2 tokens
[1714510831] main : decoded batch of 2 tokens
[1714510831] main : decoded batch of 2 tokens
[1714510831] Client 1, seq 1/ 4, prompt 15 t, response 14 t, time 0.55 s, speed 52.82 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: The best way to cook a steak is to follow these steps:
[1714510831] Client 1, seq 2, started decoding ...
[1714510831] main : decoded batch of 12 tokens
[1714510831] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510832] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510833] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] Client 0, seq 0/ 4, prompt 11 t, response 102 t, time 3.23 s, speed 35.01 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has been debated throughout history. Some people believe that the meaning of life is to seek happiness, fulfillment, and personal growth. Others believe that the meaning of life is to serve a higher purpose, such as a god or a moral principle. Still others believe that the meaning of life is to create meaning for oneself through one's experiences and relationships. Ultimately, the meaning of life is a subjective concept that varies from person to person.</s>
[1714510834] Client 0, seq 3, started decoding ...
[1714510834] main : decoded batch of 23 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510834] main : decoded batch of 2 tokens
[1714510835] main : decoded batch of 2 tokens
[1714510835] main : decoded batch of 2 tokens
[1714510835] main : decoded batch of 2 tokens
[1714510835] Client 1, seq 2/ 4, prompt 11 t, response 101 t, time 3.18 s, speed 35.21 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has been debated by scholars and thinkers for centuries. There is no one answer to this question, as it depends on each individual's personal beliefs and values. Some people believe that the meaning of life is to find happiness and fulfillment, while others believe that it is to make a positive impact on the world. Ultimately, the meaning of life is a deeply personal and subjective concept that can only be determined by each individual for themselves.
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510835] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] main : decoded batch of 1 tokens
[1714510836] Client 0, seq 3/ 4, prompt 22 t, response 100 t, time 2.02 s, speed 60.28 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: Yes, I am familiar with the Special Theory of Relativity. The Special Theory of Relativity is a theory of physics that was developed by Albert Einstein in 1905. The theory is based on two postulates: first, that the laws of physics are the same in all inertial frames of reference; and second, that the speed of light in a vacuum is the same in all inertial frames of reference, regardless of the motion of the light source or the observer.
[1714510836] main: clearing the KV cache
[1714510836]
main: n_parallel = 2, n_sequences = 4, cont_batching = 1, system tokens = 299
[1714510836] External prompt file: used built-in defaults
[1714510836] Model and path used: Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf
[1714510836] Total prompt tokens: 59, speed: 10.47 t/s
[1714510836] Total gen tokens: 317, speed: 56.25 t/s
[1714510836] Total speed (AVG): speed: 66.72 t/s
[1714510836] Cache misses: 0
[1714510836]
Raw
gpu.bias.parallel.out
Log start
llama_model_loader: loaded meta data with 27 key-value pairs and 323 tensors from Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = .
llama_model_loader: - kv 2: llama.vocab_size u32 = 32001
llama_model_loader: - kv 3: llama.context_length u32 = 32768
llama_model_loader: - kv 4: llama.embedding_length u32 = 4096
llama_model_loader: - kv 5: llama.block_count u32 = 32
llama_model_loader: - kv 6: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 7: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 8: llama.attention.head_count u32 = 32
llama_model_loader: - kv 9: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 10: llama.expert_count u32 = 8
llama_model_loader: - kv 11: llama.expert_used_count u32 = 2
llama_model_loader: - kv 12: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 13: llama.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 14: general.file_type u32 = 15
llama_model_loader: - kv 15: tokenizer.ggml.model str = llama
llama_model_loader: - kv 16: tokenizer.ggml.tokens arr[str,32001] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv 17: tokenizer.ggml.scores arr[f32,32001] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,32001] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 22: tokenizer.ggml.padding_token_id u32 = 32000
llama_model_loader: - kv 23: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 24: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 25: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
llama_model_loader: - kv 26: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type f16: 32 tensors
llama_model_loader: - type q8_0: 64 tensors
llama_model_loader: - type q4_K: 113 tensors
llama_model_loader: - type q5_K: 32 tensors
llama_model_loader: - type q6_K: 17 tensors
llm_load_vocab: special tokens definition check successful ( 260/32001 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32001
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 8
llm_load_print_meta: n_expert_used = 2
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 8x7B
llm_load_print_meta: model ftype = Q4_K - Medium
llm_load_print_meta: model params = 46.70 B
llm_load_print_meta: model size = 26.49 GiB (4.87 BPW)
llm_load_print_meta: general.name = .
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: PAD token = 32000 '[PAD]'
llm_load_print_meta: LF token = 13 '<0x0A>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size = 0.32 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors: CPU buffer size = 70.31 MiB
llm_load_tensors: CUDA0 buffer size = 27057.56 MiB
.............................................................................................
llama_new_context_with_model: n_ctx = 32768
llama_new_context_with_model: n_batch = 2048
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA0 KV buffer size = 4096.00 MiB
llama_new_context_with_model: KV self size = 4096.00 MiB, K (f16): 2048.00 MiB, V (f16): 2048.00 MiB
llama_new_context_with_model: CUDA_Host output buffer size = 0.37 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 2144.00 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 72.01 MiB
llama_new_context_with_model: graph nodes = 1510
llama_new_context_with_model: graph splits = 2
main: Simulating parallel requests from clients:
main: n_parallel = 2, n_sequences = 4, cont_batching = 1, system tokens = 299
main: Evaluating the system prompt ...
Processing requests ...
main: clearing the KV cache
Client 0, seq 0, started decoding ...
Client 1, seq 1, started decoding ...
Client 1, seq 1/ 4, prompt 15 t, response 14 t, time 0.55 s, speed 52.82 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: The best way to cook a steak is to follow these steps:
Client 1, seq 2, started decoding ...
Client 0, seq 0/ 4, prompt 11 t, response 102 t, time 3.23 s, speed 35.01 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has been debated throughout history. Some people believe that the meaning of life is to seek happiness, fulfillment, and personal growth. Others believe that the meaning of life is to serve a higher purpose, such as a god or a moral principle. Still others believe that the meaning of life is to create meaning for oneself through one's experiences and relationships. Ultimately, the meaning of life is a subjective concept that varies from person to person.</s>
Client 0, seq 3, started decoding ...
Client 1, seq 2/ 4, prompt 11 t, response 101 t, time 3.18 s, speed 35.21 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has been debated by scholars and thinkers for centuries. There is no one answer to this question, as it depends on each individual's personal beliefs and values. Some people believe that the meaning of life is to find happiness and fulfillment, while others believe that it is to make a positive impact on the world. Ultimately, the meaning of life is a deeply personal and subjective concept that can only be determined by each individual for themselves.
Client 0, seq 3/ 4, prompt 22 t, response 100 t, time 2.02 s, speed 60.28 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: Yes, I am familiar with the Special Theory of Relativity. The Special Theory of Relativity is a theory of physics that was developed by Albert Einstein in 1905. The theory is based on two postulates: first, that the laws of physics are the same in all inertial frames of reference; and second, that the speed of light in a vacuum is the same in all inertial frames of reference, regardless of the motion of the light source or the observer.
main: clearing the KV cache
main: n_parallel = 2, n_sequences = 4, cont_batching = 1, system tokens = 299
External prompt file: used built-in defaults
Model and path used: Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf
Total prompt tokens: 59, speed: 10.47 t/s
Total gen tokens: 317, speed: 56.25 t/s
Total speed (AVG): speed: 66.72 t/s
Cache misses: 0
llama_print_timings: load time = 2675.24 ms
llama_print_timings: sample time = 6.05 ms / 321 runs ( 0.02 ms per token, 53040.32 tokens per second)
llama_print_timings: prompt eval time = 4075.64 ms / 588 tokens ( 6.93 ms per token, 144.27 tokens per second)
llama_print_timings: eval time = 1507.19 ms / 87 runs ( 17.32 ms per token, 57.72 tokens per second)
llama_print_timings: total time = 5636.19 ms / 675 tokens
No new questions so proceed with build-in defaults.
run parameters as at 2024-04-30 17:00:36
Raw
gpu.bias.tasks.txt
# TASK 0
The Kessler syndrome, also known as the Kessler effect, is a scenario in which the density of objects in low Earth orbit (LEO) is high enough that collisions between objects could cause a cascade of additional collisions, creating a debris field that could make LEO unusable for future generations. Satellite constellations, such as SpaceX's Starlink, which plans to launch thousands of satellites into LEO, have the potential to accelerate the Kessler syndrome and pose a significant issue in the future.
The reason for this is that each satellite in the constellation represents a new target for collisions with other satellites, as well as with the existing debris in LEO. Additionally, the sheer number of satellites in the constellation increases the likelihood of collisions occurring. Even a small collision between two satellites can produce a significant amount of debris, which can then go on to cause further collisions.
SpaceX has stated that they are taking steps to mitigate the risk of collisions and the creation of debris, such as by designing their satellites to burn up in the atmosphere at the end of their lifespan and by implementing automated collision avoidance systems. However, it is still unclear whether these measures will be sufficient to prevent the Kessler syndrome from occurring.
Furthermore, the long-term effects of having such a large number of satellites in LEO are not yet fully understood. There are concerns that the increased brightness of the constellation could interfere with astronomical observations, and that the deployment of thousands of satellites could lead to a "tragedy of the commons" scenario in which the benefits of using LEO for communication and other purposes are outweighed by the costs of managing the resulting debris.
In summary, satellite constellations such as Starlink have the potential to accelerate the Kessler syndrome and pose a significant issue in the future. While steps are being taken to mitigate the risk of collisions and the creation of debris, the long-term effects of having such a large number of satellites in LEO are not yet fully understood.
# TASK 1
The Kessler syndrome, also known as the Kessler effect, is a scenario in which the density of objects in low Earth orbit (LEO) is high enough that collisions between objects could cause a cascade of additional collisions, creating a debris field that could make LEO unusable for future generations. Satellite constellations, such as SpaceX's Starlink, could potentially accelerate the Kessler syndrome in a few ways:
1. Increased number of satellites: Starlink plans to launch thousands of satellites into LEO, which will significantly increase the number of objects in this region of space. This will increase the likelihood of collisions between satellites, as well as between satellites and existing debris.
2. Lower altitude: Starlink satellites operate at an altitude of around 550 kilometers, which is lower than many other satellite systems. This means that the satellites (and any resulting debris) will have a shorter orbital lifetime, increasing the likelihood that they will re-enter Earth's atmosphere and burn up. However, while in orbit, the lower altitude also means that collisions at this altitude will occur at higher relative velocities, which could lead to more destructive collisions and increased debris generation.
3. Potential for operational errors: With a large constellation of satellites, there is an increased likelihood of operational errors, such as incorrect orbital maneuvers or collisions with other satellites or debris. These errors could generate new debris, further exacerbating the Kessler syndrome.
4. Difficulty in tracking and avoiding debris: As the number of objects in LEO increases, it becomes more challenging to track and predict the movements of all debris. This makes it harder to avoid collisions, increasing the risk of the Kessler syndrome.
While Starlink and other satellite constellations have the potential to accelerate the Kessler syndrome, it is important to note that the responsibility for mitigating this risk lies with both satellite operators and regulatory bodies. Satellite operators should follow best practices for debris mitigation, such as de-orbiting satellites at the end of their lifetimes, and regulatory bodies should consider implementing stricter guidelines and regulations to minimize the creation of new debris.
Additionally, there are ongoing efforts to develop technologies and strategies to mitigate the Kessler syndrome, such as active debris removal and new materials that can help reduce the likelihood of destructive collisions. By combining responsible operational practices, regulatory measures, and technological innovation, it may be possible to minimize the impact of satellite constellations on the Kessler syndrome and ensure the long-term sustainability of LEO for future generations.
# TASK 2
The Kessler syndrome, also known as the Kessler effect, is a scenario in which the density of objects in low Earth orbit (LEO) is high enough that collisions between objects could cause a cascade of additional collisions, creating a debris field that could make LEO unusable for future generations. Satellite constellations, such as SpaceX's Starlink, could potentially accelerate the Kessler syndrome in a few ways:
1. Increased number of satellites: Starlink plans to launch thousands of satellites into LEO, which will significantly increase the number of objects in this region of space. This will increase the likelihood of collisions between satellites, as well as between satellites and existing debris.
2. Lower altitude: Starlink satellites operate at an altitude of around 550 kilometers, which is lower than many other satellite systems. This means that the satellites (and any resulting debris) will have a shorter orbital lifetime, increasing the likelihood that they will re-enter Earth's atmosphere and burn up. However, while in orbit, the lower altitude also means that collisions at this altitude will occur at higher relative velocities, which could lead to more destructive collisions and increased debris generation.
3. Potential for operational errors: With a large constellation of satellites, there is an increased likelihood of operational errors, such as incorrect maneuvers or collisions with other satellites or debris. These errors could generate new debris, further contributing to the Kessler syndrome.
4. Difficulty in tracking and avoiding debris: As the number of objects in LEO increases, it becomes more challenging to track and avoid collisions with debris. This is especially true for smaller debris particles, which can be difficult to detect and may not be included in existing catalogs.
5. End-of-life disposal: Satellites in LEO eventually reach the end of their operational life and must be deorbited. If not properly managed, these end-of-life satellites could become new sources of debris, further exacerbating the Kessler syndrome.
While Starlink and other satellite constellations have the potential to accelerate the Kessler syndrome, it is essential to recognize that the majority of the debris in LEO is the result of past collisions, explosions, and fragmentations. Addressing the Kessler syndrome will require a comprehensive approach, including better tracking and management of existing debris, stricter guidelines for satellite design and operation, and the development of technologies to remove debris from orbit.
# TASK 3
The Kessler syndrome, also known as the Kessler effect, is a scenario in which the density of objects in low Earth orbit (LEO) is high enough that collisions between objects could cause a cascade of additional collisions, creating a debris field that could make LEO unusable for future generations. Satellite constellations, such as SpaceX's Starlink, could potentially accelerate the Kessler syndrome and pose a significant issue in the future due to the large number of satellites they plan to launch.
Starlink plans to launch thousands of satellites into LEO to provide global broadband internet coverage. This is an order of magnitude greater than the number of satellites currently in LEO, and could significantly increase the risk of collisions. Additionally, the low altitude of the Starlink satellites (550 km) means that they will have a shorter lifespan and will need to be replaced more frequently, further increasing the number of satellites in LEO.
The risk of collisions is not just theoretical. In 2009, the Iridium 33 and Kosmos-2251 satellites collided in orbit, creating a large debris field that is still being tracked today. The collision occurred at an altitude of 790 km, and the Starlink satellites will be orbiting at an altitude of 550 km, where the risk of collisions is even higher.
Another concern is the potential for satellites to be damaged or destroyed during launch or deployment. In 2013, a Russian anti-satellite weapon test created a large debris field in LEO, and the Starlink constellation could create similar debris fields if any of its satellites are destroyed.
The Kessler syndrome is a complex issue, and it is difficult to predict exactly how the deployment of satellite constellations will affect the risk of collisions and the creation of debris fields. However, it is clear that the deployment of thousands of satellites into LEO will increase the risk of collisions and could potentially accelerate the Kessler syndrome. It is important for the space industry to take steps to mitigate these risks, such as improving space situational awareness, developing debris removal technologies, and implementing responsible space traffic management practices.
Raw
gpu.parallel.log
[1714506692] Log start
[1714506692] Cmd: llama.cpp/build/bin/parallel -m Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf -ngl 999 -c 32768 --parallel 4 --sequences 4 --seed 25519
[1714506695] warming up the model with an empty run
[1714506695] main: Simulating parallel requests from clients:
[1714506695] main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
[1714506695]
[1714506695] main: Evaluating the system prompt ...
[1714506695]
[1714506695] Processing requests ...
[1714506695] main: clearing the KV cache
[1714506695] Client 0, seq 0, started decoding ...
[1714506695] Client 1, seq 1, started decoding ...
[1714506695] Client 2, seq 2, started decoding ...
[1714506695] Client 3, seq 3, started decoding ...
[1714506695] main : decoded batch of 59 tokens
[1714506695] main : decoded batch of 4 tokens
[1714506695] main : decoded batch of 4 tokens
[1714506695] main : decoded batch of 4 tokens
[1714506695] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506696] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506697] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506698] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506699] main : decoded batch of 4 tokens
[1714506700] main : decoded batch of 4 tokens
[1714506700] main : decoded batch of 4 tokens
[1714506700] main : decoded batch of 4 tokens
[1714506700] main : decoded batch of 4 tokens
[1714506700] main : decoded batch of 4 tokens
[1714506700] Client 0, seq 0/ 4, prompt 11 t, response 102 t, time 4.64 s, speed 24.35 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has been debated throughout history. Some people believe that the meaning of life is to seek happiness, fulfillment, and personal growth. Others believe that the meaning of life is to serve a higher purpose, such as a god or a moral principle. Still others believe that the meaning of life is to create meaning for oneself through one's experiences and relationships. Ultimately, the meaning of life is a subjective concept that varies from person to person.</s>
[1714506700] main : decoded batch of 3 tokens
[1714506700] main : decoded batch of 3 tokens
[1714506700] main : decoded batch of 3 tokens
[1714506700] main : decoded batch of 3 tokens
[1714506700] main : decoded batch of 3 tokens
[1714506700] main : decoded batch of 3 tokens
[1714506700] main : decoded batch of 3 tokens
[1714506700] main : decoded batch of 3 tokens
[1714506700] main : decoded batch of 3 tokens
[1714506700] main : decoded batch of 3 tokens
[1714506700] main : decoded batch of 3 tokens
[1714506700] main : decoded batch of 3 tokens
[1714506700] Client 2, seq 2/ 4, prompt 11 t, response 114 t, time 5.10 s, speed 24.53 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has▅p been debated▅sp for centuries.▅▅▅▅▅sp There is no one answer that is true for everyone. Some people believe that the meaning of life is to be happy, while others believe that it is to be successful. Some people believe that the meaning of life is to make a difference in the world, while others believe that it is to find inner peace. Ultimately, the meaning of▅▅▅sp life is a▅sp personal matter that each individual must decide for themselves.
[1714506700] main : decoded batch of 2 tokens
[1714506700] main : decoded batch of 2 tokens
[1714506700] main : decoded batch of 2 tokens
[1714506700] main : decoded batch of 2 tokens
[1714506700] main : decoded batch of 2 tokens
[1714506700] main : decoded batch of 2 tokens
[1714506700] main : decoded batch of 2 tokens
[1714506700] main : decoded batch of 2 tokens
[1714506700] main : decoded batch of 2 tokens
[1714506700] main : decoded batch of 2 tokens
[1714506700] main : decoded batch of 2 tokens
[1714506701] main : decoded batch of 2 tokens
[1714506701] main : decoded batch of 2 tokens
[1714506701] main : decoded batch of 2 tokens
[1714506701] Client 3, seq 3/ 4, prompt 22 t, response 128 t, time 5.52 s, speed 27.17 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: Yes, I can▅ explain the Special Theory of▅Relativity▅Relativity to▅▅Relativity you▅▅▅▅▅Relativity. The Special Theory▅Relativity of Relativity is a▅▅Rel▅▅▅▅▅▅Rel▅Relativity▅▅▅▅▅▅Relativity▅▅▅▅Rel▅Rel▅▅▅▅▅▅▅▅Rel▅Relativity▅▅▅▅▅▅▅Relativity▅ ▅Rel▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅Rel▅Relativity was proposed by Albert Einstein in 190▅</s>
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506701] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506702] main : decoded batch of 1 tokens
[1714506703] main : decoded batch of 1 tokens
[1714506703] main : decoded batch of 1 tokens
[1714506703] main : decoded batch of 1 tokens
[1714506703] main : decoded batch of 1 tokens
[1714506703] main : decoded batch of 1 tokens
[1714506703] main : decoded batch of 1 tokens
[1714506703] main : decoded batch of 1 tokens
[1714506703] main : decoded batch of 1 tokens
[1714506703] main : decoded batch of 1 tokens
[1714506703] main : decoded batch of 1 tokens
[1714506703] main : decoded batch of 1 tokens
[1714506703] main : decoded batch of 1 tokens
[1714506703] main : decoded batch of 1 tokens
[1714506703] Client 1, seq 1/ 4, prompt 15 t, response 250 t, time 7.67 s, speed 34.53 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: The best way to▅: 1. Pre▅: heat the oven▅: to 3▅▅▅▅▅▅▅▅▅:▅: ▅▅▅: 7▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅:▅▅▅▅: 5▅▅▅▅▅▅▅▅▅▅▅:▅▅: ▅▅▅▅:▅:▅▅▅▅▅▅▅▅▅▅▅▅▅: 200°F. 2. Season the steak with salt and pepper. 3. Heat a cast iron skillet over high heat. 4. Add a small amount of oil to the skillet. 5. When the oil is hot, add the steak to the skillet. 6. Sear the steak for 2-3 minutes on each side. 7. Transfer the skillet to the preheated oven. 8. Cook the steak in the oven for 5-7 minutes for medium-rare. 9. Remove the steak from the oven and let it rest for 5-10 minutes before slicing it. 10. Enjoy!
[1714506703] main: clearing the KV cache
[1714506703]
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
[1714506703] External prompt file: used built-in defaults
[1714506703] Model and path used: Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf
[1714506703] Total prompt tokens: 59, speed: 7.32 t/s
[1714506703] Total gen tokens: 594, speed: 73.67 t/s
[1714506703] Total speed (AVG): speed: 80.99 t/s
[1714506703] Cache misses: 0
[1714506703]
Raw
gpu.parallel.out
Log start
llama_model_loader: loaded meta data with 27 key-value pairs and 323 tensors from Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = .
llama_model_loader: - kv 2: llama.vocab_size u32 = 32001
llama_model_loader: - kv 3: llama.context_length u32 = 32768
llama_model_loader: - kv 4: llama.embedding_length u32 = 4096
llama_model_loader: - kv 5: llama.block_count u32 = 32
llama_model_loader: - kv 6: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 7: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 8: llama.attention.head_count u32 = 32
llama_model_loader: - kv 9: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 10: llama.expert_count u32 = 8
llama_model_loader: - kv 11: llama.expert_used_count u32 = 2
llama_model_loader: - kv 12: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 13: llama.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 14: general.file_type u32 = 15
llama_model_loader: - kv 15: tokenizer.ggml.model str = llama
llama_model_loader: - kv 16: tokenizer.ggml.tokens arr[str,32001] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv 17: tokenizer.ggml.scores arr[f32,32001] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,32001] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 22: tokenizer.ggml.padding_token_id u32 = 32000
llama_model_loader: - kv 23: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 24: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 25: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
llama_model_loader: - kv 26: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type f16: 32 tensors
llama_model_loader: - type q8_0: 64 tensors
llama_model_loader: - type q4_K: 113 tensors
llama_model_loader: - type q5_K: 32 tensors
llama_model_loader: - type q6_K: 17 tensors
llm_load_vocab: special tokens definition check successful ( 260/32001 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32001
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 8
llm_load_print_meta: n_expert_used = 2
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 8x7B
llm_load_print_meta: model ftype = Q4_K - Medium
llm_load_print_meta: model params = 46.70 B
llm_load_print_meta: model size = 26.49 GiB (4.87 BPW)
llm_load_print_meta: general.name = .
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: PAD token = 32000 '[PAD]'
llm_load_print_meta: LF token = 13 '<0x0A>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size = 0.32 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors: CPU buffer size = 70.31 MiB
llm_load_tensors: CUDA0 buffer size = 27057.56 MiB
.............................................................................................
llama_new_context_with_model: n_ctx = 32768
llama_new_context_with_model: n_batch = 2048
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA0 KV buffer size = 4096.00 MiB
llama_new_context_with_model: KV self size = 4096.00 MiB, K (f16): 2048.00 MiB, V (f16): 2048.00 MiB
llama_new_context_with_model: CUDA_Host output buffer size = 0.61 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 2144.00 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 72.01 MiB
llama_new_context_with_model: graph nodes = 1510
llama_new_context_with_model: graph splits = 2
main: Simulating parallel requests from clients:
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
main: Evaluating the system prompt ...
Processing requests ...
main: clearing the KV cache
Client 0, seq 0, started decoding ...
Client 1, seq 1, started decoding ...
Client 2, seq 2, started decoding ...
Client 3, seq 3, started decoding ...
Client 0, seq 0/ 4, prompt 11 t, response 102 t, time 4.64 s, speed 24.35 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has been debated throughout history. Some people believe that the meaning of life is to seek happiness, fulfillment, and personal growth. Others believe that the meaning of life is to serve a higher purpose, such as a god or a moral principle. Still others believe that the meaning of life is to create meaning for oneself through one's experiences and relationships. Ultimately, the meaning of life is a subjective concept that varies from person to person.</s>
Client 2, seq 2/ 4, prompt 11 t, response 114 t, time 5.10 s, speed 24.53 t/s, cache miss 0 
Input: What is the meaning of life?
Response: The meaning of life is a philosophical question that has▅p been debated▅sp for centuries.▅▅▅▅▅sp There is no one answer that is true for everyone. Some people believe that the meaning of life is to be happy, while others believe that it is to be successful. Some people believe that the meaning of life is to make a difference in the world, while others believe that it is to find inner peace. Ultimately, the meaning of▅▅▅sp life is a▅sp personal matter that each individual must decide for themselves.
Client 3, seq 3/ 4, prompt 22 t, response 128 t, time 5.52 s, speed 27.17 t/s, cache miss 0 
Input: Are you familiar with the Special Theory of Relativity and can you explain it to me?
Response: Yes, I can▅ explain the Special Theory of▅Relativity▅Relativity to▅▅Relativity you▅▅▅▅▅Relativity. The Special Theory▅Relativity of Relativity is a▅▅Rel▅▅▅▅▅▅Rel▅Relativity▅▅▅▅▅▅Relativity▅▅▅▅Rel▅Rel▅▅▅▅▅▅▅▅Rel▅Relativity▅▅▅▅▅▅▅Relativity▅ ▅Rel▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅Rel▅Relativity was proposed by Albert Einstein in 190▅</s>
Client 1, seq 1/ 4, prompt 15 t, response 250 t, time 7.67 s, speed 34.53 t/s, cache miss 0 
Input: What is the best way to cook a steak?
Response: The best way to▅: 1. Pre▅: heat the oven▅: to 3▅▅▅▅▅▅▅▅▅:▅: ▅▅▅: 7▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅:▅▅▅▅: 5▅▅▅▅▅▅▅▅▅▅▅:▅▅: ▅▅▅▅:▅:▅▅▅▅▅▅▅▅▅▅▅▅▅: 200°F. 2. Season the steak with salt and pepper. 3. Heat a cast iron skillet over high heat. 4. Add a small amount of oil to the skillet. 5. When the oil is hot, add the steak to the skillet. 6. Sear the steak for 2-3 minutes on each side. 7. Transfer the skillet to the preheated oven. 8. Cook the steak in the oven for 5-7 minutes for medium-rare. 9. Remove the steak from the oven and let it rest for 5-10 minutes before slicing it. 10. Enjoy!
main: clearing the KV cache
main: n_parallel = 4, n_sequences = 4, cont_batching = 1, system tokens = 299
External prompt file: used built-in defaults
Model and path used: Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf
Total prompt tokens: 59, speed: 7.32 t/s
Total gen tokens: 594, speed: 73.67 t/s
Total speed (AVG): speed: 80.99 t/s
Cache misses: 0
llama_print_timings: load time = 2503.70 ms
llama_print_timings: sample time = 9.93 ms / 598 runs ( 0.02 ms per token, 60233.68 tokens per second)
llama_print_timings: prompt eval time = 5834.77 ms / 830 tokens ( 7.03 ms per token, 142.25 tokens per second)
llama_print_timings: eval time = 2136.28 ms / 122 runs ( 17.51 ms per token, 57.11 tokens per second)
llama_print_timings: total time = 8063.81 ms / 952 tokens
No new questions so proceed with build-in defaults.
run parameters as at 2024-04-30 15:51:43
Raw
gpu.tasks.txt
# TASK 0
Timeout after 300 seconds
# TASK 1d
The Kessler syndrome, also known as the Kessler effect, is a scenario in which the density of objects in low Earth orbit (LEO) is high enough that collisions between objects could cause a cascade of additional collisions, creating a debris field that could make LEO unusable for future generations. Satellite constellations, such as SpaceX's Starlink, could potentially accelerate the Kessler syndrome and pose a significant issue in the future.
Starlink is a satellite constellation consisting of thousands of satellites in LEO, designed to provide global high-speed internet coverage. The large number of satellites in the constellation increases the likelihood of collisions with other satellites, spent rocket stages, and other debris in LEO. Each collision creates more debris, which increases the risk of further collisions. This chain reaction could lead to the Kessler syndrome.
Additionally, the short lifespan of Starlink satellites (approximately 5 years) means that a large number of satellites will need to be launched and deorbited regularly. The deorbiting process involves firing the satellite's engines to slow it down and cause it to re-enter the Earth's atmosphere, where it will burn up. However, if the engine fails or the satellite is not deorbited properly, it could become another piece of debris in LEO.
Furthermore, the deployment of satellite constellations at a large scale may also make it more difficult to track and avoid collisions with existing debris. The US Space Surveillance Network (SSN) currently tracks about 27,000 objects in LEO, but it is not able to track all smaller debris. The addition of thousands of satellites to LEO could make it harder to monitor and avoid collisions, increasing the risk of the Kessler syndrome.
In conclusion, satellite constellations such as Starlink have the potential to accelerate the Kessler syndrome and pose a significant issue in the future. The large number of satellites, short lifespan, and the difficulty in tracking and avoiding collisions with existing debris increase the risk of a cascading debris field in LEO. It is important for the space industry to take steps to minimize the risk of the Kessler syndrome, such as improving debris tracking and mitigation efforts, and designing satellites to safely deorbit at the end of their lifespan.
# TASK 2
Timeout after 300 seconds
# TASK 3
The Kessler syndrome,▅ing named after NASA scientist▅▅▅ Donald J. Kessler, describes a scenario▅▅▅▅ which the density of objects in low Earth▅ orbit (LEO) is▅ such that collisions between objects could cause a cascade of additional collisions, leading▅ to an exponential increase in▅ the number▅▅ of▅ debris▅ and▅▅ the▅ potential▅ to▅▅▅ render▅ space activities in▅▅▅ LEO▅▅▅▅ and▅▅▅▅▅▅▅▅▅▅
ge▅▅▅▅▅▅▅▅▅
▅▅▅▅▅▅
[PAD]▅
▅
▅▅▅▅
▅
c▅
o
n
s
t
e
l
l
a
▅
t
i
▅
on
s▅
Raw
parallel-server-request-biased.py
"""Parallel server request test script with logit bias."""
import json
import time
from multiprocessing import Process
from pathlib import Path
from typing import Final
import requests
NUM_INSTANCES: Final[int] = 4
LLM_SERVER_URL: Final[str] = "http://localhost:8510"
CONNECT_TIMEOUT: float = 300.0
RESPONSE_TIMEOUT: float = 300.0
MAXIMUM_RUNTIME: int = 300
def print_health() -> None:
"""Prints server health."""
with requests.get(f"{LLM_SERVER_URL}/health", timeout=(CONNECT_TIMEOUT, RESPONSE_TIMEOUT)) as response:
print(response.text)
def task(id_: int) -> None:
"""Basic query task.
Args:
id_ (int): task id
"""
payload: dict = {
"model": "",
"messages": [
{
"role": "user",
"content": (
"Describe how satellite constellations such as Starlink accelerate the Kessler syndrome "
"and could pose a significant issue in the future"
),
},
],
"logit_bias": [(0, False)],
"stream": True,
}
try:
with requests.post(
f"{LLM_SERVER_URL}/v1/chat/completions",
json=payload,
timeout=(CONNECT_TIMEOUT, RESPONSE_TIMEOUT),
) as response:
for line in response.iter_lines():
if not line:
continue
delta: dict = json.loads(line[len("data: ") :])["choices"][0]["delta"]
if "content" in delta:
with Path(f"task-{id_}.txt").open("a", encoding="utf-8") as file:
file.write(delta["content"])
except requests.Timeout:
with Path(f"task-{id_}.txt").open("w", encoding="utf-8") as file:
file.write(f"Timeout after {RESPONSE_TIMEOUT} seconds")
processes: list[tuple[int, Process]] = [(i, Process(target=task, args=(i,))) for i in range(NUM_INSTANCES)]
for i, process in processes:
process.start()
print("Task", i, "started")
time.sleep(1.0)
print("0 : ", end="")
print_health()
elapsed_seconds: int = 0
completed: list[bool] = [False] * len(processes)
while True:
if all(completed):
break
for i, process in processes:
if not process.is_alive() and not completed[i]:
print("Task ", i, " finished\n", elapsed_seconds, end=" : ", sep="")
print_health()
completed = [not thread.is_alive() for _i, thread in processes]
time.sleep(1)
elapsed_seconds += 1
if elapsed_seconds % 30 == 0:
print(elapsed_seconds, end=" : ")
print_health()
if elapsed_seconds > MAXIMUM_RUNTIME:
print(f"Time limit of {MAXIMUM_RUNTIME} seconds reached, killing running tasks")
for i, process in processes:
if process.is_alive():
print("Killing task", i)
process.terminate()
break
with Path("tasks.txt").open("w", encoding="utf-8") as file:
for id_, _process in processes:
task_file_path = Path(f"task-{id_}.txt")
if not task_file_path.exists():
file.write(f"# TASK {id_}\n\n")
file.write(f"Task failed to receive response within {MAXIMUM_RUNTIME} seconds")
file.write("\n\n")
continue
with task_file_path.open("r", encoding="utf-8") as task_file:
file.write(f"# TASK {id_}\n\n")
file.write(task_file.read())
file.write("\n\n")
task_file_path.unlink()
Raw
parallel-server-request.py
"""Parallel server request test script."""
import json
import time
from multiprocessing import Process
from pathlib import Path
from typing import Final
import requests
NUM_INSTANCES: Final[int] = 4
LLM_SERVER_URL: Final[str] = "http://localhost:8510"
CONNECT_TIMEOUT: float = 300.0
RESPONSE_TIMEOUT: float = 300.0
MAXIMUM_RUNTIME: int = 300
def print_health() -> None:
"""Prints server health."""
with requests.get(f"{LLM_SERVER_URL}/health", timeout=(CONNECT_TIMEOUT, RESPONSE_TIMEOUT)) as response:
print(response.text)
def task(id_: int) -> None:
"""Basic query task.
Args:
id_ (int): task id
"""
payload: dict = {
"model": "",
"messages": [
{
"role": "user",
"content": (
"Describe how satellite constellations such as Starlink accelerate the Kessler syndrome "
"and could pose a significant issue in the future"
),
},
],
"stream": True,
}
try:
with requests.post(
f"{LLM_SERVER_URL}/v1/chat/completions",
json=payload,
timeout=(CONNECT_TIMEOUT, RESPONSE_TIMEOUT),
) as response:
for line in response.iter_lines():
if not line:
continue
delta: dict = json.loads(line[len("data: ") :])["choices"][0]["delta"]
if "content" in delta:
with Path(f"task-{id_}.txt").open("a", encoding="utf-8") as file:
file.write(delta["content"])
except requests.Timeout:
with Path(f"task-{id_}.txt").open("w", encoding="utf-8") as file:
file.write(f"Timeout after {RESPONSE_TIMEOUT} seconds")
processes: list[tuple[int, Process]] = [(i, Process(target=task, args=(i,))) for i in range(NUM_INSTANCES)]
for i, process in processes:
process.start()
print("Task", i, "started")
time.sleep(1.0)
print("0 : ", end="")
print_health()
elapsed_seconds: int = 0
completed: list[bool] = [False] * len(processes)
while True:
if all(completed):
break
for i, process in processes:
if not process.is_alive() and not completed[i]:
print("Task ", i, " finished\n", elapsed_seconds, end=" : ", sep="")
print_health()
completed = [not thread.is_alive() for _i, thread in processes]
time.sleep(1)
elapsed_seconds += 1
if elapsed_seconds % 30 == 0:
print(elapsed_seconds, end=" : ")
print_health()
if elapsed_seconds > MAXIMUM_RUNTIME:
print(f"Time limit of {MAXIMUM_RUNTIME} seconds reached, killing running tasks")
for i, process in processes:
if process.is_alive():
print("Killing task", i)
process.terminate()
break
with Path("tasks.txt").open("w", encoding="utf-8") as file:
for id_, _process in processes:
task_file_path = Path(f"task-{id_}.txt")
if not task_file_path.exists():
file.write(f"# TASK {id_}\n\n")
file.write(f"Task failed to receive response within {MAXIMUM_RUNTIME} seconds")
file.write("\n\n")
continue
with task_file_path.open("r", encoding="utf-8") as task_file:
file.write(f"# TASK {id_}\n\n")
file.write(task_file.read())
file.write("\n\n")
task_file_path.unlink()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment