Skip to content

Instantly share code, notes, and snippets.

@Munsio

Munsio/gist:8dcae3ff9d58c0ccf25401287ea42619

Created May 8, 2024 08:34
Show Gist options
  • Save Munsio/8dcae3ff9d58c0ccf25401287ea42619 to your computer and use it in GitHub Desktop.
Save Munsio/8dcae3ff9d58c0ccf25401287ea42619 to your computer and use it in GitHub Desktop.
Download ZIP
deveval-ollama-output
Raw
gistfile1.txt
> sh ollama.sh
error: Cannot delete branch 'ollama' checked out at '/home/martin/Documents/projects/eval-dev-quality'
Already on 'ollama'
Your branch is up to date with 'origin/ollama'.
go install -v github.com/symflower/eval-dev-quality/...
go install -v github.com/vektra/mockery/v2@v2.40.3
go install -v gotest.tools/gotestsum@v1.11.0
eval-dev-quality install-tools
2024/05/08 10:29:01 Install "ollama" to "/home/martin/.eval-dev-quality/bin"
downloading 100% [======================================================================================================================================================================] (306/306 MB, 31 MB/s)
2024/05/08 10:29:12 $ chmod +x /home/martin/.eval-dev-quality/bin/ollama
2024/05/08 10:29:12 Checking "symflower" binary "/home/martin/.eval-dev-quality/bin/symflower" version
2024/05/08 10:29:12 $ /home/martin/.eval-dev-quality/bin/symflower version
symflower v35902 on linux/amd64
Machine ID: 047356bf1024a8f918e99450caf63ffe1992e23689ec7496be42f12c1b8089fa
Containerized: false
KernelVersion: 5.14.21-150500.55.59-default
PATH: /home/martin/.eval-dev-quality/bin:/home/martin/symflower/src/gitlab.nethead.at/symflower/symflower/bin:/home/martin/symflower/src/github.com/symflower/garden/bin:/home/martin/symflower/.devenv/kubectl@1.27.4/kubectl/bin:/home/martin/symflower/.devenv/nodejs@16.20.0/nodejs/bin:/home/martin/symflower/.devenv/maven@3.9.1/maven/bin:/home/martin/symflower/.devenv/java@11.0.21.9.1/java/bin:/home/martin/symflower/.devenv/helm@3.13.1/helm/bin:/home/martin/symflower/.devenv/gradle@8.0.2/gradle/bin:/home/martin/symflower/.devenv/java@11.0.21.9.1/java/bin:/home/martin/symflower/bin:/home/martin/symflower/.devenv/go@1.21.5/go/bin:/home/martin/symflower/bin:/home/martin/symflower/.devenv/go@1.21.5/go/bin:/home/martin/symflower/.devenv/pnpm:/home/martin/.local/bin:/home/martin/.local/bin:/home/martin/bin:/usr/local/bin:/usr/bin:/bin:/home/martin/symflower/conf/js/node_modules/.bin
ProductName: openSUSE Leap
ProductVersion: 15.5
Give us your feedback and let us know how we can improve Symflower at hello@symflower.com or https://github.com/symflower/symflower. Thanks so much for your help!
2024/05/08 10:29:12 routes.go:989: INFO server config env="map[OLLAMA_DEBUG:false OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_VRAM:0 OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:*] OLLAMA_RUNNERS_DIR: OLLAMA_TMPDIR:]"
time=2024-05-08T10:29:12.930+02:00 level=INFO source=images.go:897 msg="total blobs: 0"
time=2024-05-08T10:29:12.930+02:00 level=INFO source=images.go:904 msg="total unused blobs removed: 0"
time=2024-05-08T10:29:12.930+02:00 level=INFO source=routes.go:1034 msg="Listening on 127.0.0.1:11434 (version 0.1.34)"
time=2024-05-08T10:29:12.944+02:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama2315490584/runners
time=2024-05-08T10:29:15.621+02:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cuda_v11 rocm_v60002 cpu cpu_avx cpu_avx2]"
time=2024-05-08T10:29:15.621+02:00 level=INFO source=gpu.go:122 msg="Detecting GPUs"
time=2024-05-08T10:29:15.637+02:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
[GIN] 2024/05/08 - 10:29:15 | 200 | 53.684µs | 127.0.0.1 | HEAD "/"
pulling manifest ⠹ time=2024-05-08T10:29:18.209+02:00 level=INFO source=download.go:136 msg="downloading fad2a06e4cc7 in 4 100 MB part(s)"
pulling manifest
pulling fad2a06e4cc7... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 394 MB tpulling manifest
pulling fad2a06e4cc7... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 394 MB
pulling manifest
pulling fad2a06e4cc7... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 394 MB
pulling manifest
pulling fad2a06e4cc7... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 394 MB
pulling manifest
pulling manifest
pulling fad2a06e4cc7... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 394 MB
pulling 41c2cf8c272f... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 7.3 KB
pulling 1da0581fd4ce... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 130 B
pulling f02dd72bb242... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 59 B
pulling ea0a531a015b... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 485 B
verifying sha256 digest
writing manifest
removing any unused layers
success
[GIN] 2024/05/08 - 10:29:42 | 200 | 16.383µs | 127.0.0.1 | HEAD "/"
pulling manifest ⠧ time=2024-05-08T10:29:43.928+02:00 level=INFO source=download.go:136 msg="downloading 2af3b81862c6 in 7 100 MB part(s)"
pulling manifest
pulling 2af3b81862c6... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 637 MB tpulling manifest
pulling 2af3b81862c6... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 637 MB
pulling manifest
pulling 2af3b81862c6... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 637 MB
pulling manifest
pulling 2af3b81862c6... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 637 MB
pulling manifest
pulling manifest
pulling 2af3b81862c6... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 637 MB
pulling af0ddbdaaa26... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 70 B
pulling c8472cd9daed... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 31 B
pulling fa956ab37b8c... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 98 B
pulling 6331358be52a... 100% ▕████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 483 B
verifying sha256 digest
writing manifest
removing any unused layers
success
2024/05/08 10:30:16 Writing results to evaluation-2024-05-08-10:30:16
2024/05/08 10:30:16 skipping unavailable provider "openrouter" cause: missing access token
2024/05/08 10:30:16 $ ollama --version
Warning: could not connect to a running Ollama instance
Warning: client version is 0.1.34
2024/05/08 10:30:16 starting new Ollama server instance on "127.0.0.1:11434" using ollama
2024/05/08 10:30:16 $ ollama serve
2024/05/08 10:30:16 routes.go:989: INFO server config env="map[OLLAMA_DEBUG:false OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_VRAM:0 OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:*] OLLAMA_RUNNERS_DIR: OLLAMA_TMPDIR:]"
time=2024-05-08T10:30:16.421+02:00 level=INFO source=images.go:897 msg="total blobs: 10"
time=2024-05-08T10:30:16.421+02:00 level=INFO source=images.go:904 msg="total unused blobs removed: 0"
time=2024-05-08T10:30:16.421+02:00 level=INFO source=routes.go:1034 msg="Listening on 127.0.0.1:11434 (version 0.1.34)"
time=2024-05-08T10:30:16.467+02:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama1909733046/runners
time=2024-05-08T10:30:19.054+02:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [rocm_v60002 cpu cpu_avx cpu_avx2 cuda_v11]"
time=2024-05-08T10:30:19.054+02:00 level=INFO source=gpu.go:122 msg="Detecting GPUs"
time=2024-05-08T10:30:19.058+02:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
[GIN] 2024/05/08 - 10:30:19 | 200 | 24.723µs | 127.0.0.1 | GET "/"
[GIN] 2024/05/08 - 10:30:19 | 200 | 563.513µs | 127.0.0.1 | GET "/api/tags"
2024/05/08 10:30:19 $ symflower version
symflower v35902 on linux/amd64
Machine ID: 047356bf1024a8f918e99450caf63ffe1992e23689ec7496be42f12c1b8089fa
Containerized: false
KernelVersion: 5.14.21-150500.55.59-default
PATH: /home/martin/.eval-dev-quality/bin:/home/martin/symflower/src/gitlab.nethead.at/symflower/symflower/bin:/home/martin/symflower/src/github.com/symflower/garden/bin:/home/martin/symflower/.devenv/kubectl@1.27.4/kubectl/bin:/home/martin/symflower/.devenv/nodejs@16.20.0/nodejs/bin:/home/martin/symflower/.devenv/maven@3.9.1/maven/bin:/home/martin/symflower/.devenv/java@11.0.21.9.1/java/bin:/home/martin/symflower/.devenv/helm@3.13.1/helm/bin:/home/martin/symflower/.devenv/gradle@8.0.2/gradle/bin:/home/martin/symflower/.devenv/java@11.0.21.9.1/java/bin:/home/martin/symflower/bin:/home/martin/symflower/.devenv/go@1.21.5/go/bin:/home/martin/symflower/bin:/home/martin/symflower/.devenv/go@1.21.5/go/bin:/home/martin/symflower/.devenv/pnpm:/home/martin/.local/bin:/home/martin/.local/bin:/home/martin/bin:/usr/local/bin:/usr/bin:/bin:/home/martin/symflower/conf/js/node_modules/.bin
ProductName: openSUSE Leap
ProductVersion: 15.5
Give us your feedback and let us know how we can improve Symflower at hello@symflower.com or https://github.com/symflower/symflower. Thanks so much for your help!
2024/05/08 10:30:19 Checking "symflower" binary "/home/martin/.eval-dev-quality/bin/symflower" version
2024/05/08 10:30:19 $ /home/martin/.eval-dev-quality/bin/symflower version
symflower v35902 on linux/amd64
Machine ID: 047356bf1024a8f918e99450caf63ffe1992e23689ec7496be42f12c1b8089fa
Containerized: false
KernelVersion: 5.14.21-150500.55.59-default
PATH: /home/martin/.eval-dev-quality/bin:/home/martin/symflower/src/gitlab.nethead.at/symflower/symflower/bin:/home/martin/symflower/src/github.com/symflower/garden/bin:/home/martin/symflower/.devenv/kubectl@1.27.4/kubectl/bin:/home/martin/symflower/.devenv/nodejs@16.20.0/nodejs/bin:/home/martin/symflower/.devenv/maven@3.9.1/maven/bin:/home/martin/symflower/.devenv/java@11.0.21.9.1/java/bin:/home/martin/symflower/.devenv/helm@3.13.1/helm/bin:/home/martin/symflower/.devenv/gradle@8.0.2/gradle/bin:/home/martin/symflower/.devenv/java@11.0.21.9.1/java/bin:/home/martin/symflower/bin:/home/martin/symflower/.devenv/go@1.21.5/go/bin:/home/martin/symflower/bin:/home/martin/symflower/.devenv/go@1.21.5/go/bin:/home/martin/symflower/.devenv/pnpm:/home/martin/.local/bin:/home/martin/.local/bin:/home/martin/bin:/usr/local/bin:/usr/bin:/bin:/home/martin/symflower/conf/js/node_modules/.bin
ProductName: openSUSE Leap
ProductVersion: 15.5
Give us your feedback and let us know how we can improve Symflower at hello@symflower.com or https://github.com/symflower/symflower. Thanks so much for your help!
2024/05/08 10:30:19 Checking that models and languages can be used for evaluation
2024/05/08 10:30:19 Evaluating model "ollama/qwen:0.5b" using language "golang" and repository "golang/plain"
2024/05/08 10:30:19 Querying model "ollama/qwen:0.5b" with:
Given the following Go code file "plain.go" with package "plain", provide a test file for this code.
The tests should produce 100 percent code coverage and must compile.
The response must contain only the test code and nothing else.
```golang
package plain
func plain() {
return // This does not do anything but it gives us a line to cover.
}
```
time=2024-05-08T10:30:19.074+02:00 level=INFO source=gpu.go:122 msg="Detecting GPUs"
time=2024-05-08T10:30:19.078+02:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-05-08T10:30:19.548+02:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-05-08T10:30:19.548+02:00 level=INFO source=server.go:305 msg="starting llama server" cmd="/tmp/ollama1909733046/runners/cpu_avx2/ollama_llama_server --model /home/martin/.ollama/models/blobs/sha256-fad2a06e4cc705c2fa8bec5477ddb00dc0c859ac184c34dcc5586663774161ca --ctx-size 2048 --batch-size 512 --embedding --log-disable --parallel 1 --port 41963"
time=2024-05-08T10:30:19.549+02:00 level=INFO source=sched.go:326 msg="loaded runners" count=1
time=2024-05-08T10:30:19.549+02:00 level=INFO source=server.go:464 msg="waiting for llama runner to start responding"
{"function":"server_params_parse","level":"INFO","line":2604,"msg":"logging to file is disabled.","tid":"140534621321088","timestamp":1715157019}
{"build":1,"commit":"952d03d","function":"main","level":"INFO","line":2820,"msg":"build info","tid":"140534621321088","timestamp":1715157019}
{"function":"main","level":"INFO","line":2823,"msg":"system info","n_threads":4,"n_threads_batch":-1,"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | ","tid":"140534621321088","timestamp":1715157019,"total_threads":8}
llama_model_loader: loaded meta data with 20 key-value pairs and 291 tensors from /home/martin/.ollama/models/blobs/sha256-fad2a06e4cc705c2fa8bec5477ddb00dc0c859ac184c34dcc5586663774161ca (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen2
llama_model_loader: - kv 1: general.name str = Qwen2-beta-0_5B-Chat
llama_model_loader: - kv 2: qwen2.block_count u32 = 24
llama_model_loader: - kv 3: qwen2.context_length u32 = 32768
llama_model_loader: - kv 4: qwen2.embedding_length u32 = 1024
llama_model_loader: - kv 5: qwen2.feed_forward_length u32 = 2816
llama_model_loader: - kv 6: qwen2.attention.head_count u32 = 16
llama_model_loader: - kv 7: qwen2.attention.head_count_kv u32 = 16
llama_model_loader: - kv 8: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 9: qwen2.use_parallel_residual bool = true
llama_model_loader: - kv 10: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 11: tokenizer.ggml.tokens arr[str,151936] = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 12: tokenizer.ggml.token_type arr[i32,151936] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 13: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 14: tokenizer.ggml.eos_token_id u32 = 151643
llama_model_loader: - kv 15: tokenizer.ggml.padding_token_id u32 = 151643
llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 151643
llama_model_loader: - kv 17: tokenizer.chat_template str = {% for message in messages %}{% if lo...
llama_model_loader: - kv 18: general.quantization_version u32 = 2
llama_model_loader: - kv 19: general.file_type u32 = 2
llama_model_loader: - type f32: 121 tensors
llama_model_loader: - type q4_0: 169 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: missing pre-tokenizer type, using: 'default'
llm_load_vocab:
llm_load_vocab: ************************************
llm_load_vocab: GENERATION QUALITY WILL BE DEGRADED!
llm_load_vocab: CONSIDER REGENERATING THE MODEL
llm_load_vocab: ************************************
llm_load_vocab:
llm_load_vocab: special tokens definition check successful ( 293/151936 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = qwen2
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 151936
llm_load_print_meta: n_merges = 151387
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 1024
llm_load_print_meta: n_head = 16
llm_load_print_meta: n_head_kv = 16
llm_load_print_meta: n_layer = 24
llm_load_print_meta: n_rot = 64
llm_load_print_meta: n_embd_head_k = 64
llm_load_print_meta: n_embd_head_v = 64
llm_load_print_meta: n_gqa = 1
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-06
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 2816
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 2
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 0.5B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 619.57 M
llm_load_print_meta: model size = 371.02 MiB (5.02 BPW)
llm_load_print_meta: general.name = Qwen2-beta-0_5B-Chat
llm_load_print_meta: BOS token = 151643 '<|endoftext|>'
llm_load_print_meta: EOS token = 151643 '<|endoftext|>'
llm_load_print_meta: PAD token = 151643 '<|endoftext|>'
llm_load_print_meta: LF token = 148848 'ÄĬ'
llm_load_print_meta: EOT token = 151645 '<|im_end|>'
llm_load_tensors: ggml ctx size = 0.14 MiB
llm_load_tensors: CPU buffer size = 371.02 MiB
...............................................
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: n_batch = 512
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CPU KV buffer size = 192.00 MiB
llama_new_context_with_model: KV self size = 192.00 MiB, K (f16): 96.00 MiB, V (f16): 96.00 MiB
llama_new_context_with_model: CPU output buffer size = 0.58 MiB
llama_new_context_with_model: CPU compute buffer size = 298.75 MiB
llama_new_context_with_model: graph nodes = 846
llama_new_context_with_model: graph splits = 1
{"function":"initialize","level":"INFO","line":448,"msg":"initializing slots","n_slots":1,"tid":"140534621321088","timestamp":1715157020}
{"function":"initialize","level":"INFO","line":457,"msg":"new slot","n_ctx_slot":2048,"slot_id":0,"tid":"140534621321088","timestamp":1715157020}
{"function":"main","level":"INFO","line":3065,"msg":"model loaded","tid":"140534621321088","timestamp":1715157020}
{"function":"main","hostname":"127.0.0.1","level":"INFO","line":3268,"msg":"HTTP server listening","n_threads_http":"7","port":"41963","tid":"140534621321088","timestamp":1715157020}
{"function":"update_slots","level":"INFO","line":1579,"msg":"all slots are idle and system prompt is empty, clear the KV cache","tid":"140534621321088","timestamp":1715157020}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":0,"tid":"140534621321088","timestamp":1715157020}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":1,"tid":"140534621321088","timestamp":1715157020}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":43032,"status":200,"tid":"140533679843072","timestamp":1715157020}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":43034,"status":200,"tid":"140533663057664","timestamp":1715157020}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":2,"tid":"140534621321088","timestamp":1715157020}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":43038,"status":200,"tid":"140533654664960","timestamp":1715157020}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":3,"tid":"140534621321088","timestamp":1715157020}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":43048,"status":200,"tid":"140534621316864","timestamp":1715157020}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":4,"tid":"140534621321088","timestamp":1715157020}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":43048,"status":200,"tid":"140534621316864","timestamp":1715157020}
{"function":"log_server_request","level":"INFO","line":2735,"method":"POST","msg":"request","params":{},"path":"/tokenize","remote_addr":"127.0.0.1","remote_port":43048,"status":200,"tid":"140534621316864","timestamp":1715157020}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":5,"tid":"140534621321088","timestamp":1715157020}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":43048,"status":200,"tid":"140534621316864","timestamp":1715157020}
{"function":"launch_slot_with_data","level":"INFO","line":830,"msg":"slot is processing task","slot_id":0,"task_id":6,"tid":"140534621321088","timestamp":1715157020}
{"function":"update_slots","ga_i":0,"level":"INFO","line":1810,"msg":"slot progression","n_past":0,"n_past_se":0,"n_prompt_tokens_processed":102,"slot_id":0,"task_id":6,"tid":"140534621321088","timestamp":1715157020}
{"function":"update_slots","level":"INFO","line":1837,"msg":"kv cache rm [p0, end)","p0":0,"slot_id":0,"task_id":6,"tid":"140534621321088","timestamp":1715157020}
{"function":"print_timings","level":"INFO","line":269,"msg":"prompt eval time = 458.48 ms / 102 tokens ( 4.49 ms per token, 222.48 tokens per second)","n_prompt_tokens_processed":102,"n_tokens_second":222.47620377075356,"slot_id":0,"t_prompt_processing":458.476,"t_token":4.49486274509804,"task_id":6,"tid":"140534621321088","timestamp":1715157023}
{"function":"print_timings","level":"INFO","line":283,"msg":"generation eval time = 2353.97 ms / 93 runs ( 25.31 ms per token, 39.51 tokens per second)","n_decoded":93,"n_tokens_second":39.50780916453728,"slot_id":0,"t_token":25.311451612903227,"t_token_generation":2353.965,"task_id":6,"tid":"140534621321088","timestamp":1715157023}
{"function":"print_timings","level":"INFO","line":293,"msg":" total time = 2812.44 ms","slot_id":0,"t_prompt_processing":458.476,"t_token_generation":2353.965,"t_total":2812.4410000000003,"task_id":6,"tid":"140534621321088","timestamp":1715157023}
{"function":"update_slots","level":"INFO","line":1641,"msg":"slot released","n_cache_tokens":195,"n_ctx":2048,"n_past":194,"n_system_tokens":0,"slot_id":0,"task_id":6,"tid":"140534621321088","timestamp":1715157023,"truncated":false}
{"function":"log_server_request","level":"INFO","line":2735,"method":"POST","msg":"request","params":{},"path":"/completion","remote_addr":"127.0.0.1","remote_port":43048,"status":200,"tid":"140534621316864","timestamp":1715157023}
[GIN] 2024/05/08 - 10:30:23 | 200 | 4.044795593s | 127.0.0.1 | POST "/v1/chat/completions"
2024/05/08 10:30:23 Model "ollama/qwen:0.5b" responded with:
Here is an example test file for `plain`. The test will cover all the functions and methods provided in the `plain` package.
```
package main
import (
"fmt"
"os/exec"
)
func main() {
// Your code here goes...
}
```
This test file covers all the functions and methods provided in the `plain` package. It also checks if any of the functions and methods provide any errors when running the test code.
2024/05/08 10:30:23 $ symflower test --language golang --workspace /tmp/eval-dev-quality3460370307/plain
Checking for updates
There is a new version of symflower available! Please run `symflower update`.
found packages plain (plain.go) and main (plain_test.go) in /tmp/eval-dev-quality3460370307/plain
=== Errors
found packages plain (plain.go) and main (plain_test.go) in /tmp/eval-dev-quality3460370307/plain
DONE 0 tests, 1 error in 0.026s
Give us your feedback and let us know how we can improve Symflower at hello@symflower.com or https://github.com/symflower/symflower. Thanks so much for your help!
ERROR: execution failed: exit status 1
2024/05/08 10:30:23 Evaluated model "ollama/qwen:0.5b" using language "golang" and repository "golang/plain": encountered 1 problems: [exit status 1
github.com/symflower/eval-dev-quality/util.CommandWithResult
/home/martin/Documents/projects/eval-dev-quality/util/exec.go:39
github.com/symflower/eval-dev-quality/language/golang.(*Language).Execute
/home/martin/Documents/projects/eval-dev-quality/language/golang/language.go:85
github.com/symflower/eval-dev-quality/evaluate.Repository
/home/martin/Documents/projects/eval-dev-quality/evaluate/repository.go:68
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:233
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
/home/martin/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
github.com/symflower/eval-dev-quality/language/golang.(*Language).Execute
/home/martin/Documents/projects/eval-dev-quality/language/golang/language.go:95
github.com/symflower/eval-dev-quality/evaluate.Repository
/home/martin/Documents/projects/eval-dev-quality/evaluate/repository.go:68
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:233
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
/home/martin/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
plain.go]
2024/05/08 10:30:23 Excluding model "ollama/qwen:0.5b" since it was not able to solve the "golang/plain" repository for language "golang": [exit status 1
github.com/symflower/eval-dev-quality/util.CommandWithResult
/home/martin/Documents/projects/eval-dev-quality/util/exec.go:39
github.com/symflower/eval-dev-quality/language/golang.(*Language).Execute
/home/martin/Documents/projects/eval-dev-quality/language/golang/language.go:85
github.com/symflower/eval-dev-quality/evaluate.Repository
/home/martin/Documents/projects/eval-dev-quality/evaluate/repository.go:68
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:233
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
/home/martin/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
github.com/symflower/eval-dev-quality/language/golang.(*Language).Execute
/home/martin/Documents/projects/eval-dev-quality/language/golang/language.go:95
github.com/symflower/eval-dev-quality/evaluate.Repository
/home/martin/Documents/projects/eval-dev-quality/evaluate/repository.go:68
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:233
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
/home/martin/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
plain.go]
2024/05/08 10:30:23 Evaluating model "ollama/tinyllama:latest" using language "golang" and repository "golang/plain"
2024/05/08 10:30:23 Querying model "ollama/tinyllama:latest" with:
Given the following Go code file "plain.go" with package "plain", provide a test file for this code.
The tests should produce 100 percent code coverage and must compile.
The response must contain only the test code and nothing else.
```golang
package plain
func plain() {
return // This does not do anything but it gives us a line to cover.
}
```
time=2024-05-08T10:30:24.018+02:00 level=INFO source=gpu.go:122 msg="Detecting GPUs"
time=2024-05-08T10:30:24.026+02:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-05-08T10:30:24.219+02:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-05-08T10:30:24.219+02:00 level=INFO source=server.go:305 msg="starting llama server" cmd="/tmp/ollama1909733046/runners/cpu_avx2/ollama_llama_server --model /home/martin/.ollama/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 --ctx-size 2048 --batch-size 512 --embedding --log-disable --parallel 1 --port 41035"
time=2024-05-08T10:30:24.219+02:00 level=INFO source=sched.go:326 msg="loaded runners" count=1
time=2024-05-08T10:30:24.219+02:00 level=INFO source=server.go:464 msg="waiting for llama runner to start responding"
{"function":"server_params_parse","level":"INFO","line":2604,"msg":"logging to file is disabled.","tid":"139652420855680","timestamp":1715157024}
{"build":1,"commit":"952d03d","function":"main","level":"INFO","line":2820,"msg":"build info","tid":"139652420855680","timestamp":1715157024}
{"function":"main","level":"INFO","line":2823,"msg":"system info","n_threads":4,"n_threads_batch":-1,"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | ","tid":"139652420855680","timestamp":1715157024,"total_threads":8}
llama_model_loader: loaded meta data with 23 key-value pairs and 201 tensors from /home/martin/.ollama/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = TinyLlama
llama_model_loader: - kv 2: llama.context_length u32 = 2048
llama_model_loader: - kv 3: llama.embedding_length u32 = 2048
llama_model_loader: - kv 4: llama.block_count u32 = 22
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 5632
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 64
llama_model_loader: - kv 7: llama.attention.head_count u32 = 32
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 4
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 10: llama.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 11: general.file_type u32 = 2
llama_model_loader: - kv 12: tokenizer.ggml.model str = llama
llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 16: tokenizer.ggml.merges arr[str,61249] = ["▁ t", "e r", "i n", "▁ a", "e n...
llama_model_loader: - kv 17: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 18: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 19: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 20: tokenizer.ggml.padding_token_id u32 = 2
llama_model_loader: - kv 21: tokenizer.chat_template str = {% for message in messages %}\n{% if m...
llama_model_loader: - kv 22: general.quantization_version u32 = 2
llama_model_loader: - type f32: 45 tensors
llama_model_loader: - type q4_0: 155 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32000
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 2048
llm_load_print_meta: n_embd = 2048
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 4
llm_load_print_meta: n_layer = 22
llm_load_print_meta: n_rot = 64
llm_load_print_meta: n_embd_head_k = 64
llm_load_print_meta: n_embd_head_v = 64
llm_load_print_meta: n_gqa = 8
llm_load_print_meta: n_embd_k_gqa = 256
llm_load_print_meta: n_embd_v_gqa = 256
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 5632
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 2048
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 1B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 1.10 B
llm_load_print_meta: model size = 606.53 MiB (4.63 BPW)
llm_load_print_meta: general.name = TinyLlama
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: PAD token = 2 '</s>'
llm_load_print_meta: LF token = 13 '<0x0A>'
llm_load_tensors: ggml ctx size = 0.10 MiB
llm_load_tensors: CPU buffer size = 606.53 MiB
.....................................................................................
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: n_batch = 512
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CPU KV buffer size = 44.00 MiB
llama_new_context_with_model: KV self size = 44.00 MiB, K (f16): 22.00 MiB, V (f16): 22.00 MiB
llama_new_context_with_model: CPU output buffer size = 0.13 MiB
llama_new_context_with_model: CPU compute buffer size = 148.01 MiB
llama_new_context_with_model: graph nodes = 710
llama_new_context_with_model: graph splits = 1
{"function":"initialize","level":"INFO","line":448,"msg":"initializing slots","n_slots":1,"tid":"139652420855680","timestamp":1715157024}
{"function":"initialize","level":"INFO","line":457,"msg":"new slot","n_ctx_slot":2048,"slot_id":0,"tid":"139652420855680","timestamp":1715157024}
{"function":"main","level":"INFO","line":3065,"msg":"model loaded","tid":"139652420855680","timestamp":1715157024}
{"function":"main","hostname":"127.0.0.1","level":"INFO","line":3268,"msg":"HTTP server listening","n_threads_http":"7","port":"41035","tid":"139652420855680","timestamp":1715157024}
{"function":"update_slots","level":"INFO","line":1579,"msg":"all slots are idle and system prompt is empty, clear the KV cache","tid":"139652420855680","timestamp":1715157024}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":0,"tid":"139652420855680","timestamp":1715157024}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":36836,"status":200,"tid":"139651550799616","timestamp":1715157024}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":1,"tid":"139652420855680","timestamp":1715157024}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":36836,"status":200,"tid":"139651550799616","timestamp":1715157024}
{"function":"log_server_request","level":"INFO","line":2735,"method":"POST","msg":"request","params":{},"path":"/tokenize","remote_addr":"127.0.0.1","remote_port":36836,"status":200,"tid":"139651550799616","timestamp":1715157024}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":2,"tid":"139652420855680","timestamp":1715157024}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":36836,"status":200,"tid":"139651550799616","timestamp":1715157024}
{"function":"launch_slot_with_data","level":"INFO","line":830,"msg":"slot is processing task","slot_id":0,"task_id":3,"tid":"139652420855680","timestamp":1715157024}
{"function":"update_slots","ga_i":0,"level":"INFO","line":1810,"msg":"slot progression","n_past":0,"n_past_se":0,"n_prompt_tokens_processed":123,"slot_id":0,"task_id":3,"tid":"139652420855680","timestamp":1715157024}
{"function":"update_slots","level":"INFO","line":1837,"msg":"kv cache rm [p0, end)","p0":0,"slot_id":0,"task_id":3,"tid":"139652420855680","timestamp":1715157024}
{"function":"print_timings","level":"INFO","line":269,"msg":"prompt eval time = 1547.82 ms / 123 tokens ( 12.58 ms per token, 79.47 tokens per second)","n_prompt_tokens_processed":123,"n_tokens_second":79.46639928053835,"slot_id":0,"t_prompt_processing":1547.824,"t_token":12.583934959349595,"task_id":3,"tid":"139652420855680","timestamp":1715157036}
{"function":"print_timings","level":"INFO","line":283,"msg":"generation eval time = 10957.91 ms / 272 runs ( 40.29 ms per token, 24.82 tokens per second)","n_decoded":272,"n_tokens_second":24.82224942327055,"slot_id":0,"t_token":40.2864375,"t_token_generation":10957.911,"task_id":3,"tid":"139652420855680","timestamp":1715157036}
{"function":"print_timings","level":"INFO","line":293,"msg":" total time = 12505.74 ms","slot_id":0,"t_prompt_processing":1547.824,"t_token_generation":10957.911,"t_total":12505.735,"task_id":3,"tid":"139652420855680","timestamp":1715157036}
{"function":"update_slots","level":"INFO","line":1641,"msg":"slot released","n_cache_tokens":395,"n_ctx":2048,"n_past":394,"n_system_tokens":0,"slot_id":0,"task_id":3,"tid":"139652420855680","timestamp":1715157036,"truncated":false}
{"function":"log_server_request","level":"INFO","line":2735,"method":"POST","msg":"request","params":{},"path":"/completion","remote_addr":"127.0.0.1","remote_port":36836,"status":200,"tid":"139651550799616","timestamp":1715157036}
[GIN] 2024/05/08 - 10:30:36 | 200 | 12.942125466s | 127.0.0.1 | POST "/v1/chat/completions"
2024/05/08 10:30:36 Model "ollama/tinyllama:latest" responded with:
To test the "plain" package in Go, we need a separate file called "test.go" with only one function named "PlainTest", which must follow the same style and format as the "plain" package itself:
```go
package plain
import (
"fmt" // This imports fmt library for formatting output.
)
// PlainTest tests the "plain" package.
func PlainTest() {
fmt.Println("PlainTest")
}
```
After importing the "plain" package and running "go test", we will get a pass in coverage and a clean console output:
```
$ go test -coverprofile=coverage.out -covermode=atomic
go: downloading github.com/mdempsky/gotestsupport v0.1.2...
PlainTest
```
If the tests pass, the code coverage should be 100% and we'll see the output of our test function in the console. If the code coverage does not match, it means there are untested or uncovered lines in the code, and you can make changes to the code to make them tested using the `gotest` command or a tool such as Cover for Golang.
2024/05/08 10:30:36 $ symflower test --language golang --workspace /tmp/eval-dev-quality3603400511/plain
testing: warning: no tests to run
PASS
coverage: 0.0% of statements
ok plain 0.001s coverage: 0.0% of statements [no tests to run]
DONE 0 tests in 0.399s
Give us your feedback and let us know how we can improve Symflower at hello@symflower.com or https://github.com/symflower/symflower. Thanks so much for your help!
2024/05/08 10:30:37 Evaluated model "ollama/tinyllama:latest" using language "golang" and repository "golang/plain": encountered 1 problems: [no tests could be found
github.com/symflower/eval-dev-quality/language/golang.(*Language).Execute
/home/martin/Documents/projects/eval-dev-quality/language/golang/language.go:106
github.com/symflower/eval-dev-quality/evaluate.Repository
/home/martin/Documents/projects/eval-dev-quality/evaluate/repository.go:68
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:233
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
/home/martin/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
plain.go]
2024/05/08 10:30:37 Excluding model "ollama/tinyllama:latest" since it was not able to solve the "golang/plain" repository for language "golang": [no tests could be found
github.com/symflower/eval-dev-quality/language/golang.(*Language).Execute
/home/martin/Documents/projects/eval-dev-quality/language/golang/language.go:106
github.com/symflower/eval-dev-quality/evaluate.Repository
/home/martin/Documents/projects/eval-dev-quality/evaluate/repository.go:68
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:233
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
/home/martin/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
plain.go]
2024/05/08 10:30:37 Evaluating model "ollama/qwen:0.5b" using language "java" and repository "java/plain"
2024/05/08 10:30:37 Querying model "ollama/qwen:0.5b" with:
Given the following Java code file "src/main/java/com/eval/Plain.java" with package "com.eval", provide a test file for this code with JUnit 5 as a test framework.
The tests should produce 100 percent code coverage and must compile.
The response must contain only the test code and nothing else.
```java
package com.eval;
class Plain {
static void plain() {
}
}
```
time=2024-05-08T10:30:37.527+02:00 level=INFO source=gpu.go:122 msg="Detecting GPUs"
time=2024-05-08T10:30:37.532+02:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-05-08T10:30:38.154+02:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-05-08T10:30:38.155+02:00 level=INFO source=server.go:305 msg="starting llama server" cmd="/tmp/ollama1909733046/runners/cpu_avx2/ollama_llama_server --model /home/martin/.ollama/models/blobs/sha256-fad2a06e4cc705c2fa8bec5477ddb00dc0c859ac184c34dcc5586663774161ca --ctx-size 2048 --batch-size 512 --embedding --log-disable --parallel 1 --port 45079"
time=2024-05-08T10:30:38.155+02:00 level=INFO source=sched.go:326 msg="loaded runners" count=1
time=2024-05-08T10:30:38.155+02:00 level=INFO source=server.go:464 msg="waiting for llama runner to start responding"
{"function":"server_params_parse","level":"INFO","line":2604,"msg":"logging to file is disabled.","tid":"140509500426112","timestamp":1715157038}
{"build":1,"commit":"952d03d","function":"main","level":"INFO","line":2820,"msg":"build info","tid":"140509500426112","timestamp":1715157038}
{"function":"main","level":"INFO","line":2823,"msg":"system info","n_threads":4,"n_threads_batch":-1,"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | ","tid":"140509500426112","timestamp":1715157038,"total_threads":8}
llama_model_loader: loaded meta data with 20 key-value pairs and 291 tensors from /home/martin/.ollama/models/blobs/sha256-fad2a06e4cc705c2fa8bec5477ddb00dc0c859ac184c34dcc5586663774161ca (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen2
llama_model_loader: - kv 1: general.name str = Qwen2-beta-0_5B-Chat
llama_model_loader: - kv 2: qwen2.block_count u32 = 24
llama_model_loader: - kv 3: qwen2.context_length u32 = 32768
llama_model_loader: - kv 4: qwen2.embedding_length u32 = 1024
llama_model_loader: - kv 5: qwen2.feed_forward_length u32 = 2816
llama_model_loader: - kv 6: qwen2.attention.head_count u32 = 16
llama_model_loader: - kv 7: qwen2.attention.head_count_kv u32 = 16
llama_model_loader: - kv 8: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 9: qwen2.use_parallel_residual bool = true
llama_model_loader: - kv 10: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 11: tokenizer.ggml.tokens arr[str,151936] = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 12: tokenizer.ggml.token_type arr[i32,151936] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 13: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 14: tokenizer.ggml.eos_token_id u32 = 151643
llama_model_loader: - kv 15: tokenizer.ggml.padding_token_id u32 = 151643
llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 151643
llama_model_loader: - kv 17: tokenizer.chat_template str = {% for message in messages %}{% if lo...
llama_model_loader: - kv 18: general.quantization_version u32 = 2
llama_model_loader: - kv 19: general.file_type u32 = 2
llama_model_loader: - type f32: 121 tensors
llama_model_loader: - type q4_0: 169 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: missing pre-tokenizer type, using: 'default'
llm_load_vocab:
llm_load_vocab: ************************************
llm_load_vocab: GENERATION QUALITY WILL BE DEGRADED!
llm_load_vocab: CONSIDER REGENERATING THE MODEL
llm_load_vocab: ************************************
llm_load_vocab:
llm_load_vocab: special tokens definition check successful ( 293/151936 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = qwen2
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 151936
llm_load_print_meta: n_merges = 151387
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 1024
llm_load_print_meta: n_head = 16
llm_load_print_meta: n_head_kv = 16
llm_load_print_meta: n_layer = 24
llm_load_print_meta: n_rot = 64
llm_load_print_meta: n_embd_head_k = 64
llm_load_print_meta: n_embd_head_v = 64
llm_load_print_meta: n_gqa = 1
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-06
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 2816
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 2
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 0.5B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 619.57 M
llm_load_print_meta: model size = 371.02 MiB (5.02 BPW)
llm_load_print_meta: general.name = Qwen2-beta-0_5B-Chat
llm_load_print_meta: BOS token = 151643 '<|endoftext|>'
llm_load_print_meta: EOS token = 151643 '<|endoftext|>'
llm_load_print_meta: PAD token = 151643 '<|endoftext|>'
llm_load_print_meta: LF token = 148848 'ÄĬ'
llm_load_print_meta: EOT token = 151645 '<|im_end|>'
llm_load_tensors: ggml ctx size = 0.14 MiB
llm_load_tensors: CPU buffer size = 371.02 MiB
...............................................
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: n_batch = 512
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CPU KV buffer size = 192.00 MiB
llama_new_context_with_model: KV self size = 192.00 MiB, K (f16): 96.00 MiB, V (f16): 96.00 MiB
llama_new_context_with_model: CPU output buffer size = 0.58 MiB
llama_new_context_with_model: CPU compute buffer size = 298.75 MiB
llama_new_context_with_model: graph nodes = 846
llama_new_context_with_model: graph splits = 1
{"function":"initialize","level":"INFO","line":448,"msg":"initializing slots","n_slots":1,"tid":"140509500426112","timestamp":1715157038}
{"function":"initialize","level":"INFO","line":457,"msg":"new slot","n_ctx_slot":2048,"slot_id":0,"tid":"140509500426112","timestamp":1715157038}
{"function":"main","level":"INFO","line":3065,"msg":"model loaded","tid":"140509500426112","timestamp":1715157038}
{"function":"main","hostname":"127.0.0.1","level":"INFO","line":3268,"msg":"HTTP server listening","n_threads_http":"7","port":"45079","tid":"140509500426112","timestamp":1715157038}
{"function":"update_slots","level":"INFO","line":1579,"msg":"all slots are idle and system prompt is empty, clear the KV cache","tid":"140509500426112","timestamp":1715157038}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":0,"tid":"140509500426112","timestamp":1715157038}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":36880,"status":200,"tid":"140508558948096","timestamp":1715157038}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":1,"tid":"140509500426112","timestamp":1715157038}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":2,"tid":"140509500426112","timestamp":1715157038}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":3,"tid":"140509500426112","timestamp":1715157038}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":36908,"status":200,"tid":"140508533769984","timestamp":1715157038}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":36896,"status":200,"tid":"140508542162688","timestamp":1715157038}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":36916,"status":200,"tid":"140508437407488","timestamp":1715157038}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":4,"tid":"140509500426112","timestamp":1715157038}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":36916,"status":200,"tid":"140508437407488","timestamp":1715157038}
{"function":"log_server_request","level":"INFO","line":2735,"method":"POST","msg":"request","params":{},"path":"/tokenize","remote_addr":"127.0.0.1","remote_port":36916,"status":200,"tid":"140508437407488","timestamp":1715157038}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":5,"tid":"140509500426112","timestamp":1715157038}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":36916,"status":200,"tid":"140508437407488","timestamp":1715157038}
{"function":"launch_slot_with_data","level":"INFO","line":830,"msg":"slot is processing task","slot_id":0,"task_id":6,"tid":"140509500426112","timestamp":1715157038}
{"function":"update_slots","ga_i":0,"level":"INFO","line":1810,"msg":"slot progression","n_past":0,"n_past_se":0,"n_prompt_tokens_processed":116,"slot_id":0,"task_id":6,"tid":"140509500426112","timestamp":1715157039}
{"function":"update_slots","level":"INFO","line":1837,"msg":"kv cache rm [p0, end)","p0":0,"slot_id":0,"task_id":6,"tid":"140509500426112","timestamp":1715157039}
{"function":"print_timings","level":"INFO","line":269,"msg":"prompt eval time = 704.02 ms / 116 tokens ( 6.07 ms per token, 164.77 tokens per second)","n_prompt_tokens_processed":116,"n_tokens_second":164.76734424869642,"slot_id":0,"t_prompt_processing":704.023,"t_token":6.069163793103448,"task_id":6,"tid":"140509500426112","timestamp":1715157040}
{"function":"print_timings","level":"INFO","line":283,"msg":"generation eval time = 882.89 ms / 34 runs ( 25.97 ms per token, 38.51 tokens per second)","n_decoded":34,"n_tokens_second":38.5098936447349,"slot_id":0,"t_token":25.967352941176472,"t_token_generation":882.89,"task_id":6,"tid":"140509500426112","timestamp":1715157040}
{"function":"print_timings","level":"INFO","line":293,"msg":" total time = 1586.91 ms","slot_id":0,"t_prompt_processing":704.023,"t_token_generation":882.89,"t_total":1586.913,"task_id":6,"tid":"140509500426112","timestamp":1715157040}
{"function":"update_slots","level":"INFO","line":1641,"msg":"slot released","n_cache_tokens":150,"n_ctx":2048,"n_past":149,"n_system_tokens":0,"slot_id":0,"task_id":6,"tid":"140509500426112","timestamp":1715157040,"truncated":false}
{"function":"log_server_request","level":"INFO","line":2735,"method":"POST","msg":"request","params":{},"path":"/completion","remote_addr":"127.0.0.1","remote_port":36916,"status":200,"tid":"140508437407488","timestamp":1715157040}
[GIN] 2024/05/08 - 10:30:40 | 200 | 3.088213348s | 127.0.0.1 | POST "/v1/chat/completions"
2024/05/08 10:30:40 Model "ollama/qwen:0.5b" responded with:
```java
package com.eval;
import java.util.Arrays;
public classPlain {
static void plain() {
}
}
```
2024/05/08 10:30:40 $ symflower test --language java --workspace /tmp/eval-dev-quality2938219562/plain
Give us your feedback and let us know how we can improve Symflower at hello@symflower.com or https://github.com/symflower/symflower. Thanks so much for your help!
ERROR: All attempts fail:
#1: Internal error: exit status 1:
[]interface {}{
"/tmp/eval-dev-quality2938219562/plain/",
"[INFO] Scanning for projects...\n[WARNING] \n[WARNING] Some problems were encountered while building the effective model for eval.dev.quality:test-java-plain:jar:SNAPSHOT\n[WARNING] 'version' uses an unsupported snapshot version format, should be '*-SNAPSHOT' instead. @ line 6, column 11\n[WARNING] \n[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.\n[WARNING] \n[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.\n[WARNING] \n[INFO] \n[INFO] ------------------< eval.dev.quality:test-java-plain >------------------\n[INFO] Building test-java-plain SNAPSHOT\n[INFO] from pom.xml\n[INFO] --------------------------------[ jar ]---------------------------------\n[INFO] \n[INFO] --- clean:3.2.0:clean (default-clean) @ test-java-plain ---\n[INFO] \n[INFO] --- resources:3.3.0:resources (default-resources) @ test-java-plain ---\n[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!\n[INFO] skip non existing resourceDirectory /tmp/eval-dev-quality2938219562/plain/src/main/resources\n[INFO] \n[INFO] --- compiler:3.10.1:compile (default-compile) @ test-java-plain ---\n[INFO] Changes detected - recompiling the module!\n[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!\n[INFO] Compiling 1 source file to /tmp/eval-dev-quality2938219562/plain/target/classes\n[INFO] \n[INFO] --- resources:3.3.0:resources (default-resources) @ test-java-plain ---\n[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!\n[INFO] skip non existing resourceDirectory /tmp/eval-dev-quality2938219562/plain/src/main/resources\n[INFO] \n[INFO] --- compiler:3.10.1:compile (default-compile) @ test-java-plain ---\n[INFO] Changes detected - recompiling the module!\n[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!\n[INFO] Compiling 1 source file to /tmp/eval-dev-quality2938219562/plain/target/classes\n[INFO] \n[INFO] --- resources:3.3.0:testResources (default-testResources) @ test-java-plain ---\n[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!\n[INFO] skip non existing resourceDirectory /tmp/eval-dev-quality2938219562/plain/src/test/resources\n[INFO] \n[INFO] --- compiler:3.10.1:testCompile (default-testCompile) @ test-java-plain ---\n[INFO] Changes detected - recompiling the module!\n[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!\n[INFO] Compiling 1 source file to /tmp/eval-dev-quality2938219562/plain/target/test-classes\n[INFO] -------------------------------------------------------------\n[ERROR] COMPILATION ERROR : \n[INFO] -------------------------------------------------------------\n[ERROR] /tmp/eval-dev-quality2938219562/plain/src/test/java/com/eval/PlainTest.java:[5,8] class, interface, or enum expected\n[INFO] 1 error\n[INFO] -------------------------------------------------------------\n[INFO] \n[INFO] ------------------------------------------------------------------------\n[INFO] Skipping test-java-plain\n[INFO] This project has been banned from the build due to previous failures.\n[INFO] ------------------------------------------------------------------------\n[INFO] \n[INFO] ------------------------------------------------------------------------\n[INFO] Skipping test-java-plain\n[INFO] This project has been banned from the build due to previous failures.\n[INFO] ------------------------------------------------------------------------\n[INFO] ------------------------------------------------------------------------\n[INFO] BUILD FAILURE\n[INFO] ------------------------------------------------------------------------\n[INFO] Total time: 1.341 s\n[INFO] Finished at: 2024-05-08T10:30:43+02:00\n[INFO] ------------------------------------------------------------------------\n[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile (default-testCompile) on project test-java-plain: Compilation failure\n[ERROR] /tmp/eval-dev-quality2938219562/plain/src/test/java/com/eval/PlainTest.java:[5,8] class, interface, or enum expected\n[ERROR] \n[ERROR] -> [Help 1]\n[ERROR] \n[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.\n[ERROR] Re-run Maven using the -X switch to enable full debug logging.\n[ERROR] \n[ERROR] For more information about the errors and possible solutions, please read the following articles:\n[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException\n",
}
Stacktrace: goroutine 1 [running]:
gitlab.nethead.at/symflower/symflower/model/errors.NewInternalError({0x1ef1a40?, 0xc0000a2000}, {0xfbd660?, 0xc00019c0a8})
gitlab.nethead.at/symflower/symflower/model/errors/error.go:850 +0x59
gitlab.nethead.at/symflower/symflower/language/java/execute.MavenDependencyCommand.func1()
gitlab.nethead.at/symflower/symflower/language/java/execute/maven.go:80 +0x1ef
github.com/avast/retry-go.Do(0xc000615890, {0xc000615878, 0x3, 0x2712660?})
github.com/avast/retry-go/retry.go:135 +0x5a3
gitlab.nethead.at/symflower/symflower/language/java/execute.MavenDependencyCommand({0xc00081e240, 0x26}, 0x0, {0xc000318000, 0x8, 0x8})
gitlab.nethead.at/symflower/symflower/language/java/execute/maven.go:60 +0x1c5
gitlab.nethead.at/symflower/symflower/language/java/execute.MavenCloverCoverage({0xc00081e240, 0x26})
gitlab.nethead.at/symflower/symflower/language/java/execute/clover.go:133 +0xef
gitlab.nethead.at/symflower/symflower/cmd/symflower-local/cmd.(*commandTest).Execute(0xc00013c4a0, {0xc00013c380?, 0x7fe5786ef9b8?, 0xc00013c4a0?})
gitlab.nethead.at/symflower/symflower/cmd/symflower-local/cmd/test.go:111 +0x56f
gitlab.nethead.at/symflower/symflower/cmd.ExecuteNoExit({0x1ef12a0?, 0xc00013c380}, {0xc0000401e0?, 0xc0000061a0?, 0xc00016bed8?})
gitlab.nethead.at/symflower/symflower/cmd/execute.go:56 +0x2d0
gitlab.nethead.at/symflower/symflower/cmd.Execute({0x1ef12a0?, 0xc00013c380?}, {0xc0000401e0?, 0x26dbe00?, 0xc00013c380?})
gitlab.nethead.at/symflower/symflower/cmd/execute.go:16 +0x26
gitlab.nethead.at/symflower/symflower/cmd.ExecuteWithOSArguments(...)
gitlab.nethead.at/symflower/symflower/cmd/execute.go:75
main.main.func1()
gitlab.nethead.at/symflower/symflower/cmd/symflower-demo/main.go:40 +0x34
gitlab.nethead.at/symflower/symflower/cmd/symflower-local/cmd.(*Command).WithPanicReporter(0xc000182058?, 0x25c6968?)
gitlab.nethead.at/symflower/symflower/cmd/symflower-local/cmd/cmd.go:182 +0x48
main.main()
gitlab.nethead.at/symflower/symflower/cmd/symflower-demo/main.go:39 +0xa5
2024/05/08 10:30:43 Evaluated model "ollama/qwen:0.5b" using language "java" and repository "java/plain": encountered 1 problems: [exit status 1
github.com/symflower/eval-dev-quality/util.CommandWithResult
/home/martin/Documents/projects/eval-dev-quality/util/exec.go:39
github.com/symflower/eval-dev-quality/language/java.(*Language).Execute
/home/martin/Documents/projects/eval-dev-quality/language/java/language.go:95
github.com/symflower/eval-dev-quality/evaluate.Repository
/home/martin/Documents/projects/eval-dev-quality/evaluate/repository.go:68
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:233
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
/home/martin/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
github.com/symflower/eval-dev-quality/language/java.(*Language).Execute
/home/martin/Documents/projects/eval-dev-quality/language/java/language.go:105
github.com/symflower/eval-dev-quality/evaluate.Repository
/home/martin/Documents/projects/eval-dev-quality/evaluate/repository.go:68
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:233
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
/home/martin/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
src/main/java/com/eval/Plain.java]
2024/05/08 10:30:43 Excluding model "ollama/qwen:0.5b" since it was not able to solve the "java/plain" repository for language "java": [exit status 1
github.com/symflower/eval-dev-quality/util.CommandWithResult
/home/martin/Documents/projects/eval-dev-quality/util/exec.go:39
github.com/symflower/eval-dev-quality/language/java.(*Language).Execute
/home/martin/Documents/projects/eval-dev-quality/language/java/language.go:95
github.com/symflower/eval-dev-quality/evaluate.Repository
/home/martin/Documents/projects/eval-dev-quality/evaluate/repository.go:68
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:233
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
/home/martin/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
github.com/symflower/eval-dev-quality/language/java.(*Language).Execute
/home/martin/Documents/projects/eval-dev-quality/language/java/language.go:105
github.com/symflower/eval-dev-quality/evaluate.Repository
/home/martin/Documents/projects/eval-dev-quality/evaluate/repository.go:68
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:233
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
/home/martin/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
src/main/java/com/eval/Plain.java]
2024/05/08 10:30:43 Evaluating model "ollama/tinyllama:latest" using language "java" and repository "java/plain"
2024/05/08 10:30:43 Querying model "ollama/tinyllama:latest" with:
Given the following Java code file "src/main/java/com/eval/Plain.java" with package "com.eval", provide a test file for this code with JUnit 5 as a test framework.
The tests should produce 100 percent code coverage and must compile.
The response must contain only the test code and nothing else.
```java
package com.eval;
class Plain {
static void plain() {
}
}
```
time=2024-05-08T10:30:43.972+02:00 level=INFO source=gpu.go:122 msg="Detecting GPUs"
time=2024-05-08T10:30:43.992+02:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-05-08T10:30:44.189+02:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-05-08T10:30:44.190+02:00 level=INFO source=server.go:305 msg="starting llama server" cmd="/tmp/ollama1909733046/runners/cpu_avx2/ollama_llama_server --model /home/martin/.ollama/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 --ctx-size 2048 --batch-size 512 --embedding --log-disable --parallel 1 --port 42811"
time=2024-05-08T10:30:44.190+02:00 level=INFO source=sched.go:326 msg="loaded runners" count=1
time=2024-05-08T10:30:44.190+02:00 level=INFO source=server.go:464 msg="waiting for llama runner to start responding"
{"function":"server_params_parse","level":"INFO","line":2604,"msg":"logging to file is disabled.","tid":"140327897462656","timestamp":1715157044}
{"build":1,"commit":"952d03d","function":"main","level":"INFO","line":2820,"msg":"build info","tid":"140327897462656","timestamp":1715157044}
{"function":"main","level":"INFO","line":2823,"msg":"system info","n_threads":4,"n_threads_batch":-1,"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | ","tid":"140327897462656","timestamp":1715157044,"total_threads":8}
llama_model_loader: loaded meta data with 23 key-value pairs and 201 tensors from /home/martin/.ollama/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = TinyLlama
llama_model_loader: - kv 2: llama.context_length u32 = 2048
llama_model_loader: - kv 3: llama.embedding_length u32 = 2048
llama_model_loader: - kv 4: llama.block_count u32 = 22
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 5632
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 64
llama_model_loader: - kv 7: llama.attention.head_count u32 = 32
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 4
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 10: llama.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 11: general.file_type u32 = 2
llama_model_loader: - kv 12: tokenizer.ggml.model str = llama
llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 16: tokenizer.ggml.merges arr[str,61249] = ["▁ t", "e r", "i n", "▁ a", "e n...
llama_model_loader: - kv 17: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 18: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 19: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 20: tokenizer.ggml.padding_token_id u32 = 2
llama_model_loader: - kv 21: tokenizer.chat_template str = {% for message in messages %}\n{% if m...
llama_model_loader: - kv 22: general.quantization_version u32 = 2
llama_model_loader: - type f32: 45 tensors
llama_model_loader: - type q4_0: 155 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32000
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 2048
llm_load_print_meta: n_embd = 2048
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 4
llm_load_print_meta: n_layer = 22
llm_load_print_meta: n_rot = 64
llm_load_print_meta: n_embd_head_k = 64
llm_load_print_meta: n_embd_head_v = 64
llm_load_print_meta: n_gqa = 8
llm_load_print_meta: n_embd_k_gqa = 256
llm_load_print_meta: n_embd_v_gqa = 256
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 5632
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 2048
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 1B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 1.10 B
llm_load_print_meta: model size = 606.53 MiB (4.63 BPW)
llm_load_print_meta: general.name = TinyLlama
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: PAD token = 2 '</s>'
llm_load_print_meta: LF token = 13 '<0x0A>'
llm_load_tensors: ggml ctx size = 0.10 MiB
llm_load_tensors: CPU buffer size = 606.53 MiB
.....................................................................................
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: n_batch = 512
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CPU KV buffer size = 44.00 MiB
llama_new_context_with_model: KV self size = 44.00 MiB, K (f16): 22.00 MiB, V (f16): 22.00 MiB
llama_new_context_with_model: CPU output buffer size = 0.13 MiB
llama_new_context_with_model: CPU compute buffer size = 148.01 MiB
llama_new_context_with_model: graph nodes = 710
llama_new_context_with_model: graph splits = 1
{"function":"initialize","level":"INFO","line":448,"msg":"initializing slots","n_slots":1,"tid":"140327897462656","timestamp":1715157044}
{"function":"initialize","level":"INFO","line":457,"msg":"new slot","n_ctx_slot":2048,"slot_id":0,"tid":"140327897462656","timestamp":1715157044}
{"function":"main","level":"INFO","line":3065,"msg":"model loaded","tid":"140327897462656","timestamp":1715157044}
{"function":"main","hostname":"127.0.0.1","level":"INFO","line":3268,"msg":"HTTP server listening","n_threads_http":"7","port":"42811","tid":"140327897462656","timestamp":1715157044}
{"function":"update_slots","level":"INFO","line":1579,"msg":"all slots are idle and system prompt is empty, clear the KV cache","tid":"140327897462656","timestamp":1715157044}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":0,"tid":"140327897462656","timestamp":1715157044}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":53186,"status":200,"tid":"140327027406592","timestamp":1715157044}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":1,"tid":"140327897462656","timestamp":1715157044}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":53186,"status":200,"tid":"140327027406592","timestamp":1715157044}
{"function":"log_server_request","level":"INFO","line":2735,"method":"POST","msg":"request","params":{},"path":"/tokenize","remote_addr":"127.0.0.1","remote_port":53186,"status":200,"tid":"140327027406592","timestamp":1715157044}
{"function":"process_single_task","level":"INFO","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":2,"tid":"140327897462656","timestamp":1715157044}
{"function":"log_server_request","level":"INFO","line":2735,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":53186,"status":200,"tid":"140327027406592","timestamp":1715157044}
{"function":"launch_slot_with_data","level":"INFO","line":830,"msg":"slot is processing task","slot_id":0,"task_id":3,"tid":"140327897462656","timestamp":1715157044}
{"function":"update_slots","ga_i":0,"level":"INFO","line":1810,"msg":"slot progression","n_past":0,"n_past_se":0,"n_prompt_tokens_processed":139,"slot_id":0,"task_id":3,"tid":"140327897462656","timestamp":1715157044}
{"function":"update_slots","level":"INFO","line":1837,"msg":"kv cache rm [p0, end)","p0":0,"slot_id":0,"task_id":3,"tid":"140327897462656","timestamp":1715157044}
{"function":"print_timings","level":"INFO","line":269,"msg":"prompt eval time = 2340.70 ms / 139 tokens ( 16.84 ms per token, 59.38 tokens per second)","n_prompt_tokens_processed":139,"n_tokens_second":59.38381812317229,"slot_id":0,"t_prompt_processing":2340.705,"t_token":16.83960431654676,"task_id":3,"tid":"140327897462656","timestamp":1715157059}
{"function":"print_timings","level":"INFO","line":283,"msg":"generation eval time = 13173.53 ms / 311 runs ( 42.36 ms per token, 23.61 tokens per second)","n_decoded":311,"n_tokens_second":23.607947148562307,"slot_id":0,"t_token":42.35861736334405,"t_token_generation":13173.53,"task_id":3,"tid":"140327897462656","timestamp":1715157059}
{"function":"print_timings","level":"INFO","line":293,"msg":" total time = 15514.24 ms","slot_id":0,"t_prompt_processing":2340.705,"t_token_generation":13173.53,"t_total":15514.235,"task_id":3,"tid":"140327897462656","timestamp":1715157059}
{"function":"update_slots","level":"INFO","line":1641,"msg":"slot released","n_cache_tokens":450,"n_ctx":2048,"n_past":449,"n_system_tokens":0,"slot_id":0,"task_id":3,"tid":"140327897462656","timestamp":1715157059,"truncated":false}
{"function":"log_server_request","level":"INFO","line":2735,"method":"POST","msg":"request","params":{},"path":"/completion","remote_addr":"127.0.0.1","remote_port":53186,"status":200,"tid":"140327027406592","timestamp":1715157059}
[GIN] 2024/05/08 - 10:30:59 | 200 | 16.005203676s | 127.0.0.1 | POST "/v1/chat/completions"
2024/05/08 10:30:59 Model "ollama/tinyllama:latest" responded with:
To create a test file for this Java code, we can use JUnit 5 as a test framework. Here's an example:
**Test file "tests/ComparingPlain.java"**
```java
package com.eval;
import org.junit.jupiter.api.Test;
public class ComparingPlain {
@Test
public void plainTest() {
Plain plain = new Plai();
System.out.println(plain.plain()); // output: "Hello, world!"
}
}
```
This test file checks whether the Plain class's plain method returns "Hello, world!". The `@Test` annotation marks it as a JUnit 5 test, and the `ComparingPlain` class represents the unit under test. It declares a simple `plain()` method that we use to make sure our `Plain` class contains the same methods. The test code uses a static `plain()` method call to ensure the Plain instance has the expected value. If you're running JUnit 5 on a Linux machine, you might need to install the `junit-jupiter-engine-spi` package and use `org.junit.jupiter.api.extension.ExtendWithExtensionLoader` instead of `org.junit.jupiter.api.extension.ExtensionContext` for the `ExtensionContext` object.
2024/05/08 10:30:59 $ symflower test --language java --workspace /tmp/eval-dev-quality3716472842/plain
Give us your feedback and let us know how we can improve Symflower at hello@symflower.com or https://github.com/symflower/symflower. Thanks so much for your help!
ERROR: All attempts fail:
#1: Internal error: exit status 1:
[]interface {}{
"/tmp/eval-dev-quality3716472842/plain/",
"[INFO] Scanning for projects...\n[WARNING] \n[WARNING] Some problems were encountered while building the effective model for eval.dev.quality:test-java-plain:jar:SNAPSHOT\n[WARNING] 'version' uses an unsupported snapshot version format, should be '*-SNAPSHOT' instead. @ line 6, column 11\n[WARNING] \n[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.\n[WARNING] \n[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.\n[WARNING] \n[INFO] \n[INFO] ------------------< eval.dev.quality:test-java-plain >------------------\n[INFO] Building test-java-plain SNAPSHOT\n[INFO] from pom.xml\n[INFO] --------------------------------[ jar ]---------------------------------\n[INFO] \n[INFO] --- clean:3.2.0:clean (default-clean) @ test-java-plain ---\n[INFO] \n[INFO] --- resources:3.3.0:resources (default-resources) @ test-java-plain ---\n[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!\n[INFO] skip non existing resourceDirectory /tmp/eval-dev-quality3716472842/plain/src/main/resources\n[INFO] \n[INFO] --- compiler:3.10.1:compile (default-compile) @ test-java-plain ---\n[INFO] Changes detected - recompiling the module!\n[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!\n[INFO] Compiling 1 source file to /tmp/eval-dev-quality3716472842/plain/target/classes\n[INFO] \n[INFO] --- resources:3.3.0:resources (default-resources) @ test-java-plain ---\n[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!\n[INFO] skip non existing resourceDirectory /tmp/eval-dev-quality3716472842/plain/src/main/resources\n[INFO] \n[INFO] --- compiler:3.10.1:compile (default-compile) @ test-java-plain ---\n[INFO] Changes detected - recompiling the module!\n[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!\n[INFO] Compiling 1 source file to /tmp/eval-dev-quality3716472842/plain/target/classes\n[INFO] \n[INFO] --- resources:3.3.0:testResources (default-testResources) @ test-java-plain ---\n[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!\n[INFO] skip non existing resourceDirectory /tmp/eval-dev-quality3716472842/plain/src/test/resources\n[INFO] \n[INFO] --- compiler:3.10.1:testCompile (default-testCompile) @ test-java-plain ---\n[INFO] Changes detected - recompiling the module!\n[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!\n[INFO] Compiling 1 source file to /tmp/eval-dev-quality3716472842/plain/target/test-classes\n[INFO] -------------------------------------------------------------\n[ERROR] COMPILATION ERROR : \n[INFO] -------------------------------------------------------------\n[ERROR] /tmp/eval-dev-quality3716472842/plain/src/test/java/com/eval/PlainTest.java:[5,8] class ComparingPlain is public, should be declared in a file named ComparingPlain.java\n[ERROR] /tmp/eval-dev-quality3716472842/plain/src/test/java/com/eval/PlainTest.java:[9,27] cannot find symbol\n symbol: class Plai\n location: class com.eval.ComparingPlain\n[ERROR] /tmp/eval-dev-quality3716472842/plain/src/test/java/com/eval/PlainTest.java:[10,39] 'void' type not allowed here\n[INFO] 3 errors \n[INFO] -------------------------------------------------------------\n[INFO] \n[INFO] ------------------------------------------------------------------------\n[INFO] Skipping test-java-plain\n[INFO] This project has been banned from the build due to previous failures.\n[INFO] ------------------------------------------------------------------------\n[INFO] \n[INFO] ------------------------------------------------------------------------\n[INFO] Skipping test-java-plain\n[INFO] This project has been banned from the build due to previous failures.\n[INFO] ------------------------------------------------------------------------\n[INFO] ------------------------------------------------------------------------\n[INFO] BUILD FAILURE\n[INFO] ------------------------------------------------------------------------\n[INFO] Total time: 1.159 s\n[INFO] Finished at: 2024-05-08T10:31:02+02:00\n[INFO] ------------------------------------------------------------------------\n[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile (default-testCompile) on project test-java-plain: Compilation failure: Compilation failure: \n[ERROR] /tmp/eval-dev-quality3716472842/plain/src/test/java/com/eval/PlainTest.java:[5,8] class ComparingPlain is public, should be declared in a file named ComparingPlain.java\n[ERROR] /tmp/eval-dev-quality3716472842/plain/src/test/java/com/eval/PlainTest.java:[9,27] cannot find symbol\n[ERROR] symbol: class Plai\n[ERROR] location: class com.eval.ComparingPlain\n[ERROR] /tmp/eval-dev-quality3716472842/plain/src/test/java/com/eval/PlainTest.java:[10,39] 'void' type not allowed here\n[ERROR] -> [Help 1]\n[ERROR] \n[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.\n[ERROR] Re-run Maven using the -X switch to enable full debug logging.\n[ERROR] \n[ERROR] For more information about the errors and possible solutions, please read the following articles:\n[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException\n",
}
Stacktrace: goroutine 1 [running]:
gitlab.nethead.at/symflower/symflower/model/errors.NewInternalError({0x1ef1a40?, 0xc0000a21a0}, {0xfbd660?, 0xc00059c738})
gitlab.nethead.at/symflower/symflower/model/errors/error.go:850 +0x59
gitlab.nethead.at/symflower/symflower/language/java/execute.MavenDependencyCommand.func1()
gitlab.nethead.at/symflower/symflower/language/java/execute/maven.go:80 +0x1ef
github.com/avast/retry-go.Do(0xc000615890, {0xc000615878, 0x3, 0x2d?})
github.com/avast/retry-go/retry.go:135 +0x5a3
gitlab.nethead.at/symflower/symflower/language/java/execute.MavenDependencyCommand({0xc000574ab0, 0x26}, 0x0, {0xc0001f2080, 0x8, 0x8})
gitlab.nethead.at/symflower/symflower/language/java/execute/maven.go:60 +0x1c5
gitlab.nethead.at/symflower/symflower/language/java/execute.MavenCloverCoverage({0xc000574ab0, 0x26})
gitlab.nethead.at/symflower/symflower/language/java/execute/clover.go:133 +0xef
gitlab.nethead.at/symflower/symflower/cmd/symflower-local/cmd.(*commandTest).Execute(0xc0005b64a0, {0xc0005b6380?, 0x7ff07a71dc98?, 0xc0005b64a0?})
gitlab.nethead.at/symflower/symflower/cmd/symflower-local/cmd/test.go:111 +0x56f
gitlab.nethead.at/symflower/symflower/cmd.ExecuteNoExit({0x1ef12a0?, 0xc0005b6380}, {0xc000194120?, 0x0?, 0xc00013fed8?})
gitlab.nethead.at/symflower/symflower/cmd/execute.go:56 +0x2d0
gitlab.nethead.at/symflower/symflower/cmd.Execute({0x1ef12a0?, 0xc0005b6380?}, {0xc000194120?, 0x26dbe00?, 0xc0005b6380?})
gitlab.nethead.at/symflower/symflower/cmd/execute.go:16 +0x26
gitlab.nethead.at/symflower/symflower/cmd.ExecuteWithOSArguments(...)
gitlab.nethead.at/symflower/symflower/cmd/execute.go:75
main.main.func1()
gitlab.nethead.at/symflower/symflower/cmd/symflower-demo/main.go:40 +0x34
gitlab.nethead.at/symflower/symflower/cmd/symflower-local/cmd.(*Command).WithPanicReporter(0xc00004c1d8?, 0x25c6968?)
gitlab.nethead.at/symflower/symflower/cmd/symflower-local/cmd/cmd.go:182 +0x48
main.main()
gitlab.nethead.at/symflower/symflower/cmd/symflower-demo/main.go:39 +0xa5
2024/05/08 10:31:02 Evaluated model "ollama/tinyllama:latest" using language "java" and repository "java/plain": encountered 1 problems: [exit status 1
github.com/symflower/eval-dev-quality/util.CommandWithResult
/home/martin/Documents/projects/eval-dev-quality/util/exec.go:39
github.com/symflower/eval-dev-quality/language/java.(*Language).Execute
/home/martin/Documents/projects/eval-dev-quality/language/java/language.go:95
github.com/symflower/eval-dev-quality/evaluate.Repository
/home/martin/Documents/projects/eval-dev-quality/evaluate/repository.go:68
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:233
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
/home/martin/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
github.com/symflower/eval-dev-quality/language/java.(*Language).Execute
/home/martin/Documents/projects/eval-dev-quality/language/java/language.go:105
github.com/symflower/eval-dev-quality/evaluate.Repository
/home/martin/Documents/projects/eval-dev-quality/evaluate/repository.go:68
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:233
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
/home/martin/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
src/main/java/com/eval/Plain.java]
2024/05/08 10:31:02 Excluding model "ollama/tinyllama:latest" since it was not able to solve the "java/plain" repository for language "java": [exit status 1
github.com/symflower/eval-dev-quality/util.CommandWithResult
/home/martin/Documents/projects/eval-dev-quality/util/exec.go:39
github.com/symflower/eval-dev-quality/language/java.(*Language).Execute
/home/martin/Documents/projects/eval-dev-quality/language/java/language.go:95
github.com/symflower/eval-dev-quality/evaluate.Repository
/home/martin/Documents/projects/eval-dev-quality/evaluate/repository.go:68
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:233
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
/home/martin/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
github.com/symflower/eval-dev-quality/language/java.(*Language).Execute
/home/martin/Documents/projects/eval-dev-quality/language/java/language.go:105
github.com/symflower/eval-dev-quality/evaluate.Repository
/home/martin/Documents/projects/eval-dev-quality/evaluate/repository.go:68
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:233
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
/home/martin/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
/home/martin/Documents/projects/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
/home/martin/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
src/main/java/com/eval/Plain.java]
2024/05/08 10:31:02 Evaluating models and languages
2024/05/08 10:31:02 Evaluation score for "ollama/tinyllama:latest" ("code-invalid"): score=4, coverage-statement=0, files-executed=0, response-no-error=2, response-no-excess=0, response-with-code=2
2024/05/08 10:31:02 Evaluation score for "ollama/qwen:0.5b" ("code-invalid"): score=5, coverage-statement=0, files-executed=0, response-no-error=2, response-no-excess=1, response-with-code=2
ollama.sh: line 1: kill: (13097) - No such process
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment