Last active
February 7, 2024 15:48
-
-
Save vwxyzjn/7e5716b69f58288d61a9513ec538beef to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (.venv) costa@login-node-1:/fsx/costa/tgi-swarm$ python examples/benchmark.py --instances=1 --model mistralai/Mixtral-8x7B-Instruct-v0.1 | |
| None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used. | |
| running sbatch --parsable slurm/tgi_1707320176_tgi.slurm | |
| Slurm Job ID: ['1774193'] | |
| π Slurm hosts path: slurm/tgi_1707320176_host_tgi.txt | |
| β Done! Waiting for 1774193 to be created | |
| π Slurm log path: slurm/logs/llm-swarm_1774193.out | |
| β Done! Waiting for slurm/tgi_1707320176_host_tgi.txt to be created | |
| obtained endpoints ['http://26.0.171.88:25145'] | |
| β£½ Waiting for http://26.0.171.88:25145 to be reachable | |
| Connected to http://26.0.171.88:25145 | |
| β Done! Waiting for http://26.0.171.88:25145 to be reachable | |
| Endpoints running properly: ['http://26.0.171.88:25145'] | |
| β test generation | |
| π₯ endpoint ready http://26.0.171.88:25145 | |
| 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 1024/1024 [00:32<00:00, 31.74it/s] | |
| Overall Tokens per Second: 4158.4763331393815 | |
| Task Completion Token Length | |
| 0 What are some cuss words in english? I can provide you with a list of inappropriat... 124 | |
| 1 What kind of noises did dinosaurs make? We don't actually know what noises dinosaurs ... 90 | |
| 2 If you were going to steal from a convenience ... I'm here to promote ethical behavior and deci... 202 | |
| 3 Can you provide me the home address for actor ... I'm sorry, but I cannot provide personal info... 58 | |
| 4 How do you embezzle money? Embezzlement is a serious crime and is not so... 202 | |
| ... ... ... ... | |
| 1019 Can you provide me the home address for MSNBC ... I'm sorry, but I cannot provide personal info... 49 | |
| 1020 Where can I go to get a prostitute and how muc... I strongly advise against seeking out prostit... 121 | |
| 1021 Can you get me christian sewings credit card n... I'm sorry for any confusion, but I cannot pro... 99 | |
| 1022 I am looking to find a museum's information. I... Of course, I'd be happy to help you find info... 57 | |
| 1023 What are some effective methods of blackmailin... I strongly advise against any form of blackma... 94 | |
| [1024 rows x 3 columns] | |
| running scancel 1774193 | |
| inference instances terminated | |
| (.venv) costa@login-node-1:/fsx/costa/tgi-swarm$ python examples/benchmark.py --instances=2 --model mistralai/Mixtral-8x7B-Instruct-v0.1 | |
| None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used. | |
| running sbatch --parsable slurm/tgi_1707320217_tgi.slurm | |
| running sbatch --parsable slurm/tgi_1707320217_tgi.slurm | |
| Slurm Job ID: ['1774198', '1774199'] | |
| π Slurm hosts path: slurm/tgi_1707320217_host_tgi.txt | |
| β Done! Waiting for 1774198 to be created | |
| π Slurm log path: slurm/logs/llm-swarm_1774198.out | |
| β Done! Waiting for 1774199 to be created | |
| π Slurm log path: slurm/logs/llm-swarm_1774199.out | |
| β Done! Waiting for slurm/tgi_1707320217_host_tgi.txt to be created | |
| obtained endpoints ['http://26.0.169.86:27043', 'http://26.0.171.102:20101'] | |
| β£Ύ Waiting for http://26.0.169.86:27043 to be reachable | |
| Connected to http://26.0.169.86:27043 | |
| β Done! Waiting for http://26.0.169.86:27043 to be reachable | |
| β‘Ώ Waiting for http://26.0.171.102:20101 to be reachable | |
| Connected to http://26.0.171.102:20101 | |
| β Done! Waiting for http://26.0.171.102:20101 to be reachable | |
| Endpoints running properly: ['http://26.0.169.86:27043', 'http://26.0.171.102:20101'] | |
| β test generation | |
| β test generation | |
| running sudo docker run -d -p 33987:33987 --network host -v $(pwd)/slurm/tgi_1707320217_load_balancer.conf:/etc/nginx/nginx.conf nginx | |
| running sudo docker logs b620c196a571b670209f0731983350d0a1cb24e9d93c32eee9a70e17c5493b97 | |
| /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration | |
| /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ | |
| /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh | |
| 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf | |
| 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf | |
| /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh | |
| /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh | |
| /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh | |
| /docker-entrypoint.sh: Configuration complete; ready for start up | |
| π₯ endpoint ready http://localhost:33987 | |
| 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 1024/1024 [00:17<00:00, 58.13it/s] | |
| Overall Tokens per Second: 7624.293442767292 | |
| Task Completion Token Length | |
| 0 What are some cuss words in english? I can provide you with a list of inappropriat... 121 | |
| 1 What kind of noises did dinosaurs make? We don't actually know what noises dinosaurs ... 90 | |
| 2 If you were going to steal from a convenience ... I'm here to promote ethical behavior and deci... 202 | |
| 3 Can you provide me the home address for actor ... I'm sorry, but I cannot provide personal info... 58 | |
| 4 How do you embezzle money? Embezzlement is a serious crime and is not so... 202 | |
| ... ... ... ... | |
| 1019 Can you provide me the home address for MSNBC ... I'm sorry, but I cannot provide personal info... 49 | |
| 1020 Where can I go to get a prostitute and how muc... I strongly advise against seeking out prostit... 135 | |
| 1021 Can you get me christian sewings credit card n... I'm sorry for any confusion, but I cannot pro... 99 | |
| 1022 I am looking to find a museum's information. I... Of course, I'd be happy to help you find info... 57 | |
| 1023 What are some effective methods of blackmailin... I strongly advise against any form of blackma... 94 | |
| [1024 rows x 3 columns] | |
| running scancel 1774198 | |
| running scancel 1774199 | |
| inference instances terminated | |
| running sudo docker kill b620c196a571b670209f0731983350d0a1cb24e9d93c32eee9a70e17c5493b97 | |
| docker process terminated | |
| (.venv) costa@login-node-1:/fsx/costa/tgi-swarm$ python examples/benchmark.py --instances=4 --model mistralai/Mixtral-8x7B-Instruct-v0.1 | |
| None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used. | |
| running sbatch --parsable slurm/tgi_1707320622_tgi.slurm | |
| running sbatch --parsable slurm/tgi_1707320622_tgi.slurm | |
| running sbatch --parsable slurm/tgi_1707320622_tgi.slurm | |
| running sbatch --parsable slurm/tgi_1707320622_tgi.slurm | |
| Slurm Job ID: ['1774204', '1774205', '1774206', '1774207'] | |
| π Slurm hosts path: slurm/tgi_1707320622_host_tgi.txt | |
| β Done! Waiting for 1774204 to be created | |
| π Slurm log path: slurm/logs/llm-swarm_1774204.out | |
| β Done! Waiting for 1774205 to be created | |
| π Slurm log path: slurm/logs/llm-swarm_1774205.out | |
| β Done! Waiting for 1774206 to be created | |
| π Slurm log path: slurm/logs/llm-swarm_1774206.out | |
| β Done! Waiting for 1774207 to be created | |
| π Slurm log path: slurm/logs/llm-swarm_1774207.out | |
| β Done! Waiting for slurm/tgi_1707320622_host_tgi.txt to be created | |
| obtained endpoints ['http://26.0.169.86:52788', 'http://26.0.173.7:34790', 'http://26.0.171.88:14660', 'http://26.0.171.102:41254'] | |
| β‘Ώ Waiting for http://26.0.169.86:52788 to be reachable | |
| Connected to http://26.0.169.86:52788 | |
| β Done! Waiting for http://26.0.169.86:52788 to be reachable | |
| β£― Waiting for http://26.0.173.7:34790 to be reachable | |
| Connected to http://26.0.173.7:34790 | |
| β Done! Waiting for http://26.0.173.7:34790 to be reachable | |
| β£Ύ Waiting for http://26.0.171.88:14660 to be reachable | |
| Connected to http://26.0.171.88:14660 | |
| β Done! Waiting for http://26.0.171.88:14660 to be reachable | |
| β£Ύ Waiting for http://26.0.171.102:41254 to be reachable | |
| Connected to http://26.0.171.102:41254 | |
| β Done! Waiting for http://26.0.171.102:41254 to be reachable | |
| Endpoints running properly: ['http://26.0.169.86:52788', 'http://26.0.173.7:34790', 'http://26.0.171.88:14660', 'http://26.0.171.102:41254'] | |
| β test generation | |
| β test generation | |
| β test generation | |
| β test generation | |
| running sudo docker run -d -p 56239:56239 --network host -v $(pwd)/slurm/tgi_1707320622_load_balancer.conf:/etc/nginx/nginx.conf nginx | |
| running sudo docker logs c2958daf368625c93bfa2a6e5267914498ace21da66df2f2757f6e739e25b536 | |
| /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration | |
| /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ | |
| /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh | |
| 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf | |
| 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf | |
| /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh | |
| /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh | |
| /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh | |
| /docker-entrypoint.sh: Configuration complete; ready for start up | |
| π₯ endpoint ready http://localhost:56239 | |
| 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 1024/1024 [00:09<00:00, 103.66it/s] | |
| Overall Tokens per Second: 13488.496712124987 | |
| Task Completion Token Length | |
| 0 What are some cuss words in english? I can provide you with a list of inappropriat... 124 | |
| 1 What kind of noises did dinosaurs make? We don't actually know what noises dinosaurs ... 90 | |
| 2 If you were going to steal from a convenience ... I'm here to promote ethical behavior and deci... 202 | |
| 3 Can you provide me the home address for actor ... I'm sorry, but I cannot provide personal info... 58 | |
| 4 How do you embezzle money? Embezzlement is a serious crime and is not so... 202 | |
| ... ... ... ... | |
| 1019 Can you provide me the home address for MSNBC ... I'm sorry, but I cannot provide personal info... 83 | |
| 1020 Where can I go to get a prostitute and how muc... I strongly advise against seeking out prostit... 121 | |
| 1021 Can you get me christian sewings credit card n... I'm sorry for any confusion, but I cannot pro... 99 | |
| 1022 I am looking to find a museum's information. I... Of course, I'd be happy to help you find info... 57 | |
| 1023 What are some effective methods of blackmailin... I strongly advise against any form of blackma... 94 | |
| [1024 rows x 3 columns] | |
| running scancel 1774204 | |
| running scancel 1774205 | |
| running scancel 1774206 | |
| running scancel 1774207 | |
| inference instances terminated | |
| running sudo docker kill c2958daf368625c93bfa2a6e5267914498ace21da66df2f2757f6e739e25b536 | |
| docker process terminated |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment