jetson_containers_examples.md

Jetson Containers - Examples

jetson-containers run $(autotag nano_llm) python3 -m nano_llm.chat \
    --api=mlc \
    --model princeton-nlp/Sheared-LLaMA-2.7B-ShareGPT

>> PROMPT: I have a heart

I'm sorry to hear that you have a heart. It's a vital and essential part of the human body, but it can also be a source of pain and discomfort. Here are some basic facts and information about hearts:

* Hearts are located in the chest and are made up of two main chambers: the left and right ventricles.
* Hearts beat regularly and pump blood throughout the body.
* The heart is an important organ that helps to maintain the body's balance of essential minerals, electrolytes, and water.
* The heart</s>

┌───────────────┬─────────────┐
│ embed_time    │ 0.000127043 │
├───────────────┼─────────────┤
│ input_tokens  │     14      │
├───────────────┼─────────────┤
│ output_tokens │     128     │
├───────────────┼─────────────┤
│ prefill_time  │ 0.00976307  │
├───────────────┼─────────────┤
│ prefill_rate  │   1433.98   │
├───────────────┼─────────────┤
│ decode_time   │   4.25479   │
├───────────────┼─────────────┤
│ decode_rate   │   29.8487   │
└───────────────┴─────────────┘

>> PROMPT: how do we measure distances to other planets

To measure the distances between Earth and other planets, we use a technique called "astronomical distances." This technique involves measuring the distances between celestial bodies using a system of units called "light-years."

The light-year is defined as the distance traveled by a light-year (i.e., 1.25 meters) in one year on Earth. This system of units is based on the idea that one year on Earth equals approximately 365.25 days, which is about 3.78 days per day.

Therefore, the distance</s>

┌───────────────┬─────────────┐
│ embed_time    │ 0.000230472 │
├───────────────┼─────────────┤
│ input_tokens  │     19      │
├───────────────┼─────────────┤
│ output_tokens │     128     │
├───────────────┼─────────────┤
│ prefill_time  │  0.0252347  │
├───────────────┼─────────────┤
│ prefill_rate  │   752.932   │
├───────────────┼─────────────┤
│ decode_time   │   5.7457    │
├───────────────┼─────────────┤
│ decode_rate   │   22.1035   │
└───────────────┴─────────────┘