Skip to content

Instantly share code, notes, and snippets.

@fxmarty

fxmarty/gist:5b5dbfac7ef4a3ea746effdf146584b6

Created July 6, 2023 04:27
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save fxmarty/5b5dbfac7ef4a3ea746effdf146584b6 to your computer and use it in GitHub Desktop.
Save fxmarty/5b5dbfac7ef4a3ea746effdf146584b6 to your computer and use it in GitHub Desktop.
Download ZIP
log_dynamo_inductor_aot
Raw
gistfile1.txt
Some weights of the model checkpoint at fxmarty/tiny-llama-fast-tokenizer were not used when initializing LlamaForCausalLM: ['model.layers.1.self_attn.q_proj.weight', 'model.layers.0.self_attn.v_proj.weight', 'model.layers.1.self_attn.v_proj.weight', 'model.layers.0.self_attn.k_proj.weight', 'model.layers.1.self_attn.k_proj.weight', 'model.layers.0.self_attn.q_proj.weight']
- This IS expected if you are initializing LlamaForCausalLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing LlamaForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of LlamaForCausalLM were not initialized from the model checkpoint at fxmarty/tiny-llama-fast-tokenizer and are newly initialized: ['model.layers.0.self_attn.qkv_proj.weight', 'model.layers.1.self_attn.qkv_proj.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/1 [00:00<?, ?it/s][2023-07-06 04:20:38,056] torch._dynamo.eval_frame: [DEBUG] skipping __init__ /home/ubuntu/miniconda3/lib/python3.10/contextlib.py
[2023-07-06 04:20:38,056] torch._dynamo.eval_frame: [DEBUG] skipping __enter__ /home/ubuntu/miniconda3/lib/python3.10/contextlib.py
[2023-07-06 04:20:38,056] torch._dynamo.eval_frame: [DEBUG] skipping __init__ /home/ubuntu/miniconda3/lib/python3.10/contextlib.py
[2023-07-06 04:20:38,056] torch._dynamo.eval_frame: [DEBUG] skipping __enter__ /home/ubuntu/miniconda3/lib/python3.10/contextlib.py
[2023-07-06 04:20:38,056] torch._dynamo.eval_frame: [DEBUG] skipping enable_dynamic /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py
[2023-07-06 04:20:38,057] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:644
[2023-07-06 04:20:38,139] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['input_ids'] (1, 1000) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:38,140] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['attention_mask'] (1, 1003) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:38,141] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['position_ids'] (1, 1000) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:38,142] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['past_key_values'][0] (2, 1, 4, 1003, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None, None]
[2023-07-06 04:20:38,142] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['past_key_values'][1] (2, 1, 4, 1003, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None, None]
[2023-07-06 04:20:38,143] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['valid_past_index'] () [] []
[2023-07-06 04:20:38,144] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:685
[2023-07-06 04:20:38,144] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions []
[2023-07-06 04:20:38,144] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [ConstantVariable(NoneType)]
[2023-07-06 04:20:38,144] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [ConstantVariable(NoneType), ConstantVariable(NoneType)]
[2023-07-06 04:20:38,144] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 12 [ConstantVariable(bool)]
[2023-07-06 04:20:38,144] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,144] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR config [NNModuleVariable()]
[2023-07-06 04:20:38,145] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR output_attentions [HFPretrainedConfigVariable()]
[2023-07-06 04:20:38,145] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST output_attentions [ConstantVariable(bool)]
[2023-07-06 04:20:38,145] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:687
[2023-07-06 04:20:38,145] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_hidden_states []
[2023-07-06 04:20:38,145] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [ConstantVariable(NoneType)]
[2023-07-06 04:20:38,145] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [ConstantVariable(NoneType), ConstantVariable(NoneType)]
[2023-07-06 04:20:38,146] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 32 [ConstantVariable(bool)]
[2023-07-06 04:20:38,146] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,146] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR config [NNModuleVariable()]
[2023-07-06 04:20:38,146] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR output_hidden_states [HFPretrainedConfigVariable()]
[2023-07-06 04:20:38,146] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:686
[2023-07-06 04:20:38,146] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST output_hidden_states [ConstantVariable(bool)]
[2023-07-06 04:20:38,146] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:689
[2023-07-06 04:20:38,146] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST return_dict []
[2023-07-06 04:20:38,146] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [ConstantVariable(bool)]
[2023-07-06 04:20:38,146] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [ConstantVariable(bool), ConstantVariable(NoneType)]
[2023-07-06 04:20:38,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 52 [ConstantVariable(bool)]
[2023-07-06 04:20:38,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST return_dict []
[2023-07-06 04:20:38,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE JUMP_FORWARD 58 [ConstantVariable(bool)]
[2023-07-06 04:20:38,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST return_dict [ConstantVariable(bool)]
[2023-07-06 04:20:38,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:692
[2023-07-06 04:20:38,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR model [NNModuleVariable()]
[2023-07-06 04:20:38,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:693
[2023-07-06 04:20:38,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_ids [NNModuleVariable()]
[2023-07-06 04:20:38,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:694
[2023-07-06 04:20:38,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST attention_mask [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:695
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:696
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_key_values [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:697
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST inputs_embeds [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ListVariable()]
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:698
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ListVariable(), ConstantVariable(NoneType)]
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:699
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_hidden_states [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ListVariable(), ConstantVariable(NoneType), ConstantVariable(bool)]
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:700
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST return_dict [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ListVariable(), ConstantVariable(NoneType), ConstantVariable(bool), ConstantVariable(bool)]
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:701
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ListVariable(), ConstantVariable(NoneType), ConstantVariable(bool), ConstantVariable(bool), ConstantVariable(bool)]
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:692
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('input_ids', 'attention_mask', 'position_ids', 'past_key_values', 'inputs_embeds', 'output_attentions', 'output_hidden_states', 'return_dict', 'valid_past_index') [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ListVariable(), ConstantVariable(NoneType), ConstantVariable(bool), ConstantVariable(bool), ConstantVariable(bool), TensorVariable()]
[2023-07-06 04:20:38,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 9 [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ListVariable(), ConstantVariable(NoneType), ConstantVariable(bool), ConstantVariable(bool), ConstantVariable(bool), TensorVariable(), ConstantVariable(tuple)]
[2023-07-06 04:20:38,149] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:38,151] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:38,151] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:38,153] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:38,156] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:38,156] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:38,156] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:38,156] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,156] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:38,157] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaModel.forward at 0x7fb8d0c46cb0>, NNModuleVariable())]
[2023-07-06 04:20:38,157] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,157] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,157] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,227] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,227] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,227] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,228] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,228] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,228] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,228] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,228] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,228] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,228] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,228] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:38,228] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaModel.forward at 0x7fb8d0c46cb0>, NNModuleVariable())]
[2023-07-06 04:20:38,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaModel.forward at 0x7fb8d0c46cb0>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:38,230] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaModel.forward at 0x7fb8d0c46cb0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:38,230] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaModel.forward at 0x7fb8d0c46cb0>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:38,230] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaModel.forward at 0x7fb8d0c46cb0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:38,231] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:38,231] torch._dynamo.symbolic_convert: [DEBUG] FAILED INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:38,231] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:38,231] torch._dynamo.symbolic_convert: [DEBUG] break_graph_if_unsupported triggered compile
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2187, in inline_call_
sub_locals, closure_cells = func.bind_args(parent, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 187, in bind_args
[
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 188, in <listcomp>
wrap(val=arg, source=source)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 62, in wrap_bound_arg
return VariableBuilder(tx, source=source, **options)(val)
TypeError: VariableBuilder.__init__() got an unexpected keyword argument 'guards'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 390, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1152, in CALL_FUNCTION_KW
self.call_function(fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 559, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 333, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 595, in inline_user_function_return
result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2132, in inline_call
return cls.inline_call_(parent, func, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2229, in inline_call_
tracer.run()
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 708, in run
and self.step()
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 668, in step
getattr(self, inst.opname)(inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 390, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1140, in CALL_FUNCTION_EX
self.call_function(fn, argsvars.items, kwargsvars.items)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 559, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 333, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 287, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 120, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 595, in inline_user_function_return
result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2132, in inline_call
return cls.inline_call_(parent, func, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2190, in inline_call_
raise ArgsMismatchError(
torch._dynamo.exc.ArgsMismatchError: VariableBuilder.__init__() got an unexpected keyword argument 'guards'.
func = 'forward' /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:477, args = [<class 'trfs_fast.llama.LlamaModel'>], kwargs = OrderedDict([('input_ids', TensorVariable()), ('attention_mask', TensorVariable()), ('position_ids', TensorVariable()), ('past_key_values', ListVariable()), ('inputs_embeds', ConstantVariable(NoneType)), ('output_attentions', ConstantVariable(bool)), ('output_hidden_states', ConstantVariable(bool)), ('return_dict', ConstantVariable(bool)), ('valid_past_index', TensorVariable())])
[2023-07-06 04:20:38,235] torch._dynamo.symbolic_convert.__graph_breaks: [DEBUG] Graph break: VariableBuilder.__init__() got an unexpected keyword argument 'guards'.
func = 'forward' /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:477, args = [<class 'trfs_fast.llama.LlamaModel'>], kwargs = OrderedDict([('input_ids', TensorVariable()), ('attention_mask', TensorVariable()), ('position_ids', TensorVariable()), ('past_key_values', ListVariable()), ('inputs_embeds', ConstantVariable(NoneType)), ('output_attentions', ConstantVariable(bool)), ('output_hidden_states', ConstantVariable(bool)), ('return_dict', ConstantVariable(bool)), ('valid_past_index', TensorVariable())]) from user code at File "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 692, in forward
outputs = self.model(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1514, in _call_impl
return forward_call(*args, **kwargs)
[2023-07-06 04:20:38,235] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:38,235] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason="VariableBuilder.__init__() got an unexpected keyword argument 'guards'.\n func = 'forward' /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:477, args = [<class 'trfs_fast.llama.LlamaModel'>], kwargs = OrderedDict([('input_ids', TensorVariable()), ('attention_mask', TensorVariable()), ('position_ids', TensorVariable()), ('past_key_values', ListVariable()), ('inputs_embeds', ConstantVariable(NoneType)), ('output_attentions', ConstantVariable(bool)), ('output_hidden_states', ConstantVariable(bool)), ('return_dict', ConstantVariable(bool)), ('valid_past_index', TensorVariable())])", user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 692 in forward>, <FrameSummary file /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py, line 1514 in _call_impl>], graph_break=True)
[2023-07-06 04:20:38,295] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662761824)
hasattr(L['input_ids'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['return_dict'], 7698144)
hasattr(L['position_ids'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['inputs_embeds'], 7649088)
hasattr(L['attention_mask'], '_dynamo_dynamic_indices') == False
___check_type_id(L['past_key_values'], 7662688)
len(L['past_key_values']) == 2
hasattr(L['valid_past_index'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['output_attentions'], 7649088)
hasattr(L['past_key_values'][0], '_dynamo_dynamic_indices') == False
hasattr(L['past_key_values'][1], '_dynamo_dynamic_indices') == False
___check_obj_id(L['output_hidden_states'], 7649088)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
[2023-07-06 04:20:38,295] torch._dynamo.eval_frame: [DEBUG] skipping __getattr__ /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py
[2023-07-06 04:20:38,295] torch._dynamo.convert_frame: [DEBUG] skipping because no torch.* __getattribute__ /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/configuration_utils.py 258
[2023-07-06 04:20:38,295] torch._dynamo.eval_frame: [DEBUG] skipping _wrapped_call_impl /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py
[2023-07-06 04:20:38,295] torch._dynamo.eval_frame: [DEBUG] skipping _call_impl /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py
[2023-07-06 04:20:38,296] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:477
[2023-07-06 04:20:38,297] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['input_ids'] (1, 1000) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:38,298] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['attention_mask'] (1, 1003) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:38,299] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['position_ids'] (1, 1000) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:38,299] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['past_key_values'][0] (2, 1, 4, 1003, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None, None]
[2023-07-06 04:20:38,300] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['past_key_values'][1] (2, 1, 4, 1003, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None, None]
[2023-07-06 04:20:38,301] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['valid_past_index'] () [] []
[2023-07-06 04:20:38,301] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:490
[2023-07-06 04:20:38,301] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_DEREF output_attentions []
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [ConstantVariable(bool)]
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [ConstantVariable(bool), ConstantVariable(NoneType)]
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 12 [ConstantVariable(bool)]
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_DEREF output_attentions []
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE JUMP_FORWARD 18 [ConstantVariable(bool)]
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_DEREF output_attentions [ConstantVariable(bool)]
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:492
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_hidden_states []
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [ConstantVariable(bool)]
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [ConstantVariable(bool), ConstantVariable(NoneType)]
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 32 [ConstantVariable(bool)]
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_hidden_states []
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE JUMP_FORWARD 38 [ConstantVariable(bool)]
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:491
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST output_hidden_states [ConstantVariable(bool)]
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:494
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST return_dict []
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [ConstantVariable(bool)]
[2023-07-06 04:20:38,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [ConstantVariable(bool), ConstantVariable(NoneType)]
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 52 [ConstantVariable(bool)]
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST return_dict []
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE JUMP_FORWARD 58 [ConstantVariable(bool)]
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST return_dict [ConstantVariable(bool)]
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:497
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_ids []
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable()]
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [TensorVariable(), ConstantVariable(NoneType)]
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 84 [ConstantVariable(bool)]
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST inputs_embeds []
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [ConstantVariable(NoneType)]
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [ConstantVariable(NoneType), ConstantVariable(NoneType)]
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 84 [ConstantVariable(bool)]
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:499
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_ids []
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable()]
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [TensorVariable(), ConstantVariable(NoneType)]
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 104 [ConstantVariable(bool)]
[2023-07-06 04:20:38,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:500
[2023-07-06 04:20:38,304] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_ids []
[2023-07-06 04:20:38,304] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TensorVariable()]
[2023-07-06 04:20:38,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE UNPACK_SEQUENCE 2 [ShapeVariable()]
[2023-07-06 04:20:38,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST batch_size [ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:38,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST seq_length [ConstantVariable(int)]
[2023-07-06 04:20:38,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE JUMP_FORWARD 134 []
[2023-07-06 04:20:38,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:506
[2023-07-06 04:20:38,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST seq_length []
[2023-07-06 04:20:38,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST seq_length_with_past [ConstantVariable(int)]
[2023-07-06 04:20:38,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:507
[2023-07-06 04:20:38,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index []
[2023-07-06 04:20:38,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST past_key_values_length [TensorVariable()]
[2023-07-06 04:20:38,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:509
[2023-07-06 04:20:38,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids []
[2023-07-06 04:20:38,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable()]
[2023-07-06 04:20:38,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 0 [TensorVariable(), ConstantVariable(NoneType)]
[2023-07-06 04:20:38,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 214 [ConstantVariable(bool)]
[2023-07-06 04:20:38,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:516
[2023-07-06 04:20:38,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids []
[2023-07-06 04:20:38,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR view [TensorVariable()]
[2023-07-06 04:20:38,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [GetAttrVariable(TensorVariable(), view)]
[2023-07-06 04:20:38,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST seq_length [GetAttrVariable(TensorVariable(), view), ConstantVariable(int)]
[2023-07-06 04:20:38,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:38,322] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR long [TensorVariable()]
[2023-07-06 04:20:38,322] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), long)]
[2023-07-06 04:20:38,322] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST position_ids [TensorVariable()]
[2023-07-06 04:20:38,322] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:518
[2023-07-06 04:20:38,322] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST inputs_embeds []
[2023-07-06 04:20:38,322] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [ConstantVariable(NoneType)]
[2023-07-06 04:20:38,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 0 [ConstantVariable(NoneType), ConstantVariable(NoneType)]
[2023-07-06 04:20:38,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 248 [ConstantVariable(bool)]
[2023-07-06 04:20:38,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:519
[2023-07-06 04:20:38,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR embed_tokens [NNModuleVariable()]
[2023-07-06 04:20:38,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_ids [NNModuleVariable()]
[2023-07-06 04:20:38,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:38,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST inputs_embeds [TensorVariable()]
[2023-07-06 04:20:38,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:521
[2023-07-06 04:20:38,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST attention_mask []
[2023-07-06 04:20:38,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable()]
[2023-07-06 04:20:38,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 0 [TensorVariable(), ConstantVariable(NoneType)]
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 280 [ConstantVariable(bool)]
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:527
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST batch_size []
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [ConstantVariable(int)]
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE COMPARE_OP > [ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 308 [ConstantVariable(bool)]
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:532
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST inputs_embeds []
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:535
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_hidden_states []
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 320 [ConstantVariable(bool)]
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None []
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST all_hidden_states [ConstantVariable(NoneType)]
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:536
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_DEREF output_attentions []
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 332 [ConstantVariable(bool)]
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None []
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST all_self_attns [ConstantVariable(NoneType)]
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:537
[2023-07-06 04:20:38,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST () []
[2023-07-06 04:20:38,335] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST next_decoder_cache [TupleVariable()]
[2023-07-06 04:20:38,335] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:539
[2023-07-06 04:20:38,335] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL enumerate []
[2023-07-06 04:20:38,335] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [BuiltinVariable(enumerate)]
[2023-07-06 04:20:38,335] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR layers [BuiltinVariable(enumerate), NNModuleVariable()]
[2023-07-06 04:20:38,335] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [BuiltinVariable(enumerate), NNModuleVariable()]
[2023-07-06 04:20:38,336] torch._dynamo.symbolic_convert: [DEBUG] TRACE GET_ITER None [TupleVariable()]
[2023-07-06 04:20:38,336] torch._dynamo.symbolic_convert: [DEBUG] TRACE FOR_ITER 516 [ListIteratorVariable()]
[2023-07-06 04:20:38,336] torch._dynamo.symbolic_convert: [DEBUG] TRACE UNPACK_SEQUENCE 2 [ListIteratorVariable(), TupleVariable()]
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST idx [ListIteratorVariable(), NNModuleVariable(), ConstantVariable(int)]
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST decoder_layer [ListIteratorVariable(), NNModuleVariable()]
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:540
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_hidden_states [ListIteratorVariable()]
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 372 [ListIteratorVariable(), ConstantVariable(bool)]
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:543
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_key_values [ListIteratorVariable()]
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [ListIteratorVariable(), ListVariable()]
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [ListIteratorVariable(), ListVariable(), ConstantVariable(NoneType)]
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 388 [ListIteratorVariable(), ConstantVariable(bool)]
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_key_values [ListIteratorVariable()]
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST idx [ListIteratorVariable(), ListVariable()]
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [ListIteratorVariable(), ListVariable(), ConstantVariable(int)]
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE JUMP_FORWARD 390 [ListIteratorVariable(), TensorVariable()]
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST past_key_value [ListIteratorVariable(), TensorVariable()]
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:545
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [ListIteratorVariable()]
[2023-07-06 04:20:38,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR gradient_checkpointing [ListIteratorVariable(), NNModuleVariable()]
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 444 [ListIteratorVariable(), ConstantVariable(bool)]
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:562
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST decoder_layer [ListIteratorVariable()]
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:563
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [ListIteratorVariable(), NNModuleVariable()]
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:564
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST attention_mask [ListIteratorVariable(), NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:565
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [ListIteratorVariable(), NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:566
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_key_value [ListIteratorVariable(), NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:567
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_DEREF output_attentions [ListIteratorVariable(), NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:568
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [ListIteratorVariable(), NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(bool)]
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:562
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('attention_mask', 'position_ids', 'past_key_value', 'output_attentions', 'valid_past_index') [ListIteratorVariable(), NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(bool), TensorVariable()]
[2023-07-06 04:20:38,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 6 [ListIteratorVariable(), NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(bool), TensorVariable(), ConstantVariable(tuple)]
[2023-07-06 04:20:38,339] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:38,340] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:38,340] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:38,340] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:38,340] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:38,341] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:38,341] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:38,341] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,341] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:38,341] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaDecoderLayer.forward at 0x7fb8d0c467a0>, NNModuleVariable())]
[2023-07-06 04:20:38,341] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,341] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,341] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,342] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,342] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,342] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,342] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,342] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,342] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,342] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,342] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,342] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,343] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,343] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:38,343] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:38,343] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,343] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,343] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:38,343] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:38,343] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,343] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,343] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:38,343] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:38,343] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,343] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,343] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:38,343] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:38,344] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,344] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,344] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:38,344] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:38,344] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaDecoderLayer.forward at 0x7fb8d0c467a0>, NNModuleVariable())]
[2023-07-06 04:20:38,344] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaDecoderLayer.forward at 0x7fb8d0c467a0>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:38,344] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaDecoderLayer.forward at 0x7fb8d0c467a0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:38,344] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaDecoderLayer.forward at 0x7fb8d0c467a0>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:38,344] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaDecoderLayer.forward at 0x7fb8d0c467a0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:38,345] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:38,345] torch._dynamo.symbolic_convert: [DEBUG] FAILED INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:38,345] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:38,345] torch._dynamo.symbolic_convert: [INFO] Skipping frame because there is a graph break in a for/while loop
<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 562 in forward>
[2023-07-06 04:20:38,345] torch._dynamo.convert_frame: [DEBUG] Skipping frame Skipping frame because there is a graph break in a for/while loop
<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 562 in forward> forward /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py 477
[2023-07-06 04:20:38,345] torch._dynamo.eval_frame: [DEBUG] skipping forward /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/sparse.py
[2023-07-06 04:20:38,345] torch._dynamo.eval_frame: [DEBUG] skipping embedding /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/functional.py
[2023-07-06 04:20:38,352] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing __iter__ /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/container.py:315
[2023-07-06 04:20:38,352] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/container.py:317
[2023-07-06 04:20:38,353] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL iter []
[2023-07-06 04:20:38,354] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [BuiltinVariable(iter)]
[2023-07-06 04:20:38,354] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _modules [BuiltinVariable(iter), NNModuleVariable()]
[2023-07-06 04:20:38,354] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR values [BuiltinVariable(iter), ConstDictVariable()]
[2023-07-06 04:20:38,355] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [BuiltinVariable(iter), GetAttrVariable(ConstDictVariable(), values)]
[2023-07-06 04:20:38,355] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [BuiltinVariable(iter), TupleVariable()]
[2023-07-06 04:20:38,355] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [ListIteratorVariable()]
[2023-07-06 04:20:38,355] torch._dynamo.convert_frame: [DEBUG] Skipping frame because no content in function call __iter__ /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/container.py 315
[2023-07-06 04:20:38,355] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:261
[2023-07-06 04:20:38,356] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['hidden_states'] (1, 1000, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:38,357] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['attention_mask'] (1, 1003) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:38,357] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['position_ids'] (1, 1000) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:38,359] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['past_key_value'] (2, 1, 4, 1003, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None, None]
[2023-07-06 04:20:38,360] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['valid_past_index'] () [] []
[2023-07-06 04:20:38,360] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:281
[2023-07-06 04:20:38,360] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:38,360] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST residual [TensorVariable()]
[2023-07-06 04:20:38,360] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:283
[2023-07-06 04:20:38,360] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,360] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR input_layernorm [NNModuleVariable()]
[2023-07-06 04:20:38,361] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:38,361] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:38,361] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:38,362] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:38,363] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:38,363] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:38,363] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:38,363] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:38,363] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:38,363] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,363] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:38,364] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:38,364] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,364] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,364] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,364] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,364] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,364] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,364] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,364] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,364] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:38,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:38,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:38,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:38,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:38,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:38,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:38,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:38,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:38,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:38,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:38,367] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:38,367] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:38,367] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:38,367] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:38,367] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:38,367] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:84
[2023-07-06 04:20:38,367] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:38,367] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [TensorVariable()]
[2023-07-06 04:20:38,370] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST input_dtype [TorchVariable(torch.float16)]
[2023-07-06 04:20:38,370] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85
[2023-07-06 04:20:38,370] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:38,370] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:38,370] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:38,370] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR float32 [GetAttrVariable(TensorVariable(), to), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:38,371] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), ConstantVariable(dtype)]
[2023-07-06 04:20:38,372] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR pow [TensorVariable()]
[2023-07-06 04:20:38,372] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [GetAttrVariable(TensorVariable(), pow)]
[2023-07-06 04:20:38,372] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), pow), ConstantVariable(int)]
[2023-07-06 04:20:38,374] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mean [TensorVariable()]
[2023-07-06 04:20:38,374] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [GetAttrVariable(TensorVariable(), mean)]
[2023-07-06 04:20:38,374] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST True [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int)]
[2023-07-06 04:20:38,374] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('keepdim',) [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool)]
[2023-07-06 04:20:38,374] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 2 [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool), ConstantVariable(tuple)]
[2023-07-06 04:20:38,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST variance [TensorVariable()]
[2023-07-06 04:20:38,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86
[2023-07-06 04:20:38,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:38,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:38,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR rsqrt [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:38,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST variance [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>)]
[2023-07-06 04:20:38,377] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:38,377] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR variance_epsilon [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:38,377] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(float)]
[2023-07-06 04:20:38,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:38,380] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:38,383] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:38,383] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88
[2023-07-06 04:20:38,383] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,383] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR weight [NNModuleVariable()]
[2023-07-06 04:20:38,383] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].input_layernorm.weight (16,) [<DimDynamic.STATIC: 2>] [None]
[2023-07-06 04:20:38,383] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:38,384] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:38,384] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:38,385] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_dtype [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:38,385] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:38,386] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:38,386] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:38,386] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:38,386] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:38,386] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:38,386] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:286
[2023-07-06 04:20:38,386] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,386] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR self_attn [NNModuleVariable()]
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:287
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:288
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST attention_mask [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:289
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:290
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_key_value [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:291
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:292
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(bool)]
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:286
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('hidden_states', 'attention_mask', 'position_ids', 'past_key_value', 'output_attentions', 'valid_past_index') [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(bool), TensorVariable()]
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 6 [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(bool), TensorVariable(), ConstantVariable(tuple)]
[2023-07-06 04:20:38,387] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:38,389] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:38,389] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:38,389] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:38,389] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:38,389] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:38,389] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:38,389] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,389] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:38,390] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable())]
[2023-07-06 04:20:38,390] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,390] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,390] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,390] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,390] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,390] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,390] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:38,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:38,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:38,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:38,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:38,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:38,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable())]
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:38,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:38,393] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:38,393] torch._dynamo.symbolic_convert: [DEBUG] FAILED INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:38,393] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:38,393] torch._dynamo.symbolic_convert: [DEBUG] break_graph_if_unsupported triggered compile
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2187, in inline_call_
sub_locals, closure_cells = func.bind_args(parent, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 187, in bind_args
[
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 188, in <listcomp>
wrap(val=arg, source=source)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 62, in wrap_bound_arg
return VariableBuilder(tx, source=source, **options)(val)
TypeError: VariableBuilder.__init__() got an unexpected keyword argument 'guards'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 390, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1152, in CALL_FUNCTION_KW
self.call_function(fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 559, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 333, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 595, in inline_user_function_return
result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2132, in inline_call
return cls.inline_call_(parent, func, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2229, in inline_call_
tracer.run()
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 708, in run
and self.step()
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 668, in step
getattr(self, inst.opname)(inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 390, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1140, in CALL_FUNCTION_EX
self.call_function(fn, argsvars.items, kwargsvars.items)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 559, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 333, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 287, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 120, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 595, in inline_user_function_return
result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2132, in inline_call
return cls.inline_call_(parent, func, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2190, in inline_call_
raise ArgsMismatchError(
torch._dynamo.exc.ArgsMismatchError: VariableBuilder.__init__() got an unexpected keyword argument 'guards'.
func = 'forward' /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:190, args = [<class 'trfs_fast.llama.LlamaAttention'>], kwargs = OrderedDict([('hidden_states', TensorVariable()), ('attention_mask', TensorVariable()), ('position_ids', TensorVariable()), ('past_key_value', TensorVariable()), ('output_attentions', ConstantVariable(bool)), ('valid_past_index', TensorVariable())])
[2023-07-06 04:20:38,393] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:38,393] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason="VariableBuilder.__init__() got an unexpected keyword argument 'guards'.\n func = 'forward' /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:190, args = [<class 'trfs_fast.llama.LlamaAttention'>], kwargs = OrderedDict([('hidden_states', TensorVariable()), ('attention_mask', TensorVariable()), ('position_ids', TensorVariable()), ('past_key_value', TensorVariable()), ('output_attentions', ConstantVariable(bool)), ('valid_past_index', TensorVariable())])", user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 286 in forward>, <FrameSummary file /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py, line 1514 in _call_impl>], graph_break=True)
[2023-07-06 04:20:38,394] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['attention_mask']
[2023-07-06 04:20:38,395] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['position_ids']
[2023-07-06 04:20:38,395] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['past_key_value']
[2023-07-06 04:20:38,395] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['valid_past_index']
[2023-07-06 04:20:38,395] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_1 =====
<eval_with_key>.0 class GraphModule(torch.nn.Module):
def forward(self, L_hidden_states_ : torch.Tensor):
l_hidden_states_ = L_hidden_states_
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
to = l_hidden_states_.to(torch.float32)
pow_1 = to.pow(2); to = None
mean = pow_1.mean(-1, keepdim = True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add = mean + 1e-06; mean = None
rsqrt = torch.rsqrt(add); add = None
mul = l_hidden_states_ * rsqrt; l_hidden_states_ = rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
l__self___input_layernorm_weight = self.L__self___input_layernorm_weight
mul_1 = l__self___input_layernorm_weight * mul; l__self___input_layernorm_weight = mul = None
to_1 = mul_1.to(torch.float16); mul_1 = None
return (to_1,)
[2023-07-06 04:20:38,396] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_1 <eval_with_key>.0 opcode name target args kwargs
------------- -------------------------------- -------------------------------------------------------- --------------------------------------- -----------------
placeholder l_hidden_states_ L_hidden_states_ () {}
call_method to to (l_hidden_states_, torch.float32) {}
call_method pow_1 pow (to, 2) {}
call_method mean mean (pow_1, -1) {'keepdim': True}
call_function add <built-in function add> (mean, 1e-06) {}
call_function rsqrt <built-in method rsqrt of type object at 0x7fb940469580> (add,) {}
call_function mul <built-in function mul> (l_hidden_states_, rsqrt) {}
get_attr l__self___input_layernorm_weight L__self___input_layernorm_weight () {}
call_function mul_1 <built-in function mul> (l__self___input_layernorm_weight, mul) {}
call_method to_1 to (mul_1, torch.float16) {}
output output output ((to_1,),) {}
[2023-07-06 04:20:38,401] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
Input hash: e4edb9de
Cache preallocation: True
Warming up (2 runs)...
[2023-07-06 04:20:38,710] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 0 =====
<eval_with_key>.4 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[16], arg1_1: f16[1, 1000, 16]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
convert_element_type: f32[1, 1000, 16] = torch.ops.prims.convert_element_type.default(arg1_1, torch.float32)
pow_1: f32[1, 1000, 16] = torch.ops.aten.pow.Tensor_Scalar(convert_element_type, 2); convert_element_type = None
mean: f32[1, 1000, 1] = torch.ops.aten.mean.dim(pow_1, [-1], True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add: f32[1, 1000, 1] = torch.ops.aten.add.Tensor(mean, 1e-06); mean = None
rsqrt: f32[1, 1000, 1] = torch.ops.aten.rsqrt.default(add); add = None
mul: f32[1, 1000, 16] = torch.ops.aten.mul.Tensor(arg1_1, rsqrt); arg1_1 = rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
mul_1: f32[1, 1000, 16] = torch.ops.aten.mul.Tensor(arg0_1, mul); arg0_1 = mul = None
convert_element_type_1: f16[1, 1000, 16] = torch.ops.prims.convert_element_type.default(mul_1, torch.float16); mul_1 = None
return (convert_element_type_1,)
[2023-07-06 04:20:40,125] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 0
[2023-07-06 04:20:40,183] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 0
[2023-07-06 04:20:41,671] torch._inductor.scheduler: [DEBUG] remove_buffer('buf0')
[2023-07-06 04:20:42,300] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 4 configs
[2023-07-06 04:20:42,300] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 1, num_warps: 2, num_stages: 1
[2023-07-06 04:20:42,300] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 8, num_warps: 2, num_stages: 1
[2023-07-06 04:20:42,300] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 32, num_warps: 4, num_stages: 1
[2023-07-06 04:20:42,300] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 128, num_warps: 8, num_stages: 1
[2023-07-06 04:20:43,598] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 4 configs
[2023-07-06 04:20:43,598] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 1, num_warps: 2, num_stages: 1
[2023-07-06 04:20:43,598] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 8, num_warps: 2, num_stages: 1
[2023-07-06 04:20:43,598] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 32, num_warps: 4, num_stages: 1
[2023-07-06 04:20:43,598] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 128, num_warps: 8, num_stages: 1
[2023-07-06 04:20:43,720] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/tn/ctnjyv37bz6fvtmwelga5764xq272722mvbuxkb4qqgd3syt7hwl.py
[2023-07-06 04:20:43,725] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 0
[2023-07-06 04:20:43,727] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:43,737] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662763072)
hasattr(L['position_ids'], '_dynamo_dynamic_indices') == False
___check_type_id(L['hidden_states'], 77765680)
hasattr(L['hidden_states'], '_dynamo_dynamic_indices') == False
hasattr(L['attention_mask'], '_dynamo_dynamic_indices') == False
hasattr(L['past_key_value'], '_dynamo_dynamic_indices') == False
hasattr(L['valid_past_index'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['output_attentions'], 7698112)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
str(G['torch'].float32) == 'torch.float32'
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks.keys()) == set()
[2023-07-06 04:20:43,737] torch._dynamo.eval_frame: [DEBUG] skipping _fn /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py
[2023-07-06 04:20:43,738] torch._dynamo.eval_frame: [DEBUG] skipping nothing /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py
[2023-07-06 04:20:43,844] torch._inductor.cudagraph_trees: [DEBUG] Running warmup of function 0
[2023-07-06 04:20:44,100] torch._inductor.triton_heuristics: [DEBUG] Benchmark all input configs get:
[2023-07-06 04:20:44,100] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 1, num_warps: 2, num_stages: 1: 0.005120, nreg 16, nspill 0, #shared-mem 256
[2023-07-06 04:20:44,100] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 8, num_warps: 2, num_stages: 1: 0.005120, nreg 16, nspill 0, #shared-mem 256
[2023-07-06 04:20:44,100] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 32, num_warps: 4, num_stages: 1: 0.005120, nreg 24, nspill 0, #shared-mem 512
[2023-07-06 04:20:44,100] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 128, num_warps: 8, num_stages: 1: 0.005120, nreg 26, nspill 0, #shared-mem 1024
[2023-07-06 04:20:44,100] torch._inductor.triton_heuristics: [DEBUG] Save heuristic tuning result to /tmp/torchinductor_ubuntu/ms/cmsiodjdcnbq2hee6trx6wljayuvfvkl3gqgzjhqci2lua23fz6n.best_config
[2023-07-06 04:20:44,100] torch._dynamo.eval_frame: [DEBUG] skipping __exit__ /home/ubuntu/miniconda3/lib/python3.10/contextlib.py
[2023-07-06 04:20:44,100] torch._dynamo.eval_frame: [DEBUG] skipping __exit__ /home/ubuntu/miniconda3/lib/python3.10/contextlib.py
[2023-07-06 04:20:44,101] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:190
[2023-07-06 04:20:44,102] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['hidden_states'] (1, 1000, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:44,103] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['attention_mask'] (1, 1003) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:44,104] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['position_ids'] (1, 1000) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:44,105] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['past_key_value'] (2, 1, 4, 1003, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None, None]
[2023-07-06 04:20:44,105] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['valid_past_index'] () [] []
[2023-07-06 04:20:44,106] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:199
[2023-07-06 04:20:44,106] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions []
[2023-07-06 04:20:44,106] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST True [ConstantVariable(bool)]
[2023-07-06 04:20:44,106] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 0 [ConstantVariable(bool), ConstantVariable(bool)]
[2023-07-06 04:20:44,106] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(bool)]
[2023-07-06 04:20:44,106] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:202
[2023-07-06 04:20:44,106] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:44,106] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR size [TensorVariable()]
[2023-07-06 04:20:44,107] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), size)]
[2023-07-06 04:20:44,107] torch._dynamo.symbolic_convert: [DEBUG] TRACE UNPACK_SEQUENCE 3 [SizeVariable()]
[2023-07-06 04:20:44,107] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST bsz [ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,107] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST q_len [ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,107] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST _ [ConstantVariable(int)]
[2023-07-06 04:20:44,107] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:205
[2023-07-06 04:20:44,107] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:44,107] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR qkv_proj [NNModuleVariable()]
[2023-07-06 04:20:44,108] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:44,108] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:44,111] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR view [TensorVariable()]
[2023-07-06 04:20:44,111] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST bsz [GetAttrVariable(TensorVariable(), view)]
[2023-07-06 04:20:44,111] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_len [GetAttrVariable(TensorVariable(), view), ConstantVariable(int)]
[2023-07-06 04:20:44,111] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 3 [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,111] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,111] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR num_heads [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), NNModuleVariable()]
[2023-07-06 04:20:44,112] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,112] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR head_dim [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), NNModuleVariable()]
[2023-07-06 04:20:44,112] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 5 [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,113] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR permute [TensorVariable()]
[2023-07-06 04:20:44,113] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [GetAttrVariable(TensorVariable(), permute)]
[2023-07-06 04:20:44,113] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int)]
[2023-07-06 04:20:44,113] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 3 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,113] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,113] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 4 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,113] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 5 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,114] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST query_key_value_states [TensorVariable()]
[2023-07-06 04:20:44,114] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:207
[2023-07-06 04:20:44,114] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST query_key_value_states []
[2023-07-06 04:20:44,114] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TensorVariable()]
[2023-07-06 04:20:44,114] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,115] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST query_states [TensorVariable()]
[2023-07-06 04:20:44,115] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:208
[2023-07-06 04:20:44,115] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST query_key_value_states []
[2023-07-06 04:20:44,115] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [TensorVariable()]
[2023-07-06 04:20:44,115] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,115] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), ConstantVariable(int), ConstantVariable(NoneType)]
[2023-07-06 04:20:44,116] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), SliceVariable()]
[2023-07-06 04:20:44,117] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST key_value_states [TensorVariable()]
[2023-07-06 04:20:44,117] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:211
[2023-07-06 04:20:44,117] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:44,117] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR rotary_emb [NNModuleVariable()]
[2023-07-06 04:20:44,117] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states [NNModuleVariable()]
[2023-07-06 04:20:44,117] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:44,117] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_len [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,117] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [NNModuleVariable(), TensorVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,118] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('seq_len',) [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,118] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 2 [NNModuleVariable(), TensorVariable(), TensorVariable(), ConstantVariable(tuple)]
[2023-07-06 04:20:44,119] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:44,120] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:44,120] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:44,120] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:44,120] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:44,121] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:44,121] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:44,121] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:44,121] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:44,121] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable())]
[2023-07-06 04:20:44,121] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:44,121] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:44,121] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:44,122] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:44,122] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:44,122] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:44,122] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:44,122] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:44,122] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:44,122] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:44,122] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:44,123] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:44,123] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:44,123] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:44,123] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:44,123] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:44,123] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:44,123] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:44,123] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:44,123] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:44,123] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:44,123] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:44,123] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:44,123] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:44,124] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:44,124] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:44,124] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:44,124] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:44,124] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:44,124] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:44,124] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:44,124] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable())]
[2023-07-06 04:20:44,124] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:44,124] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:44,124] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:44,124] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:44,124] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddf80e0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 107>
[2023-07-06 04:20:44,125] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:122
[2023-07-06 04:20:44,125] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:44,125] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR cos_cached [NNModuleVariable()]
[2023-07-06 04:20:44,125] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].rotary_emb.cos_cached (1, 1, 2048, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None]
[2023-07-06 04:20:44,125] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:44,126] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:44,126] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [GetAttrVariable(TensorVariable(), to), TensorVariable()]
[2023-07-06 04:20:44,126] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('dtype',) [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:44,126] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 1 [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16), ConstantVariable(tuple)]
[2023-07-06 04:20:44,127] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:123
[2023-07-06 04:20:44,127] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable()]
[2023-07-06 04:20:44,127] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR sin_cached [TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:44,128] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].rotary_emb.sin_cached (1, 1, 2048, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None]
[2023-07-06 04:20:44,128] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,129] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TensorVariable(), GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:44,129] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [TensorVariable(), GetAttrVariable(TensorVariable(), to), TensorVariable()]
[2023-07-06 04:20:44,129] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('dtype',) [TensorVariable(), GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:44,129] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 1 [TensorVariable(), GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16), ConstantVariable(tuple)]
[2023-07-06 04:20:44,130] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:121
[2023-07-06 04:20:44,130] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 2 [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,130] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TupleVariable()]
[2023-07-06 04:20:44,130] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddf80e0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 107>
[2023-07-06 04:20:44,130] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TupleVariable()]
[2023-07-06 04:20:44,131] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:44,131] torch._dynamo.symbolic_convert: [DEBUG] TRACE UNPACK_SEQUENCE 2 [TupleVariable()]
[2023-07-06 04:20:44,131] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST cos [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,131] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST sin [TensorVariable()]
[2023-07-06 04:20:44,131] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:212
[2023-07-06 04:20:44,131] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL apply_rotary_pos_emb_opt []
[2023-07-06 04:20:44,131] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST query_states [UserFunctionVariable()]
[2023-07-06 04:20:44,131] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states [UserFunctionVariable(), TensorVariable()]
[2023-07-06 04:20:44,131] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [UserFunctionVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,131] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [UserFunctionVariable(), TensorVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,132] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos [UserFunctionVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,132] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin [UserFunctionVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,132] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [UserFunctionVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,132] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 5 [UserFunctionVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,132] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object apply_rotary_pos_emb_opt at 0x7fb8cddf82f0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 135>
[2023-07-06 04:20:44,132] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:139
[2023-07-06 04:20:44,132] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos []
[2023-07-06 04:20:44,132] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:44,133] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:44,133] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:44,134] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:44,134] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:44,134] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:44,135] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST cos [TensorVariable()]
[2023-07-06 04:20:44,135] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:140
[2023-07-06 04:20:44,135] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin []
[2023-07-06 04:20:44,135] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:44,135] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:44,135] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:44,136] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:44,136] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:44,136] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:44,137] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST sin [TensorVariable()]
[2023-07-06 04:20:44,137] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:141
[2023-07-06 04:20:44,137] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos []
[2023-07-06 04:20:44,137] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [TensorVariable()]
[2023-07-06 04:20:44,137] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,138] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR unsqueeze [TensorVariable()]
[2023-07-06 04:20:44,139] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), unsqueeze)]
[2023-07-06 04:20:44,139] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), unsqueeze), ConstantVariable(int)]
[2023-07-06 04:20:44,139] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST cos [TensorVariable()]
[2023-07-06 04:20:44,140] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:142
[2023-07-06 04:20:44,140] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin []
[2023-07-06 04:20:44,140] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [TensorVariable()]
[2023-07-06 04:20:44,140] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,141] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR unsqueeze [TensorVariable()]
[2023-07-06 04:20:44,141] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), unsqueeze)]
[2023-07-06 04:20:44,141] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), unsqueeze), ConstantVariable(int)]
[2023-07-06 04:20:44,142] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST sin [TensorVariable()]
[2023-07-06 04:20:44,142] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143
[2023-07-06 04:20:44,142] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q []
[2023-07-06 04:20:44,142] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos [TensorVariable()]
[2023-07-06 04:20:44,142] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,143] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL rotate_half [TensorVariable()]
[2023-07-06 04:20:44,143] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q [TensorVariable(), UserFunctionVariable()]
[2023-07-06 04:20:44,143] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), UserFunctionVariable(), TensorVariable()]
[2023-07-06 04:20:44,143] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:44,144] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129
[2023-07-06 04:20:44,144] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:44,144] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR roll [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:44,144] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>)]
[2023-07-06 04:20:44,144] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:44,144] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,144] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable()]
[2023-07-06 04:20:44,145] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,145] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,145] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,145] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,145] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('shifts', 'dims') [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,145] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 3 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int), ConstantVariable(tuple)]
[2023-07-06 04:20:44,146] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST x [TensorVariable()]
[2023-07-06 04:20:44,146] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130
[2023-07-06 04:20:44,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:44,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST Ellipsis [TensorVariable()]
[2023-07-06 04:20:44,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable(), ConstantVariable(ellipsis)]
[2023-07-06 04:20:44,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType)]
[2023-07-06 04:20:44,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), TensorVariable()]
[2023-07-06 04:20:44,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable()]
[2023-07-06 04:20:44,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:44,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,147] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:44,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 2 [TensorVariable(), ConstantVariable(ellipsis), SliceVariable()]
[2023-07-06 04:20:44,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE DUP_TOP_TWO None [TensorVariable(), TupleVariable()]
[2023-07-06 04:20:44,148] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TupleVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:44,149] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:44,149] torch._dynamo.symbolic_convert: [DEBUG] TRACE INPLACE_MULTIPLY None [TensorVariable(), TupleVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,149] torch._dynamo.symbolic_convert: [DEBUG] TRACE ROT_THREE None [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:44,150] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_SUBSCR None [TensorVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:44,150] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:131
[2023-07-06 04:20:44,150] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:44,150] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:44,150] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:44,150] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,150] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,151] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,152] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST q_embed [TensorVariable()]
[2023-07-06 04:20:44,152] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144
[2023-07-06 04:20:44,152] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_states []
[2023-07-06 04:20:44,152] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR copy_ [TensorVariable()]
[2023-07-06 04:20:44,152] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_states [GetAttrVariable(TensorVariable(), copy_)]
[2023-07-06 04:20:44,152] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos [GetAttrVariable(TensorVariable(), copy_), TensorVariable()]
[2023-07-06 04:20:44,152] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,153] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL rotate_half [GetAttrVariable(TensorVariable(), copy_), TensorVariable()]
[2023-07-06 04:20:44,153] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_states [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), UserFunctionVariable()]
[2023-07-06 04:20:44,153] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), UserFunctionVariable(), TensorVariable()]
[2023-07-06 04:20:44,153] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:44,154] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129
[2023-07-06 04:20:44,154] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:44,154] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR roll [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:44,154] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>)]
[2023-07-06 04:20:44,154] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:44,154] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,155] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable()]
[2023-07-06 04:20:44,155] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,155] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,155] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,155] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,155] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('shifts', 'dims') [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,155] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 3 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int), ConstantVariable(tuple)]
[2023-07-06 04:20:44,156] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST x [TensorVariable()]
[2023-07-06 04:20:44,156] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130
[2023-07-06 04:20:44,156] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:44,156] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST Ellipsis [TensorVariable()]
[2023-07-06 04:20:44,156] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable(), ConstantVariable(ellipsis)]
[2023-07-06 04:20:44,156] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType)]
[2023-07-06 04:20:44,156] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), TensorVariable()]
[2023-07-06 04:20:44,157] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable()]
[2023-07-06 04:20:44,157] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,157] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:44,157] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:44,157] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:44,157] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 2 [TensorVariable(), ConstantVariable(ellipsis), SliceVariable()]
[2023-07-06 04:20:44,157] torch._dynamo.symbolic_convert: [DEBUG] TRACE DUP_TOP_TWO None [TensorVariable(), TupleVariable()]
[2023-07-06 04:20:44,157] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TupleVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:44,158] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:44,158] torch._dynamo.symbolic_convert: [DEBUG] TRACE INPLACE_MULTIPLY None [TensorVariable(), TupleVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,159] torch._dynamo.symbolic_convert: [DEBUG] TRACE ROT_THREE None [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:44,159] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_SUBSCR None [TensorVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:44,159] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:131
[2023-07-06 04:20:44,159] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:44,159] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:44,159] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:44,159] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,159] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,160] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,161] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), copy_), TensorVariable()]
[2023-07-06 04:20:44,162] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_TOP None [TensorVariable()]
[2023-07-06 04:20:44,162] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:145
[2023-07-06 04:20:44,162] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_embed []
[2023-07-06 04:20:44,162] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:44,162] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object apply_rotary_pos_emb_opt at 0x7fb8cddf82f0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 135>
[2023-07-06 04:20:44,162] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST query_states [TensorVariable()]
[2023-07-06 04:20:44,162] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:215
[2023-07-06 04:20:44,162] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index []
[2023-07-06 04:20:44,162] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST past_kv_slice_start [TensorVariable()]
[2023-07-06 04:20:44,162] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:216
[2023-07-06 04:20:44,162] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:44,162] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR eq [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:44,163] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TorchVariable(<built-in method eq of type object at 0x7fb940469580>)]
[2023-07-06 04:20:44,163] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:44,163] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,164] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable()]
[2023-07-06 04:20:44,164] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:44,165] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_len [TensorVariable()]
[2023-07-06 04:20:44,165] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,166] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:44,166] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR ne [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:44,166] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>)]
[2023-07-06 04:20:44,166] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:44,166] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,167] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,168] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TensorVariable(), GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:44,169] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,169] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,169] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,169] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,170] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,171] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST past_kv_slice_end [TensorVariable()]
[2023-07-06 04:20:44,171] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:217
[2023-07-06 04:20:44,171] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:44,171] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR eq [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:44,172] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TorchVariable(<built-in method eq of type object at 0x7fb940469580>)]
[2023-07-06 04:20:44,172] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:44,172] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,172] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable()]
[2023-07-06 04:20:44,173] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:44,174] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states [TensorVariable()]
[2023-07-06 04:20:44,174] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,174] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -2 [TensorVariable(), ShapeVariable()]
[2023-07-06 04:20:44,174] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,174] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,175] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:44,175] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR ne [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:44,175] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>)]
[2023-07-06 04:20:44,176] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:44,176] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:44,176] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,177] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TensorVariable(), GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:44,178] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_kv_slice_end [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,178] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,178] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,179] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST past_state_slice_end [TensorVariable()]
[2023-07-06 04:20:44,179] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:218
[2023-07-06 04:20:44,179] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states []
[2023-07-06 04:20:44,179] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_key_value [TensorVariable()]
[2023-07-06 04:20:44,179] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST Ellipsis [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,179] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_kv_slice_start [TensorVariable(), TensorVariable(), ConstantVariable(ellipsis)]
[2023-07-06 04:20:44,179] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_kv_slice_end [TensorVariable(), TensorVariable(), ConstantVariable(ellipsis), TensorVariable()]
[2023-07-06 04:20:44,180] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), TensorVariable(), ConstantVariable(ellipsis), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:44,180] torch._dynamo.symbolic_convert: [DEBUG] step triggered compile
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 668, in step
getattr(self, inst.opname)(inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1251, in BUILD_SLICE
SliceVariable(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/base.py", line 27, in __call__
obj = type.__call__(cls, *args, **kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/lists.py", line 616, in __init__
unimplemented("Dynamic slicing on data-dependent value is not supported")
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 140, in unimplemented
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: Dynamic slicing on data-dependent value is not supported
[2023-07-06 04:20:44,181] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:44,182] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='step_unsupported', user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 218 in forward>], graph_break=True)
[2023-07-06 04:20:44,183] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['attention_mask']
[2023-07-06 04:20:44,183] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['past_key_value']
[2023-07-06 04:20:44,185] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_3 =====
<eval_with_key>.111 class GraphModule(torch.nn.Module):
def forward(self, L_hidden_states_ : torch.Tensor, L_position_ids_ : torch.Tensor, L_valid_past_index_ : torch.Tensor):
l_hidden_states_ = L_hidden_states_
l_position_ids_ = L_position_ids_
l_valid_past_index_ = L_valid_past_index_
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:205, code: query_key_value_states = self.qkv_proj(hidden_states).view(bsz, q_len, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4)
l__self___qkv_proj = self.L__self___qkv_proj(l_hidden_states_); l_hidden_states_ = None
view = l__self___qkv_proj.view(1, 1000, 3, 4, 4); l__self___qkv_proj = None
permute = view.permute(2, 0, 3, 1, 4); view = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:207, code: query_states = query_key_value_states[0]
getitem = permute[0]
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:208, code: key_value_states = query_key_value_states[1:]
getitem_1 = permute[slice(1, None, None)]; permute = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:211, code: cos, sin = self.rotary_emb(key_value_states, seq_len=valid_past_index + q_len)
add = l_valid_past_index_ + 1000
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:122, code: self.cos_cached.to(dtype=x.dtype),
l__self___rotary_emb_cos_cached = self.L__self___rotary_emb_cos_cached
to = l__self___rotary_emb_cos_cached.to(dtype = torch.float16); l__self___rotary_emb_cos_cached = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:123, code: self.sin_cached.to(dtype=x.dtype),
l__self___rotary_emb_sin_cached = self.L__self___rotary_emb_sin_cached
to_1 = l__self___rotary_emb_sin_cached.to(dtype = torch.float16); l__self___rotary_emb_sin_cached = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:212, code: query_states = apply_rotary_pos_emb_opt(query_states, key_value_states[0], cos, sin, position_ids)
getitem_2 = getitem_1[0]
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:139, code: cos = cos.squeeze(1).squeeze(0) # [seq_len, dim]
squeeze = to.squeeze(1); to = None
squeeze_1 = squeeze.squeeze(0); squeeze = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:140, code: sin = sin.squeeze(1).squeeze(0) # [seq_len, dim]
squeeze_2 = to_1.squeeze(1); to_1 = None
squeeze_3 = squeeze_2.squeeze(0); squeeze_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:141, code: cos = cos[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
getitem_3 = squeeze_1[l_position_ids_]; squeeze_1 = None
unsqueeze = getitem_3.unsqueeze(1); getitem_3 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:142, code: sin = sin[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
getitem_4 = squeeze_3[l_position_ids_]; squeeze_3 = l_position_ids_ = None
unsqueeze_1 = getitem_4.unsqueeze(1); getitem_4 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul = getitem * unsqueeze
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll = torch.roll(getitem, shifts = 2, dims = -1); getitem = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
getitem_5 = roll[(Ellipsis, slice(None, 2, None))]
getitem_5 *= -1; imul = getitem_5; getitem_5 = None
roll[(Ellipsis, slice(None, 2, None))] = imul; setitem = roll; imul = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul_1 = roll * unsqueeze_1; roll = None
add_1 = mul + mul_1; mul = mul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_2 = getitem_2 * unsqueeze; unsqueeze = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll_1 = torch.roll(getitem_2, shifts = 2, dims = -1)
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
getitem_6 = roll_1[(Ellipsis, slice(None, 2, None))]
getitem_6 *= -1; imul_1 = getitem_6; getitem_6 = None
roll_1[(Ellipsis, slice(None, 2, None))] = imul_1; setitem_1 = roll_1; imul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_3 = roll_1 * unsqueeze_1; roll_1 = unsqueeze_1 = None
add_2 = mul_2 + mul_3; mul_2 = mul_3 = None
copy_ = getitem_2.copy_(add_2); getitem_2 = add_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:216, code: past_kv_slice_end = torch.eq(valid_past_index, 0).int() * q_len + torch.ne(valid_past_index, 0).int() * (valid_past_index + 1)
eq = torch.eq(l_valid_past_index_, 0)
int_1 = eq.int(); eq = None
mul_4 = int_1 * 1000; int_1 = None
ne = torch.ne(l_valid_past_index_, 0)
int_2 = ne.int(); ne = None
add_3 = l_valid_past_index_ + 1
mul_5 = int_2 * add_3; int_2 = add_3 = None
add_4 = mul_4 + mul_5; mul_4 = mul_5 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:217, code: past_state_slice_end = torch.eq(valid_past_index, 0).int() * key_value_states.shape[-2] + torch.ne(valid_past_index, 0).int() * (past_kv_slice_end)
eq_1 = torch.eq(l_valid_past_index_, 0)
int_3 = eq_1.int(); eq_1 = None
mul_6 = int_3 * 1000; int_3 = None
ne_1 = torch.ne(l_valid_past_index_, 0); l_valid_past_index_ = None
int_4 = ne_1.int(); ne_1 = None
mul_7 = int_4 * add_4; int_4 = None
add_5 = mul_6 + mul_7; mul_6 = mul_7 = None
return (add_1, getitem_1, add_4, add_5)
[2023-07-06 04:20:44,186] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_3 <eval_with_key>.111 opcode name target args kwargs
------------- ------------------------------- ------------------------------------------------------- -------------------------------------------------- -------------------------
placeholder l_hidden_states_ L_hidden_states_ () {}
placeholder l_position_ids_ L_position_ids_ () {}
placeholder l_valid_past_index_ L_valid_past_index_ () {}
call_module l__self___qkv_proj L__self___qkv_proj (l_hidden_states_,) {}
call_method view view (l__self___qkv_proj, 1, 1000, 3, 4, 4) {}
call_method permute permute (view, 2, 0, 3, 1, 4) {}
call_function getitem <built-in function getitem> (permute, 0) {}
call_function getitem_1 <built-in function getitem> (permute, slice(1, None, None)) {}
call_function add <built-in function add> (l_valid_past_index_, 1000) {}
get_attr l__self___rotary_emb_cos_cached L__self___rotary_emb_cos_cached () {}
call_method to to (l__self___rotary_emb_cos_cached,) {'dtype': torch.float16}
get_attr l__self___rotary_emb_sin_cached L__self___rotary_emb_sin_cached () {}
call_method to_1 to (l__self___rotary_emb_sin_cached,) {'dtype': torch.float16}
call_function getitem_2 <built-in function getitem> (getitem_1, 0) {}
call_method squeeze squeeze (to, 1) {}
call_method squeeze_1 squeeze (squeeze, 0) {}
call_method squeeze_2 squeeze (to_1, 1) {}
call_method squeeze_3 squeeze (squeeze_2, 0) {}
call_function getitem_3 <built-in function getitem> (squeeze_1, l_position_ids_) {}
call_method unsqueeze unsqueeze (getitem_3, 1) {}
call_function getitem_4 <built-in function getitem> (squeeze_3, l_position_ids_) {}
call_method unsqueeze_1 unsqueeze (getitem_4, 1) {}
call_function mul <built-in function mul> (getitem, unsqueeze) {}
call_function roll <built-in method roll of type object at 0x7fb940469580> (getitem,) {'shifts': 2, 'dims': -1}
call_function getitem_5 <built-in function getitem> (roll, (Ellipsis, slice(None, 2, None))) {}
call_function imul <built-in function imul> (getitem_5, -1) {}
call_function setitem <built-in function setitem> (roll, (Ellipsis, slice(None, 2, None)), imul) {}
call_function mul_1 <built-in function mul> (roll, unsqueeze_1) {}
call_function add_1 <built-in function add> (mul, mul_1) {}
call_function mul_2 <built-in function mul> (getitem_2, unsqueeze) {}
call_function roll_1 <built-in method roll of type object at 0x7fb940469580> (getitem_2,) {'shifts': 2, 'dims': -1}
call_function getitem_6 <built-in function getitem> (roll_1, (Ellipsis, slice(None, 2, None))) {}
call_function imul_1 <built-in function imul> (getitem_6, -1) {}
call_function setitem_1 <built-in function setitem> (roll_1, (Ellipsis, slice(None, 2, None)), imul_1) {}
call_function mul_3 <built-in function mul> (roll_1, unsqueeze_1) {}
call_function add_2 <built-in function add> (mul_2, mul_3) {}
call_method copy_ copy_ (getitem_2, add_2) {}
call_function eq <built-in method eq of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_1 int (eq,) {}
call_function mul_4 <built-in function mul> (int_1, 1000) {}
call_function ne <built-in method ne of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_2 int (ne,) {}
call_function add_3 <built-in function add> (l_valid_past_index_, 1) {}
call_function mul_5 <built-in function mul> (int_2, add_3) {}
call_function add_4 <built-in function add> (mul_4, mul_5) {}
call_function eq_1 <built-in method eq of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_3 int (eq_1,) {}
call_function mul_6 <built-in function mul> (int_3, 1000) {}
call_function ne_1 <built-in method ne of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_4 int (ne_1,) {}
call_function mul_7 <built-in function mul> (int_4, add_4) {}
call_function add_5 <built-in function add> (mul_6, mul_7) {}
output output output ((add_1, getitem_1, add_4, add_5),) {}
[2023-07-06 04:20:44,188] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
[2023-07-06 04:20:44,314] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 14 =====
<eval_with_key>.115 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[48, 16], arg1_1: f32[1, 1, 2048, 4], arg2_1: f32[1, 1, 2048, 4], arg3_1: f16[1, 1000, 16], arg4_1: i64[1, 1000], arg5_1: i64[]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:205, code: query_key_value_states = self.qkv_proj(hidden_states).view(bsz, q_len, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4)
permute: f16[16, 48] = torch.ops.aten.permute.default(arg0_1, [1, 0]); arg0_1 = None
view: f16[1000, 16] = torch.ops.aten.view.default(arg3_1, [1000, 16]); arg3_1 = None
mm: f16[1000, 48] = torch.ops.aten.mm.default(view, permute); view = permute = None
view_1: f16[1, 1000, 48] = torch.ops.aten.view.default(mm, [1, 1000, 48])
view_2: f16[1, 1000, 3, 4, 4] = torch.ops.aten.view.default(view_1, [1, 1000, 3, 4, 4]); view_1 = None
permute_1: f16[3, 1, 4, 1000, 4] = torch.ops.aten.permute.default(view_2, [2, 0, 3, 1, 4]); view_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:207, code: query_states = query_key_value_states[0]
select: f16[1, 4, 1000, 4] = torch.ops.aten.select.int(permute_1, 0, 0)
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:208, code: key_value_states = query_key_value_states[1:]
slice_1: f16[2, 1, 4, 1000, 4] = torch.ops.aten.slice.Tensor(permute_1, 0, 1, 9223372036854775807); permute_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:122, code: self.cos_cached.to(dtype=x.dtype),
convert_element_type: f16[1, 1, 2048, 4] = torch.ops.prims.convert_element_type.default(arg1_1, torch.float16); arg1_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:123, code: self.sin_cached.to(dtype=x.dtype),
convert_element_type_1: f16[1, 1, 2048, 4] = torch.ops.prims.convert_element_type.default(arg2_1, torch.float16); arg2_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:212, code: query_states = apply_rotary_pos_emb_opt(query_states, key_value_states[0], cos, sin, position_ids)
select_1: f16[1, 4, 1000, 4] = torch.ops.aten.select.int(slice_1, 0, 0); slice_1 = None
# No stacktrace found for following nodes
squeeze: f16[1, 2048, 4] = torch.ops.aten.squeeze.dim(convert_element_type, 1); convert_element_type = None
squeeze_1: f16[2048, 4] = torch.ops.aten.squeeze.dim(squeeze, 0); squeeze = None
squeeze_2: f16[1, 2048, 4] = torch.ops.aten.squeeze.dim(convert_element_type_1, 1); convert_element_type_1 = None
squeeze_3: f16[2048, 4] = torch.ops.aten.squeeze.dim(squeeze_2, 0); squeeze_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:141, code: cos = cos[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
index: f16[1, 1000, 4] = torch.ops.aten.index.Tensor(squeeze_1, [arg4_1]); squeeze_1 = None
unsqueeze: f16[1, 1, 1000, 4] = torch.ops.aten.unsqueeze.default(index, 1); index = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:142, code: sin = sin[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
index_1: f16[1, 1000, 4] = torch.ops.aten.index.Tensor(squeeze_3, [arg4_1]); squeeze_3 = arg4_1 = None
unsqueeze_1: f16[1, 1, 1000, 4] = torch.ops.aten.unsqueeze.default(index_1, 1); index_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul: f16[1, 4, 1000, 4] = torch.ops.aten.mul.Tensor(select, unsqueeze)
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll: f16[1, 4, 1000, 4] = torch.ops.aten.roll.default(select, [2], [-1]); select = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
slice_2: f16[1, 4, 1000, 2] = torch.ops.aten.slice.Tensor(roll, 3, 0, 2)
mul_1: f16[1, 4, 1000, 2] = torch.ops.aten.mul.Tensor(slice_2, -1); slice_2 = None
slice_scatter: f16[1, 4, 1000, 4] = torch.ops.aten.slice_scatter.default(roll, mul_1, 3, 0, 2); roll = mul_1 = None
slice_3: f16[1, 4, 1000, 2] = torch.ops.aten.slice.Tensor(slice_scatter, 3, 0, 2)
slice_scatter_1: f16[1, 4, 1000, 4] = torch.ops.aten.slice_scatter.default(slice_scatter, slice_3, 3, 0, 2); slice_scatter = slice_3 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul_2: f16[1, 4, 1000, 4] = torch.ops.aten.mul.Tensor(slice_scatter_1, unsqueeze_1); slice_scatter_1 = None
add_1: f16[1, 4, 1000, 4] = torch.ops.aten.add.Tensor(mul, mul_2); mul = mul_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_3: f16[1, 4, 1000, 4] = torch.ops.aten.mul.Tensor(select_1, unsqueeze); unsqueeze = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll_1: f16[1, 4, 1000, 4] = torch.ops.aten.roll.default(select_1, [2], [-1]); select_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
slice_7: f16[1, 4, 1000, 2] = torch.ops.aten.slice.Tensor(roll_1, 3, 0, 2)
mul_4: f16[1, 4, 1000, 2] = torch.ops.aten.mul.Tensor(slice_7, -1); slice_7 = None
slice_scatter_2: f16[1, 4, 1000, 4] = torch.ops.aten.slice_scatter.default(roll_1, mul_4, 3, 0, 2); roll_1 = mul_4 = None
slice_8: f16[1, 4, 1000, 2] = torch.ops.aten.slice.Tensor(slice_scatter_2, 3, 0, 2)
slice_scatter_3: f16[1, 4, 1000, 4] = torch.ops.aten.slice_scatter.default(slice_scatter_2, slice_8, 3, 0, 2); slice_scatter_2 = slice_8 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_5: f16[1, 4, 1000, 4] = torch.ops.aten.mul.Tensor(slice_scatter_3, unsqueeze_1); slice_scatter_3 = unsqueeze_1 = None
add_2: f16[1, 4, 1000, 4] = torch.ops.aten.add.Tensor(mul_3, mul_5); mul_3 = mul_5 = None
view_3: f16[1, 1000, 48] = torch.ops.aten.view.default(mm, [1, 1000, 48]); mm = None
view_4: f16[1, 1000, 3, 4, 4] = torch.ops.aten.view.default(view_3, [1, 1000, 3, 4, 4]); view_3 = None
permute_2: f16[3, 1, 4, 1000, 4] = torch.ops.aten.permute.default(view_4, [2, 0, 3, 1, 4]); view_4 = None
slice_12: f16[2, 1, 4, 1000, 4] = torch.ops.aten.slice.Tensor(permute_2, 0, 1, 9223372036854775807)
select_scatter: f16[2, 1, 4, 1000, 4] = torch.ops.aten.select_scatter.default(slice_12, add_2, 0, 0); slice_12 = add_2 = None
slice_scatter_4: f16[3, 1, 4, 1000, 4] = torch.ops.aten.slice_scatter.default(permute_2, select_scatter, 0, 1, 9223372036854775807); permute_2 = select_scatter = None
permute_3: f16[1, 1000, 3, 4, 4] = torch.ops.aten.permute.default(slice_scatter_4, [1, 3, 0, 2, 4]); slice_scatter_4 = None
view_5: f16[1, 1000, 48] = torch.ops.aten.view.default(permute_3, [1, 1000, 48]); permute_3 = None
view_6: f16[1000, 48] = torch.ops.aten.view.default(view_5, [1000, 48]); view_5 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:216, code: past_kv_slice_end = torch.eq(valid_past_index, 0).int() * q_len + torch.ne(valid_past_index, 0).int() * (valid_past_index + 1)
eq: b8[] = torch.ops.aten.eq.Scalar(arg5_1, 0)
convert_element_type_2: i32[] = torch.ops.prims.convert_element_type.default(eq, torch.int32); eq = None
mul_6: i32[] = torch.ops.aten.mul.Tensor(convert_element_type_2, 1000); convert_element_type_2 = None
ne: b8[] = torch.ops.aten.ne.Scalar(arg5_1, 0)
convert_element_type_3: i32[] = torch.ops.prims.convert_element_type.default(ne, torch.int32); ne = None
add_3: i64[] = torch.ops.aten.add.Tensor(arg5_1, 1)
mul_7: i64[] = torch.ops.aten.mul.Tensor(convert_element_type_3, add_3); convert_element_type_3 = add_3 = None
add_4: i64[] = torch.ops.aten.add.Tensor(mul_6, mul_7); mul_6 = mul_7 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:217, code: past_state_slice_end = torch.eq(valid_past_index, 0).int() * key_value_states.shape[-2] + torch.ne(valid_past_index, 0).int() * (past_kv_slice_end)
eq_1: b8[] = torch.ops.aten.eq.Scalar(arg5_1, 0)
convert_element_type_4: i32[] = torch.ops.prims.convert_element_type.default(eq_1, torch.int32); eq_1 = None
mul_8: i32[] = torch.ops.aten.mul.Tensor(convert_element_type_4, 1000); convert_element_type_4 = None
ne_1: b8[] = torch.ops.aten.ne.Scalar(arg5_1, 0); arg5_1 = None
convert_element_type_5: i32[] = torch.ops.prims.convert_element_type.default(ne_1, torch.int32); ne_1 = None
mul_9: i64[] = torch.ops.aten.mul.Tensor(convert_element_type_5, add_4); convert_element_type_5 = None
add_5: i64[] = torch.ops.aten.add.Tensor(mul_8, mul_9); mul_8 = mul_9 = None
# No stacktrace found for following nodes
view_9: f16[1, 1000, 48] = torch.ops.aten.view.default(view_6, [1, 1000, 48]); view_6 = None
view_10: f16[1, 1000, 3, 4, 4] = torch.ops.aten.view.default(view_9, [1, 1000, 3, 4, 4]); view_9 = None
permute_5: f16[3, 1, 4, 1000, 4] = torch.ops.aten.permute.default(view_10, [2, 0, 3, 1, 4]); view_10 = None
slice_14: f16[2, 1, 4, 1000, 4] = torch.ops.aten.slice.Tensor(permute_5, 0, 1, 9223372036854775807); permute_5 = None
return (add_1, slice_14, add_4, add_5)
[2023-07-06 04:20:44,343] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 1
[2023-07-06 04:20:44,505] torch._inductor.ir: [DEBUG] convert_to_reinterpret_view failed: stride=[16000, 0, 4000, 4, 1] offset=16000 index=16000*r0 + 4000*r1 + 4*r2 + r3 + 16000*(((4*r1 + r3)//16)) + 16000
[2023-07-06 04:20:44,648] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 1
[2023-07-06 04:20:49,653] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 2 configs
[2023-07-06 04:20:49,653] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 512, num_warps: 4, num_stages: 1
[2023-07-06 04:20:49,653] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 256, num_warps: 4, num_stages: 1
[2023-07-06 04:20:49,653] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 2 configs
[2023-07-06 04:20:49,653] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 256, num_warps: 4, num_stages: 1
[2023-07-06 04:20:49,653] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 128, num_warps: 4, num_stages: 1
[2023-07-06 04:20:49,655] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 2 configs
[2023-07-06 04:20:49,655] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 256, num_warps: 4, num_stages: 1
[2023-07-06 04:20:49,655] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 128, num_warps: 4, num_stages: 1
[2023-07-06 04:20:50,220] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 2 configs
[2023-07-06 04:20:50,220] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 256, num_warps: 4, num_stages: 1
[2023-07-06 04:20:50,220] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 128, num_warps: 4, num_stages: 1
[2023-07-06 04:20:50,774] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 2 configs
[2023-07-06 04:20:50,774] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 512, num_warps: 4, num_stages: 1
[2023-07-06 04:20:50,774] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 256, num_warps: 4, num_stages: 1
[2023-07-06 04:20:50,785] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 2 configs
[2023-07-06 04:20:50,785] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 256, num_warps: 4, num_stages: 1
[2023-07-06 04:20:50,785] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 128, num_warps: 4, num_stages: 1
[2023-07-06 04:20:50,793] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/lo/clozylkw544bb7yqlyamdizaia7jo2uesdfqh4qhrfm2er5ahkkn.py
[2023-07-06 04:20:50,794] torch._inductor.compile_fx: [DEBUG] disabled cudagraphs because ['non-cuda device in graph']
skipping cudagraphs due to multiple devices
[2023-07-06 04:20:50,794] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 1
[2023-07-06 04:20:50,796] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:50,806] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662763168)
hasattr(L['position_ids'], '_dynamo_dynamic_indices') == False
hasattr(L['hidden_states'], '_dynamo_dynamic_indices') == False
hasattr(L['attention_mask'], '_dynamo_dynamic_indices') == False
hasattr(L['past_key_value'], '_dynamo_dynamic_indices') == False
hasattr(L['valid_past_index'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['output_attentions'], 7698112)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks.keys()) == set()
[2023-07-06 04:20:51,067] torch._inductor.triton_heuristics: [DEBUG] Benchmark all input configs get:
[2023-07-06 04:20:51,067] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 256, num_warps: 4, num_stages: 1: 0.006144, nreg 56, nspill 0, #shared-mem 512
[2023-07-06 04:20:51,067] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 128, num_warps: 4, num_stages: 1: 0.005120, nreg 32, nspill 0, #shared-mem 0
[2023-07-06 04:20:51,067] torch._inductor.triton_heuristics: [DEBUG] Save heuristic tuning result to /tmp/torchinductor_ubuntu/bg/cbgmas5pmuel253gz3ahwbawktqf6aqc24qu3fx2cnoj45poporq.best_config
[2023-07-06 04:20:51,154] torch._inductor.triton_heuristics: [DEBUG] Benchmark all input configs get:
[2023-07-06 04:20:51,155] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 512, num_warps: 4, num_stages: 1: 0.005120, nreg 20, nspill 0, #shared-mem 0
[2023-07-06 04:20:51,155] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 256, num_warps: 4, num_stages: 1: 0.005120, nreg 14, nspill 0, #shared-mem 0
[2023-07-06 04:20:51,155] torch._inductor.triton_heuristics: [DEBUG] Save heuristic tuning result to /tmp/torchinductor_ubuntu/5p/c5px4pduotxc7ykett376w6ivfllsxhle4zfciermbzwclsz3whs.best_config
[2023-07-06 04:20:51,242] torch._inductor.triton_heuristics: [DEBUG] Benchmark all input configs get:
[2023-07-06 04:20:51,242] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 256, num_warps: 4, num_stages: 1: 0.005120, nreg 17, nspill 0, #shared-mem 0
[2023-07-06 04:20:51,242] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 128, num_warps: 4, num_stages: 1: 0.005120, nreg 13, nspill 0, #shared-mem 0
[2023-07-06 04:20:51,242] torch._inductor.triton_heuristics: [DEBUG] Save heuristic tuning result to /tmp/torchinductor_ubuntu/tu/ctum23fv2ehp4y2k4x7ymm3ssw7lch2csweiyhwgxheu4g67rwic.best_config
[2023-07-06 04:20:51,242] torch._dynamo.eval_frame: [DEBUG] skipping __iter__ /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_tensor.py
[2023-07-06 04:20:51,279] torch._dynamo.eval_frame: [DEBUG] skipping forward /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/linear.py
[2023-07-06 04:20:51,280] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <resume in forward> /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:286
[2023-07-06 04:20:51,280] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['___stack0'][0] (1, 1000, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:51,281] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['residual'] (1, 1000, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:51,282] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST ___stack0 []
[2023-07-06 04:20:51,282] torch._dynamo.symbolic_convert: [DEBUG] TRACE JUMP_ABSOLUTE 38 [TupleVariable()]
[2023-07-06 04:20:51,282] torch._dynamo.symbolic_convert: [DEBUG] TRACE UNPACK_SEQUENCE 3 [TupleVariable()]
[2023-07-06 04:20:51,282] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [ConstantVariable(NoneType), ConstantVariable(NoneType), TensorVariable()]
[2023-07-06 04:20:51,282] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST self_attn_weights [ConstantVariable(NoneType), ConstantVariable(NoneType)]
[2023-07-06 04:20:51,282] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST present_key_value [ConstantVariable(NoneType)]
[2023-07-06 04:20:51,282] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:294
[2023-07-06 04:20:51,282] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST residual []
[2023-07-06 04:20:51,282] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:51,282] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:51,283] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:51,284] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:297
[2023-07-06 04:20:51,284] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:51,284] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST residual [TensorVariable()]
[2023-07-06 04:20:51,284] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:298
[2023-07-06 04:20:51,284] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,284] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR post_attention_layernorm [NNModuleVariable()]
[2023-07-06 04:20:51,284] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:51,284] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:51,285] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:51,286] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:51,286] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:51,286] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:51,287] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:51,287] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:51,287] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:51,287] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,287] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:51,287] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:51,287] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:51,287] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,288] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:51,288] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,288] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,288] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:51,288] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,288] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,288] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:51,289] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,289] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,289] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:51,289] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,289] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:51,289] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:51,289] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:51,289] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,289] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:51,289] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:51,289] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:51,290] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:51,291] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:51,291] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:84
[2023-07-06 04:20:51,291] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:51,291] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [TensorVariable()]
[2023-07-06 04:20:51,291] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST input_dtype [TorchVariable(torch.float16)]
[2023-07-06 04:20:51,291] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85
[2023-07-06 04:20:51,291] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:51,291] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:51,292] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:51,292] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR float32 [GetAttrVariable(TensorVariable(), to), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:51,292] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), ConstantVariable(dtype)]
[2023-07-06 04:20:51,293] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR pow [TensorVariable()]
[2023-07-06 04:20:51,293] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [GetAttrVariable(TensorVariable(), pow)]
[2023-07-06 04:20:51,293] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), pow), ConstantVariable(int)]
[2023-07-06 04:20:51,294] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mean [TensorVariable()]
[2023-07-06 04:20:51,295] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [GetAttrVariable(TensorVariable(), mean)]
[2023-07-06 04:20:51,295] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST True [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int)]
[2023-07-06 04:20:51,295] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('keepdim',) [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool)]
[2023-07-06 04:20:51,295] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 2 [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool), ConstantVariable(tuple)]
[2023-07-06 04:20:51,296] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST variance [TensorVariable()]
[2023-07-06 04:20:51,296] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86
[2023-07-06 04:20:51,296] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:51,296] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:51,296] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR rsqrt [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:51,296] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST variance [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>)]
[2023-07-06 04:20:51,296] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:51,296] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR variance_epsilon [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:51,297] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(float)]
[2023-07-06 04:20:51,297] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:51,298] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:51,299] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:51,299] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88
[2023-07-06 04:20:51,299] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,299] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR weight [NNModuleVariable()]
[2023-07-06 04:20:51,299] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].post_attention_layernorm.weight (16,) [<DimDynamic.STATIC: 2>] [None]
[2023-07-06 04:20:51,300] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:51,300] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:51,301] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:51,301] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_dtype [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:51,301] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:51,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:51,302] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:51,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:51,302] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:51,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:51,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:299
[2023-07-06 04:20:51,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,302] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mlp [NNModuleVariable()]
[2023-07-06 04:20:51,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:51,303] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:51,303] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:51,305] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:51,305] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:51,305] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:51,306] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:51,306] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:51,306] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:51,306] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,306] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:51,307] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable())]
[2023-07-06 04:20:51,307] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:51,307] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,307] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:51,307] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,307] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,307] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:51,307] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,307] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,307] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:51,308] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,308] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,308] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:51,308] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,308] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:51,308] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:51,308] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:51,308] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,308] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:51,308] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:51,308] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:51,308] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,308] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:51,308] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:51,309] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:51,309] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,309] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:51,309] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:51,309] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:51,309] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,309] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:51,309] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:51,309] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable())]
[2023-07-06 04:20:51,309] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:51,309] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:51,309] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:51,309] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:51,309] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddf85b0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 161>
[2023-07-06 04:20:51,309] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162
[2023-07-06 04:20:51,310] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,310] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR down_proj [NNModuleVariable()]
[2023-07-06 04:20:51,310] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [NNModuleVariable()]
[2023-07-06 04:20:51,310] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR act_fn [NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:51,310] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:51,310] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR gate_proj [NNModuleVariable(), NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:51,311] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [NNModuleVariable(), NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:51,311] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), NNModuleVariable(), NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:51,314] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:51,314] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:51,315] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:51,315] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:51,315] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:51,316] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:51,316] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:51,316] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:51,316] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,316] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:51,316] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable())]
[2023-07-06 04:20:51,316] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:51,316] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,316] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:51,317] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,317] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,317] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:51,317] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,317] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,317] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:51,318] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:51,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:51,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:51,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:51,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable())]
[2023-07-06 04:20:51,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:51,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:51,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:51,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:51,319] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8ccf62760, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py", line 149>
[2023-07-06 04:20:51,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py:150
[2023-07-06 04:20:51,319] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL nn []
[2023-07-06 04:20:51,320] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR functional [TorchVariable(<module 'torch.nn' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/__init__.py'>)]
[2023-07-06 04:20:51,321] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR silu [TorchVariable(<module 'torch.nn.functional' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/functional.py'>)]
[2023-07-06 04:20:51,321] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input [TorchVariable(<function silu at 0x7fb945bc5990>)]
[2023-07-06 04:20:51,321] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TorchVariable(<function silu at 0x7fb945bc5990>), TensorVariable()]
[2023-07-06 04:20:51,325] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:51,325] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8ccf62760, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py", line 149>
[2023-07-06 04:20:51,325] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:51,325] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:51,325] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:51,325] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR up_proj [NNModuleVariable(), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:51,326] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [NNModuleVariable(), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:51,326] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable(), NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:51,328] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:51,329] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:51,332] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:51,332] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddf85b0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 161>
[2023-07-06 04:20:51,332] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:51,332] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:51,332] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:51,332] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:300
[2023-07-06 04:20:51,332] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST residual []
[2023-07-06 04:20:51,332] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:51,332] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:51,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:51,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:302
[2023-07-06 04:20:51,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:51,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 1 [TensorVariable()]
[2023-07-06 04:20:51,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST outputs [TupleVariable()]
[2023-07-06 04:20:51,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:304
[2023-07-06 04:20:51,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions []
[2023-07-06 04:20:51,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 106 [ConstantVariable(bool)]
[2023-07-06 04:20:51,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:307
[2023-07-06 04:20:51,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs []
[2023-07-06 04:20:51,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST present_key_value [TupleVariable()]
[2023-07-06 04:20:51,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 1 [TupleVariable(), ConstantVariable(NoneType)]
[2023-07-06 04:20:51,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE INPLACE_ADD None [TupleVariable(), TupleVariable()]
[2023-07-06 04:20:51,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST outputs [TupleVariable()]
[2023-07-06 04:20:51,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:309
[2023-07-06 04:20:51,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs []
[2023-07-06 04:20:51,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TupleVariable()]
[2023-07-06 04:20:51,334] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing <resume in forward> (RETURN_VALUE)
[2023-07-06 04:20:51,334] torch._dynamo.symbolic_convert: [DEBUG] RETURN_VALUE triggered compile
[2023-07-06 04:20:51,334] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 309 in <resume in forward>>], graph_break=False)
[2023-07-06 04:20:51,336] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_4 =====
<eval_with_key>.120 class GraphModule(torch.nn.Module):
def forward(self, L_stack0_0_ : torch.Tensor, L_residual_ : torch.Tensor):
l_stack0_0_ = L_stack0_0_
l_residual_ = L_residual_
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:294, code: hidden_states = residual + hidden_states
add = l_residual_ + l_stack0_0_; l_residual_ = l_stack0_0_ = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
to = add.to(torch.float32)
pow_1 = to.pow(2); to = None
mean = pow_1.mean(-1, keepdim = True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add_1 = mean + 1e-06; mean = None
rsqrt = torch.rsqrt(add_1); add_1 = None
mul = add * rsqrt; rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
l__self___post_attention_layernorm_weight = self.L__self___post_attention_layernorm_weight
mul_1 = l__self___post_attention_layernorm_weight * mul; l__self___post_attention_layernorm_weight = mul = None
to_1 = mul_1.to(torch.float16); mul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
l__self___mlp_gate_proj = self.L__self___mlp_gate_proj(to_1)
# File: /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py:150, code: return nn.functional.silu(input)
silu = torch.nn.functional.silu(l__self___mlp_gate_proj); l__self___mlp_gate_proj = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
l__self___mlp_up_proj = self.L__self___mlp_up_proj(to_1); to_1 = None
mul_2 = silu * l__self___mlp_up_proj; silu = l__self___mlp_up_proj = None
l__self___mlp_down_proj = self.L__self___mlp_down_proj(mul_2); mul_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:300, code: hidden_states = residual + hidden_states
add_2 = add + l__self___mlp_down_proj; add = l__self___mlp_down_proj = None
return (add_2,)
[2023-07-06 04:20:51,336] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_4 <eval_with_key>.120 opcode name target args kwargs
------------- ----------------------------------------- -------------------------------------------------------- ------------------------------------------------ -----------------
placeholder l_stack0_0_ L_stack0_0_ () {}
placeholder l_residual_ L_residual_ () {}
call_function add <built-in function add> (l_residual_, l_stack0_0_) {}
call_method to to (add, torch.float32) {}
call_method pow_1 pow (to, 2) {}
call_method mean mean (pow_1, -1) {'keepdim': True}
call_function add_1 <built-in function add> (mean, 1e-06) {}
call_function rsqrt <built-in method rsqrt of type object at 0x7fb940469580> (add_1,) {}
call_function mul <built-in function mul> (add, rsqrt) {}
get_attr l__self___post_attention_layernorm_weight L__self___post_attention_layernorm_weight () {}
call_function mul_1 <built-in function mul> (l__self___post_attention_layernorm_weight, mul) {}
call_method to_1 to (mul_1, torch.float16) {}
call_module l__self___mlp_gate_proj L__self___mlp_gate_proj (to_1,) {}
call_function silu <function silu at 0x7fb945bc5990> (l__self___mlp_gate_proj,) {}
call_module l__self___mlp_up_proj L__self___mlp_up_proj (to_1,) {}
call_function mul_2 <built-in function mul> (silu, l__self___mlp_up_proj) {}
call_module l__self___mlp_down_proj L__self___mlp_down_proj (mul_2,) {}
call_function add_2 <built-in function add> (add, l__self___mlp_down_proj) {}
output output output ((add_2,),) {}
[2023-07-06 04:20:51,337] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
[2023-07-06 04:20:51,385] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 15 =====
<eval_with_key>.124 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[16], arg1_1: f16[64, 16], arg2_1: f16[64, 16], arg3_1: f16[16, 64], arg4_1: f16[1, 1000, 16], arg5_1: f16[1, 1000, 16]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:294, code: hidden_states = residual + hidden_states
add: f16[1, 1000, 16] = torch.ops.aten.add.Tensor(arg5_1, arg4_1); arg5_1 = arg4_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
convert_element_type: f32[1, 1000, 16] = torch.ops.prims.convert_element_type.default(add, torch.float32)
pow_1: f32[1, 1000, 16] = torch.ops.aten.pow.Tensor_Scalar(convert_element_type, 2); convert_element_type = None
mean: f32[1, 1000, 1] = torch.ops.aten.mean.dim(pow_1, [-1], True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add_1: f32[1, 1000, 1] = torch.ops.aten.add.Tensor(mean, 1e-06); mean = None
rsqrt: f32[1, 1000, 1] = torch.ops.aten.rsqrt.default(add_1); add_1 = None
mul: f32[1, 1000, 16] = torch.ops.aten.mul.Tensor(add, rsqrt); rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
mul_1: f32[1, 1000, 16] = torch.ops.aten.mul.Tensor(arg0_1, mul); arg0_1 = mul = None
convert_element_type_1: f16[1, 1000, 16] = torch.ops.prims.convert_element_type.default(mul_1, torch.float16); mul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
permute: f16[16, 64] = torch.ops.aten.permute.default(arg1_1, [1, 0]); arg1_1 = None
view: f16[1000, 16] = torch.ops.aten.view.default(convert_element_type_1, [1000, 16])
mm: f16[1000, 64] = torch.ops.aten.mm.default(view, permute); view = permute = None
view_1: f16[1, 1000, 64] = torch.ops.aten.view.default(mm, [1, 1000, 64]); mm = None
# File: /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py:150, code: return nn.functional.silu(input)
convert_element_type_2: f32[1, 1000, 64] = torch.ops.prims.convert_element_type.default(view_1, torch.float32); view_1 = None
sigmoid: f32[1, 1000, 64] = torch.ops.aten.sigmoid.default(convert_element_type_2)
mul_2: f32[1, 1000, 64] = torch.ops.aten.mul.Tensor(convert_element_type_2, sigmoid); convert_element_type_2 = sigmoid = None
convert_element_type_3: f16[1, 1000, 64] = torch.ops.prims.convert_element_type.default(mul_2, torch.float16); mul_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
permute_1: f16[16, 64] = torch.ops.aten.permute.default(arg2_1, [1, 0]); arg2_1 = None
view_2: f16[1000, 16] = torch.ops.aten.view.default(convert_element_type_1, [1000, 16]); convert_element_type_1 = None
mm_1: f16[1000, 64] = torch.ops.aten.mm.default(view_2, permute_1); view_2 = permute_1 = None
view_3: f16[1, 1000, 64] = torch.ops.aten.view.default(mm_1, [1, 1000, 64]); mm_1 = None
mul_3: f16[1, 1000, 64] = torch.ops.aten.mul.Tensor(convert_element_type_3, view_3); convert_element_type_3 = view_3 = None
permute_2: f16[64, 16] = torch.ops.aten.permute.default(arg3_1, [1, 0]); arg3_1 = None
view_4: f16[1000, 64] = torch.ops.aten.view.default(mul_3, [1000, 64]); mul_3 = None
mm_2: f16[1000, 16] = torch.ops.aten.mm.default(view_4, permute_2); view_4 = permute_2 = None
view_5: f16[1, 1000, 16] = torch.ops.aten.view.default(mm_2, [1, 1000, 16]); mm_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:300, code: hidden_states = residual + hidden_states
add_2: f16[1, 1000, 16] = torch.ops.aten.add.Tensor(add, view_5); add = view_5 = None
return (add_2,)
[2023-07-06 04:20:51,413] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 2
[2023-07-06 04:20:51,459] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 2
[2023-07-06 04:20:51,486] torch._inductor.scheduler: [DEBUG] remove_buffer('buf0')
[2023-07-06 04:20:52,116] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 2 configs
[2023-07-06 04:20:52,116] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 256, num_warps: 4, num_stages: 1
[2023-07-06 04:20:52,116] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 128, num_warps: 4, num_stages: 1
[2023-07-06 04:20:52,124] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 2 configs
[2023-07-06 04:20:52,125] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 512, num_warps: 4, num_stages: 1
[2023-07-06 04:20:52,125] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 256, num_warps: 4, num_stages: 1
[2023-07-06 04:20:52,125] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 4 configs
[2023-07-06 04:20:52,125] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 1, num_warps: 2, num_stages: 1
[2023-07-06 04:20:52,125] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 8, num_warps: 2, num_stages: 1
[2023-07-06 04:20:52,125] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 32, num_warps: 4, num_stages: 1
[2023-07-06 04:20:52,125] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 128, num_warps: 8, num_stages: 1
[2023-07-06 04:20:52,988] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 4 configs
[2023-07-06 04:20:52,988] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 1, num_warps: 2, num_stages: 1
[2023-07-06 04:20:52,988] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 8, num_warps: 2, num_stages: 1
[2023-07-06 04:20:52,988] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 32, num_warps: 4, num_stages: 1
[2023-07-06 04:20:52,988] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 128, num_warps: 8, num_stages: 1
[2023-07-06 04:20:53,009] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 2 configs
[2023-07-06 04:20:53,009] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 512, num_warps: 4, num_stages: 1
[2023-07-06 04:20:53,009] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 256, num_warps: 4, num_stages: 1
[2023-07-06 04:20:53,018] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 2 configs
[2023-07-06 04:20:53,018] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 256, num_warps: 4, num_stages: 1
[2023-07-06 04:20:53,019] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 128, num_warps: 4, num_stages: 1
[2023-07-06 04:20:53,026] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/or/cor2fodulphywxrtc7zlvp6tsjseyifwlaiv6ticnqodficz2elj.py
[2023-07-06 04:20:53,026] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 2
[2023-07-06 04:20:53,029] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:53,037] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662763072)
hasattr(L['residual'], '_dynamo_dynamic_indices') == False
___check_type_id(L['___stack0'], 7638272)
len(L['___stack0']) == 3
hasattr(L['___stack0'][0], '_dynamo_dynamic_indices') == False
___check_obj_id(L['___stack0'][1], 7649088)
___check_obj_id(L['___stack0'][2], 7649088)
___check_obj_id(L['output_attentions'], 7698112)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
str(G['torch'].float32) == 'torch.float32'
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks.keys()) == set()
[2023-07-06 04:20:53,037] torch._inductor.cudagraph_trees: [DEBUG] Running warmup of function 1
[2023-07-06 04:20:53,213] torch._inductor.triton_heuristics: [DEBUG] Benchmark all input configs get:
[2023-07-06 04:20:53,213] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 1, num_warps: 2, num_stages: 1: 0.005120, nreg 18, nspill 0, #shared-mem 256
[2023-07-06 04:20:53,213] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 8, num_warps: 2, num_stages: 1: 0.005120, nreg 18, nspill 0, #shared-mem 256
[2023-07-06 04:20:53,213] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 32, num_warps: 4, num_stages: 1: 0.005120, nreg 22, nspill 0, #shared-mem 512
[2023-07-06 04:20:53,213] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 128, num_warps: 8, num_stages: 1: 0.005120, nreg 28, nspill 0, #shared-mem 1024
[2023-07-06 04:20:53,214] torch._inductor.triton_heuristics: [DEBUG] Save heuristic tuning result to /tmp/torchinductor_ubuntu/eh/ceh7jbqrlrzcv3zqh4rrhwapliqluprt3lrh53lvvuhdpwvqyn7m.best_config
[2023-07-06 04:20:53,305] torch._inductor.triton_heuristics: [DEBUG] Benchmark all input configs get:
[2023-07-06 04:20:53,305] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 512, num_warps: 4, num_stages: 1: 0.009216, nreg 19, nspill 0, #shared-mem 0
[2023-07-06 04:20:53,305] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 256, num_warps: 4, num_stages: 1: 0.009216, nreg 16, nspill 0, #shared-mem 0
[2023-07-06 04:20:53,305] torch._inductor.triton_heuristics: [DEBUG] Save heuristic tuning result to /tmp/torchinductor_ubuntu/v5/cv57f2eqjbxmcec2s7yyljqzlj4o3t34kuus254ilu5x7nwh4o4o.best_config
[2023-07-06 04:20:53,397] torch._inductor.triton_heuristics: [DEBUG] Benchmark all input configs get:
[2023-07-06 04:20:53,397] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 256, num_warps: 4, num_stages: 1: 0.007168, nreg 14, nspill 0, #shared-mem 0
[2023-07-06 04:20:53,397] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 128, num_warps: 4, num_stages: 1: 0.007184, nreg 13, nspill 0, #shared-mem 0
[2023-07-06 04:20:53,398] torch._inductor.triton_heuristics: [DEBUG] Save heuristic tuning result to /tmp/torchinductor_ubuntu/yq/cyqg2c2iwzykrwd2ftiynpnxqx2veaxu4jqvchxyg2wxm6kg37hz.best_config
[2023-07-06 04:20:53,398] torch._dynamo.eval_frame: [DEBUG] skipping __call__ /home/ubuntu/miniconda3/lib/python3.10/weakref.py
[2023-07-06 04:20:53,398] torch._dynamo.eval_frame: [DEBUG] skipping del_ten /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_subclasses/meta_utils.py
[2023-07-06 04:20:53,398] torch._dynamo.eval_frame: [DEBUG] skipping pop /home/ubuntu/miniconda3/lib/python3.10/weakref.py
[2023-07-06 04:20:53,398] torch._dynamo.eval_frame: [DEBUG] skipping __hash__ /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/utils/weak.py
[2023-07-06 04:20:53,398] torch._dynamo.eval_frame: [DEBUG] skipping expired /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/multiprocessing/reductions.py
[2023-07-06 04:20:53,398] torch._dynamo.eval_frame: [DEBUG] skipping _expired /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/storage.py
[2023-07-06 04:20:53,398] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function forward in /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py', 'set env var TORCHDYNAMO_REPORT_GUARD_FAILURES=1 to debug further')
[2023-07-06 04:20:53,398] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:261
[2023-07-06 04:20:53,399] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['hidden_states'] (1, 1000, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:53,401] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['attention_mask'] (1, 1003) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:53,401] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['position_ids'] (1, 1000) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:53,402] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['past_key_value'] (2, 1, 4, 1003, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None, None]
[2023-07-06 04:20:53,403] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['valid_past_index'] () [] []
[2023-07-06 04:20:53,404] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:281
[2023-07-06 04:20:53,404] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:53,404] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST residual [TensorVariable()]
[2023-07-06 04:20:53,404] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:283
[2023-07-06 04:20:53,404] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,404] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR input_layernorm [NNModuleVariable()]
[2023-07-06 04:20:53,404] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:53,404] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:53,405] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:53,406] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:53,406] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:53,406] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:53,406] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:53,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:53,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:53,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:53,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:53,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:53,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:53,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:53,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:53,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:53,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:53,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:53,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:53,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:53,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:53,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:53,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:53,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:53,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:53,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:53,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:53,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:53,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:53,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:53,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:53,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:53,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:53,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:53,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:53,410] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:53,411] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:84
[2023-07-06 04:20:53,411] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:53,411] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [TensorVariable()]
[2023-07-06 04:20:53,413] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST input_dtype [TorchVariable(torch.float16)]
[2023-07-06 04:20:53,413] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85
[2023-07-06 04:20:53,413] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:53,413] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:53,414] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:53,414] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR float32 [GetAttrVariable(TensorVariable(), to), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:53,414] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), ConstantVariable(dtype)]
[2023-07-06 04:20:53,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR pow [TensorVariable()]
[2023-07-06 04:20:53,416] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [GetAttrVariable(TensorVariable(), pow)]
[2023-07-06 04:20:53,416] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), pow), ConstantVariable(int)]
[2023-07-06 04:20:53,417] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mean [TensorVariable()]
[2023-07-06 04:20:53,417] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [GetAttrVariable(TensorVariable(), mean)]
[2023-07-06 04:20:53,417] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST True [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int)]
[2023-07-06 04:20:53,417] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('keepdim',) [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool)]
[2023-07-06 04:20:53,417] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 2 [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool), ConstantVariable(tuple)]
[2023-07-06 04:20:53,418] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST variance [TensorVariable()]
[2023-07-06 04:20:53,418] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86
[2023-07-06 04:20:53,418] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:53,418] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:53,418] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR rsqrt [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:53,418] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST variance [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>)]
[2023-07-06 04:20:53,419] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:53,419] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR variance_epsilon [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:53,419] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(float)]
[2023-07-06 04:20:53,420] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:53,420] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,421] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:53,421] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88
[2023-07-06 04:20:53,421] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,421] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR weight [NNModuleVariable()]
[2023-07-06 04:20:53,422] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].input_layernorm.weight (16,) [<DimDynamic.STATIC: 2>] [None]
[2023-07-06 04:20:53,422] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:53,422] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,423] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:53,423] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_dtype [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:53,423] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:53,424] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:286
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR self_attn [NNModuleVariable()]
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:287
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:288
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST attention_mask [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:289
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:290
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_key_value [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:291
[2023-07-06 04:20:53,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:292
[2023-07-06 04:20:53,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(bool)]
[2023-07-06 04:20:53,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:286
[2023-07-06 04:20:53,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('hidden_states', 'attention_mask', 'position_ids', 'past_key_value', 'output_attentions', 'valid_past_index') [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(bool), TensorVariable()]
[2023-07-06 04:20:53,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 6 [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(bool), TensorVariable(), ConstantVariable(tuple)]
[2023-07-06 04:20:53,426] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:53,427] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:53,427] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:53,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:53,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:53,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:53,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:53,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:53,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable())]
[2023-07-06 04:20:53,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:53,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:53,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:53,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:53,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:53,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:53,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:53,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:53,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:53,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:53,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:53,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:53,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:53,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:53,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:53,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:53,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:53,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:53,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:53,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable())]
[2023-07-06 04:20:53,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:53,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:53,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:53,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:53,431] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:53,432] torch._dynamo.symbolic_convert: [DEBUG] FAILED INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:53,432] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:53,432] torch._dynamo.symbolic_convert: [DEBUG] break_graph_if_unsupported triggered compile
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2187, in inline_call_
sub_locals, closure_cells = func.bind_args(parent, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 187, in bind_args
[
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 188, in <listcomp>
wrap(val=arg, source=source)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 62, in wrap_bound_arg
return VariableBuilder(tx, source=source, **options)(val)
TypeError: VariableBuilder.__init__() got an unexpected keyword argument 'guards'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 390, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1152, in CALL_FUNCTION_KW
self.call_function(fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 559, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 333, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 595, in inline_user_function_return
result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2132, in inline_call
return cls.inline_call_(parent, func, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2229, in inline_call_
tracer.run()
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 708, in run
and self.step()
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 668, in step
getattr(self, inst.opname)(inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 390, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1140, in CALL_FUNCTION_EX
self.call_function(fn, argsvars.items, kwargsvars.items)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 559, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 333, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 287, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 120, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 595, in inline_user_function_return
result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2132, in inline_call
return cls.inline_call_(parent, func, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2190, in inline_call_
raise ArgsMismatchError(
torch._dynamo.exc.ArgsMismatchError: VariableBuilder.__init__() got an unexpected keyword argument 'guards'.
func = 'forward' /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:190, args = [<class 'trfs_fast.llama.LlamaAttention'>], kwargs = OrderedDict([('hidden_states', TensorVariable()), ('attention_mask', TensorVariable()), ('position_ids', TensorVariable()), ('past_key_value', TensorVariable()), ('output_attentions', ConstantVariable(bool)), ('valid_past_index', TensorVariable())])
[2023-07-06 04:20:53,432] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:53,432] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason="VariableBuilder.__init__() got an unexpected keyword argument 'guards'.\n func = 'forward' /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:190, args = [<class 'trfs_fast.llama.LlamaAttention'>], kwargs = OrderedDict([('hidden_states', TensorVariable()), ('attention_mask', TensorVariable()), ('position_ids', TensorVariable()), ('past_key_value', TensorVariable()), ('output_attentions', ConstantVariable(bool)), ('valid_past_index', TensorVariable())])", user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 286 in forward>, <FrameSummary file /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py, line 1514 in _call_impl>], graph_break=True)
[2023-07-06 04:20:53,433] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['attention_mask']
[2023-07-06 04:20:53,433] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['position_ids']
[2023-07-06 04:20:53,433] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['past_key_value']
[2023-07-06 04:20:53,434] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['valid_past_index']
[2023-07-06 04:20:53,434] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_5 =====
<eval_with_key>.133 class GraphModule(torch.nn.Module):
def forward(self, L_hidden_states_ : torch.Tensor):
l_hidden_states_ = L_hidden_states_
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
to = l_hidden_states_.to(torch.float32)
pow_1 = to.pow(2); to = None
mean = pow_1.mean(-1, keepdim = True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add = mean + 1e-06; mean = None
rsqrt = torch.rsqrt(add); add = None
mul = l_hidden_states_ * rsqrt; l_hidden_states_ = rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
l__self___input_layernorm_weight = self.L__self___input_layernorm_weight
mul_1 = l__self___input_layernorm_weight * mul; l__self___input_layernorm_weight = mul = None
to_1 = mul_1.to(torch.float16); mul_1 = None
return (to_1,)
[2023-07-06 04:20:53,434] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_5 <eval_with_key>.133 opcode name target args kwargs
------------- -------------------------------- -------------------------------------------------------- --------------------------------------- -----------------
placeholder l_hidden_states_ L_hidden_states_ () {}
call_method to to (l_hidden_states_, torch.float32) {}
call_method pow_1 pow (to, 2) {}
call_method mean mean (pow_1, -1) {'keepdim': True}
call_function add <built-in function add> (mean, 1e-06) {}
call_function rsqrt <built-in method rsqrt of type object at 0x7fb940469580> (add,) {}
call_function mul <built-in function mul> (l_hidden_states_, rsqrt) {}
get_attr l__self___input_layernorm_weight L__self___input_layernorm_weight () {}
call_function mul_1 <built-in function mul> (l__self___input_layernorm_weight, mul) {}
call_method to_1 to (mul_1, torch.float16) {}
output output output ((to_1,),) {}
[2023-07-06 04:20:53,435] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
[2023-07-06 04:20:53,454] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 16 =====
<eval_with_key>.137 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[16], arg1_1: f16[1, 1000, 16]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
convert_element_type: f32[1, 1000, 16] = torch.ops.prims.convert_element_type.default(arg1_1, torch.float32)
pow_1: f32[1, 1000, 16] = torch.ops.aten.pow.Tensor_Scalar(convert_element_type, 2); convert_element_type = None
mean: f32[1, 1000, 1] = torch.ops.aten.mean.dim(pow_1, [-1], True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add: f32[1, 1000, 1] = torch.ops.aten.add.Tensor(mean, 1e-06); mean = None
rsqrt: f32[1, 1000, 1] = torch.ops.aten.rsqrt.default(add); add = None
mul: f32[1, 1000, 16] = torch.ops.aten.mul.Tensor(arg1_1, rsqrt); arg1_1 = rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
mul_1: f32[1, 1000, 16] = torch.ops.aten.mul.Tensor(arg0_1, mul); arg0_1 = mul = None
convert_element_type_1: f16[1, 1000, 16] = torch.ops.prims.convert_element_type.default(mul_1, torch.float16); mul_1 = None
return (convert_element_type_1,)
[2023-07-06 04:20:53,455] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 3
[2023-07-06 04:20:53,472] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 3
[2023-07-06 04:20:53,487] torch._inductor.scheduler: [DEBUG] remove_buffer('buf0')
[2023-07-06 04:20:53,489] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/tn/ctnjyv37bz6fvtmwelga5764xq272722mvbuxkb4qqgd3syt7hwl.py
[2023-07-06 04:20:53,489] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 3
[2023-07-06 04:20:53,491] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:53,499] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662486464)
hasattr(L['position_ids'], '_dynamo_dynamic_indices') == False
___check_type_id(L['hidden_states'], 77765680)
hasattr(L['hidden_states'], '_dynamo_dynamic_indices') == False
hasattr(L['attention_mask'], '_dynamo_dynamic_indices') == False
hasattr(L['past_key_value'], '_dynamo_dynamic_indices') == False
hasattr(L['valid_past_index'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['output_attentions'], 7698112)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
str(G['torch'].float32) == 'torch.float32'
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks.keys()) == set()
[2023-07-06 04:20:53,499] torch._inductor.cudagraph_trees: [DEBUG] Running warmup of function 2
[2023-07-06 04:20:53,500] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function forward in /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py', 'set env var TORCHDYNAMO_REPORT_GUARD_FAILURES=1 to debug further')
[2023-07-06 04:20:53,501] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:190
[2023-07-06 04:20:53,502] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['hidden_states'] (1, 1000, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:53,503] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['attention_mask'] (1, 1003) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:53,503] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['position_ids'] (1, 1000) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:53,504] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['past_key_value'] (2, 1, 4, 1003, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None, None]
[2023-07-06 04:20:53,505] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['valid_past_index'] () [] []
[2023-07-06 04:20:53,506] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:199
[2023-07-06 04:20:53,506] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions []
[2023-07-06 04:20:53,506] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST True [ConstantVariable(bool)]
[2023-07-06 04:20:53,506] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 0 [ConstantVariable(bool), ConstantVariable(bool)]
[2023-07-06 04:20:53,506] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(bool)]
[2023-07-06 04:20:53,506] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:202
[2023-07-06 04:20:53,506] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:53,506] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR size [TensorVariable()]
[2023-07-06 04:20:53,506] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), size)]
[2023-07-06 04:20:53,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE UNPACK_SEQUENCE 3 [SizeVariable()]
[2023-07-06 04:20:53,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST bsz [ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST q_len [ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST _ [ConstantVariable(int)]
[2023-07-06 04:20:53,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:205
[2023-07-06 04:20:53,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR qkv_proj [NNModuleVariable()]
[2023-07-06 04:20:53,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:53,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:53,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR view [TensorVariable()]
[2023-07-06 04:20:53,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST bsz [GetAttrVariable(TensorVariable(), view)]
[2023-07-06 04:20:53,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_len [GetAttrVariable(TensorVariable(), view), ConstantVariable(int)]
[2023-07-06 04:20:53,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 3 [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR num_heads [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), NNModuleVariable()]
[2023-07-06 04:20:53,511] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,511] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR head_dim [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), NNModuleVariable()]
[2023-07-06 04:20:53,511] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 5 [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,512] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR permute [TensorVariable()]
[2023-07-06 04:20:53,512] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [GetAttrVariable(TensorVariable(), permute)]
[2023-07-06 04:20:53,512] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int)]
[2023-07-06 04:20:53,512] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 3 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,512] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,512] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 4 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,512] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 5 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,513] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST query_key_value_states [TensorVariable()]
[2023-07-06 04:20:53,513] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:207
[2023-07-06 04:20:53,513] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST query_key_value_states []
[2023-07-06 04:20:53,513] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TensorVariable()]
[2023-07-06 04:20:53,513] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,514] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST query_states [TensorVariable()]
[2023-07-06 04:20:53,514] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:208
[2023-07-06 04:20:53,514] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST query_key_value_states []
[2023-07-06 04:20:53,514] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [TensorVariable()]
[2023-07-06 04:20:53,514] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,514] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), ConstantVariable(int), ConstantVariable(NoneType)]
[2023-07-06 04:20:53,514] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), SliceVariable()]
[2023-07-06 04:20:53,515] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST key_value_states [TensorVariable()]
[2023-07-06 04:20:53,515] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:211
[2023-07-06 04:20:53,516] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,516] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR rotary_emb [NNModuleVariable()]
[2023-07-06 04:20:53,516] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states [NNModuleVariable()]
[2023-07-06 04:20:53,516] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:53,516] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_len [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,516] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [NNModuleVariable(), TensorVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,517] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('seq_len',) [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,517] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 2 [NNModuleVariable(), TensorVariable(), TensorVariable(), ConstantVariable(tuple)]
[2023-07-06 04:20:53,517] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:53,519] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:53,519] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:53,519] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:53,519] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:53,519] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:53,520] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:53,520] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,520] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:53,520] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable())]
[2023-07-06 04:20:53,520] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:53,520] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,520] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:53,520] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,520] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,520] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:53,521] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,521] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,521] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:53,521] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,521] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,521] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:53,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:53,523] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:53,523] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable())]
[2023-07-06 04:20:53,523] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:53,523] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:53,523] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:53,523] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:53,523] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddf80e0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 107>
[2023-07-06 04:20:53,523] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:122
[2023-07-06 04:20:53,523] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:53,523] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR cos_cached [NNModuleVariable()]
[2023-07-06 04:20:53,524] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].rotary_emb.cos_cached (1, 1, 2048, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None]
[2023-07-06 04:20:53,524] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:53,525] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:53,525] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [GetAttrVariable(TensorVariable(), to), TensorVariable()]
[2023-07-06 04:20:53,525] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('dtype',) [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:53,525] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 1 [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16), ConstantVariable(tuple)]
[2023-07-06 04:20:53,526] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:123
[2023-07-06 04:20:53,526] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable()]
[2023-07-06 04:20:53,526] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR sin_cached [TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:53,527] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].rotary_emb.sin_cached (1, 1, 2048, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None]
[2023-07-06 04:20:53,527] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,527] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TensorVariable(), GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:53,527] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [TensorVariable(), GetAttrVariable(TensorVariable(), to), TensorVariable()]
[2023-07-06 04:20:53,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('dtype',) [TensorVariable(), GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:53,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 1 [TensorVariable(), GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16), ConstantVariable(tuple)]
[2023-07-06 04:20:53,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:121
[2023-07-06 04:20:53,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 2 [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TupleVariable()]
[2023-07-06 04:20:53,529] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddf80e0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 107>
[2023-07-06 04:20:53,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TupleVariable()]
[2023-07-06 04:20:53,529] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:53,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE UNPACK_SEQUENCE 2 [TupleVariable()]
[2023-07-06 04:20:53,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST cos [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST sin [TensorVariable()]
[2023-07-06 04:20:53,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:212
[2023-07-06 04:20:53,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL apply_rotary_pos_emb_opt []
[2023-07-06 04:20:53,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST query_states [UserFunctionVariable()]
[2023-07-06 04:20:53,530] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states [UserFunctionVariable(), TensorVariable()]
[2023-07-06 04:20:53,530] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [UserFunctionVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,530] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [UserFunctionVariable(), TensorVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,530] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos [UserFunctionVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,530] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin [UserFunctionVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,530] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [UserFunctionVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,530] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 5 [UserFunctionVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,531] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object apply_rotary_pos_emb_opt at 0x7fb8cddf82f0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 135>
[2023-07-06 04:20:53,531] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:139
[2023-07-06 04:20:53,531] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos []
[2023-07-06 04:20:53,531] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:53,531] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:53,531] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:53,532] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:53,532] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:53,532] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:53,533] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST cos [TensorVariable()]
[2023-07-06 04:20:53,533] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:140
[2023-07-06 04:20:53,533] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin []
[2023-07-06 04:20:53,533] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:53,533] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:53,533] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:53,534] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:53,534] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:53,534] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:53,535] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST sin [TensorVariable()]
[2023-07-06 04:20:53,535] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:141
[2023-07-06 04:20:53,535] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos []
[2023-07-06 04:20:53,535] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [TensorVariable()]
[2023-07-06 04:20:53,535] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,536] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR unsqueeze [TensorVariable()]
[2023-07-06 04:20:53,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), unsqueeze)]
[2023-07-06 04:20:53,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), unsqueeze), ConstantVariable(int)]
[2023-07-06 04:20:53,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST cos [TensorVariable()]
[2023-07-06 04:20:53,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:142
[2023-07-06 04:20:53,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin []
[2023-07-06 04:20:53,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [TensorVariable()]
[2023-07-06 04:20:53,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,539] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR unsqueeze [TensorVariable()]
[2023-07-06 04:20:53,539] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), unsqueeze)]
[2023-07-06 04:20:53,539] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), unsqueeze), ConstantVariable(int)]
[2023-07-06 04:20:53,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST sin [TensorVariable()]
[2023-07-06 04:20:53,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143
[2023-07-06 04:20:53,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q []
[2023-07-06 04:20:53,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos [TensorVariable()]
[2023-07-06 04:20:53,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,541] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL rotate_half [TensorVariable()]
[2023-07-06 04:20:53,541] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q [TensorVariable(), UserFunctionVariable()]
[2023-07-06 04:20:53,541] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), UserFunctionVariable(), TensorVariable()]
[2023-07-06 04:20:53,541] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:53,541] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129
[2023-07-06 04:20:53,541] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:53,542] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR roll [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:53,542] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>)]
[2023-07-06 04:20:53,542] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:53,542] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,542] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable()]
[2023-07-06 04:20:53,542] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,543] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,543] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,543] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,543] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('shifts', 'dims') [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,543] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 3 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int), ConstantVariable(tuple)]
[2023-07-06 04:20:53,544] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST x [TensorVariable()]
[2023-07-06 04:20:53,544] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130
[2023-07-06 04:20:53,544] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:53,544] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST Ellipsis [TensorVariable()]
[2023-07-06 04:20:53,544] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable(), ConstantVariable(ellipsis)]
[2023-07-06 04:20:53,544] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType)]
[2023-07-06 04:20:53,544] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), TensorVariable()]
[2023-07-06 04:20:53,545] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable()]
[2023-07-06 04:20:53,545] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,545] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:53,545] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,545] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:53,545] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 2 [TensorVariable(), ConstantVariable(ellipsis), SliceVariable()]
[2023-07-06 04:20:53,545] torch._dynamo.symbolic_convert: [DEBUG] TRACE DUP_TOP_TWO None [TensorVariable(), TupleVariable()]
[2023-07-06 04:20:53,545] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TupleVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:53,546] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:53,546] torch._dynamo.symbolic_convert: [DEBUG] TRACE INPLACE_MULTIPLY None [TensorVariable(), TupleVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,547] torch._dynamo.symbolic_convert: [DEBUG] TRACE ROT_THREE None [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:53,547] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_SUBSCR None [TensorVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:53,547] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:131
[2023-07-06 04:20:53,547] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:53,547] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:53,547] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:53,547] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,547] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,548] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST q_embed [TensorVariable()]
[2023-07-06 04:20:53,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144
[2023-07-06 04:20:53,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_states []
[2023-07-06 04:20:53,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR copy_ [TensorVariable()]
[2023-07-06 04:20:53,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_states [GetAttrVariable(TensorVariable(), copy_)]
[2023-07-06 04:20:53,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos [GetAttrVariable(TensorVariable(), copy_), TensorVariable()]
[2023-07-06 04:20:53,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL rotate_half [GetAttrVariable(TensorVariable(), copy_), TensorVariable()]
[2023-07-06 04:20:53,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_states [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), UserFunctionVariable()]
[2023-07-06 04:20:53,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), UserFunctionVariable(), TensorVariable()]
[2023-07-06 04:20:53,551] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:53,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129
[2023-07-06 04:20:53,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:53,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR roll [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:53,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>)]
[2023-07-06 04:20:53,552] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:53,552] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,552] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable()]
[2023-07-06 04:20:53,552] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,552] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,552] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,552] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,552] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('shifts', 'dims') [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,552] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 3 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int), ConstantVariable(tuple)]
[2023-07-06 04:20:53,553] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST x [TensorVariable()]
[2023-07-06 04:20:53,553] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130
[2023-07-06 04:20:53,553] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:53,554] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST Ellipsis [TensorVariable()]
[2023-07-06 04:20:53,554] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable(), ConstantVariable(ellipsis)]
[2023-07-06 04:20:53,554] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType)]
[2023-07-06 04:20:53,554] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), TensorVariable()]
[2023-07-06 04:20:53,554] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable()]
[2023-07-06 04:20:53,554] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,554] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:53,554] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:53,554] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:53,555] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 2 [TensorVariable(), ConstantVariable(ellipsis), SliceVariable()]
[2023-07-06 04:20:53,555] torch._dynamo.symbolic_convert: [DEBUG] TRACE DUP_TOP_TWO None [TensorVariable(), TupleVariable()]
[2023-07-06 04:20:53,555] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TupleVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:53,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:53,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE INPLACE_MULTIPLY None [TensorVariable(), TupleVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE ROT_THREE None [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:53,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_SUBSCR None [TensorVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:53,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:131
[2023-07-06 04:20:53,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:53,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:53,557] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:53,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), copy_), TensorVariable()]
[2023-07-06 04:20:53,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_TOP None [TensorVariable()]
[2023-07-06 04:20:53,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:145
[2023-07-06 04:20:53,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_embed []
[2023-07-06 04:20:53,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:53,559] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object apply_rotary_pos_emb_opt at 0x7fb8cddf82f0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 135>
[2023-07-06 04:20:53,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST query_states [TensorVariable()]
[2023-07-06 04:20:53,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:215
[2023-07-06 04:20:53,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index []
[2023-07-06 04:20:53,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST past_kv_slice_start [TensorVariable()]
[2023-07-06 04:20:53,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:216
[2023-07-06 04:20:53,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:53,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR eq [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:53,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TorchVariable(<built-in method eq of type object at 0x7fb940469580>)]
[2023-07-06 04:20:53,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:53,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable()]
[2023-07-06 04:20:53,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:53,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_len [TensorVariable()]
[2023-07-06 04:20:53,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,563] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:53,563] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR ne [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:53,563] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>)]
[2023-07-06 04:20:53,563] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:53,563] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,564] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,564] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TensorVariable(), GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:53,565] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,565] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,565] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,566] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,567] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,568] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST past_kv_slice_end [TensorVariable()]
[2023-07-06 04:20:53,568] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:217
[2023-07-06 04:20:53,568] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:53,568] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR eq [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:53,568] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TorchVariable(<built-in method eq of type object at 0x7fb940469580>)]
[2023-07-06 04:20:53,568] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:53,568] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,569] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable()]
[2023-07-06 04:20:53,569] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:53,570] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states [TensorVariable()]
[2023-07-06 04:20:53,571] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,571] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -2 [TensorVariable(), ShapeVariable()]
[2023-07-06 04:20:53,571] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,571] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,572] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:53,572] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR ne [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:53,572] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>)]
[2023-07-06 04:20:53,572] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:53,572] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:53,573] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,573] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TensorVariable(), GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:53,574] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_kv_slice_end [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,574] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,575] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST past_state_slice_end [TensorVariable()]
[2023-07-06 04:20:53,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:218
[2023-07-06 04:20:53,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states []
[2023-07-06 04:20:53,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_key_value [TensorVariable()]
[2023-07-06 04:20:53,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST Ellipsis [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_kv_slice_start [TensorVariable(), TensorVariable(), ConstantVariable(ellipsis)]
[2023-07-06 04:20:53,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_kv_slice_end [TensorVariable(), TensorVariable(), ConstantVariable(ellipsis), TensorVariable()]
[2023-07-06 04:20:53,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), TensorVariable(), ConstantVariable(ellipsis), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:53,576] torch._dynamo.symbolic_convert: [DEBUG] step triggered compile
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 668, in step
getattr(self, inst.opname)(inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1251, in BUILD_SLICE
SliceVariable(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/base.py", line 27, in __call__
obj = type.__call__(cls, *args, **kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/lists.py", line 616, in __init__
unimplemented("Dynamic slicing on data-dependent value is not supported")
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 140, in unimplemented
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: Dynamic slicing on data-dependent value is not supported
[2023-07-06 04:20:53,576] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:53,576] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='step_unsupported', user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 218 in forward>], graph_break=True)
[2023-07-06 04:20:53,578] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['attention_mask']
[2023-07-06 04:20:53,578] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['past_key_value']
[2023-07-06 04:20:53,580] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_7 =====
<eval_with_key>.139 class GraphModule(torch.nn.Module):
def forward(self, L_hidden_states_ : torch.Tensor, L_position_ids_ : torch.Tensor, L_valid_past_index_ : torch.Tensor):
l_hidden_states_ = L_hidden_states_
l_position_ids_ = L_position_ids_
l_valid_past_index_ = L_valid_past_index_
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:205, code: query_key_value_states = self.qkv_proj(hidden_states).view(bsz, q_len, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4)
l__self___qkv_proj = self.L__self___qkv_proj(l_hidden_states_); l_hidden_states_ = None
view = l__self___qkv_proj.view(1, 1000, 3, 4, 4); l__self___qkv_proj = None
permute = view.permute(2, 0, 3, 1, 4); view = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:207, code: query_states = query_key_value_states[0]
getitem = permute[0]
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:208, code: key_value_states = query_key_value_states[1:]
getitem_1 = permute[slice(1, None, None)]; permute = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:211, code: cos, sin = self.rotary_emb(key_value_states, seq_len=valid_past_index + q_len)
add = l_valid_past_index_ + 1000
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:122, code: self.cos_cached.to(dtype=x.dtype),
l__self___rotary_emb_cos_cached = self.L__self___rotary_emb_cos_cached
to = l__self___rotary_emb_cos_cached.to(dtype = torch.float16); l__self___rotary_emb_cos_cached = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:123, code: self.sin_cached.to(dtype=x.dtype),
l__self___rotary_emb_sin_cached = self.L__self___rotary_emb_sin_cached
to_1 = l__self___rotary_emb_sin_cached.to(dtype = torch.float16); l__self___rotary_emb_sin_cached = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:212, code: query_states = apply_rotary_pos_emb_opt(query_states, key_value_states[0], cos, sin, position_ids)
getitem_2 = getitem_1[0]
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:139, code: cos = cos.squeeze(1).squeeze(0) # [seq_len, dim]
squeeze = to.squeeze(1); to = None
squeeze_1 = squeeze.squeeze(0); squeeze = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:140, code: sin = sin.squeeze(1).squeeze(0) # [seq_len, dim]
squeeze_2 = to_1.squeeze(1); to_1 = None
squeeze_3 = squeeze_2.squeeze(0); squeeze_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:141, code: cos = cos[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
getitem_3 = squeeze_1[l_position_ids_]; squeeze_1 = None
unsqueeze = getitem_3.unsqueeze(1); getitem_3 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:142, code: sin = sin[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
getitem_4 = squeeze_3[l_position_ids_]; squeeze_3 = l_position_ids_ = None
unsqueeze_1 = getitem_4.unsqueeze(1); getitem_4 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul = getitem * unsqueeze
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll = torch.roll(getitem, shifts = 2, dims = -1); getitem = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
getitem_5 = roll[(Ellipsis, slice(None, 2, None))]
getitem_5 *= -1; imul = getitem_5; getitem_5 = None
roll[(Ellipsis, slice(None, 2, None))] = imul; setitem = roll; imul = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul_1 = roll * unsqueeze_1; roll = None
add_1 = mul + mul_1; mul = mul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_2 = getitem_2 * unsqueeze; unsqueeze = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll_1 = torch.roll(getitem_2, shifts = 2, dims = -1)
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
getitem_6 = roll_1[(Ellipsis, slice(None, 2, None))]
getitem_6 *= -1; imul_1 = getitem_6; getitem_6 = None
roll_1[(Ellipsis, slice(None, 2, None))] = imul_1; setitem_1 = roll_1; imul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_3 = roll_1 * unsqueeze_1; roll_1 = unsqueeze_1 = None
add_2 = mul_2 + mul_3; mul_2 = mul_3 = None
copy_ = getitem_2.copy_(add_2); getitem_2 = add_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:216, code: past_kv_slice_end = torch.eq(valid_past_index, 0).int() * q_len + torch.ne(valid_past_index, 0).int() * (valid_past_index + 1)
eq = torch.eq(l_valid_past_index_, 0)
int_1 = eq.int(); eq = None
mul_4 = int_1 * 1000; int_1 = None
ne = torch.ne(l_valid_past_index_, 0)
int_2 = ne.int(); ne = None
add_3 = l_valid_past_index_ + 1
mul_5 = int_2 * add_3; int_2 = add_3 = None
add_4 = mul_4 + mul_5; mul_4 = mul_5 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:217, code: past_state_slice_end = torch.eq(valid_past_index, 0).int() * key_value_states.shape[-2] + torch.ne(valid_past_index, 0).int() * (past_kv_slice_end)
eq_1 = torch.eq(l_valid_past_index_, 0)
int_3 = eq_1.int(); eq_1 = None
mul_6 = int_3 * 1000; int_3 = None
ne_1 = torch.ne(l_valid_past_index_, 0); l_valid_past_index_ = None
int_4 = ne_1.int(); ne_1 = None
mul_7 = int_4 * add_4; int_4 = None
add_5 = mul_6 + mul_7; mul_6 = mul_7 = None
return (add_1, getitem_1, add_4, add_5)
[2023-07-06 04:20:53,581] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_7 <eval_with_key>.139 opcode name target args kwargs
------------- ------------------------------- ------------------------------------------------------- -------------------------------------------------- -------------------------
placeholder l_hidden_states_ L_hidden_states_ () {}
placeholder l_position_ids_ L_position_ids_ () {}
placeholder l_valid_past_index_ L_valid_past_index_ () {}
call_module l__self___qkv_proj L__self___qkv_proj (l_hidden_states_,) {}
call_method view view (l__self___qkv_proj, 1, 1000, 3, 4, 4) {}
call_method permute permute (view, 2, 0, 3, 1, 4) {}
call_function getitem <built-in function getitem> (permute, 0) {}
call_function getitem_1 <built-in function getitem> (permute, slice(1, None, None)) {}
call_function add <built-in function add> (l_valid_past_index_, 1000) {}
get_attr l__self___rotary_emb_cos_cached L__self___rotary_emb_cos_cached () {}
call_method to to (l__self___rotary_emb_cos_cached,) {'dtype': torch.float16}
get_attr l__self___rotary_emb_sin_cached L__self___rotary_emb_sin_cached () {}
call_method to_1 to (l__self___rotary_emb_sin_cached,) {'dtype': torch.float16}
call_function getitem_2 <built-in function getitem> (getitem_1, 0) {}
call_method squeeze squeeze (to, 1) {}
call_method squeeze_1 squeeze (squeeze, 0) {}
call_method squeeze_2 squeeze (to_1, 1) {}
call_method squeeze_3 squeeze (squeeze_2, 0) {}
call_function getitem_3 <built-in function getitem> (squeeze_1, l_position_ids_) {}
call_method unsqueeze unsqueeze (getitem_3, 1) {}
call_function getitem_4 <built-in function getitem> (squeeze_3, l_position_ids_) {}
call_method unsqueeze_1 unsqueeze (getitem_4, 1) {}
call_function mul <built-in function mul> (getitem, unsqueeze) {}
call_function roll <built-in method roll of type object at 0x7fb940469580> (getitem,) {'shifts': 2, 'dims': -1}
call_function getitem_5 <built-in function getitem> (roll, (Ellipsis, slice(None, 2, None))) {}
call_function imul <built-in function imul> (getitem_5, -1) {}
call_function setitem <built-in function setitem> (roll, (Ellipsis, slice(None, 2, None)), imul) {}
call_function mul_1 <built-in function mul> (roll, unsqueeze_1) {}
call_function add_1 <built-in function add> (mul, mul_1) {}
call_function mul_2 <built-in function mul> (getitem_2, unsqueeze) {}
call_function roll_1 <built-in method roll of type object at 0x7fb940469580> (getitem_2,) {'shifts': 2, 'dims': -1}
call_function getitem_6 <built-in function getitem> (roll_1, (Ellipsis, slice(None, 2, None))) {}
call_function imul_1 <built-in function imul> (getitem_6, -1) {}
call_function setitem_1 <built-in function setitem> (roll_1, (Ellipsis, slice(None, 2, None)), imul_1) {}
call_function mul_3 <built-in function mul> (roll_1, unsqueeze_1) {}
call_function add_2 <built-in function add> (mul_2, mul_3) {}
call_method copy_ copy_ (getitem_2, add_2) {}
call_function eq <built-in method eq of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_1 int (eq,) {}
call_function mul_4 <built-in function mul> (int_1, 1000) {}
call_function ne <built-in method ne of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_2 int (ne,) {}
call_function add_3 <built-in function add> (l_valid_past_index_, 1) {}
call_function mul_5 <built-in function mul> (int_2, add_3) {}
call_function add_4 <built-in function add> (mul_4, mul_5) {}
call_function eq_1 <built-in method eq of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_3 int (eq_1,) {}
call_function mul_6 <built-in function mul> (int_3, 1000) {}
call_function ne_1 <built-in method ne of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_4 int (ne_1,) {}
call_function mul_7 <built-in function mul> (int_4, add_4) {}
call_function add_5 <built-in function add> (mul_6, mul_7) {}
output output output ((add_1, getitem_1, add_4, add_5),) {}
[2023-07-06 04:20:53,583] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
[2023-07-06 04:20:53,705] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 17 =====
<eval_with_key>.143 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[48, 16], arg1_1: f32[1, 1, 2048, 4], arg2_1: f32[1, 1, 2048, 4], arg3_1: f16[1, 1000, 16], arg4_1: i64[1, 1000], arg5_1: i64[]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:205, code: query_key_value_states = self.qkv_proj(hidden_states).view(bsz, q_len, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4)
permute: f16[16, 48] = torch.ops.aten.permute.default(arg0_1, [1, 0]); arg0_1 = None
view: f16[1000, 16] = torch.ops.aten.view.default(arg3_1, [1000, 16]); arg3_1 = None
mm: f16[1000, 48] = torch.ops.aten.mm.default(view, permute); view = permute = None
view_1: f16[1, 1000, 48] = torch.ops.aten.view.default(mm, [1, 1000, 48])
view_2: f16[1, 1000, 3, 4, 4] = torch.ops.aten.view.default(view_1, [1, 1000, 3, 4, 4]); view_1 = None
permute_1: f16[3, 1, 4, 1000, 4] = torch.ops.aten.permute.default(view_2, [2, 0, 3, 1, 4]); view_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:207, code: query_states = query_key_value_states[0]
select: f16[1, 4, 1000, 4] = torch.ops.aten.select.int(permute_1, 0, 0)
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:208, code: key_value_states = query_key_value_states[1:]
slice_1: f16[2, 1, 4, 1000, 4] = torch.ops.aten.slice.Tensor(permute_1, 0, 1, 9223372036854775807); permute_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:122, code: self.cos_cached.to(dtype=x.dtype),
convert_element_type: f16[1, 1, 2048, 4] = torch.ops.prims.convert_element_type.default(arg1_1, torch.float16); arg1_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:123, code: self.sin_cached.to(dtype=x.dtype),
convert_element_type_1: f16[1, 1, 2048, 4] = torch.ops.prims.convert_element_type.default(arg2_1, torch.float16); arg2_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:212, code: query_states = apply_rotary_pos_emb_opt(query_states, key_value_states[0], cos, sin, position_ids)
select_1: f16[1, 4, 1000, 4] = torch.ops.aten.select.int(slice_1, 0, 0); slice_1 = None
# No stacktrace found for following nodes
squeeze: f16[1, 2048, 4] = torch.ops.aten.squeeze.dim(convert_element_type, 1); convert_element_type = None
squeeze_1: f16[2048, 4] = torch.ops.aten.squeeze.dim(squeeze, 0); squeeze = None
squeeze_2: f16[1, 2048, 4] = torch.ops.aten.squeeze.dim(convert_element_type_1, 1); convert_element_type_1 = None
squeeze_3: f16[2048, 4] = torch.ops.aten.squeeze.dim(squeeze_2, 0); squeeze_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:141, code: cos = cos[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
index: f16[1, 1000, 4] = torch.ops.aten.index.Tensor(squeeze_1, [arg4_1]); squeeze_1 = None
unsqueeze: f16[1, 1, 1000, 4] = torch.ops.aten.unsqueeze.default(index, 1); index = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:142, code: sin = sin[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
index_1: f16[1, 1000, 4] = torch.ops.aten.index.Tensor(squeeze_3, [arg4_1]); squeeze_3 = arg4_1 = None
unsqueeze_1: f16[1, 1, 1000, 4] = torch.ops.aten.unsqueeze.default(index_1, 1); index_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul: f16[1, 4, 1000, 4] = torch.ops.aten.mul.Tensor(select, unsqueeze)
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll: f16[1, 4, 1000, 4] = torch.ops.aten.roll.default(select, [2], [-1]); select = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
slice_2: f16[1, 4, 1000, 2] = torch.ops.aten.slice.Tensor(roll, 3, 0, 2)
mul_1: f16[1, 4, 1000, 2] = torch.ops.aten.mul.Tensor(slice_2, -1); slice_2 = None
slice_scatter: f16[1, 4, 1000, 4] = torch.ops.aten.slice_scatter.default(roll, mul_1, 3, 0, 2); roll = mul_1 = None
slice_3: f16[1, 4, 1000, 2] = torch.ops.aten.slice.Tensor(slice_scatter, 3, 0, 2)
slice_scatter_1: f16[1, 4, 1000, 4] = torch.ops.aten.slice_scatter.default(slice_scatter, slice_3, 3, 0, 2); slice_scatter = slice_3 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul_2: f16[1, 4, 1000, 4] = torch.ops.aten.mul.Tensor(slice_scatter_1, unsqueeze_1); slice_scatter_1 = None
add_1: f16[1, 4, 1000, 4] = torch.ops.aten.add.Tensor(mul, mul_2); mul = mul_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_3: f16[1, 4, 1000, 4] = torch.ops.aten.mul.Tensor(select_1, unsqueeze); unsqueeze = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll_1: f16[1, 4, 1000, 4] = torch.ops.aten.roll.default(select_1, [2], [-1]); select_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
slice_7: f16[1, 4, 1000, 2] = torch.ops.aten.slice.Tensor(roll_1, 3, 0, 2)
mul_4: f16[1, 4, 1000, 2] = torch.ops.aten.mul.Tensor(slice_7, -1); slice_7 = None
slice_scatter_2: f16[1, 4, 1000, 4] = torch.ops.aten.slice_scatter.default(roll_1, mul_4, 3, 0, 2); roll_1 = mul_4 = None
slice_8: f16[1, 4, 1000, 2] = torch.ops.aten.slice.Tensor(slice_scatter_2, 3, 0, 2)
slice_scatter_3: f16[1, 4, 1000, 4] = torch.ops.aten.slice_scatter.default(slice_scatter_2, slice_8, 3, 0, 2); slice_scatter_2 = slice_8 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_5: f16[1, 4, 1000, 4] = torch.ops.aten.mul.Tensor(slice_scatter_3, unsqueeze_1); slice_scatter_3 = unsqueeze_1 = None
add_2: f16[1, 4, 1000, 4] = torch.ops.aten.add.Tensor(mul_3, mul_5); mul_3 = mul_5 = None
view_3: f16[1, 1000, 48] = torch.ops.aten.view.default(mm, [1, 1000, 48]); mm = None
view_4: f16[1, 1000, 3, 4, 4] = torch.ops.aten.view.default(view_3, [1, 1000, 3, 4, 4]); view_3 = None
permute_2: f16[3, 1, 4, 1000, 4] = torch.ops.aten.permute.default(view_4, [2, 0, 3, 1, 4]); view_4 = None
slice_12: f16[2, 1, 4, 1000, 4] = torch.ops.aten.slice.Tensor(permute_2, 0, 1, 9223372036854775807)
select_scatter: f16[2, 1, 4, 1000, 4] = torch.ops.aten.select_scatter.default(slice_12, add_2, 0, 0); slice_12 = add_2 = None
slice_scatter_4: f16[3, 1, 4, 1000, 4] = torch.ops.aten.slice_scatter.default(permute_2, select_scatter, 0, 1, 9223372036854775807); permute_2 = select_scatter = None
permute_3: f16[1, 1000, 3, 4, 4] = torch.ops.aten.permute.default(slice_scatter_4, [1, 3, 0, 2, 4]); slice_scatter_4 = None
view_5: f16[1, 1000, 48] = torch.ops.aten.view.default(permute_3, [1, 1000, 48]); permute_3 = None
view_6: f16[1000, 48] = torch.ops.aten.view.default(view_5, [1000, 48]); view_5 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:216, code: past_kv_slice_end = torch.eq(valid_past_index, 0).int() * q_len + torch.ne(valid_past_index, 0).int() * (valid_past_index + 1)
eq: b8[] = torch.ops.aten.eq.Scalar(arg5_1, 0)
convert_element_type_2: i32[] = torch.ops.prims.convert_element_type.default(eq, torch.int32); eq = None
mul_6: i32[] = torch.ops.aten.mul.Tensor(convert_element_type_2, 1000); convert_element_type_2 = None
ne: b8[] = torch.ops.aten.ne.Scalar(arg5_1, 0)
convert_element_type_3: i32[] = torch.ops.prims.convert_element_type.default(ne, torch.int32); ne = None
add_3: i64[] = torch.ops.aten.add.Tensor(arg5_1, 1)
mul_7: i64[] = torch.ops.aten.mul.Tensor(convert_element_type_3, add_3); convert_element_type_3 = add_3 = None
add_4: i64[] = torch.ops.aten.add.Tensor(mul_6, mul_7); mul_6 = mul_7 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:217, code: past_state_slice_end = torch.eq(valid_past_index, 0).int() * key_value_states.shape[-2] + torch.ne(valid_past_index, 0).int() * (past_kv_slice_end)
eq_1: b8[] = torch.ops.aten.eq.Scalar(arg5_1, 0)
convert_element_type_4: i32[] = torch.ops.prims.convert_element_type.default(eq_1, torch.int32); eq_1 = None
mul_8: i32[] = torch.ops.aten.mul.Tensor(convert_element_type_4, 1000); convert_element_type_4 = None
ne_1: b8[] = torch.ops.aten.ne.Scalar(arg5_1, 0); arg5_1 = None
convert_element_type_5: i32[] = torch.ops.prims.convert_element_type.default(ne_1, torch.int32); ne_1 = None
mul_9: i64[] = torch.ops.aten.mul.Tensor(convert_element_type_5, add_4); convert_element_type_5 = None
add_5: i64[] = torch.ops.aten.add.Tensor(mul_8, mul_9); mul_8 = mul_9 = None
# No stacktrace found for following nodes
view_9: f16[1, 1000, 48] = torch.ops.aten.view.default(view_6, [1, 1000, 48]); view_6 = None
view_10: f16[1, 1000, 3, 4, 4] = torch.ops.aten.view.default(view_9, [1, 1000, 3, 4, 4]); view_9 = None
permute_5: f16[3, 1, 4, 1000, 4] = torch.ops.aten.permute.default(view_10, [2, 0, 3, 1, 4]); view_10 = None
slice_14: f16[2, 1, 4, 1000, 4] = torch.ops.aten.slice.Tensor(permute_5, 0, 1, 9223372036854775807); permute_5 = None
return (add_1, slice_14, add_4, add_5)
[2023-07-06 04:20:53,734] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 4
[2023-07-06 04:20:53,860] torch._inductor.ir: [DEBUG] convert_to_reinterpret_view failed: stride=[16000, 0, 4000, 4, 1] offset=16000 index=16000*r0 + 4000*r1 + 4*r2 + r3 + 16000*(((4*r1 + r3)//16)) + 16000
[2023-07-06 04:20:53,866] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 4
[2023-07-06 04:20:53,989] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/lo/clozylkw544bb7yqlyamdizaia7jo2uesdfqh4qhrfm2er5ahkkn.py
[2023-07-06 04:20:53,989] torch._inductor.compile_fx: [DEBUG] disabled cudagraphs because ['non-cuda device in graph']
skipping cudagraphs due to multiple devices
[2023-07-06 04:20:53,989] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 4
[2023-07-06 04:20:53,991] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:54,000] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662491744)
hasattr(L['position_ids'], '_dynamo_dynamic_indices') == False
hasattr(L['hidden_states'], '_dynamo_dynamic_indices') == False
hasattr(L['attention_mask'], '_dynamo_dynamic_indices') == False
hasattr(L['past_key_value'], '_dynamo_dynamic_indices') == False
hasattr(L['valid_past_index'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['output_attentions'], 7698112)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks.keys()) == set()
[2023-07-06 04:20:54,001] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function <resume in forward> in /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py', 'set env var TORCHDYNAMO_REPORT_GUARD_FAILURES=1 to debug further')
[2023-07-06 04:20:54,002] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <resume in forward> /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:286
[2023-07-06 04:20:54,002] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['___stack0'][0] (1, 1000, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:54,004] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['residual'] (1, 1000, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:54,005] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST ___stack0 []
[2023-07-06 04:20:54,005] torch._dynamo.symbolic_convert: [DEBUG] TRACE JUMP_ABSOLUTE 38 [TupleVariable()]
[2023-07-06 04:20:54,005] torch._dynamo.symbolic_convert: [DEBUG] TRACE UNPACK_SEQUENCE 3 [TupleVariable()]
[2023-07-06 04:20:54,005] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [ConstantVariable(NoneType), ConstantVariable(NoneType), TensorVariable()]
[2023-07-06 04:20:54,005] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST self_attn_weights [ConstantVariable(NoneType), ConstantVariable(NoneType)]
[2023-07-06 04:20:54,005] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST present_key_value [ConstantVariable(NoneType)]
[2023-07-06 04:20:54,005] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:294
[2023-07-06 04:20:54,005] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST residual []
[2023-07-06 04:20:54,005] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:54,005] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:54,006] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:54,006] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:297
[2023-07-06 04:20:54,006] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:54,006] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST residual [TensorVariable()]
[2023-07-06 04:20:54,006] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:298
[2023-07-06 04:20:54,006] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,006] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR post_attention_layernorm [NNModuleVariable()]
[2023-07-06 04:20:54,007] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:54,007] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:54,007] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:54,008] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:54,009] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:54,009] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:54,009] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:54,009] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:54,009] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:54,009] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,009] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:54,010] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:54,010] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,010] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,010] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,010] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,010] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,010] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,011] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,011] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,011] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,011] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,011] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,011] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,011] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,011] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:54,011] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:54,012] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:54,013] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,013] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,013] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,013] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:54,013] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:84
[2023-07-06 04:20:54,013] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:54,013] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [TensorVariable()]
[2023-07-06 04:20:54,013] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST input_dtype [TorchVariable(torch.float16)]
[2023-07-06 04:20:54,013] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85
[2023-07-06 04:20:54,014] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:54,014] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:54,014] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:54,014] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR float32 [GetAttrVariable(TensorVariable(), to), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:54,014] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), ConstantVariable(dtype)]
[2023-07-06 04:20:54,015] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR pow [TensorVariable()]
[2023-07-06 04:20:54,016] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [GetAttrVariable(TensorVariable(), pow)]
[2023-07-06 04:20:54,016] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), pow), ConstantVariable(int)]
[2023-07-06 04:20:54,017] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mean [TensorVariable()]
[2023-07-06 04:20:54,017] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [GetAttrVariable(TensorVariable(), mean)]
[2023-07-06 04:20:54,017] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST True [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int)]
[2023-07-06 04:20:54,017] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('keepdim',) [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool)]
[2023-07-06 04:20:54,017] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 2 [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool), ConstantVariable(tuple)]
[2023-07-06 04:20:54,018] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST variance [TensorVariable()]
[2023-07-06 04:20:54,018] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86
[2023-07-06 04:20:54,018] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:54,018] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:54,018] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR rsqrt [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:54,018] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST variance [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>)]
[2023-07-06 04:20:54,019] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:54,019] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR variance_epsilon [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:54,019] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(float)]
[2023-07-06 04:20:54,020] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:54,020] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:54,021] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:54,021] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88
[2023-07-06 04:20:54,021] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,021] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR weight [NNModuleVariable()]
[2023-07-06 04:20:54,022] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].post_attention_layernorm.weight (16,) [<DimDynamic.STATIC: 2>] [None]
[2023-07-06 04:20:54,022] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:54,022] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:54,023] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:54,023] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_dtype [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:54,023] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:54,024] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:54,025] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:54,025] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:54,025] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:54,025] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:54,025] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:299
[2023-07-06 04:20:54,025] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,025] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mlp [NNModuleVariable()]
[2023-07-06 04:20:54,025] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:54,025] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:54,026] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:54,027] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:54,027] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:54,027] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:54,027] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:54,028] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:54,028] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:54,028] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,028] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:54,028] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable())]
[2023-07-06 04:20:54,028] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,028] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,028] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,028] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,029] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,029] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,029] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,029] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,029] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,029] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,029] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,029] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable())]
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,030] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,031] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,031] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddf85b0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 161>
[2023-07-06 04:20:54,031] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162
[2023-07-06 04:20:54,031] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,031] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR down_proj [NNModuleVariable()]
[2023-07-06 04:20:54,031] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [NNModuleVariable()]
[2023-07-06 04:20:54,031] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR act_fn [NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:54,032] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:54,032] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR gate_proj [NNModuleVariable(), NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:54,032] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [NNModuleVariable(), NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:54,032] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), NNModuleVariable(), NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:54,036] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:54,036] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:54,037] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:54,037] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:54,037] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:54,038] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:54,038] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:54,038] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:54,038] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,038] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:54,038] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable())]
[2023-07-06 04:20:54,038] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,039] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,039] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,039] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,039] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,039] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,039] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,039] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,039] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,040] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,041] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:54,041] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:54,041] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable())]
[2023-07-06 04:20:54,041] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:54,041] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,041] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,041] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,041] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8ccf62760, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py", line 149>
[2023-07-06 04:20:54,041] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py:150
[2023-07-06 04:20:54,041] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL nn []
[2023-07-06 04:20:54,041] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR functional [TorchVariable(<module 'torch.nn' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/__init__.py'>)]
[2023-07-06 04:20:54,042] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR silu [TorchVariable(<module 'torch.nn.functional' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/functional.py'>)]
[2023-07-06 04:20:54,042] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input [TorchVariable(<function silu at 0x7fb945bc5990>)]
[2023-07-06 04:20:54,042] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TorchVariable(<function silu at 0x7fb945bc5990>), TensorVariable()]
[2023-07-06 04:20:54,045] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:54,045] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8ccf62760, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py", line 149>
[2023-07-06 04:20:54,045] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:54,045] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:54,045] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:54,045] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR up_proj [NNModuleVariable(), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:54,046] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [NNModuleVariable(), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:54,046] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable(), NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:54,048] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:54,049] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:54,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:54,052] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddf85b0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 161>
[2023-07-06 04:20:54,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:54,052] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:54,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:54,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:300
[2023-07-06 04:20:54,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST residual []
[2023-07-06 04:20:54,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:54,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:54,053] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:54,053] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:302
[2023-07-06 04:20:54,053] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:54,053] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 1 [TensorVariable()]
[2023-07-06 04:20:54,053] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST outputs [TupleVariable()]
[2023-07-06 04:20:54,053] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:304
[2023-07-06 04:20:54,053] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions []
[2023-07-06 04:20:54,053] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 106 [ConstantVariable(bool)]
[2023-07-06 04:20:54,053] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:307
[2023-07-06 04:20:54,053] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs []
[2023-07-06 04:20:54,053] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST present_key_value [TupleVariable()]
[2023-07-06 04:20:54,053] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 1 [TupleVariable(), ConstantVariable(NoneType)]
[2023-07-06 04:20:54,054] torch._dynamo.symbolic_convert: [DEBUG] TRACE INPLACE_ADD None [TupleVariable(), TupleVariable()]
[2023-07-06 04:20:54,054] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST outputs [TupleVariable()]
[2023-07-06 04:20:54,054] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:309
[2023-07-06 04:20:54,054] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs []
[2023-07-06 04:20:54,054] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TupleVariable()]
[2023-07-06 04:20:54,054] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing <resume in forward> (RETURN_VALUE)
[2023-07-06 04:20:54,054] torch._dynamo.symbolic_convert: [DEBUG] RETURN_VALUE triggered compile
[2023-07-06 04:20:54,054] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 309 in <resume in forward>>], graph_break=False)
[2023-07-06 04:20:54,056] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_8 =====
<eval_with_key>.148 class GraphModule(torch.nn.Module):
def forward(self, L_stack0_0_ : torch.Tensor, L_residual_ : torch.Tensor):
l_stack0_0_ = L_stack0_0_
l_residual_ = L_residual_
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:294, code: hidden_states = residual + hidden_states
add = l_residual_ + l_stack0_0_; l_residual_ = l_stack0_0_ = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
to = add.to(torch.float32)
pow_1 = to.pow(2); to = None
mean = pow_1.mean(-1, keepdim = True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add_1 = mean + 1e-06; mean = None
rsqrt = torch.rsqrt(add_1); add_1 = None
mul = add * rsqrt; rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
l__self___post_attention_layernorm_weight = self.L__self___post_attention_layernorm_weight
mul_1 = l__self___post_attention_layernorm_weight * mul; l__self___post_attention_layernorm_weight = mul = None
to_1 = mul_1.to(torch.float16); mul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
l__self___mlp_gate_proj = self.L__self___mlp_gate_proj(to_1)
# File: /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py:150, code: return nn.functional.silu(input)
silu = torch.nn.functional.silu(l__self___mlp_gate_proj); l__self___mlp_gate_proj = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
l__self___mlp_up_proj = self.L__self___mlp_up_proj(to_1); to_1 = None
mul_2 = silu * l__self___mlp_up_proj; silu = l__self___mlp_up_proj = None
l__self___mlp_down_proj = self.L__self___mlp_down_proj(mul_2); mul_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:300, code: hidden_states = residual + hidden_states
add_2 = add + l__self___mlp_down_proj; add = l__self___mlp_down_proj = None
return (add_2,)
[2023-07-06 04:20:54,056] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_8 <eval_with_key>.148 opcode name target args kwargs
------------- ----------------------------------------- -------------------------------------------------------- ------------------------------------------------ -----------------
placeholder l_stack0_0_ L_stack0_0_ () {}
placeholder l_residual_ L_residual_ () {}
call_function add <built-in function add> (l_residual_, l_stack0_0_) {}
call_method to to (add, torch.float32) {}
call_method pow_1 pow (to, 2) {}
call_method mean mean (pow_1, -1) {'keepdim': True}
call_function add_1 <built-in function add> (mean, 1e-06) {}
call_function rsqrt <built-in method rsqrt of type object at 0x7fb940469580> (add_1,) {}
call_function mul <built-in function mul> (add, rsqrt) {}
get_attr l__self___post_attention_layernorm_weight L__self___post_attention_layernorm_weight () {}
call_function mul_1 <built-in function mul> (l__self___post_attention_layernorm_weight, mul) {}
call_method to_1 to (mul_1, torch.float16) {}
call_module l__self___mlp_gate_proj L__self___mlp_gate_proj (to_1,) {}
call_function silu <function silu at 0x7fb945bc5990> (l__self___mlp_gate_proj,) {}
call_module l__self___mlp_up_proj L__self___mlp_up_proj (to_1,) {}
call_function mul_2 <built-in function mul> (silu, l__self___mlp_up_proj) {}
call_module l__self___mlp_down_proj L__self___mlp_down_proj (mul_2,) {}
call_function add_2 <built-in function add> (add, l__self___mlp_down_proj) {}
output output output ((add_2,),) {}
[2023-07-06 04:20:54,057] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
[2023-07-06 04:20:54,105] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 18 =====
<eval_with_key>.152 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[16], arg1_1: f16[64, 16], arg2_1: f16[64, 16], arg3_1: f16[16, 64], arg4_1: f16[1, 1000, 16], arg5_1: f16[1, 1000, 16]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:294, code: hidden_states = residual + hidden_states
add: f16[1, 1000, 16] = torch.ops.aten.add.Tensor(arg5_1, arg4_1); arg5_1 = arg4_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
convert_element_type: f32[1, 1000, 16] = torch.ops.prims.convert_element_type.default(add, torch.float32)
pow_1: f32[1, 1000, 16] = torch.ops.aten.pow.Tensor_Scalar(convert_element_type, 2); convert_element_type = None
mean: f32[1, 1000, 1] = torch.ops.aten.mean.dim(pow_1, [-1], True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add_1: f32[1, 1000, 1] = torch.ops.aten.add.Tensor(mean, 1e-06); mean = None
rsqrt: f32[1, 1000, 1] = torch.ops.aten.rsqrt.default(add_1); add_1 = None
mul: f32[1, 1000, 16] = torch.ops.aten.mul.Tensor(add, rsqrt); rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
mul_1: f32[1, 1000, 16] = torch.ops.aten.mul.Tensor(arg0_1, mul); arg0_1 = mul = None
convert_element_type_1: f16[1, 1000, 16] = torch.ops.prims.convert_element_type.default(mul_1, torch.float16); mul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
permute: f16[16, 64] = torch.ops.aten.permute.default(arg1_1, [1, 0]); arg1_1 = None
view: f16[1000, 16] = torch.ops.aten.view.default(convert_element_type_1, [1000, 16])
mm: f16[1000, 64] = torch.ops.aten.mm.default(view, permute); view = permute = None
view_1: f16[1, 1000, 64] = torch.ops.aten.view.default(mm, [1, 1000, 64]); mm = None
# File: /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py:150, code: return nn.functional.silu(input)
convert_element_type_2: f32[1, 1000, 64] = torch.ops.prims.convert_element_type.default(view_1, torch.float32); view_1 = None
sigmoid: f32[1, 1000, 64] = torch.ops.aten.sigmoid.default(convert_element_type_2)
mul_2: f32[1, 1000, 64] = torch.ops.aten.mul.Tensor(convert_element_type_2, sigmoid); convert_element_type_2 = sigmoid = None
convert_element_type_3: f16[1, 1000, 64] = torch.ops.prims.convert_element_type.default(mul_2, torch.float16); mul_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
permute_1: f16[16, 64] = torch.ops.aten.permute.default(arg2_1, [1, 0]); arg2_1 = None
view_2: f16[1000, 16] = torch.ops.aten.view.default(convert_element_type_1, [1000, 16]); convert_element_type_1 = None
mm_1: f16[1000, 64] = torch.ops.aten.mm.default(view_2, permute_1); view_2 = permute_1 = None
view_3: f16[1, 1000, 64] = torch.ops.aten.view.default(mm_1, [1, 1000, 64]); mm_1 = None
mul_3: f16[1, 1000, 64] = torch.ops.aten.mul.Tensor(convert_element_type_3, view_3); convert_element_type_3 = view_3 = None
permute_2: f16[64, 16] = torch.ops.aten.permute.default(arg3_1, [1, 0]); arg3_1 = None
view_4: f16[1000, 64] = torch.ops.aten.view.default(mul_3, [1000, 64]); mul_3 = None
mm_2: f16[1000, 16] = torch.ops.aten.mm.default(view_4, permute_2); view_4 = permute_2 = None
view_5: f16[1, 1000, 16] = torch.ops.aten.view.default(mm_2, [1, 1000, 16]); mm_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:300, code: hidden_states = residual + hidden_states
add_2: f16[1, 1000, 16] = torch.ops.aten.add.Tensor(add, view_5); add = view_5 = None
return (add_2,)
[2023-07-06 04:20:54,133] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 5
[2023-07-06 04:20:54,177] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 5
[2023-07-06 04:20:54,201] torch._inductor.scheduler: [DEBUG] remove_buffer('buf0')
[2023-07-06 04:20:54,210] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/or/cor2fodulphywxrtc7zlvp6tsjseyifwlaiv6ticnqodficz2elj.py
[2023-07-06 04:20:54,210] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 5
[2023-07-06 04:20:54,212] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:54,220] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662486464)
hasattr(L['residual'], '_dynamo_dynamic_indices') == False
___check_type_id(L['___stack0'], 7638272)
len(L['___stack0']) == 3
hasattr(L['___stack0'][0], '_dynamo_dynamic_indices') == False
___check_obj_id(L['___stack0'][1], 7649088)
___check_obj_id(L['___stack0'][2], 7649088)
___check_obj_id(L['output_attentions'], 7698112)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
str(G['torch'].float32) == 'torch.float32'
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks.keys()) == set()
[2023-07-06 04:20:54,220] torch._inductor.cudagraph_trees: [DEBUG] Running warmup of function 3
[2023-07-06 04:20:54,221] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:83
[2023-07-06 04:20:54,222] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['hidden_states'] (1, 1000, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:54,223] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:84
[2023-07-06 04:20:54,223] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:54,223] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [TensorVariable()]
[2023-07-06 04:20:54,226] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST input_dtype [TorchVariable(torch.float16)]
[2023-07-06 04:20:54,226] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85
[2023-07-06 04:20:54,226] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:54,226] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:54,226] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:54,226] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR float32 [GetAttrVariable(TensorVariable(), to), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:54,226] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), ConstantVariable(dtype)]
[2023-07-06 04:20:54,228] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR pow [TensorVariable()]
[2023-07-06 04:20:54,228] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [GetAttrVariable(TensorVariable(), pow)]
[2023-07-06 04:20:54,228] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), pow), ConstantVariable(int)]
[2023-07-06 04:20:54,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mean [TensorVariable()]
[2023-07-06 04:20:54,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [GetAttrVariable(TensorVariable(), mean)]
[2023-07-06 04:20:54,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST True [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int)]
[2023-07-06 04:20:54,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('keepdim',) [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool)]
[2023-07-06 04:20:54,229] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 2 [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool), ConstantVariable(tuple)]
[2023-07-06 04:20:54,230] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST variance [TensorVariable()]
[2023-07-06 04:20:54,230] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86
[2023-07-06 04:20:54,230] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:54,230] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:54,230] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR rsqrt [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:54,231] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST variance [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>)]
[2023-07-06 04:20:54,231] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:54,231] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR variance_epsilon [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:54,231] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(float)]
[2023-07-06 04:20:54,232] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:54,232] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:54,233] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:54,233] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88
[2023-07-06 04:20:54,233] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,233] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR weight [NNModuleVariable()]
[2023-07-06 04:20:54,234] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].weight (16,) [<DimDynamic.STATIC: 2>] [None]
[2023-07-06 04:20:54,235] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:54,235] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:54,236] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:54,236] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_dtype [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:54,236] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:54,237] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:54,237] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-07-06 04:20:54,238] torch._dynamo.symbolic_convert: [DEBUG] RETURN_VALUE triggered compile
[2023-07-06 04:20:54,238] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 88 in forward>], graph_break=False)
[2023-07-06 04:20:54,238] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_9 =====
<eval_with_key>.161 class GraphModule(torch.nn.Module):
def forward(self, L_hidden_states_ : torch.Tensor):
l_hidden_states_ = L_hidden_states_
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
to = l_hidden_states_.to(torch.float32)
pow_1 = to.pow(2); to = None
mean = pow_1.mean(-1, keepdim = True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add = mean + 1e-06; mean = None
rsqrt = torch.rsqrt(add); add = None
mul = l_hidden_states_ * rsqrt; l_hidden_states_ = rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
l__self___weight = self.L__self___weight
mul_1 = l__self___weight * mul; l__self___weight = mul = None
to_1 = mul_1.to(torch.float16); mul_1 = None
return (to_1,)
[2023-07-06 04:20:54,239] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_9 <eval_with_key>.161 opcode name target args kwargs
------------- ---------------- -------------------------------------------------------- --------------------------------- -----------------
placeholder l_hidden_states_ L_hidden_states_ () {}
call_method to to (l_hidden_states_, torch.float32) {}
call_method pow_1 pow (to, 2) {}
call_method mean mean (pow_1, -1) {'keepdim': True}
call_function add <built-in function add> (mean, 1e-06) {}
call_function rsqrt <built-in method rsqrt of type object at 0x7fb940469580> (add,) {}
call_function mul <built-in function mul> (l_hidden_states_, rsqrt) {}
get_attr l__self___weight L__self___weight () {}
call_function mul_1 <built-in function mul> (l__self___weight, mul) {}
call_method to_1 to (mul_1, torch.float16) {}
output output output ((to_1,),) {}
[2023-07-06 04:20:54,239] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
[2023-07-06 04:20:54,258] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 19 =====
<eval_with_key>.165 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[16], arg1_1: f16[1, 1000, 16]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
convert_element_type: f32[1, 1000, 16] = torch.ops.prims.convert_element_type.default(arg1_1, torch.float32)
pow_1: f32[1, 1000, 16] = torch.ops.aten.pow.Tensor_Scalar(convert_element_type, 2); convert_element_type = None
mean: f32[1, 1000, 1] = torch.ops.aten.mean.dim(pow_1, [-1], True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add: f32[1, 1000, 1] = torch.ops.aten.add.Tensor(mean, 1e-06); mean = None
rsqrt: f32[1, 1000, 1] = torch.ops.aten.rsqrt.default(add); add = None
mul: f32[1, 1000, 16] = torch.ops.aten.mul.Tensor(arg1_1, rsqrt); arg1_1 = rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
mul_1: f32[1, 1000, 16] = torch.ops.aten.mul.Tensor(arg0_1, mul); arg0_1 = mul = None
convert_element_type_1: f16[1, 1000, 16] = torch.ops.prims.convert_element_type.default(mul_1, torch.float16); mul_1 = None
return (convert_element_type_1,)
[2023-07-06 04:20:54,259] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 6
[2023-07-06 04:20:54,276] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 6
[2023-07-06 04:20:54,291] torch._inductor.scheduler: [DEBUG] remove_buffer('buf0')
[2023-07-06 04:20:54,292] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/tn/ctnjyv37bz6fvtmwelga5764xq272722mvbuxkb4qqgd3syt7hwl.py
[2023-07-06 04:20:54,293] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 6
[2023-07-06 04:20:54,295] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:54,298] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662764176)
___check_type_id(L['hidden_states'], 77765680)
hasattr(L['hidden_states'], '_dynamo_dynamic_indices') == False
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
str(G['torch'].float32) == 'torch.float32'
[2023-07-06 04:20:54,298] torch._inductor.cudagraph_trees: [DEBUG] Running warmup of function 4
[2023-07-06 04:20:54,299] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing __init__ <string>:2
[2023-07-06 04:20:54,299] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['last_hidden_state'] (1, 1000, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:54,300] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line <string>:3
[2023-07-06 04:20:54,300] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST last_hidden_state []
[2023-07-06 04:20:54,300] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable()]
[2023-07-06 04:20:54,300] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_ATTR last_hidden_state [TensorVariable(), DataClassVariable()]
[2023-07-06 04:20:54,301] torch._dynamo.symbolic_convert: [DEBUG] STORE_ATTR triggered compile
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1195, in STORE_ATTR
.call_function(self, [obj, ConstantVariable(inst.argval), val], {})
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 591, in call_function
result = handler(tx, *args, **kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1103, in call_setattr
return obj.call_method(tx, "__setattr__", [name_var, val], {})
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/dicts.py", line 417, in call_method
return super().call_method(tx, name, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/dicts.py", line 198, in call_method
return super().call_method(tx, name, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/base.py", line 260, in call_method
raise unimplemented(f"call_method {self} {name} {args} {kwargs}")
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 140, in unimplemented
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: call_method DataClassVariable() __setitem__ [ConstantVariable(str), TensorVariable()] {}
[2023-07-06 04:20:54,304] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:54,304] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='store_attr', user_stack=[<FrameSummary file <string>, line 3 in __init__>], graph_break=True)
[2023-07-06 04:20:54,309] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_type_id(L['self'], 110084784)
___check_type_id(L['past_key_values'], 7638272)
len(L['past_key_values']) == 2
___check_obj_id(L['self'].attentions, 7649088)
hasattr(L['last_hidden_state'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['self'].hidden_states, 7649088)
___check_obj_id(L['self'].past_key_values, 7649088)
___check_obj_id(L['self'].last_hidden_state, 7649088)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
[2023-07-06 04:20:54,309] torch._dynamo.convert_frame: [DEBUG] skipping because no torch.* <resume in __init__> <string> 3
[2023-07-06 04:20:54,309] torch._dynamo.convert_frame: [DEBUG] skipping because no torch.* __post_init__ /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/utils/generic.py 251
[2023-07-06 04:20:54,309] torch._dynamo.eval_frame: [DEBUG] skipping fields /home/ubuntu/miniconda3/lib/python3.10/dataclasses.py
[2023-07-06 04:20:54,309] torch._dynamo.eval_frame: [DEBUG] skipping <genexpr> /home/ubuntu/miniconda3/lib/python3.10/dataclasses.py
[2023-07-06 04:20:54,310] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing __setitem__ /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/utils/generic.py:328
[2023-07-06 04:20:54,311] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].last_hidden_state (1, 1000, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:54,312] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/utils/generic.py:330
[2023-07-06 04:20:54,312] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL super []
[2023-07-06 04:20:54,312] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_DEREF __class__ [BuiltinVariable(super)]
[2023-07-06 04:20:54,312] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [BuiltinVariable(super), UserDefinedClassVariable()]
[2023-07-06 04:20:54,312] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [BuiltinVariable(super), UserDefinedClassVariable(), DataClassVariable()]
[2023-07-06 04:20:54,313] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR __setitem__ [SuperVariable()]
[2023-07-06 04:20:54,313] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key [GetAttrVariable(SuperVariable(), __setitem__)]
[2023-07-06 04:20:54,313] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST value [GetAttrVariable(SuperVariable(), __setitem__), ConstantVariable(str)]
[2023-07-06 04:20:54,313] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [GetAttrVariable(SuperVariable(), __setitem__), ConstantVariable(str), TensorVariable()]
[2023-07-06 04:20:54,313] torch._dynamo.symbolic_convert: [DEBUG] break_graph_if_unsupported triggered compile
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 390, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1100, in CALL_FUNCTION
self.call_function(fn, args, {})
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 559, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 588, in call_function
return self.obj.call_method(tx, self.name, args, kwargs).add_options(self)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 115, in call_method
unimplemented(f"non-function or method super: {inner_fn}")
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 140, in unimplemented
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: non-function or method super: <slot wrapper '__setitem__' of 'collections.OrderedDict' objects>
[2023-07-06 04:20:54,315] torch._dynamo.symbolic_convert.__graph_breaks: [DEBUG] Graph break: non-function or method super: <slot wrapper '__setitem__' of 'collections.OrderedDict' objects> from user code at File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/utils/generic.py", line 330, in __setitem__
super().__setitem__(key, value)
[2023-07-06 04:20:54,315] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:54,315] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason="non-function or method super: <slot wrapper '__setitem__' of 'collections.OrderedDict' objects>", user_stack=[<FrameSummary file /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/utils/generic.py, line 330 in __setitem__>], graph_break=True)
[2023-07-06 04:20:54,321] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_type_id(L['key'], 7626144)
L['key'] == 'last_hidden_state'
___check_type_id(L['self'], 110084784)
L['self'].last_hidden_state is L['value']
___check_obj_id(L['__class__'], 107106000)
___check_obj_id(L['self'].attentions, 7649088)
___check_obj_id(L['self'].hidden_states, 7649088)
___check_type_id(L['self'].past_key_values, 7638272)
len(L['self'].past_key_values) == 2
hasattr(L['self'].last_hidden_state, '_dynamo_dynamic_indices') == False
___check_obj_id(L['self'].past_key_values[0], 7649088)
___check_obj_id(L['self'].past_key_values[1], 7649088)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
[2023-07-06 04:20:54,321] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <resume in __setitem__> /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/utils/generic.py:330
[2023-07-06 04:20:54,322] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].last_hidden_state (1, 1000, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:54,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST ___stack0 []
[2023-07-06 04:20:54,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE JUMP_ABSOLUTE 20 [ConstantVariable(NoneType)]
[2023-07-06 04:20:54,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_TOP None [ConstantVariable(NoneType)]
[2023-07-06 04:20:54,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/utils/generic.py:332
[2023-07-06 04:20:54,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL super []
[2023-07-06 04:20:54,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_DEREF __class__ [BuiltinVariable(super)]
[2023-07-06 04:20:54,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [BuiltinVariable(super), UserDefinedClassVariable()]
[2023-07-06 04:20:54,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [BuiltinVariable(super), UserDefinedClassVariable(), DataClassVariable()]
[2023-07-06 04:20:54,323] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR __setattr__ [SuperVariable()]
[2023-07-06 04:20:54,324] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key [GetAttrVariable(SuperVariable(), __setattr__)]
[2023-07-06 04:20:54,324] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST value [GetAttrVariable(SuperVariable(), __setattr__), ConstantVariable(str)]
[2023-07-06 04:20:54,324] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [GetAttrVariable(SuperVariable(), __setattr__), ConstantVariable(str), TensorVariable()]
[2023-07-06 04:20:54,324] torch._dynamo.symbolic_convert: [DEBUG] break_graph_if_unsupported triggered compile
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 390, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1100, in CALL_FUNCTION
self.call_function(fn, args, {})
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 559, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 588, in call_function
return self.obj.call_method(tx, self.name, args, kwargs).add_options(self)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 115, in call_method
unimplemented(f"non-function or method super: {inner_fn}")
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 140, in unimplemented
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: non-function or method super: <slot wrapper '__setattr__' of 'object' objects>
[2023-07-06 04:20:54,324] torch._dynamo.symbolic_convert.__graph_breaks: [DEBUG] Graph break: non-function or method super: <slot wrapper '__setattr__' of 'object' objects> from user code at File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/utils/generic.py", line 332, in <resume in __setitem__>
super().__setattr__(key, value)
[2023-07-06 04:20:54,324] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:54,324] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason="non-function or method super: <slot wrapper '__setattr__' of 'object' objects>", user_stack=[<FrameSummary file /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/utils/generic.py, line 332 in <resume in __setitem__>>], graph_break=True)
[2023-07-06 04:20:54,330] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_type_id(L['key'], 7626144)
L['key'] == 'last_hidden_state'
___check_type_id(L['self'], 110084784)
L['self'].last_hidden_state is L['value']
___check_obj_id(L['__class__'], 107106000)
___check_obj_id(L['self'].attentions, 7649088)
___check_obj_id(L['self'].hidden_states, 7649088)
___check_type_id(L['self'].past_key_values, 7638272)
len(L['self'].past_key_values) == 2
hasattr(L['self'].last_hidden_state, '_dynamo_dynamic_indices') == False
___check_obj_id(L['self'].past_key_values[0], 7649088)
___check_obj_id(L['self'].past_key_values[1], 7649088)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
[2023-07-06 04:20:54,330] torch._dynamo.convert_frame: [DEBUG] skipping because no torch.* <resume in __setitem__> /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/utils/generic.py 332
[2023-07-06 04:20:54,330] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function __setitem__ in /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/utils/generic.py', 'set env var TORCHDYNAMO_REPORT_GUARD_FAILURES=1 to debug further')
[2023-07-06 04:20:54,330] torch._dynamo.convert_frame: [DEBUG] skipping because no torch.* __setitem__ /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/utils/generic.py 328
[2023-07-06 04:20:54,331] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <resume in forward> /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:692
[2023-07-06 04:20:54,332] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['___stack0'].last_hidden_state (1, 1000, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:54,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST ___stack0 []
[2023-07-06 04:20:54,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE JUMP_ABSOLUTE 90 [DataClassVariable()]
[2023-07-06 04:20:54,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST outputs [DataClassVariable()]
[2023-07-06 04:20:54,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:704
[2023-07-06 04:20:54,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs []
[2023-07-06 04:20:54,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [DataClassVariable()]
[2023-07-06 04:20:54,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [DataClassVariable(), ConstantVariable(int)]
[2023-07-06 04:20:54,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:54,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:705
[2023-07-06 04:20:54,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR lm_head [NNModuleVariable()]
[2023-07-06 04:20:54,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:54,334] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:54,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST logits [TensorVariable()]
[2023-07-06 04:20:54,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:707
[2023-07-06 04:20:54,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None []
[2023-07-06 04:20:54,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST loss [ConstantVariable(NoneType)]
[2023-07-06 04:20:54,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:708
[2023-07-06 04:20:54,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST labels []
[2023-07-06 04:20:54,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [ConstantVariable(NoneType)]
[2023-07-06 04:20:54,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [ConstantVariable(NoneType), ConstantVariable(NoneType)]
[2023-07-06 04:20:54,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 222 [ConstantVariable(bool)]
[2023-07-06 04:20:54,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:720
[2023-07-06 04:20:54,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST return_dict []
[2023-07-06 04:20:54,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 266 [ConstantVariable(bool)]
[2023-07-06 04:20:54,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:724
[2023-07-06 04:20:54,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL CausalLMOutputWithPast []
[2023-07-06 04:20:54,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:725
[2023-07-06 04:20:54,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST loss [UserDefinedClassVariable()]
[2023-07-06 04:20:54,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:726
[2023-07-06 04:20:54,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST logits [UserDefinedClassVariable(), ConstantVariable(NoneType)]
[2023-07-06 04:20:54,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:727
[2023-07-06 04:20:54,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable()]
[2023-07-06 04:20:54,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR past_key_values [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable(), DataClassVariable()]
[2023-07-06 04:20:54,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:728
[2023-07-06 04:20:54,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:54,338] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR hidden_states [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable(), TupleVariable(), DataClassVariable()]
[2023-07-06 04:20:54,339] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:729
[2023-07-06 04:20:54,339] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable(), TupleVariable(), ConstantVariable(NoneType)]
[2023-07-06 04:20:54,339] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR attentions [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable(), TupleVariable(), ConstantVariable(NoneType), DataClassVariable()]
[2023-07-06 04:20:54,339] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:724
[2023-07-06 04:20:54,339] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('loss', 'logits', 'past_key_values', 'hidden_states', 'attentions') [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable(), TupleVariable(), ConstantVariable(NoneType), ConstantVariable(NoneType)]
[2023-07-06 04:20:54,339] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 5 [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable(), TupleVariable(), ConstantVariable(NoneType), ConstantVariable(NoneType), ConstantVariable(tuple)]
[2023-07-06 04:20:54,339] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [DataClassVariable()]
[2023-07-06 04:20:54,339] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing <resume in forward> (RETURN_VALUE)
[2023-07-06 04:20:54,339] torch._dynamo.symbolic_convert: [DEBUG] RETURN_VALUE triggered compile
[2023-07-06 04:20:54,340] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 724 in <resume in forward>>], graph_break=False)
[2023-07-06 04:20:54,341] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_13 =====
<eval_with_key>.167 class GraphModule(torch.nn.Module):
def forward(self, L_stack0_last_hidden_state : torch.Tensor):
l_stack0_last_hidden_state = L_stack0_last_hidden_state
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:705, code: logits = self.lm_head(hidden_states)
l__self___lm_head = self.L__self___lm_head(l_stack0_last_hidden_state); l_stack0_last_hidden_state = None
return (l__self___lm_head,)
[2023-07-06 04:20:54,341] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_13 <eval_with_key>.167 opcode name target args kwargs
----------- -------------------------- -------------------------- ----------------------------- --------
placeholder l_stack0_last_hidden_state L_stack0_last_hidden_state () {}
call_module l__self___lm_head L__self___lm_head (l_stack0_last_hidden_state,) {}
output output output ((l__self___lm_head,),) {}
[2023-07-06 04:20:54,341] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
[2023-07-06 04:20:54,352] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 20 =====
<eval_with_key>.171 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[32000, 16], arg1_1: f16[1, 1000, 16]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:705, code: logits = self.lm_head(hidden_states)
permute: f16[16, 32000] = torch.ops.aten.permute.default(arg0_1, [1, 0]); arg0_1 = None
view: f16[1000, 16] = torch.ops.aten.view.default(arg1_1, [1000, 16]); arg1_1 = None
mm: f16[1000, 32000] = torch.ops.aten.mm.default(view, permute); view = permute = None
view_1: f16[1, 1000, 32000] = torch.ops.aten.view.default(mm, [1, 1000, 32000]); mm = None
return (view_1,)
[2023-07-06 04:20:54,361] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 7
[2023-07-06 04:20:54,367] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 7
[2023-07-06 04:20:54,370] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/kk/ckkkx2tf6bgblkt5hisgibketyheywf6ccyhzdrdgv6pbbtyn5rf.py
[2023-07-06 04:20:54,370] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 7
[2023-07-06 04:20:54,372] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:54,377] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662761824)
___check_obj_id(L['labels'], 7649088)
___check_type_id(L['___stack0'], 110084784)
___check_obj_id(L['return_dict'], 7698144)
___check_obj_id(L['___stack0'].attentions, 7649088)
___check_obj_id(L['___stack0'].hidden_states, 7649088)
___check_type_id(L['___stack0'].past_key_values, 7638272)
len(L['___stack0'].past_key_values) == 2
hasattr(L['___stack0'].last_hidden_state, '_dynamo_dynamic_indices') == False
___check_obj_id(L['___stack0'].past_key_values[0], 7649088)
___check_obj_id(L['___stack0'].past_key_values[1], 7649088)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
[2023-07-06 04:20:54,378] torch._inductor.cudagraph_trees: [DEBUG] Running warmup of function 5
[2023-07-06 04:20:54,418] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function forward in /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py', 'set env var TORCHDYNAMO_REPORT_GUARD_FAILURES=1 to debug further')
[2023-07-06 04:20:54,419] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:644
[2023-07-06 04:20:54,420] torch._dynamo.variables.builder: [DEBUG] automatic dynamic L['input_ids'] size(1) 1 != 1000
[2023-07-06 04:20:54,420] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['input_ids'] (1, 1) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:54,421] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['attention_mask'] (1, 1003) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:54,422] torch._dynamo.variables.builder: [DEBUG] automatic dynamic L['position_ids'] size(1) 1 != 1000
[2023-07-06 04:20:54,422] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['position_ids'] (1, 1) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:54,422] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['past_key_values'][0] (2, 1, 4, 1003, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None, None]
[2023-07-06 04:20:54,423] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['past_key_values'][1] (2, 1, 4, 1003, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None, None]
[2023-07-06 04:20:54,424] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['valid_past_index'] () [] []
[2023-07-06 04:20:54,424] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:685
[2023-07-06 04:20:54,424] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions []
[2023-07-06 04:20:54,424] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [ConstantVariable(NoneType)]
[2023-07-06 04:20:54,424] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [ConstantVariable(NoneType), ConstantVariable(NoneType)]
[2023-07-06 04:20:54,424] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 12 [ConstantVariable(bool)]
[2023-07-06 04:20:54,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR config [NNModuleVariable()]
[2023-07-06 04:20:54,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR output_attentions [HFPretrainedConfigVariable()]
[2023-07-06 04:20:54,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST output_attentions [ConstantVariable(bool)]
[2023-07-06 04:20:54,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:687
[2023-07-06 04:20:54,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_hidden_states []
[2023-07-06 04:20:54,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [ConstantVariable(NoneType)]
[2023-07-06 04:20:54,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [ConstantVariable(NoneType), ConstantVariable(NoneType)]
[2023-07-06 04:20:54,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 32 [ConstantVariable(bool)]
[2023-07-06 04:20:54,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR config [NNModuleVariable()]
[2023-07-06 04:20:54,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR output_hidden_states [HFPretrainedConfigVariable()]
[2023-07-06 04:20:54,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:686
[2023-07-06 04:20:54,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST output_hidden_states [ConstantVariable(bool)]
[2023-07-06 04:20:54,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:689
[2023-07-06 04:20:54,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST return_dict []
[2023-07-06 04:20:54,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [ConstantVariable(bool)]
[2023-07-06 04:20:54,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [ConstantVariable(bool), ConstantVariable(NoneType)]
[2023-07-06 04:20:54,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 52 [ConstantVariable(bool)]
[2023-07-06 04:20:54,427] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST return_dict []
[2023-07-06 04:20:54,427] torch._dynamo.symbolic_convert: [DEBUG] TRACE JUMP_FORWARD 58 [ConstantVariable(bool)]
[2023-07-06 04:20:54,427] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST return_dict [ConstantVariable(bool)]
[2023-07-06 04:20:54,427] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:692
[2023-07-06 04:20:54,427] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,427] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR model [NNModuleVariable()]
[2023-07-06 04:20:54,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:693
[2023-07-06 04:20:54,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_ids [NNModuleVariable()]
[2023-07-06 04:20:54,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:694
[2023-07-06 04:20:54,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST attention_mask [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:54,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:695
[2023-07-06 04:20:54,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:54,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:696
[2023-07-06 04:20:54,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_key_values [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:54,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:697
[2023-07-06 04:20:54,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST inputs_embeds [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ListVariable()]
[2023-07-06 04:20:54,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:698
[2023-07-06 04:20:54,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ListVariable(), ConstantVariable(NoneType)]
[2023-07-06 04:20:54,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:699
[2023-07-06 04:20:54,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_hidden_states [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ListVariable(), ConstantVariable(NoneType), ConstantVariable(bool)]
[2023-07-06 04:20:54,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:700
[2023-07-06 04:20:54,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST return_dict [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ListVariable(), ConstantVariable(NoneType), ConstantVariable(bool), ConstantVariable(bool)]
[2023-07-06 04:20:54,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:701
[2023-07-06 04:20:54,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ListVariable(), ConstantVariable(NoneType), ConstantVariable(bool), ConstantVariable(bool), ConstantVariable(bool)]
[2023-07-06 04:20:54,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:692
[2023-07-06 04:20:54,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('input_ids', 'attention_mask', 'position_ids', 'past_key_values', 'inputs_embeds', 'output_attentions', 'output_hidden_states', 'return_dict', 'valid_past_index') [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ListVariable(), ConstantVariable(NoneType), ConstantVariable(bool), ConstantVariable(bool), ConstantVariable(bool), TensorVariable()]
[2023-07-06 04:20:54,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 9 [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ListVariable(), ConstantVariable(NoneType), ConstantVariable(bool), ConstantVariable(bool), ConstantVariable(bool), TensorVariable(), ConstantVariable(tuple)]
[2023-07-06 04:20:54,429] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:54,536] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:54,536] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:54,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:54,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:54,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:54,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:54,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:54,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaModel.forward at 0x7fb8d0c46cb0>, NNModuleVariable())]
[2023-07-06 04:20:54,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,539] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,539] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,539] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,539] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,539] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,539] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,539] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,539] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:54,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:54,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:54,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:54,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:54,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:54,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:54,540] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:54,541] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,541] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,541] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:54,541] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:54,541] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaModel.forward at 0x7fb8d0c46cb0>, NNModuleVariable())]
[2023-07-06 04:20:54,541] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaModel.forward at 0x7fb8d0c46cb0>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:54,541] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaModel.forward at 0x7fb8d0c46cb0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,541] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaModel.forward at 0x7fb8d0c46cb0>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,541] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaModel.forward at 0x7fb8d0c46cb0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,542] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:54,542] torch._dynamo.symbolic_convert: [DEBUG] FAILED INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:54,542] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:54,542] torch._dynamo.symbolic_convert: [DEBUG] break_graph_if_unsupported triggered compile
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2187, in inline_call_
sub_locals, closure_cells = func.bind_args(parent, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 187, in bind_args
[
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 188, in <listcomp>
wrap(val=arg, source=source)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 62, in wrap_bound_arg
return VariableBuilder(tx, source=source, **options)(val)
TypeError: VariableBuilder.__init__() got an unexpected keyword argument 'guards'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 390, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1152, in CALL_FUNCTION_KW
self.call_function(fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 559, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 333, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 595, in inline_user_function_return
result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2132, in inline_call
return cls.inline_call_(parent, func, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2229, in inline_call_
tracer.run()
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 708, in run
and self.step()
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 668, in step
getattr(self, inst.opname)(inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 390, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1140, in CALL_FUNCTION_EX
self.call_function(fn, argsvars.items, kwargsvars.items)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 559, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 333, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 287, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 120, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 595, in inline_user_function_return
result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2132, in inline_call
return cls.inline_call_(parent, func, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2190, in inline_call_
raise ArgsMismatchError(
torch._dynamo.exc.ArgsMismatchError: VariableBuilder.__init__() got an unexpected keyword argument 'guards'.
func = 'forward' /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:477, args = [<class 'trfs_fast.llama.LlamaModel'>], kwargs = OrderedDict([('input_ids', TensorVariable()), ('attention_mask', TensorVariable()), ('position_ids', TensorVariable()), ('past_key_values', ListVariable()), ('inputs_embeds', ConstantVariable(NoneType)), ('output_attentions', ConstantVariable(bool)), ('output_hidden_states', ConstantVariable(bool)), ('return_dict', ConstantVariable(bool)), ('valid_past_index', TensorVariable())])
[2023-07-06 04:20:54,542] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:54,542] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason="VariableBuilder.__init__() got an unexpected keyword argument 'guards'.\n func = 'forward' /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:477, args = [<class 'trfs_fast.llama.LlamaModel'>], kwargs = OrderedDict([('input_ids', TensorVariable()), ('attention_mask', TensorVariable()), ('position_ids', TensorVariable()), ('past_key_values', ListVariable()), ('inputs_embeds', ConstantVariable(NoneType)), ('output_attentions', ConstantVariable(bool)), ('output_hidden_states', ConstantVariable(bool)), ('return_dict', ConstantVariable(bool)), ('valid_past_index', TensorVariable())])", user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 692 in forward>, <FrameSummary file /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py, line 1514 in _call_impl>], graph_break=True)
[2023-07-06 04:20:54,551] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662761824)
hasattr(L['input_ids'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['return_dict'], 7698144)
hasattr(L['position_ids'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['inputs_embeds'], 7649088)
hasattr(L['attention_mask'], '_dynamo_dynamic_indices') == False
___check_type_id(L['past_key_values'], 7662688)
len(L['past_key_values']) == 2
hasattr(L['valid_past_index'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['output_attentions'], 7649088)
hasattr(L['past_key_values'][0], '_dynamo_dynamic_indices') == False
hasattr(L['past_key_values'][1], '_dynamo_dynamic_indices') == False
___check_obj_id(L['output_hidden_states'], 7649088)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
[2023-07-06 04:20:54,551] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function forward in /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py', 'set env var TORCHDYNAMO_REPORT_GUARD_FAILURES=1 to debug further')
[2023-07-06 04:20:54,552] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:261
[2023-07-06 04:20:54,552] torch._dynamo.variables.builder: [DEBUG] automatic dynamic L['hidden_states'] size(1) 1 != 1000
[2023-07-06 04:20:54,552] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['hidden_states'] (1, 1, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:54,553] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['attention_mask'] (1, 1003) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:54,554] torch._dynamo.variables.builder: [DEBUG] automatic dynamic L['position_ids'] size(1) 1 != 1000
[2023-07-06 04:20:54,554] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['position_ids'] (1, 1) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:54,555] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['past_key_value'] (2, 1, 4, 1003, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None, None]
[2023-07-06 04:20:54,556] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['valid_past_index'] () [] []
[2023-07-06 04:20:54,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:281
[2023-07-06 04:20:54,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:54,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST residual [TensorVariable()]
[2023-07-06 04:20:54,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:283
[2023-07-06 04:20:54,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR input_layernorm [NNModuleVariable()]
[2023-07-06 04:20:54,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:54,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:54,557] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:54,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:54,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:54,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:54,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:54,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:54,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:54,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:54,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:54,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:54,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:54,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:54,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:54,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:54,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:54,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:54,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:54,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:54,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:54,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:54,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:54,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,563] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:54,563] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:84
[2023-07-06 04:20:54,563] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:54,563] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [TensorVariable()]
[2023-07-06 04:20:54,566] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST input_dtype [TorchVariable(torch.float16)]
[2023-07-06 04:20:54,566] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85
[2023-07-06 04:20:54,566] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:54,566] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:54,566] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:54,566] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR float32 [GetAttrVariable(TensorVariable(), to), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:54,566] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), ConstantVariable(dtype)]
[2023-07-06 04:20:54,568] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR pow [TensorVariable()]
[2023-07-06 04:20:54,568] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [GetAttrVariable(TensorVariable(), pow)]
[2023-07-06 04:20:54,568] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), pow), ConstantVariable(int)]
[2023-07-06 04:20:54,569] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mean [TensorVariable()]
[2023-07-06 04:20:54,569] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [GetAttrVariable(TensorVariable(), mean)]
[2023-07-06 04:20:54,569] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST True [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int)]
[2023-07-06 04:20:54,569] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('keepdim',) [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool)]
[2023-07-06 04:20:54,569] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 2 [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool), ConstantVariable(tuple)]
[2023-07-06 04:20:54,570] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST variance [TensorVariable()]
[2023-07-06 04:20:54,570] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86
[2023-07-06 04:20:54,570] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:54,570] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:54,570] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR rsqrt [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:54,571] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST variance [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>)]
[2023-07-06 04:20:54,571] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:54,571] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR variance_epsilon [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:54,571] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(float)]
[2023-07-06 04:20:54,572] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:54,572] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:54,573] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:54,573] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88
[2023-07-06 04:20:54,573] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,573] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR weight [NNModuleVariable()]
[2023-07-06 04:20:54,574] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].input_layernorm.weight (16,) [<DimDynamic.STATIC: 2>] [None]
[2023-07-06 04:20:54,574] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:54,574] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:54,575] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:54,575] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_dtype [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:54,575] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:54,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:54,576] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:286
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR self_attn [NNModuleVariable()]
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:287
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:288
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST attention_mask [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:289
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:290
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_key_value [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:291
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:292
[2023-07-06 04:20:54,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(bool)]
[2023-07-06 04:20:54,578] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:286
[2023-07-06 04:20:54,578] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('hidden_states', 'attention_mask', 'position_ids', 'past_key_value', 'output_attentions', 'valid_past_index') [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(bool), TensorVariable()]
[2023-07-06 04:20:54,578] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 6 [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(bool), TensorVariable(), ConstantVariable(tuple)]
[2023-07-06 04:20:54,578] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:54,579] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:54,579] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:54,580] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:54,580] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:54,580] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:54,580] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:54,580] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,580] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:54,581] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable())]
[2023-07-06 04:20:54,581] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,581] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,581] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,581] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,581] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,581] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,581] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,581] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,581] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:54,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:54,583] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:54,583] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:54,583] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:54,583] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:54,583] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable())]
[2023-07-06 04:20:54,583] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:54,583] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,583] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,583] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:54,583] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:54,583] torch._dynamo.symbolic_convert: [DEBUG] FAILED INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:54,583] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:54,584] torch._dynamo.symbolic_convert: [DEBUG] break_graph_if_unsupported triggered compile
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2187, in inline_call_
sub_locals, closure_cells = func.bind_args(parent, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 187, in bind_args
[
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 188, in <listcomp>
wrap(val=arg, source=source)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 62, in wrap_bound_arg
return VariableBuilder(tx, source=source, **options)(val)
TypeError: VariableBuilder.__init__() got an unexpected keyword argument 'guards'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 390, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1152, in CALL_FUNCTION_KW
self.call_function(fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 559, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 333, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 595, in inline_user_function_return
result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2132, in inline_call
return cls.inline_call_(parent, func, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2229, in inline_call_
tracer.run()
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 708, in run
and self.step()
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 668, in step
getattr(self, inst.opname)(inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 390, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1140, in CALL_FUNCTION_EX
self.call_function(fn, argsvars.items, kwargsvars.items)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 559, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 333, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 287, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 120, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 595, in inline_user_function_return
result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2132, in inline_call
return cls.inline_call_(parent, func, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2190, in inline_call_
raise ArgsMismatchError(
torch._dynamo.exc.ArgsMismatchError: VariableBuilder.__init__() got an unexpected keyword argument 'guards'.
func = 'forward' /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:190, args = [<class 'trfs_fast.llama.LlamaAttention'>], kwargs = OrderedDict([('hidden_states', TensorVariable()), ('attention_mask', TensorVariable()), ('position_ids', TensorVariable()), ('past_key_value', TensorVariable()), ('output_attentions', ConstantVariable(bool)), ('valid_past_index', TensorVariable())])
[2023-07-06 04:20:54,584] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:54,584] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason="VariableBuilder.__init__() got an unexpected keyword argument 'guards'.\n func = 'forward' /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:190, args = [<class 'trfs_fast.llama.LlamaAttention'>], kwargs = OrderedDict([('hidden_states', TensorVariable()), ('attention_mask', TensorVariable()), ('position_ids', TensorVariable()), ('past_key_value', TensorVariable()), ('output_attentions', ConstantVariable(bool)), ('valid_past_index', TensorVariable())])", user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 286 in forward>, <FrameSummary file /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py, line 1514 in _call_impl>], graph_break=True)
[2023-07-06 04:20:54,585] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['attention_mask']
[2023-07-06 04:20:54,585] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['position_ids']
[2023-07-06 04:20:54,585] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['past_key_value']
[2023-07-06 04:20:54,585] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['valid_past_index']
[2023-07-06 04:20:54,586] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_15 =====
<eval_with_key>.176 class GraphModule(torch.nn.Module):
def forward(self, L_hidden_states_ : torch.Tensor):
l_hidden_states_ = L_hidden_states_
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
to = l_hidden_states_.to(torch.float32)
pow_1 = to.pow(2); to = None
mean = pow_1.mean(-1, keepdim = True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add = mean + 1e-06; mean = None
rsqrt = torch.rsqrt(add); add = None
mul = l_hidden_states_ * rsqrt; l_hidden_states_ = rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
l__self___input_layernorm_weight = self.L__self___input_layernorm_weight
mul_1 = l__self___input_layernorm_weight * mul; l__self___input_layernorm_weight = mul = None
to_1 = mul_1.to(torch.float16); mul_1 = None
return (to_1,)
[2023-07-06 04:20:54,586] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_15 <eval_with_key>.176 opcode name target args kwargs
------------- -------------------------------- -------------------------------------------------------- --------------------------------------- -----------------
placeholder l_hidden_states_ L_hidden_states_ () {}
call_method to to (l_hidden_states_, torch.float32) {}
call_method pow_1 pow (to, 2) {}
call_method mean mean (pow_1, -1) {'keepdim': True}
call_function add <built-in function add> (mean, 1e-06) {}
call_function rsqrt <built-in method rsqrt of type object at 0x7fb940469580> (add,) {}
call_function mul <built-in function mul> (l_hidden_states_, rsqrt) {}
get_attr l__self___input_layernorm_weight L__self___input_layernorm_weight () {}
call_function mul_1 <built-in function mul> (l__self___input_layernorm_weight, mul) {}
call_method to_1 to (mul_1, torch.float16) {}
output output output ((to_1,),) {}
[2023-07-06 04:20:54,587] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
[2023-07-06 04:20:54,605] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 21 =====
<eval_with_key>.180 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[16], arg1_1: f16[1, 1, 16]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
convert_element_type: f32[1, 1, 16] = torch.ops.prims.convert_element_type.default(arg1_1, torch.float32)
pow_1: f32[1, 1, 16] = torch.ops.aten.pow.Tensor_Scalar(convert_element_type, 2); convert_element_type = None
mean: f32[1, 1, 1] = torch.ops.aten.mean.dim(pow_1, [-1], True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add: f32[1, 1, 1] = torch.ops.aten.add.Tensor(mean, 1e-06); mean = None
rsqrt: f32[1, 1, 1] = torch.ops.aten.rsqrt.default(add); add = None
mul: f32[1, 1, 16] = torch.ops.aten.mul.Tensor(arg1_1, rsqrt); arg1_1 = rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
mul_1: f32[1, 1, 16] = torch.ops.aten.mul.Tensor(arg0_1, mul); arg0_1 = mul = None
convert_element_type_1: f16[1, 1, 16] = torch.ops.prims.convert_element_type.default(mul_1, torch.float16); mul_1 = None
return (convert_element_type_1,)
[2023-07-06 04:20:54,607] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 8
[2023-07-06 04:20:54,623] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 8
[2023-07-06 04:20:54,632] torch._inductor.scheduler: [DEBUG] remove_buffer('buf0')
[2023-07-06 04:20:55,244] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 1 configs
[2023-07-06 04:20:55,244] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 1, num_warps: 2, num_stages: 1
[2023-07-06 04:20:55,339] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 1 configs
[2023-07-06 04:20:55,339] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 1, num_warps: 2, num_stages: 1
[2023-07-06 04:20:55,344] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/vz/cvzk75oefn2tjj3ekdj3gg5a7rpdjqti5tysf4vatbmcjiobtp5i.py
[2023-07-06 04:20:55,344] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 8
[2023-07-06 04:20:55,347] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:55,355] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662763072)
hasattr(L['position_ids'], '_dynamo_dynamic_indices') == False
___check_type_id(L['hidden_states'], 77765680)
hasattr(L['hidden_states'], '_dynamo_dynamic_indices') == False
hasattr(L['attention_mask'], '_dynamo_dynamic_indices') == False
hasattr(L['past_key_value'], '_dynamo_dynamic_indices') == False
hasattr(L['valid_past_index'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['output_attentions'], 7698112)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
str(G['torch'].float32) == 'torch.float32'
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks.keys()) == set()
[2023-07-06 04:20:55,355] torch._inductor.cudagraph_trees: [DEBUG] Running warmup of function 6
[2023-07-06 04:20:55,355] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function forward in /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py', 'set env var TORCHDYNAMO_REPORT_GUARD_FAILURES=1 to debug further')
[2023-07-06 04:20:55,356] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:190
[2023-07-06 04:20:55,357] torch._dynamo.variables.builder: [DEBUG] automatic dynamic L['hidden_states'] size(1) 1 != 1000
[2023-07-06 04:20:55,357] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['hidden_states'] (1, 1, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:55,358] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['attention_mask'] (1, 1003) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:55,358] torch._dynamo.variables.builder: [DEBUG] automatic dynamic L['position_ids'] size(1) 1 != 1000
[2023-07-06 04:20:55,358] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['position_ids'] (1, 1) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:55,359] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['past_key_value'] (2, 1, 4, 1003, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None, None]
[2023-07-06 04:20:55,360] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['valid_past_index'] () [] []
[2023-07-06 04:20:55,361] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:199
[2023-07-06 04:20:55,361] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions []
[2023-07-06 04:20:55,361] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST True [ConstantVariable(bool)]
[2023-07-06 04:20:55,361] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 0 [ConstantVariable(bool), ConstantVariable(bool)]
[2023-07-06 04:20:55,361] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(bool)]
[2023-07-06 04:20:55,361] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:202
[2023-07-06 04:20:55,361] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:55,361] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR size [TensorVariable()]
[2023-07-06 04:20:55,361] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), size)]
[2023-07-06 04:20:55,362] torch._dynamo.symbolic_convert: [DEBUG] TRACE UNPACK_SEQUENCE 3 [SizeVariable()]
[2023-07-06 04:20:55,362] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST bsz [ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,362] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST q_len [ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,362] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST _ [ConstantVariable(int)]
[2023-07-06 04:20:55,362] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:205
[2023-07-06 04:20:55,362] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:55,362] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR qkv_proj [NNModuleVariable()]
[2023-07-06 04:20:55,362] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:55,362] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:55,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR view [TensorVariable()]
[2023-07-06 04:20:55,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST bsz [GetAttrVariable(TensorVariable(), view)]
[2023-07-06 04:20:55,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_len [GetAttrVariable(TensorVariable(), view), ConstantVariable(int)]
[2023-07-06 04:20:55,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 3 [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,365] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR num_heads [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), NNModuleVariable()]
[2023-07-06 04:20:55,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR head_dim [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), NNModuleVariable()]
[2023-07-06 04:20:55,366] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 5 [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,367] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR permute [TensorVariable()]
[2023-07-06 04:20:55,367] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [GetAttrVariable(TensorVariable(), permute)]
[2023-07-06 04:20:55,367] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int)]
[2023-07-06 04:20:55,367] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 3 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,367] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,367] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 4 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,367] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 5 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,368] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST query_key_value_states [TensorVariable()]
[2023-07-06 04:20:55,368] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:207
[2023-07-06 04:20:55,368] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST query_key_value_states []
[2023-07-06 04:20:55,368] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TensorVariable()]
[2023-07-06 04:20:55,368] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,369] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST query_states [TensorVariable()]
[2023-07-06 04:20:55,369] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:208
[2023-07-06 04:20:55,369] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST query_key_value_states []
[2023-07-06 04:20:55,369] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [TensorVariable()]
[2023-07-06 04:20:55,369] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,369] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), ConstantVariable(int), ConstantVariable(NoneType)]
[2023-07-06 04:20:55,370] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), SliceVariable()]
[2023-07-06 04:20:55,370] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST key_value_states [TensorVariable()]
[2023-07-06 04:20:55,370] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:211
[2023-07-06 04:20:55,371] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:55,371] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR rotary_emb [NNModuleVariable()]
[2023-07-06 04:20:55,371] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states [NNModuleVariable()]
[2023-07-06 04:20:55,371] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:55,371] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_len [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,371] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [NNModuleVariable(), TensorVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,372] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('seq_len',) [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,372] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 2 [NNModuleVariable(), TensorVariable(), TensorVariable(), ConstantVariable(tuple)]
[2023-07-06 04:20:55,373] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:55,374] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:55,374] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:55,375] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:55,375] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:55,375] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:55,375] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:55,375] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:55,375] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:55,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable())]
[2023-07-06 04:20:55,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:55,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:55,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:55,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:55,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:55,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:55,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:55,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:55,376] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:55,377] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:55,377] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:55,377] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:55,377] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:55,377] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:55,377] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:55,377] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:55,377] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:55,377] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:55,377] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable())]
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:55,378] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:55,379] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:55,379] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddf80e0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 107>
[2023-07-06 04:20:55,379] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:122
[2023-07-06 04:20:55,379] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:55,379] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR cos_cached [NNModuleVariable()]
[2023-07-06 04:20:55,379] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].rotary_emb.cos_cached (1, 1, 2048, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None]
[2023-07-06 04:20:55,380] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:55,380] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:55,380] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [GetAttrVariable(TensorVariable(), to), TensorVariable()]
[2023-07-06 04:20:55,381] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('dtype',) [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:55,381] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 1 [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16), ConstantVariable(tuple)]
[2023-07-06 04:20:55,382] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:123
[2023-07-06 04:20:55,382] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable()]
[2023-07-06 04:20:55,382] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR sin_cached [TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:55,382] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].rotary_emb.sin_cached (1, 1, 2048, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None]
[2023-07-06 04:20:55,383] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,383] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TensorVariable(), GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:55,383] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [TensorVariable(), GetAttrVariable(TensorVariable(), to), TensorVariable()]
[2023-07-06 04:20:55,383] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('dtype',) [TensorVariable(), GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:55,383] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 1 [TensorVariable(), GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16), ConstantVariable(tuple)]
[2023-07-06 04:20:55,385] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:121
[2023-07-06 04:20:55,385] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 2 [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,385] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TupleVariable()]
[2023-07-06 04:20:55,385] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddf80e0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 107>
[2023-07-06 04:20:55,385] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TupleVariable()]
[2023-07-06 04:20:55,385] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:55,385] torch._dynamo.symbolic_convert: [DEBUG] TRACE UNPACK_SEQUENCE 2 [TupleVariable()]
[2023-07-06 04:20:55,385] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST cos [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,385] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST sin [TensorVariable()]
[2023-07-06 04:20:55,385] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:212
[2023-07-06 04:20:55,385] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL apply_rotary_pos_emb_opt []
[2023-07-06 04:20:55,385] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST query_states [UserFunctionVariable()]
[2023-07-06 04:20:55,385] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states [UserFunctionVariable(), TensorVariable()]
[2023-07-06 04:20:55,385] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [UserFunctionVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,385] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [UserFunctionVariable(), TensorVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,386] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos [UserFunctionVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,386] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin [UserFunctionVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,386] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [UserFunctionVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,386] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 5 [UserFunctionVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,386] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object apply_rotary_pos_emb_opt at 0x7fb8cddf82f0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 135>
[2023-07-06 04:20:55,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:139
[2023-07-06 04:20:55,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos []
[2023-07-06 04:20:55,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:55,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:55,387] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:55,388] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:55,388] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:55,388] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:55,389] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST cos [TensorVariable()]
[2023-07-06 04:20:55,389] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:140
[2023-07-06 04:20:55,389] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin []
[2023-07-06 04:20:55,389] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:55,389] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:55,389] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:55,390] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:55,390] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:55,390] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:55,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST sin [TensorVariable()]
[2023-07-06 04:20:55,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:141
[2023-07-06 04:20:55,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos []
[2023-07-06 04:20:55,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [TensorVariable()]
[2023-07-06 04:20:55,391] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,392] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR unsqueeze [TensorVariable()]
[2023-07-06 04:20:55,393] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), unsqueeze)]
[2023-07-06 04:20:55,393] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), unsqueeze), ConstantVariable(int)]
[2023-07-06 04:20:55,393] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST cos [TensorVariable()]
[2023-07-06 04:20:55,393] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:142
[2023-07-06 04:20:55,393] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin []
[2023-07-06 04:20:55,393] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [TensorVariable()]
[2023-07-06 04:20:55,393] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,395] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR unsqueeze [TensorVariable()]
[2023-07-06 04:20:55,395] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), unsqueeze)]
[2023-07-06 04:20:55,395] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), unsqueeze), ConstantVariable(int)]
[2023-07-06 04:20:55,396] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST sin [TensorVariable()]
[2023-07-06 04:20:55,396] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143
[2023-07-06 04:20:55,396] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q []
[2023-07-06 04:20:55,396] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos [TensorVariable()]
[2023-07-06 04:20:55,396] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,397] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL rotate_half [TensorVariable()]
[2023-07-06 04:20:55,397] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q [TensorVariable(), UserFunctionVariable()]
[2023-07-06 04:20:55,397] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), UserFunctionVariable(), TensorVariable()]
[2023-07-06 04:20:55,397] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:55,397] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129
[2023-07-06 04:20:55,397] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:55,398] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR roll [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:55,398] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>)]
[2023-07-06 04:20:55,398] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:55,398] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,398] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable()]
[2023-07-06 04:20:55,399] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,399] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,399] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,399] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,399] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('shifts', 'dims') [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,399] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 3 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int), ConstantVariable(tuple)]
[2023-07-06 04:20:55,400] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST x [TensorVariable()]
[2023-07-06 04:20:55,400] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130
[2023-07-06 04:20:55,400] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:55,400] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST Ellipsis [TensorVariable()]
[2023-07-06 04:20:55,400] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable(), ConstantVariable(ellipsis)]
[2023-07-06 04:20:55,400] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType)]
[2023-07-06 04:20:55,400] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), TensorVariable()]
[2023-07-06 04:20:55,401] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable()]
[2023-07-06 04:20:55,401] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,401] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:55,401] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,401] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:55,401] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 2 [TensorVariable(), ConstantVariable(ellipsis), SliceVariable()]
[2023-07-06 04:20:55,401] torch._dynamo.symbolic_convert: [DEBUG] TRACE DUP_TOP_TWO None [TensorVariable(), TupleVariable()]
[2023-07-06 04:20:55,401] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TupleVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:55,402] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:55,402] torch._dynamo.symbolic_convert: [DEBUG] TRACE INPLACE_MULTIPLY None [TensorVariable(), TupleVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,403] torch._dynamo.symbolic_convert: [DEBUG] TRACE ROT_THREE None [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:55,403] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_SUBSCR None [TensorVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:55,403] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:131
[2023-07-06 04:20:55,403] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:55,403] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:55,403] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:55,403] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,403] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,404] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,405] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST q_embed [TensorVariable()]
[2023-07-06 04:20:55,405] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144
[2023-07-06 04:20:55,405] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_states []
[2023-07-06 04:20:55,405] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR copy_ [TensorVariable()]
[2023-07-06 04:20:55,406] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_states [GetAttrVariable(TensorVariable(), copy_)]
[2023-07-06 04:20:55,406] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos [GetAttrVariable(TensorVariable(), copy_), TensorVariable()]
[2023-07-06 04:20:55,406] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL rotate_half [GetAttrVariable(TensorVariable(), copy_), TensorVariable()]
[2023-07-06 04:20:55,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_states [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), UserFunctionVariable()]
[2023-07-06 04:20:55,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), UserFunctionVariable(), TensorVariable()]
[2023-07-06 04:20:55,407] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:55,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129
[2023-07-06 04:20:55,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:55,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR roll [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:55,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>)]
[2023-07-06 04:20:55,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:55,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable()]
[2023-07-06 04:20:55,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,408] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('shifts', 'dims') [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 3 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int), ConstantVariable(tuple)]
[2023-07-06 04:20:55,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST x [TensorVariable()]
[2023-07-06 04:20:55,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130
[2023-07-06 04:20:55,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:55,409] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST Ellipsis [TensorVariable()]
[2023-07-06 04:20:55,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable(), ConstantVariable(ellipsis)]
[2023-07-06 04:20:55,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType)]
[2023-07-06 04:20:55,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), TensorVariable()]
[2023-07-06 04:20:55,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable()]
[2023-07-06 04:20:55,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:55,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:55,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:55,411] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 2 [TensorVariable(), ConstantVariable(ellipsis), SliceVariable()]
[2023-07-06 04:20:55,411] torch._dynamo.symbolic_convert: [DEBUG] TRACE DUP_TOP_TWO None [TensorVariable(), TupleVariable()]
[2023-07-06 04:20:55,411] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TupleVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:55,412] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:55,412] torch._dynamo.symbolic_convert: [DEBUG] TRACE INPLACE_MULTIPLY None [TensorVariable(), TupleVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,412] torch._dynamo.symbolic_convert: [DEBUG] TRACE ROT_THREE None [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:55,412] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_SUBSCR None [TensorVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:55,413] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:131
[2023-07-06 04:20:55,413] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:55,413] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:55,413] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:55,413] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,413] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,414] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), copy_), TensorVariable()]
[2023-07-06 04:20:55,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_TOP None [TensorVariable()]
[2023-07-06 04:20:55,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:145
[2023-07-06 04:20:55,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_embed []
[2023-07-06 04:20:55,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:55,415] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object apply_rotary_pos_emb_opt at 0x7fb8cddf82f0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 135>
[2023-07-06 04:20:55,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST query_states [TensorVariable()]
[2023-07-06 04:20:55,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:215
[2023-07-06 04:20:55,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index []
[2023-07-06 04:20:55,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST past_kv_slice_start [TensorVariable()]
[2023-07-06 04:20:55,416] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:216
[2023-07-06 04:20:55,416] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:55,416] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR eq [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:55,416] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TorchVariable(<built-in method eq of type object at 0x7fb940469580>)]
[2023-07-06 04:20:55,416] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:55,416] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,417] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable()]
[2023-07-06 04:20:55,417] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:55,418] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_len [TensorVariable()]
[2023-07-06 04:20:55,418] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,419] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:55,419] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR ne [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:55,419] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>)]
[2023-07-06 04:20:55,419] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:55,419] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,420] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,420] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TensorVariable(), GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:55,421] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,421] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,422] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,422] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,423] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,424] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST past_kv_slice_end [TensorVariable()]
[2023-07-06 04:20:55,424] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:217
[2023-07-06 04:20:55,424] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:55,424] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR eq [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:55,424] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TorchVariable(<built-in method eq of type object at 0x7fb940469580>)]
[2023-07-06 04:20:55,424] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:55,424] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable()]
[2023-07-06 04:20:55,425] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:55,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states [TensorVariable()]
[2023-07-06 04:20:55,426] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,427] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -2 [TensorVariable(), ShapeVariable()]
[2023-07-06 04:20:55,427] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,427] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:55,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR ne [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:55,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>)]
[2023-07-06 04:20:55,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:55,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:55,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,429] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TensorVariable(), GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:55,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_kv_slice_end [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,432] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST past_state_slice_end [TensorVariable()]
[2023-07-06 04:20:55,432] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:218
[2023-07-06 04:20:55,432] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states []
[2023-07-06 04:20:55,432] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_key_value [TensorVariable()]
[2023-07-06 04:20:55,432] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST Ellipsis [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,432] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_kv_slice_start [TensorVariable(), TensorVariable(), ConstantVariable(ellipsis)]
[2023-07-06 04:20:55,432] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_kv_slice_end [TensorVariable(), TensorVariable(), ConstantVariable(ellipsis), TensorVariable()]
[2023-07-06 04:20:55,432] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), TensorVariable(), ConstantVariable(ellipsis), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:55,432] torch._dynamo.symbolic_convert: [DEBUG] step triggered compile
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 668, in step
getattr(self, inst.opname)(inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1251, in BUILD_SLICE
SliceVariable(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/base.py", line 27, in __call__
obj = type.__call__(cls, *args, **kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/lists.py", line 616, in __init__
unimplemented("Dynamic slicing on data-dependent value is not supported")
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 140, in unimplemented
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: Dynamic slicing on data-dependent value is not supported
[2023-07-06 04:20:55,432] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:55,432] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='step_unsupported', user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 218 in forward>], graph_break=True)
[2023-07-06 04:20:55,434] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['attention_mask']
[2023-07-06 04:20:55,434] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['past_key_value']
[2023-07-06 04:20:55,436] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_17 =====
<eval_with_key>.182 class GraphModule(torch.nn.Module):
def forward(self, L_hidden_states_ : torch.Tensor, L_position_ids_ : torch.Tensor, L_valid_past_index_ : torch.Tensor):
l_hidden_states_ = L_hidden_states_
l_position_ids_ = L_position_ids_
l_valid_past_index_ = L_valid_past_index_
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:205, code: query_key_value_states = self.qkv_proj(hidden_states).view(bsz, q_len, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4)
l__self___qkv_proj = self.L__self___qkv_proj(l_hidden_states_); l_hidden_states_ = None
view = l__self___qkv_proj.view(1, 1, 3, 4, 4); l__self___qkv_proj = None
permute = view.permute(2, 0, 3, 1, 4); view = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:207, code: query_states = query_key_value_states[0]
getitem = permute[0]
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:208, code: key_value_states = query_key_value_states[1:]
getitem_1 = permute[slice(1, None, None)]; permute = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:211, code: cos, sin = self.rotary_emb(key_value_states, seq_len=valid_past_index + q_len)
add = l_valid_past_index_ + 1
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:122, code: self.cos_cached.to(dtype=x.dtype),
l__self___rotary_emb_cos_cached = self.L__self___rotary_emb_cos_cached
to = l__self___rotary_emb_cos_cached.to(dtype = torch.float16); l__self___rotary_emb_cos_cached = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:123, code: self.sin_cached.to(dtype=x.dtype),
l__self___rotary_emb_sin_cached = self.L__self___rotary_emb_sin_cached
to_1 = l__self___rotary_emb_sin_cached.to(dtype = torch.float16); l__self___rotary_emb_sin_cached = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:212, code: query_states = apply_rotary_pos_emb_opt(query_states, key_value_states[0], cos, sin, position_ids)
getitem_2 = getitem_1[0]
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:139, code: cos = cos.squeeze(1).squeeze(0) # [seq_len, dim]
squeeze = to.squeeze(1); to = None
squeeze_1 = squeeze.squeeze(0); squeeze = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:140, code: sin = sin.squeeze(1).squeeze(0) # [seq_len, dim]
squeeze_2 = to_1.squeeze(1); to_1 = None
squeeze_3 = squeeze_2.squeeze(0); squeeze_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:141, code: cos = cos[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
getitem_3 = squeeze_1[l_position_ids_]; squeeze_1 = None
unsqueeze = getitem_3.unsqueeze(1); getitem_3 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:142, code: sin = sin[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
getitem_4 = squeeze_3[l_position_ids_]; squeeze_3 = l_position_ids_ = None
unsqueeze_1 = getitem_4.unsqueeze(1); getitem_4 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul = getitem * unsqueeze
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll = torch.roll(getitem, shifts = 2, dims = -1); getitem = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
getitem_5 = roll[(Ellipsis, slice(None, 2, None))]
getitem_5 *= -1; imul = getitem_5; getitem_5 = None
roll[(Ellipsis, slice(None, 2, None))] = imul; setitem = roll; imul = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul_1 = roll * unsqueeze_1; roll = None
add_1 = mul + mul_1; mul = mul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_2 = getitem_2 * unsqueeze; unsqueeze = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll_1 = torch.roll(getitem_2, shifts = 2, dims = -1)
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
getitem_6 = roll_1[(Ellipsis, slice(None, 2, None))]
getitem_6 *= -1; imul_1 = getitem_6; getitem_6 = None
roll_1[(Ellipsis, slice(None, 2, None))] = imul_1; setitem_1 = roll_1; imul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_3 = roll_1 * unsqueeze_1; roll_1 = unsqueeze_1 = None
add_2 = mul_2 + mul_3; mul_2 = mul_3 = None
copy_ = getitem_2.copy_(add_2); getitem_2 = add_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:216, code: past_kv_slice_end = torch.eq(valid_past_index, 0).int() * q_len + torch.ne(valid_past_index, 0).int() * (valid_past_index + 1)
eq = torch.eq(l_valid_past_index_, 0)
int_1 = eq.int(); eq = None
mul_4 = int_1 * 1; int_1 = None
ne = torch.ne(l_valid_past_index_, 0)
int_2 = ne.int(); ne = None
add_3 = l_valid_past_index_ + 1
mul_5 = int_2 * add_3; int_2 = add_3 = None
add_4 = mul_4 + mul_5; mul_4 = mul_5 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:217, code: past_state_slice_end = torch.eq(valid_past_index, 0).int() * key_value_states.shape[-2] + torch.ne(valid_past_index, 0).int() * (past_kv_slice_end)
eq_1 = torch.eq(l_valid_past_index_, 0)
int_3 = eq_1.int(); eq_1 = None
mul_6 = int_3 * 1; int_3 = None
ne_1 = torch.ne(l_valid_past_index_, 0); l_valid_past_index_ = None
int_4 = ne_1.int(); ne_1 = None
mul_7 = int_4 * add_4; int_4 = None
add_5 = mul_6 + mul_7; mul_6 = mul_7 = None
return (add_1, getitem_1, add_4, add_5)
[2023-07-06 04:20:55,437] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_17 <eval_with_key>.182 opcode name target args kwargs
------------- ------------------------------- ------------------------------------------------------- -------------------------------------------------- -------------------------
placeholder l_hidden_states_ L_hidden_states_ () {}
placeholder l_position_ids_ L_position_ids_ () {}
placeholder l_valid_past_index_ L_valid_past_index_ () {}
call_module l__self___qkv_proj L__self___qkv_proj (l_hidden_states_,) {}
call_method view view (l__self___qkv_proj, 1, 1, 3, 4, 4) {}
call_method permute permute (view, 2, 0, 3, 1, 4) {}
call_function getitem <built-in function getitem> (permute, 0) {}
call_function getitem_1 <built-in function getitem> (permute, slice(1, None, None)) {}
call_function add <built-in function add> (l_valid_past_index_, 1) {}
get_attr l__self___rotary_emb_cos_cached L__self___rotary_emb_cos_cached () {}
call_method to to (l__self___rotary_emb_cos_cached,) {'dtype': torch.float16}
get_attr l__self___rotary_emb_sin_cached L__self___rotary_emb_sin_cached () {}
call_method to_1 to (l__self___rotary_emb_sin_cached,) {'dtype': torch.float16}
call_function getitem_2 <built-in function getitem> (getitem_1, 0) {}
call_method squeeze squeeze (to, 1) {}
call_method squeeze_1 squeeze (squeeze, 0) {}
call_method squeeze_2 squeeze (to_1, 1) {}
call_method squeeze_3 squeeze (squeeze_2, 0) {}
call_function getitem_3 <built-in function getitem> (squeeze_1, l_position_ids_) {}
call_method unsqueeze unsqueeze (getitem_3, 1) {}
call_function getitem_4 <built-in function getitem> (squeeze_3, l_position_ids_) {}
call_method unsqueeze_1 unsqueeze (getitem_4, 1) {}
call_function mul <built-in function mul> (getitem, unsqueeze) {}
call_function roll <built-in method roll of type object at 0x7fb940469580> (getitem,) {'shifts': 2, 'dims': -1}
call_function getitem_5 <built-in function getitem> (roll, (Ellipsis, slice(None, 2, None))) {}
call_function imul <built-in function imul> (getitem_5, -1) {}
call_function setitem <built-in function setitem> (roll, (Ellipsis, slice(None, 2, None)), imul) {}
call_function mul_1 <built-in function mul> (roll, unsqueeze_1) {}
call_function add_1 <built-in function add> (mul, mul_1) {}
call_function mul_2 <built-in function mul> (getitem_2, unsqueeze) {}
call_function roll_1 <built-in method roll of type object at 0x7fb940469580> (getitem_2,) {'shifts': 2, 'dims': -1}
call_function getitem_6 <built-in function getitem> (roll_1, (Ellipsis, slice(None, 2, None))) {}
call_function imul_1 <built-in function imul> (getitem_6, -1) {}
call_function setitem_1 <built-in function setitem> (roll_1, (Ellipsis, slice(None, 2, None)), imul_1) {}
call_function mul_3 <built-in function mul> (roll_1, unsqueeze_1) {}
call_function add_2 <built-in function add> (mul_2, mul_3) {}
call_method copy_ copy_ (getitem_2, add_2) {}
call_function eq <built-in method eq of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_1 int (eq,) {}
call_function mul_4 <built-in function mul> (int_1, 1) {}
call_function ne <built-in method ne of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_2 int (ne,) {}
call_function add_3 <built-in function add> (l_valid_past_index_, 1) {}
call_function mul_5 <built-in function mul> (int_2, add_3) {}
call_function add_4 <built-in function add> (mul_4, mul_5) {}
call_function eq_1 <built-in method eq of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_3 int (eq_1,) {}
call_function mul_6 <built-in function mul> (int_3, 1) {}
call_function ne_1 <built-in method ne of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_4 int (ne_1,) {}
call_function mul_7 <built-in function mul> (int_4, add_4) {}
call_function add_5 <built-in function add> (mul_6, mul_7) {}
output output output ((add_1, getitem_1, add_4, add_5),) {}
[2023-07-06 04:20:55,439] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
[2023-07-06 04:20:55,561] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 22 =====
<eval_with_key>.186 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[48, 16], arg1_1: f32[1, 1, 2048, 4], arg2_1: f32[1, 1, 2048, 4], arg3_1: f16[1, 1, 16], arg4_1: i64[1, 1], arg5_1: i64[]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:205, code: query_key_value_states = self.qkv_proj(hidden_states).view(bsz, q_len, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4)
permute: f16[16, 48] = torch.ops.aten.permute.default(arg0_1, [1, 0]); arg0_1 = None
view: f16[1, 16] = torch.ops.aten.view.default(arg3_1, [1, 16]); arg3_1 = None
mm: f16[1, 48] = torch.ops.aten.mm.default(view, permute); view = permute = None
view_1: f16[1, 1, 48] = torch.ops.aten.view.default(mm, [1, 1, 48])
view_2: f16[1, 1, 3, 4, 4] = torch.ops.aten.view.default(view_1, [1, 1, 3, 4, 4]); view_1 = None
permute_1: f16[3, 1, 4, 1, 4] = torch.ops.aten.permute.default(view_2, [2, 0, 3, 1, 4]); view_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:207, code: query_states = query_key_value_states[0]
select: f16[1, 4, 1, 4] = torch.ops.aten.select.int(permute_1, 0, 0)
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:208, code: key_value_states = query_key_value_states[1:]
slice_1: f16[2, 1, 4, 1, 4] = torch.ops.aten.slice.Tensor(permute_1, 0, 1, 9223372036854775807); permute_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:122, code: self.cos_cached.to(dtype=x.dtype),
convert_element_type: f16[1, 1, 2048, 4] = torch.ops.prims.convert_element_type.default(arg1_1, torch.float16); arg1_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:123, code: self.sin_cached.to(dtype=x.dtype),
convert_element_type_1: f16[1, 1, 2048, 4] = torch.ops.prims.convert_element_type.default(arg2_1, torch.float16); arg2_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:212, code: query_states = apply_rotary_pos_emb_opt(query_states, key_value_states[0], cos, sin, position_ids)
select_1: f16[1, 4, 1, 4] = torch.ops.aten.select.int(slice_1, 0, 0); slice_1 = None
# No stacktrace found for following nodes
squeeze: f16[1, 2048, 4] = torch.ops.aten.squeeze.dim(convert_element_type, 1); convert_element_type = None
squeeze_1: f16[2048, 4] = torch.ops.aten.squeeze.dim(squeeze, 0); squeeze = None
squeeze_2: f16[1, 2048, 4] = torch.ops.aten.squeeze.dim(convert_element_type_1, 1); convert_element_type_1 = None
squeeze_3: f16[2048, 4] = torch.ops.aten.squeeze.dim(squeeze_2, 0); squeeze_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:141, code: cos = cos[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
index: f16[1, 1, 4] = torch.ops.aten.index.Tensor(squeeze_1, [arg4_1]); squeeze_1 = None
unsqueeze: f16[1, 1, 1, 4] = torch.ops.aten.unsqueeze.default(index, 1); index = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:142, code: sin = sin[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
index_1: f16[1, 1, 4] = torch.ops.aten.index.Tensor(squeeze_3, [arg4_1]); squeeze_3 = arg4_1 = None
unsqueeze_1: f16[1, 1, 1, 4] = torch.ops.aten.unsqueeze.default(index_1, 1); index_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul: f16[1, 4, 1, 4] = torch.ops.aten.mul.Tensor(select, unsqueeze)
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll: f16[1, 4, 1, 4] = torch.ops.aten.roll.default(select, [2], [-1]); select = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
slice_2: f16[1, 4, 1, 2] = torch.ops.aten.slice.Tensor(roll, 3, 0, 2)
mul_1: f16[1, 4, 1, 2] = torch.ops.aten.mul.Tensor(slice_2, -1); slice_2 = None
slice_scatter: f16[1, 4, 1, 4] = torch.ops.aten.slice_scatter.default(roll, mul_1, 3, 0, 2); roll = mul_1 = None
slice_3: f16[1, 4, 1, 2] = torch.ops.aten.slice.Tensor(slice_scatter, 3, 0, 2)
slice_scatter_1: f16[1, 4, 1, 4] = torch.ops.aten.slice_scatter.default(slice_scatter, slice_3, 3, 0, 2); slice_scatter = slice_3 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul_2: f16[1, 4, 1, 4] = torch.ops.aten.mul.Tensor(slice_scatter_1, unsqueeze_1); slice_scatter_1 = None
add_1: f16[1, 4, 1, 4] = torch.ops.aten.add.Tensor(mul, mul_2); mul = mul_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_3: f16[1, 4, 1, 4] = torch.ops.aten.mul.Tensor(select_1, unsqueeze); unsqueeze = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll_1: f16[1, 4, 1, 4] = torch.ops.aten.roll.default(select_1, [2], [-1]); select_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
slice_7: f16[1, 4, 1, 2] = torch.ops.aten.slice.Tensor(roll_1, 3, 0, 2)
mul_4: f16[1, 4, 1, 2] = torch.ops.aten.mul.Tensor(slice_7, -1); slice_7 = None
slice_scatter_2: f16[1, 4, 1, 4] = torch.ops.aten.slice_scatter.default(roll_1, mul_4, 3, 0, 2); roll_1 = mul_4 = None
slice_8: f16[1, 4, 1, 2] = torch.ops.aten.slice.Tensor(slice_scatter_2, 3, 0, 2)
slice_scatter_3: f16[1, 4, 1, 4] = torch.ops.aten.slice_scatter.default(slice_scatter_2, slice_8, 3, 0, 2); slice_scatter_2 = slice_8 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_5: f16[1, 4, 1, 4] = torch.ops.aten.mul.Tensor(slice_scatter_3, unsqueeze_1); slice_scatter_3 = unsqueeze_1 = None
add_2: f16[1, 4, 1, 4] = torch.ops.aten.add.Tensor(mul_3, mul_5); mul_3 = mul_5 = None
view_3: f16[1, 1, 48] = torch.ops.aten.view.default(mm, [1, 1, 48]); mm = None
view_4: f16[1, 1, 3, 4, 4] = torch.ops.aten.view.default(view_3, [1, 1, 3, 4, 4]); view_3 = None
permute_2: f16[3, 1, 4, 1, 4] = torch.ops.aten.permute.default(view_4, [2, 0, 3, 1, 4]); view_4 = None
slice_12: f16[2, 1, 4, 1, 4] = torch.ops.aten.slice.Tensor(permute_2, 0, 1, 9223372036854775807)
select_scatter: f16[2, 1, 4, 1, 4] = torch.ops.aten.select_scatter.default(slice_12, add_2, 0, 0); slice_12 = add_2 = None
slice_scatter_4: f16[3, 1, 4, 1, 4] = torch.ops.aten.slice_scatter.default(permute_2, select_scatter, 0, 1, 9223372036854775807); permute_2 = select_scatter = None
permute_3: f16[1, 1, 3, 4, 4] = torch.ops.aten.permute.default(slice_scatter_4, [1, 3, 0, 2, 4]); slice_scatter_4 = None
view_5: f16[1, 1, 48] = torch.ops.aten.view.default(permute_3, [1, 1, 48]); permute_3 = None
view_6: f16[1, 48] = torch.ops.aten.view.default(view_5, [1, 48]); view_5 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:216, code: past_kv_slice_end = torch.eq(valid_past_index, 0).int() * q_len + torch.ne(valid_past_index, 0).int() * (valid_past_index + 1)
eq: b8[] = torch.ops.aten.eq.Scalar(arg5_1, 0)
convert_element_type_2: i32[] = torch.ops.prims.convert_element_type.default(eq, torch.int32); eq = None
mul_6: i32[] = torch.ops.aten.mul.Tensor(convert_element_type_2, 1); convert_element_type_2 = None
ne: b8[] = torch.ops.aten.ne.Scalar(arg5_1, 0)
convert_element_type_3: i32[] = torch.ops.prims.convert_element_type.default(ne, torch.int32); ne = None
add_3: i64[] = torch.ops.aten.add.Tensor(arg5_1, 1)
mul_7: i64[] = torch.ops.aten.mul.Tensor(convert_element_type_3, add_3); convert_element_type_3 = add_3 = None
add_4: i64[] = torch.ops.aten.add.Tensor(mul_6, mul_7); mul_6 = mul_7 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:217, code: past_state_slice_end = torch.eq(valid_past_index, 0).int() * key_value_states.shape[-2] + torch.ne(valid_past_index, 0).int() * (past_kv_slice_end)
eq_1: b8[] = torch.ops.aten.eq.Scalar(arg5_1, 0)
convert_element_type_4: i32[] = torch.ops.prims.convert_element_type.default(eq_1, torch.int32); eq_1 = None
mul_8: i32[] = torch.ops.aten.mul.Tensor(convert_element_type_4, 1); convert_element_type_4 = None
ne_1: b8[] = torch.ops.aten.ne.Scalar(arg5_1, 0); arg5_1 = None
convert_element_type_5: i32[] = torch.ops.prims.convert_element_type.default(ne_1, torch.int32); ne_1 = None
mul_9: i64[] = torch.ops.aten.mul.Tensor(convert_element_type_5, add_4); convert_element_type_5 = None
add_5: i64[] = torch.ops.aten.add.Tensor(mul_8, mul_9); mul_8 = mul_9 = None
# No stacktrace found for following nodes
view_9: f16[1, 1, 48] = torch.ops.aten.view.default(view_6, [1, 1, 48]); view_6 = None
view_10: f16[1, 1, 3, 4, 4] = torch.ops.aten.view.default(view_9, [1, 1, 3, 4, 4]); view_9 = None
permute_5: f16[3, 1, 4, 1, 4] = torch.ops.aten.permute.default(view_10, [2, 0, 3, 1, 4]); view_10 = None
slice_14: f16[2, 1, 4, 1, 4] = torch.ops.aten.slice.Tensor(permute_5, 0, 1, 9223372036854775807); permute_5 = None
return (add_1, slice_14, add_4, add_5)
[2023-07-06 04:20:55,596] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 9
[2023-07-06 04:20:55,706] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 9
[2023-07-06 04:20:55,887] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 1 configs
[2023-07-06 04:20:55,887] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 32, num_warps: 1, num_stages: 1
[2023-07-06 04:20:55,888] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 1 configs
[2023-07-06 04:20:55,888] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 16, num_warps: 1, num_stages: 1
[2023-07-06 04:20:56,144] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 1 configs
[2023-07-06 04:20:56,144] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 16, num_warps: 1, num_stages: 1
[2023-07-06 04:20:56,156] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 1 configs
[2023-07-06 04:20:56,156] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 32, num_warps: 1, num_stages: 1
[2023-07-06 04:20:57,507] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/hj/chjdplcvb6oibk7aigc4ou76dlz6yn4gxfu3dxactvqvnkkqmftz.py
[2023-07-06 04:20:57,508] torch._inductor.compile_fx: [DEBUG] disabled cudagraphs because ['non-cuda device in graph']
skipping cudagraphs due to multiple devices
[2023-07-06 04:20:57,508] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 9
[2023-07-06 04:20:57,510] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:57,520] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662763168)
hasattr(L['position_ids'], '_dynamo_dynamic_indices') == False
hasattr(L['hidden_states'], '_dynamo_dynamic_indices') == False
hasattr(L['attention_mask'], '_dynamo_dynamic_indices') == False
hasattr(L['past_key_value'], '_dynamo_dynamic_indices') == False
hasattr(L['valid_past_index'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['output_attentions'], 7698112)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks.keys()) == set()
[2023-07-06 04:20:57,546] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function <resume in forward> in /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py', 'set env var TORCHDYNAMO_REPORT_GUARD_FAILURES=1 to debug further')
[2023-07-06 04:20:57,547] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <resume in forward> /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:286
[2023-07-06 04:20:57,547] torch._dynamo.variables.builder: [DEBUG] automatic dynamic L['___stack0'][0] size(1) 1 != 1000
[2023-07-06 04:20:57,547] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['___stack0'][0] (1, 1, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:57,549] torch._dynamo.variables.builder: [DEBUG] automatic dynamic L['residual'] size(1) 1 != 1000
[2023-07-06 04:20:57,549] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['residual'] (1, 1, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:57,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST ___stack0 []
[2023-07-06 04:20:57,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE JUMP_ABSOLUTE 38 [TupleVariable()]
[2023-07-06 04:20:57,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE UNPACK_SEQUENCE 3 [TupleVariable()]
[2023-07-06 04:20:57,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [ConstantVariable(NoneType), ConstantVariable(NoneType), TensorVariable()]
[2023-07-06 04:20:57,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST self_attn_weights [ConstantVariable(NoneType), ConstantVariable(NoneType)]
[2023-07-06 04:20:57,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST present_key_value [ConstantVariable(NoneType)]
[2023-07-06 04:20:57,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:294
[2023-07-06 04:20:57,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST residual []
[2023-07-06 04:20:57,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:57,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:57,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:57,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:297
[2023-07-06 04:20:57,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:57,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST residual [TensorVariable()]
[2023-07-06 04:20:57,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:298
[2023-07-06 04:20:57,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR post_attention_layernorm [NNModuleVariable()]
[2023-07-06 04:20:57,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:57,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:57,553] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:57,554] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:57,554] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:57,554] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:57,555] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:57,555] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:57,555] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:57,555] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,555] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:57,555] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:57,555] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,555] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,555] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,556] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:57,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:57,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:57,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:57,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,557] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:57,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:57,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:57,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:57,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:57,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:57,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:57,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:57,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:57,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:57,558] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:57,559] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:57,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:84
[2023-07-06 04:20:57,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:57,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [TensorVariable()]
[2023-07-06 04:20:57,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST input_dtype [TorchVariable(torch.float16)]
[2023-07-06 04:20:57,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85
[2023-07-06 04:20:57,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:57,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:57,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:57,559] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR float32 [GetAttrVariable(TensorVariable(), to), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:57,560] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), ConstantVariable(dtype)]
[2023-07-06 04:20:57,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR pow [TensorVariable()]
[2023-07-06 04:20:57,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [GetAttrVariable(TensorVariable(), pow)]
[2023-07-06 04:20:57,561] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), pow), ConstantVariable(int)]
[2023-07-06 04:20:57,562] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mean [TensorVariable()]
[2023-07-06 04:20:57,563] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [GetAttrVariable(TensorVariable(), mean)]
[2023-07-06 04:20:57,563] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST True [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int)]
[2023-07-06 04:20:57,563] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('keepdim',) [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool)]
[2023-07-06 04:20:57,563] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 2 [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool), ConstantVariable(tuple)]
[2023-07-06 04:20:57,564] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST variance [TensorVariable()]
[2023-07-06 04:20:57,564] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86
[2023-07-06 04:20:57,564] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:57,564] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:57,564] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR rsqrt [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:57,564] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST variance [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>)]
[2023-07-06 04:20:57,564] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:57,564] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR variance_epsilon [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:57,565] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(float)]
[2023-07-06 04:20:57,565] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:57,566] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:57,567] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:57,567] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88
[2023-07-06 04:20:57,567] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,567] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR weight [NNModuleVariable()]
[2023-07-06 04:20:57,567] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].post_attention_layernorm.weight (16,) [<DimDynamic.STATIC: 2>] [None]
[2023-07-06 04:20:57,568] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:57,568] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:57,569] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:57,569] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_dtype [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:57,569] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:57,570] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:57,570] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:57,570] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:57,570] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:57,570] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:57,570] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:299
[2023-07-06 04:20:57,571] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,571] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mlp [NNModuleVariable()]
[2023-07-06 04:20:57,571] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:57,571] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:57,571] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:57,573] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:57,573] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:57,573] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:57,573] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:57,573] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:57,573] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:57,574] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,574] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:57,574] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable())]
[2023-07-06 04:20:57,574] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,574] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,574] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,574] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,574] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,574] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,575] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,575] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,575] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,575] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,575] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,575] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,575] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable())]
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:57,576] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:57,577] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddf85b0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 161>
[2023-07-06 04:20:57,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162
[2023-07-06 04:20:57,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR down_proj [NNModuleVariable()]
[2023-07-06 04:20:57,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [NNModuleVariable()]
[2023-07-06 04:20:57,577] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR act_fn [NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:57,578] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:57,578] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR gate_proj [NNModuleVariable(), NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:57,578] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [NNModuleVariable(), NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:57,578] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), NNModuleVariable(), NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:57,581] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:57,581] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:57,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:57,582] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:57,583] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:57,583] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:57,583] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:57,583] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:57,583] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,583] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:57,584] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable())]
[2023-07-06 04:20:57,584] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,584] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,584] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,584] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,584] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,584] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,584] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,584] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,584] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,585] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,586] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:57,586] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:57,586] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,586] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,586] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:57,586] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:57,586] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable())]
[2023-07-06 04:20:57,586] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:57,586] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:57,586] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:57,586] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:57,586] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8ccf62760, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py", line 149>
[2023-07-06 04:20:57,586] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py:150
[2023-07-06 04:20:57,586] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL nn []
[2023-07-06 04:20:57,587] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR functional [TorchVariable(<module 'torch.nn' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/__init__.py'>)]
[2023-07-06 04:20:57,587] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR silu [TorchVariable(<module 'torch.nn.functional' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/functional.py'>)]
[2023-07-06 04:20:57,587] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input [TorchVariable(<function silu at 0x7fb945bc5990>)]
[2023-07-06 04:20:57,587] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TorchVariable(<function silu at 0x7fb945bc5990>), TensorVariable()]
[2023-07-06 04:20:57,590] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:57,590] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8ccf62760, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py", line 149>
[2023-07-06 04:20:57,590] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:57,590] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:57,590] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:57,591] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR up_proj [NNModuleVariable(), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:57,591] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [NNModuleVariable(), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:57,591] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable(), NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:57,593] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:57,594] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:57,597] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:57,597] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddf85b0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 161>
[2023-07-06 04:20:57,597] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:57,597] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:57,597] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:57,597] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:300
[2023-07-06 04:20:57,597] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST residual []
[2023-07-06 04:20:57,597] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:57,597] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:57,598] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:57,598] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:302
[2023-07-06 04:20:57,598] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:57,598] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 1 [TensorVariable()]
[2023-07-06 04:20:57,598] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST outputs [TupleVariable()]
[2023-07-06 04:20:57,599] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:304
[2023-07-06 04:20:57,599] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions []
[2023-07-06 04:20:57,599] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 106 [ConstantVariable(bool)]
[2023-07-06 04:20:57,599] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:307
[2023-07-06 04:20:57,599] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs []
[2023-07-06 04:20:57,599] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST present_key_value [TupleVariable()]
[2023-07-06 04:20:57,599] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 1 [TupleVariable(), ConstantVariable(NoneType)]
[2023-07-06 04:20:57,599] torch._dynamo.symbolic_convert: [DEBUG] TRACE INPLACE_ADD None [TupleVariable(), TupleVariable()]
[2023-07-06 04:20:57,599] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST outputs [TupleVariable()]
[2023-07-06 04:20:57,599] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:309
[2023-07-06 04:20:57,599] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs []
[2023-07-06 04:20:57,599] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TupleVariable()]
[2023-07-06 04:20:57,599] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing <resume in forward> (RETURN_VALUE)
[2023-07-06 04:20:57,599] torch._dynamo.symbolic_convert: [DEBUG] RETURN_VALUE triggered compile
[2023-07-06 04:20:57,599] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 309 in <resume in forward>>], graph_break=False)
[2023-07-06 04:20:57,601] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_18 =====
<eval_with_key>.191 class GraphModule(torch.nn.Module):
def forward(self, L_stack0_0_ : torch.Tensor, L_residual_ : torch.Tensor):
l_stack0_0_ = L_stack0_0_
l_residual_ = L_residual_
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:294, code: hidden_states = residual + hidden_states
add = l_residual_ + l_stack0_0_; l_residual_ = l_stack0_0_ = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
to = add.to(torch.float32)
pow_1 = to.pow(2); to = None
mean = pow_1.mean(-1, keepdim = True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add_1 = mean + 1e-06; mean = None
rsqrt = torch.rsqrt(add_1); add_1 = None
mul = add * rsqrt; rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
l__self___post_attention_layernorm_weight = self.L__self___post_attention_layernorm_weight
mul_1 = l__self___post_attention_layernorm_weight * mul; l__self___post_attention_layernorm_weight = mul = None
to_1 = mul_1.to(torch.float16); mul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
l__self___mlp_gate_proj = self.L__self___mlp_gate_proj(to_1)
# File: /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py:150, code: return nn.functional.silu(input)
silu = torch.nn.functional.silu(l__self___mlp_gate_proj); l__self___mlp_gate_proj = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
l__self___mlp_up_proj = self.L__self___mlp_up_proj(to_1); to_1 = None
mul_2 = silu * l__self___mlp_up_proj; silu = l__self___mlp_up_proj = None
l__self___mlp_down_proj = self.L__self___mlp_down_proj(mul_2); mul_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:300, code: hidden_states = residual + hidden_states
add_2 = add + l__self___mlp_down_proj; add = l__self___mlp_down_proj = None
return (add_2,)
[2023-07-06 04:20:57,602] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_18 <eval_with_key>.191 opcode name target args kwargs
------------- ----------------------------------------- -------------------------------------------------------- ------------------------------------------------ -----------------
placeholder l_stack0_0_ L_stack0_0_ () {}
placeholder l_residual_ L_residual_ () {}
call_function add <built-in function add> (l_residual_, l_stack0_0_) {}
call_method to to (add, torch.float32) {}
call_method pow_1 pow (to, 2) {}
call_method mean mean (pow_1, -1) {'keepdim': True}
call_function add_1 <built-in function add> (mean, 1e-06) {}
call_function rsqrt <built-in method rsqrt of type object at 0x7fb940469580> (add_1,) {}
call_function mul <built-in function mul> (add, rsqrt) {}
get_attr l__self___post_attention_layernorm_weight L__self___post_attention_layernorm_weight () {}
call_function mul_1 <built-in function mul> (l__self___post_attention_layernorm_weight, mul) {}
call_method to_1 to (mul_1, torch.float16) {}
call_module l__self___mlp_gate_proj L__self___mlp_gate_proj (to_1,) {}
call_function silu <function silu at 0x7fb945bc5990> (l__self___mlp_gate_proj,) {}
call_module l__self___mlp_up_proj L__self___mlp_up_proj (to_1,) {}
call_function mul_2 <built-in function mul> (silu, l__self___mlp_up_proj) {}
call_module l__self___mlp_down_proj L__self___mlp_down_proj (mul_2,) {}
call_function add_2 <built-in function add> (add, l__self___mlp_down_proj) {}
output output output ((add_2,),) {}
[2023-07-06 04:20:57,603] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
[2023-07-06 04:20:57,651] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 23 =====
<eval_with_key>.195 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[16], arg1_1: f16[64, 16], arg2_1: f16[64, 16], arg3_1: f16[16, 64], arg4_1: f16[1, 1, 16], arg5_1: f16[1, 1, 16]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:294, code: hidden_states = residual + hidden_states
add: f16[1, 1, 16] = torch.ops.aten.add.Tensor(arg5_1, arg4_1); arg5_1 = arg4_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
convert_element_type: f32[1, 1, 16] = torch.ops.prims.convert_element_type.default(add, torch.float32)
pow_1: f32[1, 1, 16] = torch.ops.aten.pow.Tensor_Scalar(convert_element_type, 2); convert_element_type = None
mean: f32[1, 1, 1] = torch.ops.aten.mean.dim(pow_1, [-1], True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add_1: f32[1, 1, 1] = torch.ops.aten.add.Tensor(mean, 1e-06); mean = None
rsqrt: f32[1, 1, 1] = torch.ops.aten.rsqrt.default(add_1); add_1 = None
mul: f32[1, 1, 16] = torch.ops.aten.mul.Tensor(add, rsqrt); rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
mul_1: f32[1, 1, 16] = torch.ops.aten.mul.Tensor(arg0_1, mul); arg0_1 = mul = None
convert_element_type_1: f16[1, 1, 16] = torch.ops.prims.convert_element_type.default(mul_1, torch.float16); mul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
permute: f16[16, 64] = torch.ops.aten.permute.default(arg1_1, [1, 0]); arg1_1 = None
view: f16[1, 16] = torch.ops.aten.view.default(convert_element_type_1, [1, 16])
mm: f16[1, 64] = torch.ops.aten.mm.default(view, permute); view = permute = None
view_1: f16[1, 1, 64] = torch.ops.aten.view.default(mm, [1, 1, 64]); mm = None
# File: /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py:150, code: return nn.functional.silu(input)
convert_element_type_2: f32[1, 1, 64] = torch.ops.prims.convert_element_type.default(view_1, torch.float32); view_1 = None
sigmoid: f32[1, 1, 64] = torch.ops.aten.sigmoid.default(convert_element_type_2)
mul_2: f32[1, 1, 64] = torch.ops.aten.mul.Tensor(convert_element_type_2, sigmoid); convert_element_type_2 = sigmoid = None
convert_element_type_3: f16[1, 1, 64] = torch.ops.prims.convert_element_type.default(mul_2, torch.float16); mul_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
permute_1: f16[16, 64] = torch.ops.aten.permute.default(arg2_1, [1, 0]); arg2_1 = None
view_2: f16[1, 16] = torch.ops.aten.view.default(convert_element_type_1, [1, 16]); convert_element_type_1 = None
mm_1: f16[1, 64] = torch.ops.aten.mm.default(view_2, permute_1); view_2 = permute_1 = None
view_3: f16[1, 1, 64] = torch.ops.aten.view.default(mm_1, [1, 1, 64]); mm_1 = None
mul_3: f16[1, 1, 64] = torch.ops.aten.mul.Tensor(convert_element_type_3, view_3); convert_element_type_3 = view_3 = None
permute_2: f16[64, 16] = torch.ops.aten.permute.default(arg3_1, [1, 0]); arg3_1 = None
view_4: f16[1, 64] = torch.ops.aten.view.default(mul_3, [1, 64]); mul_3 = None
mm_2: f16[1, 16] = torch.ops.aten.mm.default(view_4, permute_2); view_4 = permute_2 = None
view_5: f16[1, 1, 16] = torch.ops.aten.view.default(mm_2, [1, 1, 16]); mm_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:300, code: hidden_states = residual + hidden_states
add_2: f16[1, 1, 16] = torch.ops.aten.add.Tensor(add, view_5); add = view_5 = None
return (add_2,)
[2023-07-06 04:20:57,695] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 10
[2023-07-06 04:20:57,737] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 10
[2023-07-06 04:20:57,754] torch._inductor.scheduler: [DEBUG] remove_buffer('buf0')
[2023-07-06 04:20:57,768] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 1 configs
[2023-07-06 04:20:57,768] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 1, num_warps: 2, num_stages: 1
[2023-07-06 04:20:57,768] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 1 configs
[2023-07-06 04:20:57,768] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 1 configs
[2023-07-06 04:20:57,768] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 64, num_warps: 1, num_stages: 1
[2023-07-06 04:20:57,768] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 16, num_warps: 1, num_stages: 1
[2023-07-06 04:20:57,864] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 1 configs
[2023-07-06 04:20:57,864] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 1, num_warps: 2, num_stages: 1
[2023-07-06 04:20:57,928] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 1 configs
[2023-07-06 04:20:57,928] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 64, num_warps: 1, num_stages: 1
[2023-07-06 04:20:57,934] torch._inductor.triton_heuristics: [DEBUG] CachingAutotuner gets 1 configs
[2023-07-06 04:20:57,934] torch._inductor.triton_heuristics: [DEBUG] XBLOCK: 16, num_warps: 1, num_stages: 1
[2023-07-06 04:20:57,938] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/oo/cooxrverhcp2snw6sgvi3xmyupj2eewevtsplvvj674cqwmsnljq.py
[2023-07-06 04:20:57,939] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 10
[2023-07-06 04:20:57,941] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:57,949] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662763072)
hasattr(L['residual'], '_dynamo_dynamic_indices') == False
___check_type_id(L['___stack0'], 7638272)
len(L['___stack0']) == 3
hasattr(L['___stack0'][0], '_dynamo_dynamic_indices') == False
___check_obj_id(L['___stack0'][1], 7649088)
___check_obj_id(L['___stack0'][2], 7649088)
___check_obj_id(L['output_attentions'], 7698112)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
str(G['torch'].float32) == 'torch.float32'
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks.keys()) == set()
[2023-07-06 04:20:57,949] torch._inductor.cudagraph_trees: [DEBUG] Running warmup of function 7
[2023-07-06 04:20:57,950] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function forward in /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py', 'set env var TORCHDYNAMO_REPORT_GUARD_FAILURES=1 to debug further')
[2023-07-06 04:20:57,950] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:261
[2023-07-06 04:20:57,951] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['hidden_states'] (1, 1, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:57,952] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['attention_mask'] (1, 1003) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:57,953] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['position_ids'] (1, 1) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:57,954] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['past_key_value'] (2, 1, 4, 1003, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None, None]
[2023-07-06 04:20:57,954] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['valid_past_index'] () [] []
[2023-07-06 04:20:57,955] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:281
[2023-07-06 04:20:57,955] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:57,955] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST residual [TensorVariable()]
[2023-07-06 04:20:57,955] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:283
[2023-07-06 04:20:57,955] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,955] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR input_layernorm [NNModuleVariable()]
[2023-07-06 04:20:57,956] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:57,956] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:57,956] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:57,957] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:57,957] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:57,958] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:57,958] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:57,958] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:57,958] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:57,958] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,958] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:57,959] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:57,959] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,959] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,959] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,959] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,959] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,959] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,959] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,960] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,960] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,960] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,960] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,960] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,960] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,960] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:57,960] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:57,960] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,960] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:57,961] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:57,962] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:57,962] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:57,962] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:84
[2023-07-06 04:20:57,962] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:57,962] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [TensorVariable()]
[2023-07-06 04:20:57,965] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST input_dtype [TorchVariable(torch.float16)]
[2023-07-06 04:20:57,965] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85
[2023-07-06 04:20:57,965] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:57,965] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:57,965] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:57,965] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR float32 [GetAttrVariable(TensorVariable(), to), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:57,966] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), ConstantVariable(dtype)]
[2023-07-06 04:20:57,967] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR pow [TensorVariable()]
[2023-07-06 04:20:57,967] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [GetAttrVariable(TensorVariable(), pow)]
[2023-07-06 04:20:57,967] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), pow), ConstantVariable(int)]
[2023-07-06 04:20:57,968] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mean [TensorVariable()]
[2023-07-06 04:20:57,968] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [GetAttrVariable(TensorVariable(), mean)]
[2023-07-06 04:20:57,968] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST True [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int)]
[2023-07-06 04:20:57,968] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('keepdim',) [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool)]
[2023-07-06 04:20:57,969] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 2 [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool), ConstantVariable(tuple)]
[2023-07-06 04:20:57,969] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST variance [TensorVariable()]
[2023-07-06 04:20:57,970] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86
[2023-07-06 04:20:57,970] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:57,970] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:57,970] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR rsqrt [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:57,970] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST variance [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>)]
[2023-07-06 04:20:57,970] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:57,970] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR variance_epsilon [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:57,970] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(float)]
[2023-07-06 04:20:57,971] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:57,972] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:57,973] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:57,973] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88
[2023-07-06 04:20:57,973] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,973] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR weight [NNModuleVariable()]
[2023-07-06 04:20:57,973] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].input_layernorm.weight (16,) [<DimDynamic.STATIC: 2>] [None]
[2023-07-06 04:20:57,974] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:57,974] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:57,975] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:57,975] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_dtype [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:57,975] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:57,976] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:57,976] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:57,976] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:57,976] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:57,976] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:57,976] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:286
[2023-07-06 04:20:57,976] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,976] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR self_attn [NNModuleVariable()]
[2023-07-06 04:20:57,977] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:287
[2023-07-06 04:20:57,977] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:57,977] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:288
[2023-07-06 04:20:57,977] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST attention_mask [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:57,977] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:289
[2023-07-06 04:20:57,977] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:57,977] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:290
[2023-07-06 04:20:57,977] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_key_value [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:57,977] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:291
[2023-07-06 04:20:57,977] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:57,977] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:292
[2023-07-06 04:20:57,977] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(bool)]
[2023-07-06 04:20:57,977] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:286
[2023-07-06 04:20:57,977] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('hidden_states', 'attention_mask', 'position_ids', 'past_key_value', 'output_attentions', 'valid_past_index') [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(bool), TensorVariable()]
[2023-07-06 04:20:57,977] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 6 [NNModuleVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(bool), TensorVariable(), ConstantVariable(tuple)]
[2023-07-06 04:20:57,978] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:57,979] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:57,979] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:57,979] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:57,979] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:57,980] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:57,980] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:57,980] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,980] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:57,980] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable())]
[2023-07-06 04:20:57,980] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,980] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,980] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,981] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,981] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,981] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,981] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,981] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,981] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,981] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,981] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:57,981] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable())]
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:57,982] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:57,983] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:57,983] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaAttention.forward at 0x7fb8d0c46680>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:57,983] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:57,983] torch._dynamo.symbolic_convert: [DEBUG] FAILED INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:57,983] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:57,983] torch._dynamo.symbolic_convert: [DEBUG] break_graph_if_unsupported triggered compile
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2187, in inline_call_
sub_locals, closure_cells = func.bind_args(parent, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 187, in bind_args
[
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 188, in <listcomp>
wrap(val=arg, source=source)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 62, in wrap_bound_arg
return VariableBuilder(tx, source=source, **options)(val)
TypeError: VariableBuilder.__init__() got an unexpected keyword argument 'guards'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 390, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1152, in CALL_FUNCTION_KW
self.call_function(fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 559, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 333, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 595, in inline_user_function_return
result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2132, in inline_call
return cls.inline_call_(parent, func, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2229, in inline_call_
tracer.run()
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 708, in run
and self.step()
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 668, in step
getattr(self, inst.opname)(inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 390, in wrapper
return inner_fn(self, inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1140, in CALL_FUNCTION_EX
self.call_function(fn, argsvars.items, kwargsvars.items)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 559, in call_function
self.push(fn.call_function(self, args, kwargs))
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 333, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 287, in call_function
return super().call_function(tx, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 120, in call_function
return tx.inline_user_function_return(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 595, in inline_user_function_return
result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2132, in inline_call
return cls.inline_call_(parent, func, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2190, in inline_call_
raise ArgsMismatchError(
torch._dynamo.exc.ArgsMismatchError: VariableBuilder.__init__() got an unexpected keyword argument 'guards'.
func = 'forward' /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:190, args = [<class 'trfs_fast.llama.LlamaAttention'>], kwargs = OrderedDict([('hidden_states', TensorVariable()), ('attention_mask', TensorVariable()), ('position_ids', TensorVariable()), ('past_key_value', TensorVariable()), ('output_attentions', ConstantVariable(bool)), ('valid_past_index', TensorVariable())])
[2023-07-06 04:20:57,984] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:57,984] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason="VariableBuilder.__init__() got an unexpected keyword argument 'guards'.\n func = 'forward' /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:190, args = [<class 'trfs_fast.llama.LlamaAttention'>], kwargs = OrderedDict([('hidden_states', TensorVariable()), ('attention_mask', TensorVariable()), ('position_ids', TensorVariable()), ('past_key_value', TensorVariable()), ('output_attentions', ConstantVariable(bool)), ('valid_past_index', TensorVariable())])", user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 286 in forward>, <FrameSummary file /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py, line 1514 in _call_impl>], graph_break=True)
[2023-07-06 04:20:57,985] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['attention_mask']
[2023-07-06 04:20:57,985] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['position_ids']
[2023-07-06 04:20:57,985] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['past_key_value']
[2023-07-06 04:20:57,985] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['valid_past_index']
[2023-07-06 04:20:57,986] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_19 =====
<eval_with_key>.204 class GraphModule(torch.nn.Module):
def forward(self, L_hidden_states_ : torch.Tensor):
l_hidden_states_ = L_hidden_states_
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
to = l_hidden_states_.to(torch.float32)
pow_1 = to.pow(2); to = None
mean = pow_1.mean(-1, keepdim = True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add = mean + 1e-06; mean = None
rsqrt = torch.rsqrt(add); add = None
mul = l_hidden_states_ * rsqrt; l_hidden_states_ = rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
l__self___input_layernorm_weight = self.L__self___input_layernorm_weight
mul_1 = l__self___input_layernorm_weight * mul; l__self___input_layernorm_weight = mul = None
to_1 = mul_1.to(torch.float16); mul_1 = None
return (to_1,)
[2023-07-06 04:20:57,986] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_19 <eval_with_key>.204 opcode name target args kwargs
------------- -------------------------------- -------------------------------------------------------- --------------------------------------- -----------------
placeholder l_hidden_states_ L_hidden_states_ () {}
call_method to to (l_hidden_states_, torch.float32) {}
call_method pow_1 pow (to, 2) {}
call_method mean mean (pow_1, -1) {'keepdim': True}
call_function add <built-in function add> (mean, 1e-06) {}
call_function rsqrt <built-in method rsqrt of type object at 0x7fb940469580> (add,) {}
call_function mul <built-in function mul> (l_hidden_states_, rsqrt) {}
get_attr l__self___input_layernorm_weight L__self___input_layernorm_weight () {}
call_function mul_1 <built-in function mul> (l__self___input_layernorm_weight, mul) {}
call_method to_1 to (mul_1, torch.float16) {}
output output output ((to_1,),) {}
[2023-07-06 04:20:57,987] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
[2023-07-06 04:20:58,005] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 24 =====
<eval_with_key>.208 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[16], arg1_1: f16[1, 1, 16]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
convert_element_type: f32[1, 1, 16] = torch.ops.prims.convert_element_type.default(arg1_1, torch.float32)
pow_1: f32[1, 1, 16] = torch.ops.aten.pow.Tensor_Scalar(convert_element_type, 2); convert_element_type = None
mean: f32[1, 1, 1] = torch.ops.aten.mean.dim(pow_1, [-1], True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add: f32[1, 1, 1] = torch.ops.aten.add.Tensor(mean, 1e-06); mean = None
rsqrt: f32[1, 1, 1] = torch.ops.aten.rsqrt.default(add); add = None
mul: f32[1, 1, 16] = torch.ops.aten.mul.Tensor(arg1_1, rsqrt); arg1_1 = rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
mul_1: f32[1, 1, 16] = torch.ops.aten.mul.Tensor(arg0_1, mul); arg0_1 = mul = None
convert_element_type_1: f16[1, 1, 16] = torch.ops.prims.convert_element_type.default(mul_1, torch.float16); mul_1 = None
return (convert_element_type_1,)
[2023-07-06 04:20:58,007] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 11
[2023-07-06 04:20:58,023] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 11
[2023-07-06 04:20:58,033] torch._inductor.scheduler: [DEBUG] remove_buffer('buf0')
[2023-07-06 04:20:58,034] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/vz/cvzk75oefn2tjj3ekdj3gg5a7rpdjqti5tysf4vatbmcjiobtp5i.py
[2023-07-06 04:20:58,035] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 11
[2023-07-06 04:20:58,037] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:58,045] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662486464)
hasattr(L['position_ids'], '_dynamo_dynamic_indices') == False
___check_type_id(L['hidden_states'], 77765680)
hasattr(L['hidden_states'], '_dynamo_dynamic_indices') == False
hasattr(L['attention_mask'], '_dynamo_dynamic_indices') == False
hasattr(L['past_key_value'], '_dynamo_dynamic_indices') == False
hasattr(L['valid_past_index'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['output_attentions'], 7698112)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
str(G['torch'].float32) == 'torch.float32'
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks.keys()) == set()
[2023-07-06 04:20:58,045] torch._inductor.cudagraph_trees: [DEBUG] Running warmup of function 8
[2023-07-06 04:20:58,045] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function forward in /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py', 'set env var TORCHDYNAMO_REPORT_GUARD_FAILURES=1 to debug further')
[2023-07-06 04:20:58,046] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:190
[2023-07-06 04:20:58,047] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['hidden_states'] (1, 1, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:58,048] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['attention_mask'] (1, 1003) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:58,048] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['position_ids'] (1, 1) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-07-06 04:20:58,049] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['past_key_value'] (2, 1, 4, 1003, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None, None]
[2023-07-06 04:20:58,050] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['valid_past_index'] () [] []
[2023-07-06 04:20:58,051] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:199
[2023-07-06 04:20:58,051] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions []
[2023-07-06 04:20:58,051] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST True [ConstantVariable(bool)]
[2023-07-06 04:20:58,051] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 0 [ConstantVariable(bool), ConstantVariable(bool)]
[2023-07-06 04:20:58,051] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(bool)]
[2023-07-06 04:20:58,051] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:202
[2023-07-06 04:20:58,051] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:58,051] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR size [TensorVariable()]
[2023-07-06 04:20:58,051] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), size)]
[2023-07-06 04:20:58,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE UNPACK_SEQUENCE 3 [SizeVariable()]
[2023-07-06 04:20:58,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST bsz [ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST q_len [ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST _ [ConstantVariable(int)]
[2023-07-06 04:20:58,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:205
[2023-07-06 04:20:58,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR qkv_proj [NNModuleVariable()]
[2023-07-06 04:20:58,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:58,052] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:58,055] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR view [TensorVariable()]
[2023-07-06 04:20:58,055] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST bsz [GetAttrVariable(TensorVariable(), view)]
[2023-07-06 04:20:58,055] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_len [GetAttrVariable(TensorVariable(), view), ConstantVariable(int)]
[2023-07-06 04:20:58,055] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 3 [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,055] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,055] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR num_heads [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), NNModuleVariable()]
[2023-07-06 04:20:58,056] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,056] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR head_dim [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), NNModuleVariable()]
[2023-07-06 04:20:58,056] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 5 [GetAttrVariable(TensorVariable(), view), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,057] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR permute [TensorVariable()]
[2023-07-06 04:20:58,057] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [GetAttrVariable(TensorVariable(), permute)]
[2023-07-06 04:20:58,057] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int)]
[2023-07-06 04:20:58,057] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 3 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,057] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,057] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 4 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,057] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 5 [GetAttrVariable(TensorVariable(), permute), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,058] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST query_key_value_states [TensorVariable()]
[2023-07-06 04:20:58,058] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:207
[2023-07-06 04:20:58,058] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST query_key_value_states []
[2023-07-06 04:20:58,058] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TensorVariable()]
[2023-07-06 04:20:58,058] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,059] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST query_states [TensorVariable()]
[2023-07-06 04:20:58,059] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:208
[2023-07-06 04:20:58,059] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST query_key_value_states []
[2023-07-06 04:20:58,059] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [TensorVariable()]
[2023-07-06 04:20:58,059] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,059] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), ConstantVariable(int), ConstantVariable(NoneType)]
[2023-07-06 04:20:58,059] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), SliceVariable()]
[2023-07-06 04:20:58,061] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST key_value_states [TensorVariable()]
[2023-07-06 04:20:58,061] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:211
[2023-07-06 04:20:58,061] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,061] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR rotary_emb [NNModuleVariable()]
[2023-07-06 04:20:58,061] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states [NNModuleVariable()]
[2023-07-06 04:20:58,061] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:58,061] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_len [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,061] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [NNModuleVariable(), TensorVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,062] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('seq_len',) [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,062] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 2 [NNModuleVariable(), TensorVariable(), TensorVariable(), ConstantVariable(tuple)]
[2023-07-06 04:20:58,062] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:58,064] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:58,064] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:58,064] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:58,064] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:58,064] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:58,065] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:58,065] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,065] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:58,065] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable())]
[2023-07-06 04:20:58,065] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,065] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,065] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,065] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,065] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,065] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,066] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,066] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,066] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,066] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,066] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,066] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,067] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:58,068] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:58,068] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable())]
[2023-07-06 04:20:58,068] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:58,068] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:58,068] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:58,068] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaRotaryEmbedding.forward at 0x7fb8d0c46290>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:58,068] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddf80e0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 107>
[2023-07-06 04:20:58,068] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:122
[2023-07-06 04:20:58,068] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,068] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR cos_cached [NNModuleVariable()]
[2023-07-06 04:20:58,069] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].rotary_emb.cos_cached (1, 1, 2048, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None]
[2023-07-06 04:20:58,069] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:58,070] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:58,070] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [GetAttrVariable(TensorVariable(), to), TensorVariable()]
[2023-07-06 04:20:58,070] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('dtype',) [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:58,070] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 1 [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16), ConstantVariable(tuple)]
[2023-07-06 04:20:58,071] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:123
[2023-07-06 04:20:58,071] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable()]
[2023-07-06 04:20:58,071] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR sin_cached [TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:58,072] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].rotary_emb.sin_cached (1, 1, 2048, 4) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None, None]
[2023-07-06 04:20:58,072] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,072] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TensorVariable(), GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:58,072] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [TensorVariable(), GetAttrVariable(TensorVariable(), to), TensorVariable()]
[2023-07-06 04:20:58,073] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('dtype',) [TensorVariable(), GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:58,073] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 1 [TensorVariable(), GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16), ConstantVariable(tuple)]
[2023-07-06 04:20:58,074] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:121
[2023-07-06 04:20:58,074] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 2 [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,074] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TupleVariable()]
[2023-07-06 04:20:58,074] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddf80e0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 107>
[2023-07-06 04:20:58,074] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TupleVariable()]
[2023-07-06 04:20:58,074] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:58,074] torch._dynamo.symbolic_convert: [DEBUG] TRACE UNPACK_SEQUENCE 2 [TupleVariable()]
[2023-07-06 04:20:58,074] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST cos [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,074] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST sin [TensorVariable()]
[2023-07-06 04:20:58,074] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:212
[2023-07-06 04:20:58,074] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL apply_rotary_pos_emb_opt []
[2023-07-06 04:20:58,075] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST query_states [UserFunctionVariable()]
[2023-07-06 04:20:58,075] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states [UserFunctionVariable(), TensorVariable()]
[2023-07-06 04:20:58,075] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [UserFunctionVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,075] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [UserFunctionVariable(), TensorVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,075] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos [UserFunctionVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,075] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin [UserFunctionVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,075] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [UserFunctionVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,075] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 5 [UserFunctionVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,076] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object apply_rotary_pos_emb_opt at 0x7fb8cddf82f0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 135>
[2023-07-06 04:20:58,076] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:139
[2023-07-06 04:20:58,076] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos []
[2023-07-06 04:20:58,076] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:58,076] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:58,076] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:58,077] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:58,077] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:58,077] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:58,078] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST cos [TensorVariable()]
[2023-07-06 04:20:58,078] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:140
[2023-07-06 04:20:58,078] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin []
[2023-07-06 04:20:58,078] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:58,078] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:58,078] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:58,079] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR squeeze [TensorVariable()]
[2023-07-06 04:20:58,079] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [GetAttrVariable(TensorVariable(), squeeze)]
[2023-07-06 04:20:58,079] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), squeeze), ConstantVariable(int)]
[2023-07-06 04:20:58,080] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST sin [TensorVariable()]
[2023-07-06 04:20:58,080] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:141
[2023-07-06 04:20:58,080] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos []
[2023-07-06 04:20:58,080] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [TensorVariable()]
[2023-07-06 04:20:58,080] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,081] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR unsqueeze [TensorVariable()]
[2023-07-06 04:20:58,082] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), unsqueeze)]
[2023-07-06 04:20:58,082] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), unsqueeze), ConstantVariable(int)]
[2023-07-06 04:20:58,082] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST cos [TensorVariable()]
[2023-07-06 04:20:58,082] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:142
[2023-07-06 04:20:58,082] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin []
[2023-07-06 04:20:58,082] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST position_ids [TensorVariable()]
[2023-07-06 04:20:58,083] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,084] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR unsqueeze [TensorVariable()]
[2023-07-06 04:20:58,084] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [GetAttrVariable(TensorVariable(), unsqueeze)]
[2023-07-06 04:20:58,084] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), unsqueeze), ConstantVariable(int)]
[2023-07-06 04:20:58,085] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST sin [TensorVariable()]
[2023-07-06 04:20:58,085] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143
[2023-07-06 04:20:58,085] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q []
[2023-07-06 04:20:58,085] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos [TensorVariable()]
[2023-07-06 04:20:58,085] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,086] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL rotate_half [TensorVariable()]
[2023-07-06 04:20:58,086] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q [TensorVariable(), UserFunctionVariable()]
[2023-07-06 04:20:58,086] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), UserFunctionVariable(), TensorVariable()]
[2023-07-06 04:20:58,086] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:58,086] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129
[2023-07-06 04:20:58,087] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:58,087] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR roll [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:58,087] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>)]
[2023-07-06 04:20:58,087] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:58,087] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,087] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable()]
[2023-07-06 04:20:58,087] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,088] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,088] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,088] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,088] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('shifts', 'dims') [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,088] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 3 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int), ConstantVariable(tuple)]
[2023-07-06 04:20:58,089] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST x [TensorVariable()]
[2023-07-06 04:20:58,089] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130
[2023-07-06 04:20:58,089] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:58,089] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST Ellipsis [TensorVariable()]
[2023-07-06 04:20:58,089] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable(), ConstantVariable(ellipsis)]
[2023-07-06 04:20:58,089] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType)]
[2023-07-06 04:20:58,089] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), TensorVariable()]
[2023-07-06 04:20:58,090] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable()]
[2023-07-06 04:20:58,090] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,090] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:58,090] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,090] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:58,090] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 2 [TensorVariable(), ConstantVariable(ellipsis), SliceVariable()]
[2023-07-06 04:20:58,090] torch._dynamo.symbolic_convert: [DEBUG] TRACE DUP_TOP_TWO None [TensorVariable(), TupleVariable()]
[2023-07-06 04:20:58,090] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TupleVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:58,091] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:58,091] torch._dynamo.symbolic_convert: [DEBUG] TRACE INPLACE_MULTIPLY None [TensorVariable(), TupleVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,092] torch._dynamo.symbolic_convert: [DEBUG] TRACE ROT_THREE None [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:58,092] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_SUBSCR None [TensorVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:58,092] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:131
[2023-07-06 04:20:58,092] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:58,092] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:58,092] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:58,092] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,092] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,093] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,094] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST q_embed [TensorVariable()]
[2023-07-06 04:20:58,094] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144
[2023-07-06 04:20:58,094] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_states []
[2023-07-06 04:20:58,094] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR copy_ [TensorVariable()]
[2023-07-06 04:20:58,094] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_states [GetAttrVariable(TensorVariable(), copy_)]
[2023-07-06 04:20:58,094] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST cos [GetAttrVariable(TensorVariable(), copy_), TensorVariable()]
[2023-07-06 04:20:58,094] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,095] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL rotate_half [GetAttrVariable(TensorVariable(), copy_), TensorVariable()]
[2023-07-06 04:20:58,095] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_states [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), UserFunctionVariable()]
[2023-07-06 04:20:58,095] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), UserFunctionVariable(), TensorVariable()]
[2023-07-06 04:20:58,096] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:58,096] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129
[2023-07-06 04:20:58,096] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:58,096] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR roll [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:58,096] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>)]
[2023-07-06 04:20:58,096] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:58,096] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,097] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable()]
[2023-07-06 04:20:58,097] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,097] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,097] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,097] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,097] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('shifts', 'dims') [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,097] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 3 [TorchVariable(<built-in method roll of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int), ConstantVariable(int), ConstantVariable(tuple)]
[2023-07-06 04:20:58,098] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST x [TensorVariable()]
[2023-07-06 04:20:58,098] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130
[2023-07-06 04:20:58,098] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:58,098] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST Ellipsis [TensorVariable()]
[2023-07-06 04:20:58,098] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable(), ConstantVariable(ellipsis)]
[2023-07-06 04:20:58,098] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType)]
[2023-07-06 04:20:58,098] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), TensorVariable()]
[2023-07-06 04:20:58,099] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable()]
[2023-07-06 04:20:58,099] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,099] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:58,099] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_FLOOR_DIVIDE None [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int), ConstantVariable(int)]
[2023-07-06 04:20:58,099] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), ConstantVariable(ellipsis), ConstantVariable(NoneType), ConstantVariable(int)]
[2023-07-06 04:20:58,099] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 2 [TensorVariable(), ConstantVariable(ellipsis), SliceVariable()]
[2023-07-06 04:20:58,099] torch._dynamo.symbolic_convert: [DEBUG] TRACE DUP_TOP_TWO None [TensorVariable(), TupleVariable()]
[2023-07-06 04:20:58,099] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), TupleVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:58,100] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:58,100] torch._dynamo.symbolic_convert: [DEBUG] TRACE INPLACE_MULTIPLY None [TensorVariable(), TupleVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,101] torch._dynamo.symbolic_convert: [DEBUG] TRACE ROT_THREE None [TensorVariable(), TupleVariable(), TensorVariable()]
[2023-07-06 04:20:58,101] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_SUBSCR None [TensorVariable(), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:58,101] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:131
[2023-07-06 04:20:58,101] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-07-06 04:20:58,101] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:58,101] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object rotate_half at 0x7fb8cddf8240, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 127>
[2023-07-06 04:20:58,101] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST sin [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,101] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,102] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [GetAttrVariable(TensorVariable(), copy_), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,103] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), copy_), TensorVariable()]
[2023-07-06 04:20:58,104] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_TOP None [TensorVariable()]
[2023-07-06 04:20:58,104] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:145
[2023-07-06 04:20:58,104] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_embed []
[2023-07-06 04:20:58,104] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:58,104] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object apply_rotary_pos_emb_opt at 0x7fb8cddf82f0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 135>
[2023-07-06 04:20:58,104] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST query_states [TensorVariable()]
[2023-07-06 04:20:58,104] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:215
[2023-07-06 04:20:58,104] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index []
[2023-07-06 04:20:58,104] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST past_kv_slice_start [TensorVariable()]
[2023-07-06 04:20:58,104] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:216
[2023-07-06 04:20:58,104] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:58,104] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR eq [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:58,105] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TorchVariable(<built-in method eq of type object at 0x7fb940469580>)]
[2023-07-06 04:20:58,105] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:58,105] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,105] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable()]
[2023-07-06 04:20:58,106] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:58,107] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST q_len [TensorVariable()]
[2023-07-06 04:20:58,107] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,107] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:58,108] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR ne [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:58,108] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>)]
[2023-07-06 04:20:58,108] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:58,108] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,109] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,109] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TensorVariable(), GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:58,110] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,110] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,110] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable(), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,111] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,112] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,112] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST past_kv_slice_end [TensorVariable()]
[2023-07-06 04:20:58,112] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:217
[2023-07-06 04:20:58,112] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:58,113] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR eq [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:58,113] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TorchVariable(<built-in method eq of type object at 0x7fb940469580>)]
[2023-07-06 04:20:58,113] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:58,113] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TorchVariable(<built-in method eq of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,114] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable()]
[2023-07-06 04:20:58,114] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:58,115] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states [TensorVariable()]
[2023-07-06 04:20:58,115] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR shape [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,115] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -2 [TensorVariable(), ShapeVariable()]
[2023-07-06 04:20:58,115] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), ShapeVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,116] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,116] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:58,116] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR ne [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:58,117] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST valid_past_index [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>)]
[2023-07-06 04:20:58,117] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:58,117] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TensorVariable(), TorchVariable(<built-in method ne of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,118] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR int [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,118] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TensorVariable(), GetAttrVariable(TensorVariable(), int)]
[2023-07-06 04:20:58,119] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_kv_slice_end [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,119] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,120] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,120] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST past_state_slice_end [TensorVariable()]
[2023-07-06 04:20:58,120] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:218
[2023-07-06 04:20:58,120] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST key_value_states []
[2023-07-06 04:20:58,121] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_key_value [TensorVariable()]
[2023-07-06 04:20:58,121] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST Ellipsis [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,121] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_kv_slice_start [TensorVariable(), TensorVariable(), ConstantVariable(ellipsis)]
[2023-07-06 04:20:58,121] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST past_kv_slice_end [TensorVariable(), TensorVariable(), ConstantVariable(ellipsis), TensorVariable()]
[2023-07-06 04:20:58,121] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), TensorVariable(), ConstantVariable(ellipsis), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,121] torch._dynamo.symbolic_convert: [DEBUG] step triggered compile
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 668, in step
getattr(self, inst.opname)(inst)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1251, in BUILD_SLICE
SliceVariable(
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/base.py", line 27, in __call__
obj = type.__call__(cls, *args, **kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/lists.py", line 616, in __init__
unimplemented("Dynamic slicing on data-dependent value is not supported")
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 140, in unimplemented
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: Dynamic slicing on data-dependent value is not supported
[2023-07-06 04:20:58,121] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:58,121] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='step_unsupported', user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 218 in forward>], graph_break=True)
[2023-07-06 04:20:58,123] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['attention_mask']
[2023-07-06 04:20:58,123] torch._dynamo.output_graph: [DEBUG] REMOVE UNUSED GRAPHARG L['past_key_value']
[2023-07-06 04:20:58,124] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_21 =====
<eval_with_key>.210 class GraphModule(torch.nn.Module):
def forward(self, L_hidden_states_ : torch.Tensor, L_position_ids_ : torch.Tensor, L_valid_past_index_ : torch.Tensor):
l_hidden_states_ = L_hidden_states_
l_position_ids_ = L_position_ids_
l_valid_past_index_ = L_valid_past_index_
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:205, code: query_key_value_states = self.qkv_proj(hidden_states).view(bsz, q_len, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4)
l__self___qkv_proj = self.L__self___qkv_proj(l_hidden_states_); l_hidden_states_ = None
view = l__self___qkv_proj.view(1, 1, 3, 4, 4); l__self___qkv_proj = None
permute = view.permute(2, 0, 3, 1, 4); view = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:207, code: query_states = query_key_value_states[0]
getitem = permute[0]
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:208, code: key_value_states = query_key_value_states[1:]
getitem_1 = permute[slice(1, None, None)]; permute = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:211, code: cos, sin = self.rotary_emb(key_value_states, seq_len=valid_past_index + q_len)
add = l_valid_past_index_ + 1
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:122, code: self.cos_cached.to(dtype=x.dtype),
l__self___rotary_emb_cos_cached = self.L__self___rotary_emb_cos_cached
to = l__self___rotary_emb_cos_cached.to(dtype = torch.float16); l__self___rotary_emb_cos_cached = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:123, code: self.sin_cached.to(dtype=x.dtype),
l__self___rotary_emb_sin_cached = self.L__self___rotary_emb_sin_cached
to_1 = l__self___rotary_emb_sin_cached.to(dtype = torch.float16); l__self___rotary_emb_sin_cached = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:212, code: query_states = apply_rotary_pos_emb_opt(query_states, key_value_states[0], cos, sin, position_ids)
getitem_2 = getitem_1[0]
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:139, code: cos = cos.squeeze(1).squeeze(0) # [seq_len, dim]
squeeze = to.squeeze(1); to = None
squeeze_1 = squeeze.squeeze(0); squeeze = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:140, code: sin = sin.squeeze(1).squeeze(0) # [seq_len, dim]
squeeze_2 = to_1.squeeze(1); to_1 = None
squeeze_3 = squeeze_2.squeeze(0); squeeze_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:141, code: cos = cos[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
getitem_3 = squeeze_1[l_position_ids_]; squeeze_1 = None
unsqueeze = getitem_3.unsqueeze(1); getitem_3 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:142, code: sin = sin[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
getitem_4 = squeeze_3[l_position_ids_]; squeeze_3 = l_position_ids_ = None
unsqueeze_1 = getitem_4.unsqueeze(1); getitem_4 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul = getitem * unsqueeze
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll = torch.roll(getitem, shifts = 2, dims = -1); getitem = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
getitem_5 = roll[(Ellipsis, slice(None, 2, None))]
getitem_5 *= -1; imul = getitem_5; getitem_5 = None
roll[(Ellipsis, slice(None, 2, None))] = imul; setitem = roll; imul = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul_1 = roll * unsqueeze_1; roll = None
add_1 = mul + mul_1; mul = mul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_2 = getitem_2 * unsqueeze; unsqueeze = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll_1 = torch.roll(getitem_2, shifts = 2, dims = -1)
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
getitem_6 = roll_1[(Ellipsis, slice(None, 2, None))]
getitem_6 *= -1; imul_1 = getitem_6; getitem_6 = None
roll_1[(Ellipsis, slice(None, 2, None))] = imul_1; setitem_1 = roll_1; imul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_3 = roll_1 * unsqueeze_1; roll_1 = unsqueeze_1 = None
add_2 = mul_2 + mul_3; mul_2 = mul_3 = None
copy_ = getitem_2.copy_(add_2); getitem_2 = add_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:216, code: past_kv_slice_end = torch.eq(valid_past_index, 0).int() * q_len + torch.ne(valid_past_index, 0).int() * (valid_past_index + 1)
eq = torch.eq(l_valid_past_index_, 0)
int_1 = eq.int(); eq = None
mul_4 = int_1 * 1; int_1 = None
ne = torch.ne(l_valid_past_index_, 0)
int_2 = ne.int(); ne = None
add_3 = l_valid_past_index_ + 1
mul_5 = int_2 * add_3; int_2 = add_3 = None
add_4 = mul_4 + mul_5; mul_4 = mul_5 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:217, code: past_state_slice_end = torch.eq(valid_past_index, 0).int() * key_value_states.shape[-2] + torch.ne(valid_past_index, 0).int() * (past_kv_slice_end)
eq_1 = torch.eq(l_valid_past_index_, 0)
int_3 = eq_1.int(); eq_1 = None
mul_6 = int_3 * 1; int_3 = None
ne_1 = torch.ne(l_valid_past_index_, 0); l_valid_past_index_ = None
int_4 = ne_1.int(); ne_1 = None
mul_7 = int_4 * add_4; int_4 = None
add_5 = mul_6 + mul_7; mul_6 = mul_7 = None
return (add_1, getitem_1, add_4, add_5)
[2023-07-06 04:20:58,125] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_21 <eval_with_key>.210 opcode name target args kwargs
------------- ------------------------------- ------------------------------------------------------- -------------------------------------------------- -------------------------
placeholder l_hidden_states_ L_hidden_states_ () {}
placeholder l_position_ids_ L_position_ids_ () {}
placeholder l_valid_past_index_ L_valid_past_index_ () {}
call_module l__self___qkv_proj L__self___qkv_proj (l_hidden_states_,) {}
call_method view view (l__self___qkv_proj, 1, 1, 3, 4, 4) {}
call_method permute permute (view, 2, 0, 3, 1, 4) {}
call_function getitem <built-in function getitem> (permute, 0) {}
call_function getitem_1 <built-in function getitem> (permute, slice(1, None, None)) {}
call_function add <built-in function add> (l_valid_past_index_, 1) {}
get_attr l__self___rotary_emb_cos_cached L__self___rotary_emb_cos_cached () {}
call_method to to (l__self___rotary_emb_cos_cached,) {'dtype': torch.float16}
get_attr l__self___rotary_emb_sin_cached L__self___rotary_emb_sin_cached () {}
call_method to_1 to (l__self___rotary_emb_sin_cached,) {'dtype': torch.float16}
call_function getitem_2 <built-in function getitem> (getitem_1, 0) {}
call_method squeeze squeeze (to, 1) {}
call_method squeeze_1 squeeze (squeeze, 0) {}
call_method squeeze_2 squeeze (to_1, 1) {}
call_method squeeze_3 squeeze (squeeze_2, 0) {}
call_function getitem_3 <built-in function getitem> (squeeze_1, l_position_ids_) {}
call_method unsqueeze unsqueeze (getitem_3, 1) {}
call_function getitem_4 <built-in function getitem> (squeeze_3, l_position_ids_) {}
call_method unsqueeze_1 unsqueeze (getitem_4, 1) {}
call_function mul <built-in function mul> (getitem, unsqueeze) {}
call_function roll <built-in method roll of type object at 0x7fb940469580> (getitem,) {'shifts': 2, 'dims': -1}
call_function getitem_5 <built-in function getitem> (roll, (Ellipsis, slice(None, 2, None))) {}
call_function imul <built-in function imul> (getitem_5, -1) {}
call_function setitem <built-in function setitem> (roll, (Ellipsis, slice(None, 2, None)), imul) {}
call_function mul_1 <built-in function mul> (roll, unsqueeze_1) {}
call_function add_1 <built-in function add> (mul, mul_1) {}
call_function mul_2 <built-in function mul> (getitem_2, unsqueeze) {}
call_function roll_1 <built-in method roll of type object at 0x7fb940469580> (getitem_2,) {'shifts': 2, 'dims': -1}
call_function getitem_6 <built-in function getitem> (roll_1, (Ellipsis, slice(None, 2, None))) {}
call_function imul_1 <built-in function imul> (getitem_6, -1) {}
call_function setitem_1 <built-in function setitem> (roll_1, (Ellipsis, slice(None, 2, None)), imul_1) {}
call_function mul_3 <built-in function mul> (roll_1, unsqueeze_1) {}
call_function add_2 <built-in function add> (mul_2, mul_3) {}
call_method copy_ copy_ (getitem_2, add_2) {}
call_function eq <built-in method eq of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_1 int (eq,) {}
call_function mul_4 <built-in function mul> (int_1, 1) {}
call_function ne <built-in method ne of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_2 int (ne,) {}
call_function add_3 <built-in function add> (l_valid_past_index_, 1) {}
call_function mul_5 <built-in function mul> (int_2, add_3) {}
call_function add_4 <built-in function add> (mul_4, mul_5) {}
call_function eq_1 <built-in method eq of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_3 int (eq_1,) {}
call_function mul_6 <built-in function mul> (int_3, 1) {}
call_function ne_1 <built-in method ne of type object at 0x7fb940469580> (l_valid_past_index_, 0) {}
call_method int_4 int (ne_1,) {}
call_function mul_7 <built-in function mul> (int_4, add_4) {}
call_function add_5 <built-in function add> (mul_6, mul_7) {}
output output output ((add_1, getitem_1, add_4, add_5),) {}
[2023-07-06 04:20:58,127] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
[2023-07-06 04:20:58,248] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 25 =====
<eval_with_key>.214 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[48, 16], arg1_1: f32[1, 1, 2048, 4], arg2_1: f32[1, 1, 2048, 4], arg3_1: f16[1, 1, 16], arg4_1: i64[1, 1], arg5_1: i64[]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:205, code: query_key_value_states = self.qkv_proj(hidden_states).view(bsz, q_len, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4)
permute: f16[16, 48] = torch.ops.aten.permute.default(arg0_1, [1, 0]); arg0_1 = None
view: f16[1, 16] = torch.ops.aten.view.default(arg3_1, [1, 16]); arg3_1 = None
mm: f16[1, 48] = torch.ops.aten.mm.default(view, permute); view = permute = None
view_1: f16[1, 1, 48] = torch.ops.aten.view.default(mm, [1, 1, 48])
view_2: f16[1, 1, 3, 4, 4] = torch.ops.aten.view.default(view_1, [1, 1, 3, 4, 4]); view_1 = None
permute_1: f16[3, 1, 4, 1, 4] = torch.ops.aten.permute.default(view_2, [2, 0, 3, 1, 4]); view_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:207, code: query_states = query_key_value_states[0]
select: f16[1, 4, 1, 4] = torch.ops.aten.select.int(permute_1, 0, 0)
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:208, code: key_value_states = query_key_value_states[1:]
slice_1: f16[2, 1, 4, 1, 4] = torch.ops.aten.slice.Tensor(permute_1, 0, 1, 9223372036854775807); permute_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:122, code: self.cos_cached.to(dtype=x.dtype),
convert_element_type: f16[1, 1, 2048, 4] = torch.ops.prims.convert_element_type.default(arg1_1, torch.float16); arg1_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:123, code: self.sin_cached.to(dtype=x.dtype),
convert_element_type_1: f16[1, 1, 2048, 4] = torch.ops.prims.convert_element_type.default(arg2_1, torch.float16); arg2_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:212, code: query_states = apply_rotary_pos_emb_opt(query_states, key_value_states[0], cos, sin, position_ids)
select_1: f16[1, 4, 1, 4] = torch.ops.aten.select.int(slice_1, 0, 0); slice_1 = None
# No stacktrace found for following nodes
squeeze: f16[1, 2048, 4] = torch.ops.aten.squeeze.dim(convert_element_type, 1); convert_element_type = None
squeeze_1: f16[2048, 4] = torch.ops.aten.squeeze.dim(squeeze, 0); squeeze = None
squeeze_2: f16[1, 2048, 4] = torch.ops.aten.squeeze.dim(convert_element_type_1, 1); convert_element_type_1 = None
squeeze_3: f16[2048, 4] = torch.ops.aten.squeeze.dim(squeeze_2, 0); squeeze_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:141, code: cos = cos[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
index: f16[1, 1, 4] = torch.ops.aten.index.Tensor(squeeze_1, [arg4_1]); squeeze_1 = None
unsqueeze: f16[1, 1, 1, 4] = torch.ops.aten.unsqueeze.default(index, 1); index = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:142, code: sin = sin[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim]
index_1: f16[1, 1, 4] = torch.ops.aten.index.Tensor(squeeze_3, [arg4_1]); squeeze_3 = arg4_1 = None
unsqueeze_1: f16[1, 1, 1, 4] = torch.ops.aten.unsqueeze.default(index_1, 1); index_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul: f16[1, 4, 1, 4] = torch.ops.aten.mul.Tensor(select, unsqueeze)
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll: f16[1, 4, 1, 4] = torch.ops.aten.roll.default(select, [2], [-1]); select = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
slice_2: f16[1, 4, 1, 2] = torch.ops.aten.slice.Tensor(roll, 3, 0, 2)
mul_1: f16[1, 4, 1, 2] = torch.ops.aten.mul.Tensor(slice_2, -1); slice_2 = None
slice_scatter: f16[1, 4, 1, 4] = torch.ops.aten.slice_scatter.default(roll, mul_1, 3, 0, 2); roll = mul_1 = None
slice_3: f16[1, 4, 1, 2] = torch.ops.aten.slice.Tensor(slice_scatter, 3, 0, 2)
slice_scatter_1: f16[1, 4, 1, 4] = torch.ops.aten.slice_scatter.default(slice_scatter, slice_3, 3, 0, 2); slice_scatter = slice_3 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:143, code: q_embed = (q * cos) + (rotate_half(q) * sin)
mul_2: f16[1, 4, 1, 4] = torch.ops.aten.mul.Tensor(slice_scatter_1, unsqueeze_1); slice_scatter_1 = None
add_1: f16[1, 4, 1, 4] = torch.ops.aten.add.Tensor(mul, mul_2); mul = mul_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_3: f16[1, 4, 1, 4] = torch.ops.aten.mul.Tensor(select_1, unsqueeze); unsqueeze = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:129, code: x = torch.roll(x, shifts=x.shape[-1] // 2, dims=-1)
roll_1: f16[1, 4, 1, 4] = torch.ops.aten.roll.default(select_1, [2], [-1]); select_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:130, code: x[..., : x.shape[-1] // 2] *= -1
slice_7: f16[1, 4, 1, 2] = torch.ops.aten.slice.Tensor(roll_1, 3, 0, 2)
mul_4: f16[1, 4, 1, 2] = torch.ops.aten.mul.Tensor(slice_7, -1); slice_7 = None
slice_scatter_2: f16[1, 4, 1, 4] = torch.ops.aten.slice_scatter.default(roll_1, mul_4, 3, 0, 2); roll_1 = mul_4 = None
slice_8: f16[1, 4, 1, 2] = torch.ops.aten.slice.Tensor(slice_scatter_2, 3, 0, 2)
slice_scatter_3: f16[1, 4, 1, 4] = torch.ops.aten.slice_scatter.default(slice_scatter_2, slice_8, 3, 0, 2); slice_scatter_2 = slice_8 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:144, code: key_states.copy_((key_states * cos) + (rotate_half(key_states) * sin))
mul_5: f16[1, 4, 1, 4] = torch.ops.aten.mul.Tensor(slice_scatter_3, unsqueeze_1); slice_scatter_3 = unsqueeze_1 = None
add_2: f16[1, 4, 1, 4] = torch.ops.aten.add.Tensor(mul_3, mul_5); mul_3 = mul_5 = None
view_3: f16[1, 1, 48] = torch.ops.aten.view.default(mm, [1, 1, 48]); mm = None
view_4: f16[1, 1, 3, 4, 4] = torch.ops.aten.view.default(view_3, [1, 1, 3, 4, 4]); view_3 = None
permute_2: f16[3, 1, 4, 1, 4] = torch.ops.aten.permute.default(view_4, [2, 0, 3, 1, 4]); view_4 = None
slice_12: f16[2, 1, 4, 1, 4] = torch.ops.aten.slice.Tensor(permute_2, 0, 1, 9223372036854775807)
select_scatter: f16[2, 1, 4, 1, 4] = torch.ops.aten.select_scatter.default(slice_12, add_2, 0, 0); slice_12 = add_2 = None
slice_scatter_4: f16[3, 1, 4, 1, 4] = torch.ops.aten.slice_scatter.default(permute_2, select_scatter, 0, 1, 9223372036854775807); permute_2 = select_scatter = None
permute_3: f16[1, 1, 3, 4, 4] = torch.ops.aten.permute.default(slice_scatter_4, [1, 3, 0, 2, 4]); slice_scatter_4 = None
view_5: f16[1, 1, 48] = torch.ops.aten.view.default(permute_3, [1, 1, 48]); permute_3 = None
view_6: f16[1, 48] = torch.ops.aten.view.default(view_5, [1, 48]); view_5 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:216, code: past_kv_slice_end = torch.eq(valid_past_index, 0).int() * q_len + torch.ne(valid_past_index, 0).int() * (valid_past_index + 1)
eq: b8[] = torch.ops.aten.eq.Scalar(arg5_1, 0)
convert_element_type_2: i32[] = torch.ops.prims.convert_element_type.default(eq, torch.int32); eq = None
mul_6: i32[] = torch.ops.aten.mul.Tensor(convert_element_type_2, 1); convert_element_type_2 = None
ne: b8[] = torch.ops.aten.ne.Scalar(arg5_1, 0)
convert_element_type_3: i32[] = torch.ops.prims.convert_element_type.default(ne, torch.int32); ne = None
add_3: i64[] = torch.ops.aten.add.Tensor(arg5_1, 1)
mul_7: i64[] = torch.ops.aten.mul.Tensor(convert_element_type_3, add_3); convert_element_type_3 = add_3 = None
add_4: i64[] = torch.ops.aten.add.Tensor(mul_6, mul_7); mul_6 = mul_7 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:217, code: past_state_slice_end = torch.eq(valid_past_index, 0).int() * key_value_states.shape[-2] + torch.ne(valid_past_index, 0).int() * (past_kv_slice_end)
eq_1: b8[] = torch.ops.aten.eq.Scalar(arg5_1, 0)
convert_element_type_4: i32[] = torch.ops.prims.convert_element_type.default(eq_1, torch.int32); eq_1 = None
mul_8: i32[] = torch.ops.aten.mul.Tensor(convert_element_type_4, 1); convert_element_type_4 = None
ne_1: b8[] = torch.ops.aten.ne.Scalar(arg5_1, 0); arg5_1 = None
convert_element_type_5: i32[] = torch.ops.prims.convert_element_type.default(ne_1, torch.int32); ne_1 = None
mul_9: i64[] = torch.ops.aten.mul.Tensor(convert_element_type_5, add_4); convert_element_type_5 = None
add_5: i64[] = torch.ops.aten.add.Tensor(mul_8, mul_9); mul_8 = mul_9 = None
# No stacktrace found for following nodes
view_9: f16[1, 1, 48] = torch.ops.aten.view.default(view_6, [1, 1, 48]); view_6 = None
view_10: f16[1, 1, 3, 4, 4] = torch.ops.aten.view.default(view_9, [1, 1, 3, 4, 4]); view_9 = None
permute_5: f16[3, 1, 4, 1, 4] = torch.ops.aten.permute.default(view_10, [2, 0, 3, 1, 4]); view_10 = None
slice_14: f16[2, 1, 4, 1, 4] = torch.ops.aten.slice.Tensor(permute_5, 0, 1, 9223372036854775807); permute_5 = None
return (add_1, slice_14, add_4, add_5)
[2023-07-06 04:20:58,284] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 12
[2023-07-06 04:20:58,391] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 12
[2023-07-06 04:20:58,487] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/hj/chjdplcvb6oibk7aigc4ou76dlz6yn4gxfu3dxactvqvnkkqmftz.py
[2023-07-06 04:20:58,487] torch._inductor.compile_fx: [DEBUG] disabled cudagraphs because ['non-cuda device in graph']
skipping cudagraphs due to multiple devices
[2023-07-06 04:20:58,487] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 12
[2023-07-06 04:20:58,490] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:58,499] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662491744)
hasattr(L['position_ids'], '_dynamo_dynamic_indices') == False
hasattr(L['hidden_states'], '_dynamo_dynamic_indices') == False
hasattr(L['attention_mask'], '_dynamo_dynamic_indices') == False
hasattr(L['past_key_value'], '_dynamo_dynamic_indices') == False
hasattr(L['valid_past_index'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['output_attentions'], 7698112)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks.keys()) == set()
[2023-07-06 04:20:58,500] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function <resume in forward> in /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py', 'set env var TORCHDYNAMO_REPORT_GUARD_FAILURES=1 to debug further')
[2023-07-06 04:20:58,500] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <resume in forward> /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:286
[2023-07-06 04:20:58,500] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['___stack0'][0] (1, 1, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:58,501] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['residual'] (1, 1, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:58,503] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST ___stack0 []
[2023-07-06 04:20:58,503] torch._dynamo.symbolic_convert: [DEBUG] TRACE JUMP_ABSOLUTE 38 [TupleVariable()]
[2023-07-06 04:20:58,503] torch._dynamo.symbolic_convert: [DEBUG] TRACE UNPACK_SEQUENCE 3 [TupleVariable()]
[2023-07-06 04:20:58,503] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [ConstantVariable(NoneType), ConstantVariable(NoneType), TensorVariable()]
[2023-07-06 04:20:58,503] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST self_attn_weights [ConstantVariable(NoneType), ConstantVariable(NoneType)]
[2023-07-06 04:20:58,503] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST present_key_value [ConstantVariable(NoneType)]
[2023-07-06 04:20:58,503] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:294
[2023-07-06 04:20:58,503] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST residual []
[2023-07-06 04:20:58,503] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:58,503] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,504] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:58,504] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:297
[2023-07-06 04:20:58,504] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:58,504] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST residual [TensorVariable()]
[2023-07-06 04:20:58,504] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:298
[2023-07-06 04:20:58,504] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,504] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR post_attention_layernorm [NNModuleVariable()]
[2023-07-06 04:20:58,504] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:58,505] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:58,505] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:58,506] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:58,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:58,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:58,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:58,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:58,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:58,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,507] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:58,508] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:58,508] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,508] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,508] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,508] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,508] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,508] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,509] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,509] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,509] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,509] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,509] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,509] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,509] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,509] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:58,509] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:58,509] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable())]
[2023-07-06 04:20:58,510] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:58,511] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:58,511] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:58,511] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaRMSNorm.forward at 0x7fb8d0c460e0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:58,511] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:58,511] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:84
[2023-07-06 04:20:58,511] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:58,511] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [TensorVariable()]
[2023-07-06 04:20:58,511] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST input_dtype [TorchVariable(torch.float16)]
[2023-07-06 04:20:58,511] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85
[2023-07-06 04:20:58,511] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:58,512] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:58,512] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:58,512] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR float32 [GetAttrVariable(TensorVariable(), to), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:58,512] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), ConstantVariable(dtype)]
[2023-07-06 04:20:58,513] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR pow [TensorVariable()]
[2023-07-06 04:20:58,514] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [GetAttrVariable(TensorVariable(), pow)]
[2023-07-06 04:20:58,514] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), pow), ConstantVariable(int)]
[2023-07-06 04:20:58,515] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mean [TensorVariable()]
[2023-07-06 04:20:58,515] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [GetAttrVariable(TensorVariable(), mean)]
[2023-07-06 04:20:58,515] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST True [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int)]
[2023-07-06 04:20:58,515] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('keepdim',) [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool)]
[2023-07-06 04:20:58,515] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 2 [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool), ConstantVariable(tuple)]
[2023-07-06 04:20:58,516] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST variance [TensorVariable()]
[2023-07-06 04:20:58,516] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86
[2023-07-06 04:20:58,516] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:58,516] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:58,516] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR rsqrt [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:58,516] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST variance [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>)]
[2023-07-06 04:20:58,516] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:58,516] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR variance_epsilon [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:58,517] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(float)]
[2023-07-06 04:20:58,517] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:58,518] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,519] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:58,519] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88
[2023-07-06 04:20:58,519] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,519] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR weight [NNModuleVariable()]
[2023-07-06 04:20:58,519] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].post_attention_layernorm.weight (16,) [<DimDynamic.STATIC: 2>] [None]
[2023-07-06 04:20:58,520] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:58,520] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,521] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:58,521] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_dtype [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:58,521] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:58,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:58,522] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddd7ec0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 83>
[2023-07-06 04:20:58,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:58,522] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:58,522] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:58,523] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:299
[2023-07-06 04:20:58,523] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,523] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mlp [NNModuleVariable()]
[2023-07-06 04:20:58,523] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:58,523] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:58,523] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:58,525] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:58,525] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:58,525] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:58,525] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:58,525] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:58,525] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:58,525] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,526] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:58,526] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable())]
[2023-07-06 04:20:58,526] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,526] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,526] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,526] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,526] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,526] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,527] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,527] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,527] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,527] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,527] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,527] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,527] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,527] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:58,527] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:58,527] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,527] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable())]
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:58,528] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function LlamaMLP.forward at 0x7fb8d0c464d0>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:58,529] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8cddf85b0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 161>
[2023-07-06 04:20:58,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162
[2023-07-06 04:20:58,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR down_proj [NNModuleVariable()]
[2023-07-06 04:20:58,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [NNModuleVariable()]
[2023-07-06 04:20:58,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR act_fn [NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:58,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:58,529] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR gate_proj [NNModuleVariable(), NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:58,530] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [NNModuleVariable(), NNModuleVariable(), NNModuleVariable()]
[2023-07-06 04:20:58,530] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), NNModuleVariable(), NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:58,533] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:58,533] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:58,534] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1508
[2023-07-06 04:20:58,534] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
[2023-07-06 04:20:58,534] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _C [TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:58,535] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _get_tracing_state [TorchVariable(<module 'torch._C' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so'>)]
[2023-07-06 04:20:58,535] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method _get_tracing_state of PyCapsule object at 0x7fb974963060>)]
[2023-07-06 04:20:58,535] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(NoneType)]
[2023-07-06 04:20:58,535] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,535] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2023-07-06 04:20:58,535] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST forward_call [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable())]
[2023-07-06 04:20:58,535] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,535] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,535] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,536] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,536] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,536] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _backward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,536] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,536] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,536] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _forward_pre_hooks [NNModuleVariable()]
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_pre_hooks []
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1512
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_backward_hooks []
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_hooks []
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1513
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL _global_forward_pre_hooks []
[2023-07-06 04:20:58,537] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1511
[2023-07-06 04:20:58,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 76 [ConstDictVariable()]
[2023-07-06 04:20:58,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1514
[2023-07-06 04:20:58,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST forward_call []
[2023-07-06 04:20:58,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST args [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable())]
[2023-07-06 04:20:58,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable()]
[2023-07-06 04:20:58,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST kwargs [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:58,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE DICT_MERGE 1 [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable(), ConstDictVariable(), ConstDictVariable()]
[2023-07-06 04:20:58,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_EX 1 [UserMethodVariable(<function SiLUActivation.forward at 0x7fb8ccf64280>, NNModuleVariable()), TupleVariable(), ConstDictVariable()]
[2023-07-06 04:20:58,538] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fb8ccf62760, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py", line 149>
[2023-07-06 04:20:58,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py:150
[2023-07-06 04:20:58,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL nn []
[2023-07-06 04:20:58,538] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR functional [TorchVariable(<module 'torch.nn' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/__init__.py'>)]
[2023-07-06 04:20:58,539] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR silu [TorchVariable(<module 'torch.nn.functional' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/functional.py'>)]
[2023-07-06 04:20:58,539] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input [TorchVariable(<function silu at 0x7fb945bc5990>)]
[2023-07-06 04:20:58,539] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TorchVariable(<function silu at 0x7fb945bc5990>), TensorVariable()]
[2023-07-06 04:20:58,543] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:58,543] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8ccf62760, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py", line 149>
[2023-07-06 04:20:58,543] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:58,543] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:58,543] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:58,543] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR up_proj [NNModuleVariable(), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:58,543] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [NNModuleVariable(), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:58,543] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable(), NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:58,546] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [NNModuleVariable(), TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,547] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:58,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:58,549] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7fb8cddf85b0, file "/home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py", line 161>
[2023-07-06 04:20:58,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:58,549] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object _call_impl at 0x7fb97409ec30, file "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1507>
[2023-07-06 04:20:58,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:58,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:300
[2023-07-06 04:20:58,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST residual []
[2023-07-06 04:20:58,549] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:58,550] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,550] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:58,550] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:302
[2023-07-06 04:20:58,550] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 1 [TensorVariable()]
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST outputs [TupleVariable()]
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:304
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST output_attentions []
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 106 [ConstantVariable(bool)]
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:307
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs []
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST present_key_value [TupleVariable()]
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 1 [TupleVariable(), ConstantVariable(NoneType)]
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE INPLACE_ADD None [TupleVariable(), TupleVariable()]
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST outputs [TupleVariable()]
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:309
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs []
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TupleVariable()]
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing <resume in forward> (RETURN_VALUE)
[2023-07-06 04:20:58,551] torch._dynamo.symbolic_convert: [DEBUG] RETURN_VALUE triggered compile
[2023-07-06 04:20:58,551] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 309 in <resume in forward>>], graph_break=False)
[2023-07-06 04:20:58,553] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_22 =====
<eval_with_key>.219 class GraphModule(torch.nn.Module):
def forward(self, L_stack0_0_ : torch.Tensor, L_residual_ : torch.Tensor):
l_stack0_0_ = L_stack0_0_
l_residual_ = L_residual_
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:294, code: hidden_states = residual + hidden_states
add = l_residual_ + l_stack0_0_; l_residual_ = l_stack0_0_ = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
to = add.to(torch.float32)
pow_1 = to.pow(2); to = None
mean = pow_1.mean(-1, keepdim = True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add_1 = mean + 1e-06; mean = None
rsqrt = torch.rsqrt(add_1); add_1 = None
mul = add * rsqrt; rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
l__self___post_attention_layernorm_weight = self.L__self___post_attention_layernorm_weight
mul_1 = l__self___post_attention_layernorm_weight * mul; l__self___post_attention_layernorm_weight = mul = None
to_1 = mul_1.to(torch.float16); mul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
l__self___mlp_gate_proj = self.L__self___mlp_gate_proj(to_1)
# File: /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py:150, code: return nn.functional.silu(input)
silu = torch.nn.functional.silu(l__self___mlp_gate_proj); l__self___mlp_gate_proj = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
l__self___mlp_up_proj = self.L__self___mlp_up_proj(to_1); to_1 = None
mul_2 = silu * l__self___mlp_up_proj; silu = l__self___mlp_up_proj = None
l__self___mlp_down_proj = self.L__self___mlp_down_proj(mul_2); mul_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:300, code: hidden_states = residual + hidden_states
add_2 = add + l__self___mlp_down_proj; add = l__self___mlp_down_proj = None
return (add_2,)
[2023-07-06 04:20:58,554] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_22 <eval_with_key>.219 opcode name target args kwargs
------------- ----------------------------------------- -------------------------------------------------------- ------------------------------------------------ -----------------
placeholder l_stack0_0_ L_stack0_0_ () {}
placeholder l_residual_ L_residual_ () {}
call_function add <built-in function add> (l_residual_, l_stack0_0_) {}
call_method to to (add, torch.float32) {}
call_method pow_1 pow (to, 2) {}
call_method mean mean (pow_1, -1) {'keepdim': True}
call_function add_1 <built-in function add> (mean, 1e-06) {}
call_function rsqrt <built-in method rsqrt of type object at 0x7fb940469580> (add_1,) {}
call_function mul <built-in function mul> (add, rsqrt) {}
get_attr l__self___post_attention_layernorm_weight L__self___post_attention_layernorm_weight () {}
call_function mul_1 <built-in function mul> (l__self___post_attention_layernorm_weight, mul) {}
call_method to_1 to (mul_1, torch.float16) {}
call_module l__self___mlp_gate_proj L__self___mlp_gate_proj (to_1,) {}
call_function silu <function silu at 0x7fb945bc5990> (l__self___mlp_gate_proj,) {}
call_module l__self___mlp_up_proj L__self___mlp_up_proj (to_1,) {}
call_function mul_2 <built-in function mul> (silu, l__self___mlp_up_proj) {}
call_module l__self___mlp_down_proj L__self___mlp_down_proj (mul_2,) {}
call_function add_2 <built-in function add> (add, l__self___mlp_down_proj) {}
output output output ((add_2,),) {}
[2023-07-06 04:20:58,555] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
[2023-07-06 04:20:58,602] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 26 =====
<eval_with_key>.223 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[16], arg1_1: f16[64, 16], arg2_1: f16[64, 16], arg3_1: f16[16, 64], arg4_1: f16[1, 1, 16], arg5_1: f16[1, 1, 16]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:294, code: hidden_states = residual + hidden_states
add: f16[1, 1, 16] = torch.ops.aten.add.Tensor(arg5_1, arg4_1); arg5_1 = arg4_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
convert_element_type: f32[1, 1, 16] = torch.ops.prims.convert_element_type.default(add, torch.float32)
pow_1: f32[1, 1, 16] = torch.ops.aten.pow.Tensor_Scalar(convert_element_type, 2); convert_element_type = None
mean: f32[1, 1, 1] = torch.ops.aten.mean.dim(pow_1, [-1], True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add_1: f32[1, 1, 1] = torch.ops.aten.add.Tensor(mean, 1e-06); mean = None
rsqrt: f32[1, 1, 1] = torch.ops.aten.rsqrt.default(add_1); add_1 = None
mul: f32[1, 1, 16] = torch.ops.aten.mul.Tensor(add, rsqrt); rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
mul_1: f32[1, 1, 16] = torch.ops.aten.mul.Tensor(arg0_1, mul); arg0_1 = mul = None
convert_element_type_1: f16[1, 1, 16] = torch.ops.prims.convert_element_type.default(mul_1, torch.float16); mul_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
permute: f16[16, 64] = torch.ops.aten.permute.default(arg1_1, [1, 0]); arg1_1 = None
view: f16[1, 16] = torch.ops.aten.view.default(convert_element_type_1, [1, 16])
mm: f16[1, 64] = torch.ops.aten.mm.default(view, permute); view = permute = None
view_1: f16[1, 1, 64] = torch.ops.aten.view.default(mm, [1, 1, 64]); mm = None
# File: /home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/activations.py:150, code: return nn.functional.silu(input)
convert_element_type_2: f32[1, 1, 64] = torch.ops.prims.convert_element_type.default(view_1, torch.float32); view_1 = None
sigmoid: f32[1, 1, 64] = torch.ops.aten.sigmoid.default(convert_element_type_2)
mul_2: f32[1, 1, 64] = torch.ops.aten.mul.Tensor(convert_element_type_2, sigmoid); convert_element_type_2 = sigmoid = None
convert_element_type_3: f16[1, 1, 64] = torch.ops.prims.convert_element_type.default(mul_2, torch.float16); mul_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:162, code: return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
permute_1: f16[16, 64] = torch.ops.aten.permute.default(arg2_1, [1, 0]); arg2_1 = None
view_2: f16[1, 16] = torch.ops.aten.view.default(convert_element_type_1, [1, 16]); convert_element_type_1 = None
mm_1: f16[1, 64] = torch.ops.aten.mm.default(view_2, permute_1); view_2 = permute_1 = None
view_3: f16[1, 1, 64] = torch.ops.aten.view.default(mm_1, [1, 1, 64]); mm_1 = None
mul_3: f16[1, 1, 64] = torch.ops.aten.mul.Tensor(convert_element_type_3, view_3); convert_element_type_3 = view_3 = None
permute_2: f16[64, 16] = torch.ops.aten.permute.default(arg3_1, [1, 0]); arg3_1 = None
view_4: f16[1, 64] = torch.ops.aten.view.default(mul_3, [1, 64]); mul_3 = None
mm_2: f16[1, 16] = torch.ops.aten.mm.default(view_4, permute_2); view_4 = permute_2 = None
view_5: f16[1, 1, 16] = torch.ops.aten.view.default(mm_2, [1, 1, 16]); mm_2 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:300, code: hidden_states = residual + hidden_states
add_2: f16[1, 1, 16] = torch.ops.aten.add.Tensor(add, view_5); add = view_5 = None
return (add_2,)
[2023-07-06 04:20:58,645] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 13
[2023-07-06 04:20:58,687] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 13
[2023-07-06 04:20:58,704] torch._inductor.scheduler: [DEBUG] remove_buffer('buf0')
[2023-07-06 04:20:58,713] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/oo/cooxrverhcp2snw6sgvi3xmyupj2eewevtsplvvj674cqwmsnljq.py
[2023-07-06 04:20:58,713] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 13
[2023-07-06 04:20:58,715] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:58,723] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662486464)
hasattr(L['residual'], '_dynamo_dynamic_indices') == False
___check_type_id(L['___stack0'], 7638272)
len(L['___stack0']) == 3
hasattr(L['___stack0'][0], '_dynamo_dynamic_indices') == False
___check_obj_id(L['___stack0'][1], 7649088)
___check_obj_id(L['___stack0'][2], 7649088)
___check_obj_id(L['output_attentions'], 7698112)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
str(G['torch'].float32) == 'torch.float32'
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks.keys()) == set()
___check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks, 7509984)
set(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks.keys()) == set()
[2023-07-06 04:20:58,723] torch._inductor.cudagraph_trees: [DEBUG] Running warmup of function 9
[2023-07-06 04:20:58,724] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function forward in /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py', 'set env var TORCHDYNAMO_REPORT_GUARD_FAILURES=1 to debug further')
[2023-07-06 04:20:58,724] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:83
[2023-07-06 04:20:58,725] torch._dynamo.variables.builder: [DEBUG] automatic dynamic L['hidden_states'] size(1) 1 != 1000
[2023-07-06 04:20:58,725] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['hidden_states'] (1, 1, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:58,726] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:84
[2023-07-06 04:20:58,726] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:58,726] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR dtype [TensorVariable()]
[2023-07-06 04:20:58,729] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST input_dtype [TorchVariable(torch.float16)]
[2023-07-06 04:20:58,729] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85
[2023-07-06 04:20:58,729] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:58,729] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:58,729] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:58,729] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR float32 [GetAttrVariable(TensorVariable(), to), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:58,730] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), ConstantVariable(dtype)]
[2023-07-06 04:20:58,731] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR pow [TensorVariable()]
[2023-07-06 04:20:58,731] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 2 [GetAttrVariable(TensorVariable(), pow)]
[2023-07-06 04:20:58,731] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), pow), ConstantVariable(int)]
[2023-07-06 04:20:58,732] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mean [TensorVariable()]
[2023-07-06 04:20:58,732] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST -1 [GetAttrVariable(TensorVariable(), mean)]
[2023-07-06 04:20:58,732] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST True [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int)]
[2023-07-06 04:20:58,732] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('keepdim',) [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool)]
[2023-07-06 04:20:58,732] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 2 [GetAttrVariable(TensorVariable(), mean), ConstantVariable(int), ConstantVariable(bool), ConstantVariable(tuple)]
[2023-07-06 04:20:58,733] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST variance [TensorVariable()]
[2023-07-06 04:20:58,733] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86
[2023-07-06 04:20:58,733] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states []
[2023-07-06 04:20:58,733] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [TensorVariable()]
[2023-07-06 04:20:58,734] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR rsqrt [TensorVariable(), TorchVariable(<module 'torch' from '/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2023-07-06 04:20:58,734] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST variance [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>)]
[2023-07-06 04:20:58,734] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:58,734] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR variance_epsilon [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), NNModuleVariable()]
[2023-07-06 04:20:58,734] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable(), ConstantVariable(float)]
[2023-07-06 04:20:58,735] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), TorchVariable(<built-in method rsqrt of type object at 0x7fb940469580>), TensorVariable()]
[2023-07-06 04:20:58,736] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,736] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:58,737] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88
[2023-07-06 04:20:58,737] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,737] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR weight [NNModuleVariable()]
[2023-07-06 04:20:58,737] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].weight (16,) [<DimDynamic.STATIC: 2>] [None]
[2023-07-06 04:20:58,738] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:58,738] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-07-06 04:20:58,738] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR to [TensorVariable()]
[2023-07-06 04:20:58,739] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST input_dtype [GetAttrVariable(TensorVariable(), to)]
[2023-07-06 04:20:58,739] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [GetAttrVariable(TensorVariable(), to), TorchVariable(torch.float16)]
[2023-07-06 04:20:58,740] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-07-06 04:20:58,740] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-07-06 04:20:58,740] torch._dynamo.symbolic_convert: [DEBUG] RETURN_VALUE triggered compile
[2023-07-06 04:20:58,740] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 88 in forward>], graph_break=False)
[2023-07-06 04:20:58,741] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_23 =====
<eval_with_key>.232 class GraphModule(torch.nn.Module):
def forward(self, L_hidden_states_ : torch.Tensor):
l_hidden_states_ = L_hidden_states_
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
to = l_hidden_states_.to(torch.float32)
pow_1 = to.pow(2); to = None
mean = pow_1.mean(-1, keepdim = True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add = mean + 1e-06; mean = None
rsqrt = torch.rsqrt(add); add = None
mul = l_hidden_states_ * rsqrt; l_hidden_states_ = rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
l__self___weight = self.L__self___weight
mul_1 = l__self___weight * mul; l__self___weight = mul = None
to_1 = mul_1.to(torch.float16); mul_1 = None
return (to_1,)
[2023-07-06 04:20:58,741] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_23 <eval_with_key>.232 opcode name target args kwargs
------------- ---------------- -------------------------------------------------------- --------------------------------- -----------------
placeholder l_hidden_states_ L_hidden_states_ () {}
call_method to to (l_hidden_states_, torch.float32) {}
call_method pow_1 pow (to, 2) {}
call_method mean mean (pow_1, -1) {'keepdim': True}
call_function add <built-in function add> (mean, 1e-06) {}
call_function rsqrt <built-in method rsqrt of type object at 0x7fb940469580> (add,) {}
call_function mul <built-in function mul> (l_hidden_states_, rsqrt) {}
get_attr l__self___weight L__self___weight () {}
call_function mul_1 <built-in function mul> (l__self___weight, mul) {}
call_method to_1 to (mul_1, torch.float16) {}
output output output ((to_1,),) {}
[2023-07-06 04:20:58,742] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
[2023-07-06 04:20:58,760] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 27 =====
<eval_with_key>.236 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[16], arg1_1: f16[1, 1, 16]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:85, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
convert_element_type: f32[1, 1, 16] = torch.ops.prims.convert_element_type.default(arg1_1, torch.float32)
pow_1: f32[1, 1, 16] = torch.ops.aten.pow.Tensor_Scalar(convert_element_type, 2); convert_element_type = None
mean: f32[1, 1, 1] = torch.ops.aten.mean.dim(pow_1, [-1], True); pow_1 = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:86, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
add: f32[1, 1, 1] = torch.ops.aten.add.Tensor(mean, 1e-06); mean = None
rsqrt: f32[1, 1, 1] = torch.ops.aten.rsqrt.default(add); add = None
mul: f32[1, 1, 16] = torch.ops.aten.mul.Tensor(arg1_1, rsqrt); arg1_1 = rsqrt = None
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:88, code: return (self.weight * hidden_states).to(input_dtype)
mul_1: f32[1, 1, 16] = torch.ops.aten.mul.Tensor(arg0_1, mul); arg0_1 = mul = None
convert_element_type_1: f16[1, 1, 16] = torch.ops.prims.convert_element_type.default(mul_1, torch.float16); mul_1 = None
return (convert_element_type_1,)
[2023-07-06 04:20:58,762] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 14
[2023-07-06 04:20:58,778] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 14
[2023-07-06 04:20:58,788] torch._inductor.scheduler: [DEBUG] remove_buffer('buf0')
[2023-07-06 04:20:58,789] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/vz/cvzk75oefn2tjj3ekdj3gg5a7rpdjqti5tysf4vatbmcjiobtp5i.py
[2023-07-06 04:20:58,789] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 14
[2023-07-06 04:20:58,792] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:58,795] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662764176)
___check_type_id(L['hidden_states'], 77765680)
hasattr(L['hidden_states'], '_dynamo_dynamic_indices') == False
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
str(G['torch'].float32) == 'torch.float32'
[2023-07-06 04:20:58,795] torch._inductor.cudagraph_trees: [DEBUG] Running warmup of function 10
[2023-07-06 04:20:58,795] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function __init__ in <string>', 'set env var TORCHDYNAMO_REPORT_GUARD_FAILURES=1 to debug further')
[2023-07-06 04:20:58,796] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing __init__ <string>:2
[2023-07-06 04:20:58,796] torch._dynamo.variables.builder: [DEBUG] automatic dynamic L['last_hidden_state'] size(1) 1 != 1000
[2023-07-06 04:20:58,796] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['last_hidden_state'] (1, 1, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:58,797] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line <string>:3
[2023-07-06 04:20:58,797] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST last_hidden_state []
[2023-07-06 04:20:58,797] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable()]
[2023-07-06 04:20:58,797] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_ATTR last_hidden_state [TensorVariable(), DataClassVariable()]
[2023-07-06 04:20:58,798] torch._dynamo.symbolic_convert: [DEBUG] STORE_ATTR triggered compile
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1195, in STORE_ATTR
.call_function(self, [obj, ConstantVariable(inst.argval), val], {})
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 591, in call_function
result = handler(tx, *args, **kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1103, in call_setattr
return obj.call_method(tx, "__setattr__", [name_var, val], {})
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/dicts.py", line 417, in call_method
return super().call_method(tx, name, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/dicts.py", line 198, in call_method
return super().call_method(tx, name, args, kwargs)
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/variables/base.py", line 260, in call_method
raise unimplemented(f"call_method {self} {name} {args} {kwargs}")
File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 140, in unimplemented
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: call_method DataClassVariable() __setitem__ [ConstantVariable(str), TensorVariable()] {}
[2023-07-06 04:20:58,798] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-07-06 04:20:58,798] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='store_attr', user_stack=[<FrameSummary file <string>, line 3 in __init__>], graph_break=True)
[2023-07-06 04:20:58,802] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_type_id(L['self'], 110084784)
___check_type_id(L['past_key_values'], 7638272)
len(L['past_key_values']) == 2
___check_obj_id(L['self'].attentions, 7649088)
hasattr(L['last_hidden_state'], '_dynamo_dynamic_indices') == False
___check_obj_id(L['self'].hidden_states, 7649088)
___check_obj_id(L['self'].past_key_values, 7649088)
___check_obj_id(L['self'].last_hidden_state, 7649088)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
[2023-07-06 04:20:58,802] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function <resume in forward> in /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py', 'set env var TORCHDYNAMO_REPORT_GUARD_FAILURES=1 to debug further')
[2023-07-06 04:20:58,803] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <resume in forward> /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:692
[2023-07-06 04:20:58,803] torch._dynamo.variables.builder: [DEBUG] automatic dynamic L['___stack0'].last_hidden_state size(1) 1 != 1000
[2023-07-06 04:20:58,803] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['___stack0'].last_hidden_state (1, 1, 16) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None, None]
[2023-07-06 04:20:58,805] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST ___stack0 []
[2023-07-06 04:20:58,805] torch._dynamo.symbolic_convert: [DEBUG] TRACE JUMP_ABSOLUTE 90 [DataClassVariable()]
[2023-07-06 04:20:58,805] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST outputs [DataClassVariable()]
[2023-07-06 04:20:58,805] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:704
[2023-07-06 04:20:58,805] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs []
[2023-07-06 04:20:58,805] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [DataClassVariable()]
[2023-07-06 04:20:58,805] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [DataClassVariable(), ConstantVariable(int)]
[2023-07-06 04:20:58,805] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST hidden_states [TensorVariable()]
[2023-07-06 04:20:58,805] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:705
[2023-07-06 04:20:58,805] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-07-06 04:20:58,805] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR lm_head [NNModuleVariable()]
[2023-07-06 04:20:58,806] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST hidden_states [NNModuleVariable()]
[2023-07-06 04:20:58,806] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2023-07-06 04:20:58,808] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST logits [TensorVariable()]
[2023-07-06 04:20:58,808] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:707
[2023-07-06 04:20:58,808] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None []
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST loss [ConstantVariable(NoneType)]
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:708
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST labels []
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [ConstantVariable(NoneType)]
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [ConstantVariable(NoneType), ConstantVariable(NoneType)]
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 222 [ConstantVariable(bool)]
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:720
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST return_dict []
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 266 [ConstantVariable(bool)]
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:724
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL CausalLMOutputWithPast []
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:725
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST loss [UserDefinedClassVariable()]
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:726
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST logits [UserDefinedClassVariable(), ConstantVariable(NoneType)]
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:727
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable()]
[2023-07-06 04:20:58,809] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR past_key_values [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable(), DataClassVariable()]
[2023-07-06 04:20:58,810] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:728
[2023-07-06 04:20:58,810] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable(), TupleVariable()]
[2023-07-06 04:20:58,810] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR hidden_states [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable(), TupleVariable(), DataClassVariable()]
[2023-07-06 04:20:58,810] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:729
[2023-07-06 04:20:58,810] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST outputs [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable(), TupleVariable(), ConstantVariable(NoneType)]
[2023-07-06 04:20:58,810] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR attentions [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable(), TupleVariable(), ConstantVariable(NoneType), DataClassVariable()]
[2023-07-06 04:20:58,810] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:724
[2023-07-06 04:20:58,810] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('loss', 'logits', 'past_key_values', 'hidden_states', 'attentions') [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable(), TupleVariable(), ConstantVariable(NoneType), ConstantVariable(NoneType)]
[2023-07-06 04:20:58,810] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 5 [UserDefinedClassVariable(), ConstantVariable(NoneType), TensorVariable(), TupleVariable(), ConstantVariable(NoneType), ConstantVariable(NoneType), ConstantVariable(tuple)]
[2023-07-06 04:20:58,811] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [DataClassVariable()]
[2023-07-06 04:20:58,811] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing <resume in forward> (RETURN_VALUE)
[2023-07-06 04:20:58,811] torch._dynamo.symbolic_convert: [DEBUG] RETURN_VALUE triggered compile
[2023-07-06 04:20:58,811] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py, line 724 in <resume in forward>>], graph_break=False)
[2023-07-06 04:20:58,812] torch._dynamo.output_graph.__graph_code: [DEBUG] TRACED GRAPH
===== __compiled_fn_25 =====
<eval_with_key>.238 class GraphModule(torch.nn.Module):
def forward(self, L_stack0_last_hidden_state : torch.Tensor):
l_stack0_last_hidden_state = L_stack0_last_hidden_state
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:705, code: logits = self.lm_head(hidden_states)
l__self___lm_head = self.L__self___lm_head(l_stack0_last_hidden_state); l_stack0_last_hidden_state = None
return (l__self___lm_head,)
[2023-07-06 04:20:58,812] torch._dynamo.output_graph.__graph: [DEBUG] TRACED GRAPH
__compiled_fn_25 <eval_with_key>.238 opcode name target args kwargs
----------- -------------------------- -------------------------- ----------------------------- --------
placeholder l_stack0_last_hidden_state L_stack0_last_hidden_state () {}
call_module l__self___lm_head L__self___lm_head (l_stack0_last_hidden_state,) {}
output output output ((l__self___lm_head,),) {}
[2023-07-06 04:20:58,812] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function inductor
[2023-07-06 04:20:58,823] torch._functorch.aot_autograd.__aot_graphs: [INFO] TRACED GRAPH
===== Forward graph 28 =====
<eval_with_key>.242 class <lambda>(torch.nn.Module):
def forward(self, arg0_1: f16[32000, 16], arg1_1: f16[1, 1, 16]):
# File: /home/ubuntu/accelerated-pytorch-transformers-generation/src/trfs_fast/llama.py:705, code: logits = self.lm_head(hidden_states)
permute: f16[16, 32000] = torch.ops.aten.permute.default(arg0_1, [1, 0]); arg0_1 = None
view: f16[1, 16] = torch.ops.aten.view.default(arg1_1, [1, 16]); arg1_1 = None
mm: f16[1, 32000] = torch.ops.aten.mm.default(view, permute); view = permute = None
view_1: f16[1, 1, 32000] = torch.ops.aten.view.default(mm, [1, 1, 32000]); mm = None
return (view_1,)
[2023-07-06 04:20:58,838] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 15
[2023-07-06 04:20:58,844] torch._inductor.graph: [DEBUG] Force channels last inputs for 0 conv for the current graph with id 15
[2023-07-06 04:20:58,846] torch._inductor.graph: [DEBUG] Output code written to: /tmp/torchinductor_ubuntu/7y/c7ycehmc443da67tdrrtaosxvw4ksnvhbehw2tj6bp2msphljdvi.py
[2023-07-06 04:20:58,846] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 15
[2023-07-06 04:20:58,848] torch._dynamo.output_graph: [INFO] Step 2: done compiler function inductor
[2023-07-06 04:20:58,854] torch._dynamo.convert_frame.__guards: [DEBUG] GUARDS:
___check_obj_id(L['self'], 140431662761824)
___check_obj_id(L['labels'], 7649088)
___check_type_id(L['___stack0'], 110084784)
___check_obj_id(L['return_dict'], 7698144)
___check_obj_id(L['___stack0'].attentions, 7649088)
___check_obj_id(L['___stack0'].hidden_states, 7649088)
___check_type_id(L['___stack0'].past_key_values, 7638272)
len(L['___stack0'].past_key_values) == 2
hasattr(L['___stack0'].last_hidden_state, '_dynamo_dynamic_indices') == False
___check_obj_id(L['___stack0'].past_key_values[0], 7649088)
___check_obj_id(L['___stack0'].past_key_values[1], 7649088)
not ___is_grad_enabled()
not ___are_deterministic_algorithms_enabled()
utils_device.CURRENT_DEVICE == None
[2023-07-06 04:20:58,854] torch._inductor.cudagraph_trees: [DEBUG] Running warmup of function 11
[2023-07-06 04:20:58,855] torch._inductor.cudagraph_trees: [DEBUG] Recording function 6 of graph recording id 0
[2023-07-06 04:20:58,976] torch._inductor.cudagraph_trees: [DEBUG] Recording function 7 of graph recording id 1
[2023-07-06 04:20:59,088] torch._inductor.cudagraph_trees: [DEBUG] Recording function 8 of graph recording id 2
[2023-07-06 04:20:59,200] torch._inductor.cudagraph_trees: [DEBUG] Recording function 9 of graph recording id 3
[2023-07-06 04:20:59,312] torch._inductor.cudagraph_trees: [DEBUG] Recording function 10 of graph recording id 4
[2023-07-06 04:20:59,423] torch._inductor.cudagraph_trees: [DEBUG] Recording function 11 of graph recording id 5
[2023-07-06 04:20:59,537] torch._inductor.cudagraph_trees: [DEBUG] Recording function 0 of graph recording id 6
[2023-07-06 04:20:59,653] torch._inductor.cudagraph_trees: [DEBUG] Recording function 1 of graph recording id 7
[2023-07-06 04:20:59,771] torch._inductor.cudagraph_trees: [DEBUG] Recording function 2 of graph recording id 8
[2023-07-06 04:20:59,886] torch._inductor.cudagraph_trees: [DEBUG] Recording function 3 of graph recording id 9
[2023-07-06 04:20:59,999] torch._inductor.cudagraph_trees: [DEBUG] Recording function 4 of graph recording id 10
[2023-07-06 04:21:00,110] torch._inductor.cudagraph_trees: [DEBUG] Recording function 5 of graph recording id 11
Warmup/compilation time: 22.32 seconds (2 generate calls)
Measuring generate: 0%| | 0/1 [00:00<?, ?it/s]
Measuring generate: 100%|██████████| 1/1 [00:00<00:00, 167.56it/s]
100%|██████████| 1/1 [00:22<00:00, 22.36s/it]
100%|██████████| 1/1 [00:22<00:00, 22.36s/it]
100%|██████████| 1/1 [00:22<00:00, 22.36s/it]
100%|██████████| 1/1 [00:22<00:00, 22.36s/it]
100%|██████████| 1/1 [00:22<00:00, 22.36s/it]
100%|██████████| 1/1 [00:22<00:00, 22.36s/it]
Preallocate Compile Batch size ... Max GPU memory (MB) Results hash Git hash
0 True static 1 ... 29.06 efd05c5d cd00bdb
[1 rows x 11 columns]
Results also appended to ./results/results_llama.csv
[2023-07-06 04:21:00,294] torch._dynamo.utils: [INFO] TorchDynamo compilation metrics:
Function Runtimes (s)
-------------------------------------- --------------
_compile 19.5125
OutputGraph.call_user_compiler 18.1622
create_aot_dispatcher_function 18.7711
compile_fx.<locals>.fw_compiler_base 17.0007
GraphLowering.run 0.5331
GraphLowering.compile_to_module 14.3102
Scheduler.__init__ 6.1754
Scheduler.codegen 0.2503
WrapperCodeGen.generate 0.0188
cudagraphify 0.0031
CachingAutotuner.benchmark_all_configs 0.8992
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment