Skip to content

Instantly share code, notes, and snippets.

@ezyang

ezyang/final.csv Secret

Created December 5, 2022 22:57
Show Gist options
  • Save ezyang/e03fce6e07ecdc184893e15755f7c63e to your computer and use it in GitHub Desktop.
Save ezyang/e03fce6e07ecdc184893e15755f7c63e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
final.csv
a05b7b1c73247ff562a82aac0edca79bbaebc2bd https://gist.github.com/fa65798977b0d018ec5637e601ce2b47
torchbench BERT_pytorch PASS
torchbench Background_Matting FAIL benchmarks/dynamo/common.py NotImplementedError: Eager model failed to run
torchbench LearningToPaint PASS
torchbench Super_SloMo PASS
torchbench alexnet PASS
torchbench attention_is_all_you_need_pytorch PASS
torchbench dcgan PASS
torchbench densenet121 FAIL detectron2/layers/deform_conv.py ImportError: /home/ezyang/local/b/pytorch-env/lib/python3.9/site-packages/detectron2/_C.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops10select_int4callERKNS_6TensorEll
torchbench dlrm PASS
torchbench drq FAIL functorch/_src/aot_autograd.py AssertionError on line 1554: assert type(inner_out) == type(user_out)
torchbench fastNLP_Bert PASS
torchbench functorch_dp_cifar10 PASS
torchbench functorch_maml_omniglot PASS
torchbench hf_Albert PASS
torchbench hf_Bart PASS
torchbench hf_Bert PASS
torchbench hf_BigBird FAIL transformers/models/big_bird/modeling_big_bird.py RuntimeError: Output 0 of CompiledFunctionBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.
torchbench hf_DistilBert PASS
torchbench hf_GPT2 PASS
torchbench hf_GPT2_large PASS
torchbench hf_Longformer FAIL functorch/_src/aot_autograd.py AssertionError on line 1554: assert type(inner_out) == type(user_out)
torchbench hf_Reformer FAIL sympy-1.11.1-py3.9.egg/sympy/core/mod.py ZeroDivisionError: Modulo by zero
torchbench hf_T5 PASS
torchbench hf_T5_base PASS
torchbench hf_T5_large PASS
torchbench lennard_jones PASS
torchbench maml_omniglot PASS
torchbench mnasnet1_0 PASS
torchbench mobilenet_v2 PASS
torchbench mobilenet_v2_quantized_qat FAIL functorch/_src/compilers.py AssertionError: output 2 where {s0: 2, s2: 224}: torch.Size([0]) aka (0,) != torch.Size([32])
torchbench mobilenet_v3_large PASS
torchbench moco FAIL torch/nn/modules/module.py AttributeError: 'ResNet' object has no attribute 'layer1[0]'
torchbench nvidia_deeprecommender PASS
torchbench pytorch_CycleGAN_and_pix2pix PASS
torchbench pytorch_stargan PASS
torchbench pytorch_struct PASS
torchbench pytorch_unet PASS
torchbench resnet152 PASS
torchbench resnet18 PASS
torchbench resnet50 PASS
torchbench resnet50_quantized_qat FAIL functorch/_src/compilers.py AssertionError: output 2 where {s0: 2, s2: 224}: torch.Size([0]) aka (0,) != torch.Size([64])
torchbench resnext50_32x4d PASS
torchbench shufflenet_v2_x1_0 PASS
torchbench soft_actor_critic FAIL functorch/_src/aot_autograd.py AssertionError on line 1554: assert type(inner_out) == type(user_out)
torchbench speech_transformer PASS
torchbench squeezenet1_1 PASS
torchbench tacotron2 FAIL torch/fx/interpreter.py RuntimeError: Cannot call sizes() on tensor with symbolic sizes/strides
torchbench timm_efficientdet FAIL torch/nn/functional.py RuntimeError: Cannot call sizes() on tensor with symbolic sizes/strides
torchbench timm_efficientnet PASS
torchbench timm_regnet FAIL torch/_dynamo/guards.py AssertionError: Unknown shape symbol s2.
torchbench timm_resnest PASS
torchbench timm_vision_transformer PASS
torchbench timm_vision_transformer_large PASS
torchbench timm_vovnet PASS
torchbench tts_angular FAIL torch/nn/modules/rnn.py RuntimeError: Cannot call sizes() on tensor with symbolic sizes/strides
torchbench vgg16 PASS
torchbench vision_maskrcnn FAIL torch/fx/node.py AttributeError: 'SymInt' object has no attribute 'size'
torchbench yolov3 PASS
huggingface AlbertForMaskedLM PASS
huggingface AlbertForQuestionAnswering FAIL transformers/configuration_utils.py OSError: Can't load config for 'allenai/longformer-base-4096'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'allenai/longformer-base-4096' is the correct path to a directory containing a config.json file
huggingface BartForCausalLM PASS
huggingface BartForConditionalGeneration PASS
huggingface BertForMaskedLM PASS
huggingface BertForQuestionAnswering PASS
huggingface BlenderbotForCausalLM PASS
huggingface BlenderbotSmallForCausalLM PASS
huggingface BlenderbotSmallForConditionalGeneration PASS
huggingface CamemBert PASS
huggingface DebertaForMaskedLM PASS
huggingface DebertaForQuestionAnswering PASS
huggingface DebertaV2ForMaskedLM PASS
huggingface DebertaV2ForQuestionAnswering PASS
huggingface DistilBertForMaskedLM PASS
huggingface DistilBertForQuestionAnswering PASS
huggingface DistillGPT2 PASS
huggingface ElectraForCausalLM PASS
huggingface ElectraForQuestionAnswering PASS
huggingface GPT2ForSequenceClassification PASS
huggingface GoogleFnet PASS
huggingface LayoutLMForMaskedLM PASS
huggingface LayoutLMForSequenceClassification PASS
huggingface M2M100ForConditionalGeneration PASS
huggingface MBartForCausalLM PASS
huggingface MBartForConditionalGeneration PASS
huggingface MT5ForConditionalGeneration PASS
huggingface MegatronBertForCausalLM PASS
huggingface MegatronBertForQuestionAnswering PASS
huggingface MobileBertForMaskedLM PASS
huggingface MobileBertForQuestionAnswering PASS
huggingface OPTForCausalLM PASS
huggingface PLBartForCausalLM PASS
huggingface PLBartForConditionalGeneration PASS
huggingface PegasusForCausalLM PASS
huggingface PegasusForConditionalGeneration PASS
huggingface RobertaForCausalLM PASS
huggingface RobertaForQuestionAnswering PASS
huggingface Speech2Text2ForCausalLM PASS
huggingface T5ForConditionalGeneration PASS
huggingface T5Small PASS
huggingface TrOCRForCausalLM PASS
huggingface XGLMForCausalLM PASS
huggingface XLNetLMHeadModel PASS
huggingface YituTechConvBert PASS
timm_models adv_inception_v3 PASS
timm_models beit_base_patch16_224 PASS
timm_models botnet26t_256 FAIL torch/fx/experimental/symbolic_shapes.py AssertionError: Failed to find s3 allocated at File "/data/users/ezyang/b/pytorch/benchmarks/dynamo/timm_models.py", line 322, in <module>
timm_models cait_m36_384 PASS
timm_models coat_lite_mini PASS
timm_models convit_base PASS
timm_models convmixer_768_32 PASS
timm_models convnext_base PASS
timm_models crossvit_9_240 FAIL torch/nn/functional.py RuntimeError: Cannot call sizes() on tensor with symbolic sizes/strides
timm_models cspdarknet53 FAIL torch/fx/experimental/symbolic_shapes.py AssertionError: Failed to find s3 allocated at File "/data/users/ezyang/b/pytorch/benchmarks/dynamo/timm_models.py", line 322, in <module>
timm_models deit_base_distilled_patch16_224 PASS
timm_models dla102 PASS
timm_models dm_nfnet_f0 PASS
timm_models dpn107 FAIL torch/_dynamo/guards.py AssertionError: Unknown shape symbol s2.
timm_models eca_botnext26ts_256 FAIL torch/fx/experimental/symbolic_shapes.py AssertionError: Failed to find s3 allocated at File "/data/users/ezyang/b/pytorch/benchmarks/dynamo/timm_models.py", line 322, in <module>
timm_models eca_halonext26ts FAIL torch/fx/experimental/symbolic_shapes.py AssertionError: Failed to find s3 allocated at File "/data/users/ezyang/b/pytorch/benchmarks/dynamo/timm_models.py", line 322, in <module>
timm_models ese_vovnet19b_dw PASS
timm_models fbnetc_100 PASS
timm_models fbnetv3_b PASS
timm_models gernet_l PASS
timm_models ghostnet_100 PASS
timm_models gluon_inception_v3 PASS
timm_models gluon_xception65 PASS
timm_models gmixer_24_224 PASS
timm_models gmlp_s16_224 PASS
timm_models hrnet_w18 FAIL torch/nn/functional.py RuntimeError: Cannot call sizes() on tensor with symbolic sizes/strides
timm_models inception_v3 PASS
timm_models jx_nest_base PASS
timm_models lcnet_050 PASS
timm_models levit_128 FAIL torch/autograd/__init__.py RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.
timm_models mixer_b16_224 PASS
timm_models mixnet_l PASS
timm_models mnasnet_100 PASS
timm_models mobilenetv2_100 PASS
timm_models mobilenetv3_large_100 PASS
timm_models mobilevit_s PASS
timm_models nfnet_l0 PASS
timm_models pit_b_224 PASS
timm_models pnasnet5large PASS
timm_models poolformer_m36 PASS
timm_models regnety_002 FAIL torch/_dynamo/guards.py AssertionError: Unknown shape symbol s1.
timm_models repvgg_a2 PASS
timm_models res2net101_26w_4s PASS
timm_models res2net50_14w_8s PASS
timm_models res2next50 PASS
timm_models resmlp_12_224 PASS
timm_models resnest101e PASS
timm_models rexnet_100 PASS
timm_models sebotnet33ts_256 FAIL torch/fx/experimental/symbolic_shapes.py AssertionError: Failed to find s3 allocated at File "/data/users/ezyang/b/pytorch/benchmarks/dynamo/timm_models.py", line 322, in <module>
timm_models selecsls42b PASS
timm_models spnasnet_100 PASS
timm_models swin_base_patch4_window7_224 FAIL torch/fx/experimental/symbolic_shapes.py AssertionError: Failed to find s3 allocated at File "/data/users/ezyang/b/pytorch/benchmarks/dynamo/timm_models.py", line 322, in <module>
timm_models swsl_resnext101_32x16d PASS
timm_models tf_efficientnet_b0 PASS
timm_models tf_mixnet_l PASS
timm_models tinynet_a PASS
timm_models tnt_s_patch16_224 PASS
timm_models twins_pcpvt_base FAIL sympy-1.11.1-py3.9.egg/sympy/core/mod.py ZeroDivisionError: Modulo by zero
timm_models visformer_small PASS
timm_models vit_base_patch16_224 PASS
timm_models volo_d1_224 PASS
timm_models xcit_large_24_p8_224 FAIL functorch/_src/compilers.py AssertionError: output 0 where {s2: 2, s3: 28, s10: 602880, s12: 3, s14: 384, s15: 56, s16: 192, s17: 112, s19: 64, s20: 224}: (768*s3**2, 1, 768*s3, 768) aka (602112, 1, 21504, 768) != (602112, 784, 28, 1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment