SNPE Performance Comparison between 1.61 and 1.64

fast.svg

large.svg

<svg width="1000" viewBox="0 0 2787 1189" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xml:space="preserve" overflow="hidden"><defs><clipPath id="clip0"><rect x="1364" y="171" width="2787" height="1189"/></clipPath><clipPath id="clip1"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip2"><rect x="1537" y="420" width="2565" height="730"/></clipPath><clipPath id="clip3"><rect x="1537" y="420" width="2565" height="730"/></clipPath><clipPath id="clip4"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip5"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip6"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip7"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip8"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip9"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip10"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip11"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip12"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip13"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip14"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip15"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip16"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip17"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip18"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip19"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip20"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip21"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip22"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip23"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip24"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip25"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip26"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip27"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip28"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip29"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip30"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip31"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip32"><rect x="1364" y="171" width="2785" height="1186"/></clipPath><clipPath id="clip33"><rect x="1364" y="171" width="2785" height="1186"/></clipPath></defs><g clip-path="url(#clip0)" transform="translate(-1364 -171)"><rect x="1365" y="172" width="2784" height="1186" fill="#FFFFFF"/><g clip-path="url(#clip1)"><path d="M1537.5 1025.53 4098.5 1025.53M1537.5 904.53 4098.5 904.53M1537.5 783.526 4098.5 783.526M1537.5 662.522 4098.5 662.522M1537.5 541.518 4098.5 541.518M1537.5 420.5 4098.5 420.5" stroke="#D9D9D9" stroke-width="3.4375" stroke-linejoin="round" stroke-miterlimit="10" fill="none"/></g><g clip-path="url(#clip2)"><path d="M1576.05 1011.03 1612.05 1011.03 1612.05 1147 1576.05 1147ZM1737.06 1011.03 1772.06 1011.03 1772.06 1147 1737.06 1147ZM1897.06 1005.03 1933.06 1005.03 1933.06 1147 1897.06 1147ZM2057.07 907.03 2093.07 907.03 2093.07 1147 2057.07 1147ZM2217.07 943.031 2253.07 943.031 2253.07 1147 2217.07 1147ZM2377.08 926.03 2413.08 926.03 2413.08 1147 2377.08 1147ZM2537.08 801.026 2573.08 801.026 2573.08 1147 2537.08 1147ZM2697.09 587.019 2733.09 587.019 2733.09 1147 2697.09 1147ZM2857.09 667.022 2893.09 667.022 2893.09 1147 2857.09 1147ZM3017.1 650.021 3053.1 650.021 3053.1 1147 3017.1 1147ZM3177.1 644.021 3213.11 644.021 3213.11 1147 3177.1 1147ZM3337.11 640.021 3373.11 640.021 3373.11 1147 3337.11 1147ZM3497.11 653.021 3533.12 653.021 3533.12 1147 3497.11 1147ZM3658.12 580.019 3693.12 580.019 3693.12 1147 3658.12 1147ZM3818.12 768.025 3854.13 768.025 3854.13 1147 3818.12 1147ZM3978.13 557.018 4014.13 557.018 4014.13 1147 3978.13 1147Z" fill="#4472C4"/></g><g clip-path="url(#clip3)"><path d="M1622.05 993.033 1658.05 993.033 1658.05 1147 1622.05 1147ZM1782.06 1003.03 1818.06 1003.03 1818.06 1147 1782.06 1147ZM1942.06 997.033 1978.06 997.033 1978.06 1147 1942.06 1147ZM2102.07 874.029 2138.07 874.029 2138.07 1147 2102.07 1147ZM2262.07 829.027 2298.08 829.027 2298.08 1147 2262.07 1147ZM2422.08 868.028 2458.08 868.028 2458.08 1147 2422.08 1147ZM2583.08 857.028 2618.09 857.028 2618.09 1147 2583.08 1147ZM2743.09 591.019 2779.09 591.019 2779.09 1147 2743.09 1147ZM2903.1 714.023 2939.1 714.023 2939.1 1147 2903.1 1147ZM3063.1 704.023 3099.1 704.023 3099.1 1147 3063.1 1147ZM3223.11 704.023 3259.11 704.023 3259.11 1147 3223.11 1147ZM3383.11 654.021 3419.11 654.021 3419.11 1147 3383.11 1147ZM3543.12 643.021 3579.12 643.021 3579.12 1147 3543.12 1147ZM3703.12 542.018 3739.12 542.018 3739.12 1147 3703.12 1147ZM3863.13 780.026 3899.13 780.026 3899.13 1147 3863.13 1147ZM4023.13 548.018 4059.13 548.018 4059.13 1147 4023.13 1147Z" fill="#ED7D31"/></g><g clip-path="url(#clip4)"><path d="M1537.5 1146.5 4098.5 1146.5" stroke="#D9D9D9" stroke-width="3.4375" stroke-linejoin="round" stroke-miterlimit="10" fill="none" fill-rule="evenodd"/></g><g clip-path="url(#clip5)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 1478.08 1158)">0</text></g><g clip-path="url(#clip6)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 1415.38 1037)">2000</text></g><g clip-path="url(#clip7)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 1415.38 916)">4000</text></g><g clip-path="url(#clip8)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 1415.38 795)">6000</text></g><g clip-path="url(#clip9)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 1415.38 674)">8000</text></g><g clip-path="url(#clip10)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 1394.48 553)">10000</text></g><g clip-path="url(#clip11)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 1394.48 432)">12000</text></g><g clip-path="url(#clip12)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 1606.84 1212)">1</text></g><g clip-path="url(#clip13)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 1766.93 1212)">2</text></g><g clip-path="url(#clip14)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 1927.01 1212)">3</text></g><g clip-path="url(#clip15)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 2087.1 1212)">4</text></g><g clip-path="url(#clip16)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 2247.19 1212)">5</text></g><g clip-path="url(#clip17)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 2407.28 1212)">6</text></g><g clip-path="url(#clip18)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 2567.36 1212)">7</text></g><g clip-path="url(#clip19)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 2727.45 1212)">8</text></g><g clip-path="url(#clip20)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 2887.54 1212)">9</text></g><g clip-path="url(#clip21)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 3037.18 1212)">10</text></g><g clip-path="url(#clip22)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 3197.26 1212)">11</text></g><g clip-path="url(#clip23)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 3357.35 1212)">12</text></g><g clip-path="url(#clip24)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 3517.44 1212)">13</text></g><g clip-path="url(#clip25)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 3677.53 1212)">14</text></g><g clip-path="url(#clip26)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 3837.61 1212)">15</text></g><g clip-path="url(#clip27)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 3997.7 1212)">16</text></g><g clip-path="url(#clip28)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="64" transform="matrix(1 0 0 1 1735.11 267)">snpe_bench inference time comparison between 1.61 and 1.64 on larger more </text></g><g clip-path="url(#clip29)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="64" transform="matrix(1 0 0 1 2543.02 345)">accurate models</text></g><g clip-path="url(#clip30)"><rect x="2534" y="1280" width="23" height="23" fill="#4472C4"/></g><g clip-path="url(#clip31)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 2566.4 1303)">mean 1.61</text></g><g clip-path="url(#clip32)"><rect x="2786" y="1280" width="23" height="23" fill="#ED7D31"/></g><g clip-path="url(#clip33)"><text fill="#595959" font-family="Calibri,Calibri_MSFontService,sans-serif" font-weight="400" font-size="41" transform="matrix(1 0 0 1 2818.75 1303)">mean 1.64</text></g><rect x="1365.5" y="172.5" width="2784" height="1186" stroke="#D9D9D9" stroke-width="3.4375" stroke-linejoin="round" stroke-miterlimit="10" fill="none"/></g></svg>

medium.svg

perf.md

SNPE Performance Comparison between 1.61 and 1.64

I upgraded my Qualcomm 888 testing pipeline to a new version of the SNPE SDK moving from snpe-1.61.0 to snpe-1.64.0. Then re-ran all our 8000 models we have been testing. The results are mixed. On the one hand snpe-1.64.0 seems to have fixed some quantization bugs such that a lot more models are getting an interesting non-zero F1 accuracy score and all scores are the same or better than before, but on the other hand the mean inference times have slowed down some. The slow down is more evident in the smaller faster models. So my question is for smaller models the perf hit is pretty bad, is this a known issue and something that will likely improve in future versions of the SDK ?

Here's the results on the smaller faster models:

Here's the result on medium sized models:

Here's the results on larger models: