Skip to content

Instantly share code, notes, and snippets.

@codecat
Last active May 11, 2018 16:28
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save codecat/9128e290b7f3fc74374a3035ff79e9af to your computer and use it in GitHub Desktop.
Save codecat/9128e290b7f3fc74374a3035ff79e9af to your computer and use it in GitHub Desktop.
MSVC 2017 optimization bug

This is reproduced on Visual Studio 2017 15.7.1.

  1. Download Angelscript from here: http://www.angelcode.com/angelscript/sdk/files/angelscript_2.32.0.zip
  2. Extract it to some folder, for me this is D:\Dev\msvc_bug_test
  3. Open sdk/angelscript/projects/msvc2015/angelscript.sln in Visual Studio 2017
  4. It will ask to upgrade the project to SDK 10.0.17134.0 and toolset v141, so we do that
  5. We switch the build to Release and build the project
  6. Close the Visual Studio solution, and make a new solution/project, I'll call it bugtest
  7. Switch to Release configuration
  8. Switch the runtime library to /MT in order to match Angelscript
  9. Add include/library directories to the project:
d:\Dev\msvc_bug_test\sdk\angelscript\include\
d:\Dev\msvc_bug_test\sdk\angelscript\lib\
  1. Create a new file main.cpp and add this code:
#include <cstdio>
#include <cassert>

#include <angelscript.h>
#pragma comment(lib, "angelscript.lib")

static void MessageCallback(const asSMessageInfo* msg, void* param)
{
  printf("%s\n", msg->message);
}

int main()
{
  int r = 0;

  asIScriptEngine* engine = asCreateScriptEngine();

  r = engine->SetMessageCallback(asFUNCTION(MessageCallback), nullptr, asCALL_CDECL); assert(r >= 0);

  asIScriptModule* mod = engine->GetModule("Scripts", asGM_CREATE_IF_NOT_EXISTS);
  r = mod->AddScriptSection("Test", "class Foo { void Bar() { } } void Main() { Foo f; f.DoesNotExist(); }"); assert(r >= 0);
  r = mod->Build();

  return 0;
}
  1. Put a breakpoint on the printf call in MessageCallback
  2. Run the application, the breakpoint should get hit
  3. Switch to disassembly view
  4. Step out of the callback function
  5. Observe return assembly: (this is where the optimization bug occurs)
00340EB8 FF 75 0C             push        dword ptr [param2]  
00340EBB FF 75 08             push        dword ptr [param1]  
00340EBE FF D6                call        esi  
  }
}
00340EC0 8B 4D F4             mov         ecx,dword ptr [ebp-0Ch]  
00340EC3 64 89 0D 00 00 00 00 mov         dword ptr fs:[0],ecx  
00340ECA 5E                   pop         esi  
00340ECB 8B E5                mov         esp,ebp  
00340ECD 5D                   pop         ebp  
00340ECE C2 10 00             ret         10h  
  1. Note that pop esi is supposed to pop the thisptr back. Instead, it's the value of param1
  2. Step out of the function
  3. Observe the return assembly:
	preMessage.isSet = false;
00348A86 8B 45 08             mov         eax,dword ptr [section]  
00348A89 C6 86 BC 0B 00 00 00 mov         byte ptr [esi+0BBCh],0  
  1. Note that preMessage is a field in the class, and esi here is used as the thisptr to set it to false, but esi is not the actual thisptr, it's some value on the stack
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment