adamsmith/ms-outlook-addin-advice.txt

## ms-outlook-addin-advice.txt
(Hi, I'm Adam Smith.  I started Xobni in 2006, which has written the most sophisticated and successful MS Outlook addin in history.  This took millions of dollars and several years of work by some of the best programmers in the valley.  Other startups have thrown millions of dollars at building Outlook addins and failed.  This is my take on the idea of building a new MS Outlook addin as of 2012.)

My general advice is avoid any Outlook addins or desktop software if at all possible.  More and more stuff is moving to the web, and the table stakes of doing anything reasonably well in Outlook are remarkably high.  Consider it to be on the same level of making hardware with regards to the cost of bugs/mistakes, amount of black art, etc., but maybe a little higher since there's a much weaker ecosystem of people doing it.  I pasted below in this email an old list of "gotchas" or things-to-know I made after leaving Xobni.  It's not comprehensive but will include some of the biggest ones from a pain perspective.

The biggest thing it will take is an ability to recruit and filter experienced Windows developers.  From there, expect long product cycles and expect to invest over a long period of time as you run into a slew of bug fighting as Outlook 2003 v.s. 2007 v.s. 2010 v.s. 2013 on top of different Windows versions, antivirus configs, machine hardware (performance).

The biggest "shortcuts" are in knowing how to touch the relevant API's.  Minimize the use of COM API's (e.g. use MAPI), and don't use them from managed code; COM objects need their hands held through the entire lifecycle by unmanaged code.  Otherwise you run into performance and weird stability issues.

Another shortcut is to target later versions of Outlook, e.g. 2007+.

If you can avoid any kind of "syncing", ie maintaining your own synced representation of the email data in Outlook, that's a huge shortcut. Syncing with Outlook has tons and tons of pain points, depending on how real time you need to be, as backends can be Exchange, or IMAP, or POP/PST's.

But the biggest shortcut of all is avoiding Outlook.  : )  I would build my business around every other platform first, and then if it really is a huge pain point then start thinking about strategies.

Because MSFT open sourced a PST file format SDK it's also now possible to extract information from PST/OST files without using any Outlook API's.  If that works for your application then consider yourself extremely lucky!

The newer versions of Outlook/Exchange also have some cloud functionality.  If there are any cloud-to-cloud sync options they will be better.  Client software should be a last resort.


Adam
http://adamsmith.cc/


P.S. for more flavor of working with Outlook, see this 2009 blog post: http://adamsmith.cc/outlook-performance


P.P.S. the notes below are rather brief and cryptic.  Sorry about that.  I may go in and clean them up in the future but likely won't : /


MS Office tips
* Resiliancy registry keys -- deleting them
* VSTO -- avoid it (doesn't handle UI/non-UI modes properly, at least as of a previous version, but in general you want control, e.g. with resiliancy keys)
* PIAs -- 2002 had individual ones, 2003 is in an MSI; just use XP and add shit as necessary.  the 2003 PIA's are surprisingly hard to dissect and ship standalone.  or you could just ship the whole MSI.
* auto updating -- admin privs, windows service, I gently recommend not relying on an underlying CA store/use your own CA
* re-registering COM component and LoadBehavior via windows service
* Regfree COM doesn't work unless you create Outlook.exe.config or whatever
* In general avoid COM.  Avoid COM.  Avoid COM.  Except for your actual addin.
* Use a shim for your own appdomain, from the LATEST shim wizard.  authenticode sign it (and any other unmanaged assemblies.)
* You can do cool things in the shim, like measuring startup time, delaying startup, etc
* object model -- adding memory pressure trick
* object model & thread 0 -- UI responsiveness/non-responsiveness
* PIAs -- it's surprisingly hard to get the CLR to use YOUR PIA's -- FULLY customize all the GUIDs, and I mean fully, and custom Startup(object) (Instance) marshaling.  also, explain why you want the CLR to use your interop assemblies.
* shutting down -- UI mode, non-UI mode, releasing trick
* ReleaseCOMObject -- suggest using mixed-mode C++ to directly handle all object model objects; those objects are recycled across addins (at least in later versions), which is another great example of non-sandboxing of addins, and it's not always clear when the .net com wrapper AddRef()'s so you don't know how many times to ReleaseCOMObject()
* don't fork a process from an office process
* some older antivirus apps are thought to have false positives on the CLR's gzip libraries (data execution)
* load speed -- use the GAC.  once you're in the GAC you can run ngen (is this still best practice?)
* load speed -- async .net/etc load from the shim, progressive loading based on time, cpu/disk activity, and user behavior
* contractors -- for intense COM/win32 work, use separate repo
* app crashes -- top level catch all ThreadPool exceptions, and register the standard .NET exception handlers
* winqual for error reports, but of low value for us/managed apps, use esp if lots of unmanaged logic, inc potentially the installer etc
* windbg intro/cheat sheet, plus using debugview and OutputDebugString()
* if you're going to use a 3rd party COM component with xyz threading model that will marshal calls to its original thread, initialize it away from thread 0 so it doesn't block UI
* in general avoid using thread 0 as much as humanly possible.  e.g. create your own thread for UI drawing/updates/etc, that's not thread 0
* to make sure the CLR is using your interop assemblies, register the distro PIAs in the GAC, launch Outlook, attach with WinDbg, and then list out the modules and look at the path of any loaded IA's
* use preservesig if a com component returns error codes often that get marshaled by the clr as exceptions
* certain office api calls can block for unbounded amounts of time.  make sure your locks won't timeout on these.
* obfuscating managed code, deobfuscating stacktraces manually and automatically, inlining-aware, line number strings in exceptions
* detecting outlook version from out of app / installer -- most appropriate approach is to look in the windows installer database for the right guid(s) (?), but you can get by with checking registry keys
* exception watcher/webapp -- find some way to count and aggregate the most common exceptions.  dropbox for example has an in-house exception tracker.  more on implementation.
* installing the .net framework -- use xobni's open sourced NSIS plugin
* installer -- have a high res icon too for thumbnail viewing, authenticode signing in addition to strong signing assemblies, making sure UAC elevates you
* installer -- editing the template/etc file to get rid of or modify the "Nullsoft Installer" grey text, and to optimally move other stuff/strings/etc around (e.g. the xobni team picture at the end of installer flow)
* I'd suggest having some way to get to a simple debug command line or list of debug commands, available but hidden in field.  could be to a python prompt or something similar.
* emailing / working with MSFT office teams


Outlook specific ones
* hooking store modified events can cause an async infinite loop (only if there's a ghost store for a missing PST file?)
* don't forget outlook profiles / to handle them.  should be straightforward.
* multiple explorers, too (activeexplorer / etc)
* replying to messages -- use object model
* using reflection on OOM (current) .Selection object to get IsOutgoing for whatever status without marking it as "don't send yet"
* Redemption -- re-registering via windows service if corrupted (even if personalized, which is broken)
* out of process -- using OOM from OOP and avoiding security dialogs
* redemption had a significant problem with RPC handles and left-open objects.  (might also apply to other helper com objects)
* (generally avoiding redemption if possible.  as a theme many of the shortcuts one might take to speed up development like using Redemption or the OOM in managed code end up being not worth it in the end.)


More
* see comments (https://gist.github.com/3940222#comments) for more
	(Hi, I'm Adam Smith. I started Xobni in 2006, which has written the most sophisticated and successful MS Outlook addin in history. This took millions of dollars and several years of work by some of the best programmers in the valley. Other startups have thrown millions of dollars at building Outlook addins and failed. This is my take on the idea of building a new MS Outlook addin as of 2012.)

	My general advice is avoid any Outlook addins or desktop software if at all possible. More and more stuff is moving to the web, and the table stakes of doing anything reasonably well in Outlook are remarkably high. Consider it to be on the same level of making hardware with regards to the cost of bugs/mistakes, amount of black art, etc., but maybe a little higher since there's a much weaker ecosystem of people doing it. I pasted below in this email an old list of "gotchas" or things-to-know I made after leaving Xobni. It's not comprehensive but will include some of the biggest ones from a pain perspective.

	The biggest thing it will take is an ability to recruit and filter experienced Windows developers. From there, expect long product cycles and expect to invest over a long period of time as you run into a slew of bug fighting as Outlook 2003 v.s. 2007 v.s. 2010 v.s. 2013 on top of different Windows versions, antivirus configs, machine hardware (performance).

	The biggest "shortcuts" are in knowing how to touch the relevant API's. Minimize the use of COM API's (e.g. use MAPI), and don't use them from managed code; COM objects need their hands held through the entire lifecycle by unmanaged code. Otherwise you run into performance and weird stability issues.

	Another shortcut is to target later versions of Outlook, e.g. 2007+.

	If you can avoid any kind of "syncing", ie maintaining your own synced representation of the email data in Outlook, that's a huge shortcut. Syncing with Outlook has tons and tons of pain points, depending on how real time you need to be, as backends can be Exchange, or IMAP, or POP/PST's.

	But the biggest shortcut of all is avoiding Outlook. : ) I would build my business around every other platform first, and then if it really is a huge pain point then start thinking about strategies.

	Because MSFT open sourced a PST file format SDK it's also now possible to extract information from PST/OST files without using any Outlook API's. If that works for your application then consider yourself extremely lucky!

	The newer versions of Outlook/Exchange also have some cloud functionality. If there are any cloud-to-cloud sync options they will be better. Client software should be a last resort.


	Adam
	http://adamsmith.cc/


	P.S. for more flavor of working with Outlook, see this 2009 blog post: http://adamsmith.cc/outlook-performance


	P.P.S. the notes below are rather brief and cryptic. Sorry about that. I may go in and clean them up in the future but likely won't : /


	MS Office tips
	* Resiliancy registry keys -- deleting them
	* VSTO -- avoid it (doesn't handle UI/non-UI modes properly, at least as of a previous version, but in general you want control, e.g. with resiliancy keys)
	* PIAs -- 2002 had individual ones, 2003 is in an MSI; just use XP and add shit as necessary. the 2003 PIA's are surprisingly hard to dissect and ship standalone. or you could just ship the whole MSI.
	* auto updating -- admin privs, windows service, I gently recommend not relying on an underlying CA store/use your own CA
	* re-registering COM component and LoadBehavior via windows service
	* Regfree COM doesn't work unless you create Outlook.exe.config or whatever
	* In general avoid COM. Avoid COM. Avoid COM. Except for your actual addin.
	* Use a shim for your own appdomain, from the LATEST shim wizard. authenticode sign it (and any other unmanaged assemblies.)
	* You can do cool things in the shim, like measuring startup time, delaying startup, etc
	* object model -- adding memory pressure trick
	* object model & thread 0 -- UI responsiveness/non-responsiveness
	* PIAs -- it's surprisingly hard to get the CLR to use YOUR PIA's -- FULLY customize all the GUIDs, and I mean fully, and custom Startup(object) (Instance) marshaling. also, explain why you want the CLR to use your interop assemblies.
	* shutting down -- UI mode, non-UI mode, releasing trick
	* ReleaseCOMObject -- suggest using mixed-mode C++ to directly handle all object model objects; those objects are recycled across addins (at least in later versions), which is another great example of non-sandboxing of addins, and it's not always clear when the .net com wrapper AddRef()'s so you don't know how many times to ReleaseCOMObject()
	* don't fork a process from an office process
	* some older antivirus apps are thought to have false positives on the CLR's gzip libraries (data execution)
	* load speed -- use the GAC. once you're in the GAC you can run ngen (is this still best practice?)
	* load speed -- async .net/etc load from the shim, progressive loading based on time, cpu/disk activity, and user behavior
	* contractors -- for intense COM/win32 work, use separate repo
	* app crashes -- top level catch all ThreadPool exceptions, and register the standard .NET exception handlers
	* winqual for error reports, but of low value for us/managed apps, use esp if lots of unmanaged logic, inc potentially the installer etc
	* windbg intro/cheat sheet, plus using debugview and OutputDebugString()
	* if you're going to use a 3rd party COM component with xyz threading model that will marshal calls to its original thread, initialize it away from thread 0 so it doesn't block UI
	* in general avoid using thread 0 as much as humanly possible. e.g. create your own thread for UI drawing/updates/etc, that's not thread 0
	* to make sure the CLR is using your interop assemblies, register the distro PIAs in the GAC, launch Outlook, attach with WinDbg, and then list out the modules and look at the path of any loaded IA's
	* use preservesig if a com component returns error codes often that get marshaled by the clr as exceptions
	* certain office api calls can block for unbounded amounts of time. make sure your locks won't timeout on these.
	* obfuscating managed code, deobfuscating stacktraces manually and automatically, inlining-aware, line number strings in exceptions
	* detecting outlook version from out of app / installer -- most appropriate approach is to look in the windows installer database for the right guid(s) (?), but you can get by with checking registry keys
	* exception watcher/webapp -- find some way to count and aggregate the most common exceptions. dropbox for example has an in-house exception tracker. more on implementation.
	* installing the .net framework -- use xobni's open sourced NSIS plugin
	* installer -- have a high res icon too for thumbnail viewing, authenticode signing in addition to strong signing assemblies, making sure UAC elevates you
	* installer -- editing the template/etc file to get rid of or modify the "Nullsoft Installer" grey text, and to optimally move other stuff/strings/etc around (e.g. the xobni team picture at the end of installer flow)
	* I'd suggest having some way to get to a simple debug command line or list of debug commands, available but hidden in field. could be to a python prompt or something similar.
	* emailing / working with MSFT office teams


	Outlook specific ones
	* hooking store modified events can cause an async infinite loop (only if there's a ghost store for a missing PST file?)
	* don't forget outlook profiles / to handle them. should be straightforward.
	* multiple explorers, too (activeexplorer / etc)
	* replying to messages -- use object model
	* using reflection on OOM (current) .Selection object to get IsOutgoing for whatever status without marking it as "don't send yet"
	* Redemption -- re-registering via windows service if corrupted (even if personalized, which is broken)
	* out of process -- using OOM from OOP and avoiding security dialogs
	* redemption had a significant problem with RPC handles and left-open objects. (might also apply to other helper com objects)
	* (generally avoiding redemption if possible. as a theme many of the shortcuts one might take to speed up development like using Redemption or the OOM in managed code end up being not worth it in the end.)


	More
	* see comments (https://gist.github.com/3940222#comments) for more