Background on .NET Framework and .NET Core proxy support

DotNetProxySupport.cs

// WebRequest started it all. HttpWebRequest, FtpWebRequest, FileWebRequest, etc.

// Initial values when starting. The represents an object wrapping the platform/system proxy configuration settings.
// On Windows, these settings are from the IE setting dialog ("Wininet settings").
Console.WriteLine(WebRequest.DefaultWebProxy); // Internal type "System.Net.WebRequest+WebProxyWrapper"
Console.WriteLine(WebRequest.DefaultWebProxy.Credentials); // "null"

// HttpWebRequest has a single property, 'Proxy'. If null, it should not use a proxy. If non-null, then it
// should use a proxy. The initial value is non-null and the proxy to use is the platform/system proxy.
HttpWebRequest hwr = WebRequest.CreateHttp("http://www.contoso.com");
Console.WriteLine(hwr.Proxy); // Internal type "System.Net.WebRequest+WebProxyWrapper"
Console.WriteLine(hwr.Proxy.Credentials); // "null"

// WebClient was invented after HttpWebRequest. It was meant as a simpler API compared to HttpWebRequest.
// It also supported a different async pattern (event based) compared with APM (Asynchronous Programming
// model, i.e. Begin/End) of HttpWebRequest.
// 
// WebClient also has a single property, 'Proxy'.
WebClient client = new WebClient();
Console.WriteLine(client.Proxy); // Internal type "System.Net.WebRequest+WebProxyWrapper"
Console.WriteLine(client.Proxy.Credentials); // "null"

// Changing the properties of WebRequest.DefaultWebProxy affect all WebRequest and WebClient based objects.
var credential = new NetworkCredential("username", "password");
Console.WriteLine($"{credential.UserName} {credential.Password}"); // "username password"
WebRequest.DefaultWebProxy.Credentials = new NetworkCredential("username", "password");

HttpWebRequest hwr2 = WebRequest.CreateHttp("http://www.contoso.com");
var hwrProxyCred = (NetworkCredential)hwr2.Proxy.Credentials;
Console.WriteLine($"{hwrProxyCred.UserName} {hwrProxyCred.Password}"); // "username password"
WebClient client2 = new WebClient();
var clientProxyCred = (NetworkCredential)client2.Proxy.Credentials;
Console.WriteLine($"{clientProxyCred.UserName} {clientProxyCred.Password}"); // "username password"

// HttpClient was introduced in .NET Framework 4.5. It was the next iteration of HTTP APIs. It introduced a
// asynchronous API pattern (Task, async, await). It was the first new API family to not have any synchronous
// networking or I/O APIs.
//
// During this time, we wanted to reduce the dependency of .NET having to do complex proxy calculations on the
// machine. Not only was it reading IE settings from the registry, but it was also using WPAD protocol to find
// a JavaScript file, PAC (Proxy Auto Config) file. It then was running a JavaScript interpreter in-process of
// .NET Framework. .NET Framework moved to a new model which was introduced across Windows by using a new
// service process call WinHttp AutoProxy Discover Service. This was an out-of-process service that could
// find and process PAC files. Using this service was considered safer and faster and could be used by all
// processes on the system.
//
// As part of that plan, we considered that when designing HttpClient and HttpClientHandler and the properties
// which controlled proxy usage. We wanted to move away from exposing the system/platform proxy object. 
// HttpClient introduced 2 properties to control proxy usage. 'UseProxy' was a boolean.  If 'true' then
// use the proxy specified by the 'Proxy' property. The 'Proxy' property was an IWebProxy object. The default
// value is null.  For HttpClient/HttpClientHandler, 'null' means use the platform/system proxy.  Non-null
// means to use that specific custom proxy that a developer might set.
var handler = new HttpClientHandler();
Console.WriteLine(handler.UseProxy); // 'true'
Console.WriteLine(handler.Proxy == null ? "null" : handler.Proxy.ToString()); // "null"

// But where to store credentials when using platform/system proxy?

// Since HttpClientHandler was using HttpWebRequest underneath, it ended up having a dependency on the static
// WebRequest.DefaultWebProxy property. So, this became a pattern we documented for users.
WebRequest.DefaultWebProxy.Credentials = new NetworkCredential("username", "password");
// or
WebRequest.DefaultWebProxy.Credentials = CredentialCache.DefaultCredentials;

// So, now the creation/use of HttpClient/HttpClientHandler would still have:
//HttpClientHandler.UseProxy == true
//HttpClientHandler.Proxy == null
// which means to use the platform/system proxy. But now, it will have a set of credentials that can be
// passed to it.

// We later discovered that doing this new design added confusion. We should have kept the single property for 'Proxy'.

// During .NET Core development, we realized this and added a new property, HttpClientHandler.DefaultProxyCredentials
// which is only used in cases where 'UseProxy==true', 'Proxy==null', i.e. using the platform/system proxy.
//

// This is the current state of the HttpClient APIs.

shouldn't this be on docs. somewhere?

shouldn't this be on docs. somewhere?

Thanks for your comment. Yes. We are working on draft text to add to our conceptual docs on the set of HTTP APIs regarding these different APIs.

ah awesome!

Can you elaborate on how to make a direct (no proxy) request using HttpClient/HttpClientHandler? This is a scenario that seems to be overlooked above and in most of my reading today. As this is one of the best resources for explaining the various mechanisms for using proxies, I'm asking here as it will make this more complete and a solid reference.

I need to migrate an old library that is mostly based on HttpWebRequest to purely HttpClient/HttpClientHandler usage and still support multi targeting net48 and netstandard2.0 (netcoreapp3.1 too with a forward view)

I need to support 4 scenarios: (excuse the code blocks; markdown seems to have a problem with the '#' symbol in my code blocks as single blocks)

1. System settings:
   httpClientHandler.UseProxy = true;
httpClientHandler.Proxy = null;

2. User specified proxy (with and without credentials)
   #if NETSTANDARD2_0
httpClientHandler.UseProxy = true;
httpClientHandler.Proxy = new WebProxy(proxyUri);
#elif NETCOREAPP3_1 //probably worth setting when we can
HttpClient.DefaultProxy = new WebProxy(proxyUri);
#endif

3. System settings but override credentials - use WebRequest.DefaultWebProxy.Credentials for net48 and netstandard2.0,
   httpClientHandler.UseProxy = true;
httpClientHandler.Proxy = null;
#if NETSTANDARD2_0
WebRequest.DefaultWebProxy.Credentials = myCredentials;
#elif NETCOREAPP3_? //vNext? - not yet available
HttpClientHandler.DefaultProxyCredentials = myCredentials;
#endif

4. No proxy at all - direct to the internet overriding any system settings.
   ???

Just to compound the confusion:
In HttpWebRequest it appears that it used to be the case that you would set the proxy to the result of GlobalProxySelection.GetEmptyWebProxy()
IWebProxy myProxy = GlobalProxySelection.GetEmptyWebProxy();
GlobalProxySelection.Select = myProxy;
but GlobalProxySelection is now deprecated saying "Use 'null' instead of GetEmptyWebProxy" however the docs for HttpWebRequest.Proxy says it throws a ArgumentNullException if Proxy is set to null.

In WebClient it also says that we should use the value 'null' to indicate not to use a proxy and go direct. but on the very same page it also says that it throws an ArgumentNullException if Proxy is set to null.

But in HttpClient, "Proxy = null;" means "Use System settings"? How then do I say, "don't use a proxy regardless of what system settings say"?

Can you elaborate on how to make a direct (no proxy) request using HttpClient/HttpClientHandler?

If you never want to use a proxy then simply do this:

var handler = new HttpClientHandler();
handler.UseProxy = false;
var client = new HttpClient(handler);

// Do stuff
HttpResponseMessage response = await client.GetAsync("http://example.com");

Also, I recommend if you have further questions about .NET Core, that you open up an issue on https://github.com/dotnet/runtime/issues.

wow, I feel pretty stupid for missing that, it could hardly be more obvious :)