Skip to content

Instantly share code, notes, and snippets.

@spevans

spevans/urlsession_proposal.md

Last active February 7, 2021 16:57
Show Gist options
  • Save spevans/6e0fe946506b564cb57fa31ea1d451b3 to your computer and use it in GitHub Desktop.
Save spevans/6e0fe946506b564cb57fa31ea1d451b3 to your computer and use it in GitHub Desktop.
Download ZIP
Proposal: swift-corelibs-foundation: Replace libcurl with SwiftNIO and AsyncHttpClient
Raw
urlsession_proposal.md

Proposal: swift-corelibs-foundation: Replace libcurl with SwiftNIO and AsyncHttpClient

[cc millenomi Tony_Parker]

Motivation

The swift-corelibs-foundation's (scl-f) URLSession implementation for protocols http and ftp currently uses libcurl, provided by the host's package system or built separately as part of the toolchain build (Windows). At the current time these 2 protocols still have bugs and require more work to fix up authentication, cookies, session metrics and some concurrency bugs in the general URLSession codebase. It will take significant effort to fully implement everything that is still outstanding.

Proposed Solution

I am proposing to replace the libcurl implementation with async-http-client (AHC) based on swift-nio.

URLSession/http would become a thin wrapper around AHC, which already implements most of the functionality required, although its delegate does need expanding to allow mapping to all of the URLSession delegate methods. This would allow using the development effort of AHC/NIO and not splitting off resources to maintain two full HTTP clients.

FTP support would be dropped unless an alternative FTP NIO client is developed.

AHC and NIO sources would be vended by scl-f meaning that a copy of the individual packages would be copied into scl-f's source tree and then the NIO packages renamed to avoid clashing with the normal NIO modules using a script. The modules would then be imported @_implementationOnly. This would allow Foundation / FoundationNetworking to still be used by applications using either NIO1 or NIO2 packages.

The source of the packages in the scl-f source tree would be manually updated as necessary.

Although a complete copy of the NIO sources would be built as part of scl-f, the extra binary size of including NIO in FoundationNetworking will still be smaller than linking to libcurl, which on Ubunutu currently links to about 40 non-system libraries

CMake would still be used to build scl-f so the NIO packages would require CMakefiles. The following packages would be used:

This would mean that NIO support would need to be finished on Windows and would be a future requirement for any platform that wanted to be supported by Swift.

This will not be a quick change to make as there are many parts that need to be implemented and the URLSession code that remains will be ported to async/await if that is found to be the best solution to fix concurrency issues. This will probably take at least a year.

Please note that even though FoundationNetworking will use NIO code this does not mean that FoundationNetworking or Foundation will become a Swift package itself. It will still remain part of the toolchain and be built with CMake as part of the toolchain build.

Main tasks

  • Convert scl-f from an Xcode project to Package.swift to make it easier to develop with packages. This DOES NOT mean scl-f itself will be a package, only that it is easier for development and testing builds of scl-f itself on Linux and macOS with packages.

  • Enhance AHC's client and delegate methods to provide all of the functionality that URLSession/http requires. This will also benefit clients of AHC.

  • Convert scl-f's _HTTPURLProtocol to use AHC.

  • Add CMakefiles for all of the NIO projects.

Acknowledgements

Thanks to johannesweiss, lukasa and graskind for ideas on using NIO/AHC and feedback on this proposal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment