Concept: DDEV WSL Integration based on Interop

ddev-inside-wsl.md

DDEV Interop Based WSL Integration

Contents

• 1 Introduction and Objectives
  • 1.1 Terms of Reference
  • 1.2 Context Delimitation
  • 1.3 Quality Objectives
  • 1.4 Quality Scenarios
• 2 Boundary Conditions
• 3 Context
  • 3.1 Involved Components
  • 3.2 Technical / Distribution Context
• 4 Consideration of the actual state
  • 4.1 Description
  • 4.2 Evaluation
• 5 Recommendations for action
  • 5.1 Install DDEV in Windows
  • 5.2 Simplify the installation by suggesting the Windows Installer
  • 5.3 Stay with Docker Desktop
• 6 Conclusion
• 7 Appendix
  • 7.1 Sources and References

1 Introduction and Objectives

1.1 Terms of Reference

The goal of this concept is to simplify the installation process and use of DDEV in a WSL Distribution hosted on Windows.

1.2 Context Delimitation

This concept focuses on the installation and usage of DDEV inside WSL. The installation and use of DDEV on any other operating system than Windows is NOT part of this concept.

The usage of DDEV on Windows without WSL is mentioned but also not part of this concept.

Windows Versions below Windows 10 21H1 are not covered by this concept.

1.3 Quality Objectives

Priority	Quality objective	Description
1	Complexity	The complexity should be reduced to allow a simple setup and usage of DDEV.
2	Stability	All included components have to be stable to guarantee productivity when using DDEV.
3	Interoperability	One of the main concepts of WSL is the interoperability between the sub-system and the host-system. This principle should be should be preserved.
4	Maintainability	This concept should reduce maintenance efforts of DDEV.
5	Performance	The performance of DDEV may not be affected negatively.
6	Licenses	When possible open source and "free to use" licensed software should be used.

1.4 Quality Scenarios

• As a first time user I want to be able to install and use ddev inside WSL within less than 5 Minutes.
• As a web-developer I want to be able to use ddev inside WSL without any drawbacks compared to other installation methods.
• As a web-developer working on both native Windows and WSL I want to use ddev without administration overhead.
• As a web-developer working with ddev inside WSL I expect nearly the same performance as on a native Linux.
• As a maintainer of ddev I do not want to have additional efforts for supporting WSL beside native Windows support.

2 Boundary Conditions

DDEV has a large ecosystem, proposed changes should not cause any backwards compatibility issues.

3 Context

3.1 Involved Components

DDEV

DDEV is an open source tool which allows you to get local PHP development environments up and running within minutes. It is based on per-project environment configurations, which can be extended, version controlled, and shared.

Windows

Microsoft Windows is a graphical operating system developed and published by Microsoft.

WSL

The Windows Subsystem for Linux lets developers run a GNU/Linux environment -- including most command-line tools, utilities, and applications -- directly on Windows, unmodified, without the overhead of a traditional virtual machine or dualboot setup.

Interop

WSL Interop is a software component that allows the interoperability between the WSL and the Windows host system. It allows the translation of paths and the exchange of environment variables and network settings, also it allows you to run Windows executables inside WSL and vice versa. It is a WSL feature that is enabled by default.

Windows Terminal

Windows Terminal is a terminal application that allows you to easily start Powershell or CMD sessions on your Windows host, or start a shell session inside one of your WSL distributions. Since Windows 11 it is the default terminal.

Docker

Docker is an open platform for developing, shipping, and running applications. Docker provides the ability to package and run an application in a loosely isolated environment called a container. The isolation and security allows you to run many containers simultaneously on a given host.

Docker Desktop

Docker Desktop is an application for Mac or Windows environments, that enables you to build and share containerized applications and microservices. Docker Desktop includes Docker Engine, Docker CLI client, Docker Compose, Docker Content Trust, Kubernetes, and Credential Helper. Docker Desktop ships with a WSL Backend for running Docker.

Chocolatey

Chocolatey is an open source project that provides a package manager for Windows targeting developers and admins.

mkcert

mkcert is a tool to generate locally-trusted development certificates.

Homebrew on Linux

Homebrew on Linux is a package manager for linux.

3.2 Technical / Distribution Context

The considered setup is one or more WSL Distributions installed on a Windows Host. Docker Desktop is installed in Windows with enabled WSL Backend.

The WSL Interop Feature is enabled (System default) as an API Interface for communication between the Windows and the WSL environment.

The DDEV project resides in the Linux file system of a WSL distribution.

4 Consideration of the actual state

4.1 Description

The DDEV documentation currently suggests installing Docker for Desktop on Windows and mkcert and ddev inside the WSL distribution.

For installing ddev inside WSL the documentation suggest using Homebrew on Linux .

Consideration is being given to adapting the documentation so that instead of installing Docker Desktop on Windows, Docker is installed inside the WSL distribution.

Due to the fact, that ddev is installed in the WSL distribution, it is not possible to execute any commands from a Windows command line.

Running ddev inside the WSL distribution causes the adaptions of the hosts file to happen in the /etc/hosts file of the WSL distribution instead of the Windows one. The /etc/hosts file will be regenerated on WSL restart by representing the state of the Windows equivalent and the changes will be lost.

DDEV has to be managed inside the WSL using Homebrew on Linux and can not be managed from the Windows settings.

mkcert needs to be installed in both Windows and WSL to make SSL work properly when accessing a DDEV environment from a Client from Windows environment, additionally environment variable exports are necessary.

Installing docker inside the WSL distribution will increase the isolation of the system.

The setup needs to be done for every newly installed WSL distribution.

4.2 Evaluation

The current setup workflow does work but contains some problems.

One problem is, that host entries have to be manually done, if you are using ddev inside wsl.

There are optional requirements like mkcert that should be declared optional. Also package managers like Homebrew on Linux and Chocolatey are not mandatory for the usage of ddev and should be declared as helper utilities.

The current isolation of ddev inside WSL conflicts with the paradigm of interoperability, allowing you to easily use Windows executables in WSL and vice versa. The management of your installed software should mainly happen in Windows, as this is the main system you are working with.

5 Recommendations for action

5.1 Install DDEV in Windows

Instead of installing ddev and its dependencies inside WSL this should be done inside Windows. Execution of ddev commands is then possible inside WSL by executing ddev.exe, as interop will proxy them to the ddev binary installed in Windows. Docker Desktop will use its WSL Backend for running the containers.

The advantages of this approach include

• Host entries will be made to the Windows host file, so no manual step is necessary
• DDEV will be available in all WSL Distributions with zero configuration
• DDEV is available on Windows command lines
• DDEV can be managed in Windows, including the "Apps and Features" Windows settings.
• The WSL specific mkcert setup steps are optional (Still required if the certificate needs to be trusted in WSL environment)

Known limitations are

• DDEV has to be accessed by ddev.exe inside WSL to use Interop, this can be bypassed by an alias.
• Docker Desktop is required to run docker from Windows, which has commercial licensing for bigger companies.

5.2 Simplify the installation by suggesting the Windows Installer

Using a package manager like Chocolatey is a good way to install your software and keep it up to date, The possibility to use Chocolatey should definitely be mentioned but is relevant only for advanced Windows users.

Suggesting the installer over a package manager will narrow down the documentation a lot, as the installation of Chocolatey does not need to be explained and also the setup of mkcert is included in the installer. Setting the required environment variables for using mkcert inside WSL can be included in the installer to.

This will also result in a GUI only setup workflow as an alternative to the CLI only workflow using Chocolatey.

Setting the mkcert environment variables could also be done in a Chocolatey post-install script, if mkcert is declared as a dependency of ddev.

5.3 Stay with Docker Desktop

Docker Desktop provides a good implementation of docker on Windows with good performance and WSL integration. Even WSL will advise you to install Docker Desktop and to enable it four this distribution, if you run "docker -v" in a WSL distribution without docker available.

6 Conclusion

Following these recommendations can reduce the required amount of documentation for the setup of DDEV inside WSL. It will also reduce the complexity, as ddev has not to be installed multiples times, when using DDEV in Windows and WSL or in multiple WSL distributions, which could result in different versions across the system.

In this way DDEV does follow the idea of interoperability as it allows its use in Windows and WSL seamlessly.

Homebrew for Linux is no longer part of the setup, which will reduce complexity too.

The changes do not require any changes to the ddev binary, as most of the proposed changes concern the documentation only.

The recommended setup has a hard dependency on Docker Desktop which is not open source. This seems to be tolerable as it will be free to use for most DDEV users, also it is a defacto standard for any non Linux operating system. Docker Desktop is already the suggested way of providing docker functionality in the DDEV documentation at this point in time.

An open topic is the requirement of the .exe extension when executing ddev.exe inside WSL. To be compatible to existing documentation an alias has to be created, which is still a manual step. This might be automated in some way too.

The user experience could be further improved by automating the WSL setup for mkcert by updating the installer and / or extend the installation script for the Chocolatey installer. This may also reduce some lines of documentation.

The performance will still be the same as with the currently suggested setup, as Docker Desktop will still use the WSL backend and the project will still reside in the Linux file system.

7 Appendix

7.1 Sources and References

• DDEV-Local Documentation
• What is Windows Subsystem for Linux | Microsoft Docs
• Working across file systems | Microsoft Docs
• Docker overview | Docker Documentation
• Docker Desktop overview | Docker Documentation
• Chocolatey Software
• FiloSottile/mkcert
• Homebrew on Linux — Homebrew Documentation

ddev-wsl-context.drawio.svg

ddev-wsl-recommendation.drawio.svg

logo.png

You say "install ddev on Windows". Are you saying to install the Linux binary in Windows (which is an interesting concept). It looks like you're talking about ddev.exe though...

So doesn't this approach lose all the speed and usage advantages of WSL2?

How is this actually different from current Windows support? Where is WSL2 used except as a terminal?

Sorry if I was not precise. WSL Interop allows you to run Windows binaries (.exe) files from within the WSL. All paths will be "translated".
Available in WSL are all binaries that are in the Path in Windows.
I think they decided that the .exe extension is required when calling the binary to avoid conflicts with existing linux binaries in path.
The performance will be great, as the project is still in WSL (Linux file system) and you do NOT have any windows file system mounted.

@rfay (just to make sure you get notified ;))

There are a lot of complexities that are associated with Windows only. So when running ddev.exe (windows binary) you'd end up with some pathing issues and such, and the OS would be detected wrong.

Have you actually tried this? It's an amazing concept.

Yes I did try it. It worked well so far. But there might be edge cases I didn't think of.
Just install a new WSL Distro (Enable it in Docker Desktop, as only the default distro is supported by default).
Check out a project in there and run (ddev.exe config if not yet initialized) and ddev.exe start from a bash in wsl.

Edit:
After setting up DDEV in Windows with installer or choco

I don't think that anything about this approach argues for Docker Desktop, interestingly enough. Using ddev.exe should work exactly the same with docker-inside. (or basically either way).

Regarding Performance: I noticed that in the docker-compose file there will be the windows representation of the path to the WSL. So I was concerned that docker might try to mount the windows file system in which the linux one is mounted which would have caused bad performance.
Sadly I have on my private computer only small projects, but there is no negative performance impact noticeable. In both setups the loading time is about 20ms. Yes, we need to check with something bigger with more file system access.

In case these paths become a problem this might be fixed in ddev.

I don't think that anything about this approach argues for Docker Desktop, interestingly enough. Using ddev.exe should work exactly the same with docker-inside. (or basically either way).

This would break the system, as the ddev.exe will return with error code 1 as no docker deamon is available. As you can run multiple WSL Distributions, binaries from WSL can not be exported in your windows path. To call a linux binary from windows you have to call wsl -d DISTRIBUTION docker which needs to be implemented in the application.
Also this would destroy the zero configuration setup for new WSL Distros as you will then have to setup and configure docker in there, which will cause following problems.

Please note the diagram in 5.1 which shows the importance of the Docker Desktop WSL Backend to work, as all communicaton with docker is done on the windows side. Only the mount is happening for the Linux File System to preserve a good performance. As noted above this has to be verified, as the behavior is slightly different then expected.

One error I came across is a permission error when executing yarn:

nl@XPS-9510-NL:~/projects/acol$ ddev.exe yarn build
Unable to remove \\wsl.localhost\Ubuntu-22.04\home\nl\projects\acol\.ddev\.global_commands: CreateFile \\wsl.localhost\Ubuntu-22.04\home\nl\projects\acol\.ddev\.global_commands\host\yarn: Access is denied.

But it doesn't seem to have any impact. Running the commands works fine.

All-clear regarding performance, from my side. I've used the administration of some CMS system with some file system IO, but it looks like my concep has proven right as there are no notable differences in performance.
We still mount the linux file system, even the paths are in windows format. :)

ddev (WSL)

ddev.exe (Windows called from WSL)

I have to say... it will take a lot of time and a lot of convincing to move from running linux-native (linux ddev inside wsl2) to running ddev.exe inside WSL2. There are so very many workarounds in the Windows code, all awkward, all different from everywhere else. The number of workarounds for WSL2 is minor, because it's linux.

So I'll keep thinking about this, but it won't be an easy sell. It's just backwards, I'm afraid. What I'd rather be doing would be proxying Windows into WSL2.

Yes, I understand that. I also prefer Linux and Linux binaries in general. It's easier to develop software that is Linux-centric, and even easier to develop software that is Linux-only.
But currently from a perspective of a Web-Developer working on Windows with WSL, the proposed approach does offer less configuration with more features and same performance.
As long as you don't want to give up Windows support in general, you won't get rid of Windows workarounds anyway.

What would you think of a pull request that updates the documentation as recommended in the concept and lets people try it out to gather some feedback?
(It will not be merged until there is a decision of course.)

No, we'd have to completely set up testing to validate that it can even pass tests. I doubt that it can. And it would require another permutation of testing. Currently we only test WSL2 with Docker Desktop and Ubuntu WSL2 ddev.

You're welcome to publicize a technique using something less formal than the ddev docs, Stack Overflow, a gist like you have, or whatever. I don't mind people trying out something and seeing if they like it, but in general things that go into the docs need full testing too. Which, of course, means we need a full testing permutation for WSL2 with docker inside, which we don't have yet.

I've created some small documentation for the workflow I described in the concept here.
You can find it here: https://gist.github.com/nico-loeber/bbdae25d1bf6862c791d7ac6cf6fce58

I will try to find some people who are willing to help me test it in production conditions.
Maybe you find some time to try it out yourself, @rfay.

I have to confess that the more I think about this, the less likely I am to suggest using ddev.exe inside WSL2. You have no idea how many special cases there are for windows, that are already really hard to maintain. And for WSL2... very few. Because it's linux.

Yes you already mentioned those special cases. But as I already tried to say, this way of installation should not introduce any new special cases. And the only way to get rid of maintaining specials for windows is dropping the support for windows. I really hope you do not concider this an option.

And you have to admit that this documentation is really short and simple.
Yes, it is less detailed as the official one and does not mention choco and trusting mkcert in WSL, but it will produce a minimal working setup.

This introduces all of the problems of Windows (and all the special cases) into WSL2 usage! And in the process convinces the code that it's running in Windows, not Linux.

You might try running make test in your setup and see how far you get. You'll probably have to adapt it because it's built to build where it is. I'm not optimistic.

I spent a little time with this concept today and I definitely find that a golang program compiled to exe will think it's Windows, so all the places that do IsWindows() or IsWSL2() will get the wrong idea. Golang's runtime.GOOS gets the go OS rather than the contextual OS I guess. There might be an alternate way to detect, but currently there would be a significant bit of misbehavior I think.

there would be a significant bit of misbehavior I think.

I would be really interested in the misbehavior you would expect.

If everything works as expected, isWindows() returning true, is right and shouldn't be a problem, as WSL Interop will care about the conversion of paths and environment variables and stuff like that. isWsl2() should no longer be needed theoretically.

A WSL CI Pipeline for tests would be really great. It is hard for me the confirm it is working, without knowing which features I have to test.