Proposal for handling directories outside of "web" in pub build and pub serve.

gistfile1.md

Pub has two commands for working with transformers, build and serve. Both of those currently are hardcoded to only see stuff in your package's web/, asset/, and lib/ directories. We've been wanting to have support for test/, example, and others for a while (see #14673 and #15924). This sketches out what I'm thinking to handle this. Feedback is welcome!

The basic idea is that build and serve will be able to see of these directories: asset/, benchmark/, bin/, example/, test/, and web/. Transformers will be able to run on assets in any of those.

The build/ directory

Right now, pub build creates a build/ directory containing the output of the build process. That directory only contains the outputs whose path is within web/. If we start building tests and examples into there, stuff could start colliding.

So the first change is that we'll reorganize the build/ directory to match your package. Outputs within web/ will be within web/ in the build/ directory. Instead of:

build/index.html
build/myapp.dart.js

You'll have:

build/web/index.html
build/web/myapp.dart.js

This means if you have deploy scripts that read stuff out of build/, you'll need to change them to look in build/web/ when/if we make this change.

The pub build command

The main problem with running pub build on all of those directories is performance. If your test/ directory has hundreds of test files, you'll be waiting a long time for dart2js to compile all of them to JavaScript. If you're just trying to build and ship your app, and not run the tests as JS, that's a waste.

To handle that, we'll allow an argument to pub build to specify which directory (or directories) to build. I'm thinking something like:

pub build                   # Just build "web", like current behavior.
pub build web               # Equivalent to above.
pub build *                 # Build everything.
pub build test              # Build tests.
pub build web test example  # Build all three of those.
pub build example/foo       # Just build the "foo" example subdirectory.

I just made this up, though, so it probably needs some hashing out.

When you run this, it will still completely clean out your build directory. If you run:

pub build test

and then run:

pub build example

Your build/ directory will only have example in it. You could have made changes to stuff between those two pub build runs. If we left the previous build in there, you'd end up with a mixture of inconsistent outputs from different points in time.

The pub serve command

The dev server will follow this. It will serve files out of all of the above directories. This means the URL structure needs to change a bit. To hit your myapp.html file in web/ instead of:

localhost:8080/myapp.html

You'll do:

localhost:8080/web/myapp.html

That way we can have non-colliding URLs to reach stuff outside of web/ too.

Again, the challenge here is dart2js compile times. Pub serve waits until everything has been built once before it serves any assets (since it doesn't know which outputs will exist until the transformers have run). If we ran dart2js on all of your tests, the server would take forever to start up. Worse, touching a file imported by any of those tests would trigger a rebuild on all of them.

Our plan for that is to make the dart2js transformer run lazily. By default, it won't run the dart2js transformer on any entrypoints. You'll get a nice, fast startup, and you can keep iterating in dartium quickly without dart2js being in the loop at all.

When you do want to compile something to JS, it will determine this automatically. The first time your browser requests the compiled JS for some file (i.e. the first time pub serve gets a request for a .dart.js URL), pub serve will say, "OK, looks like I need to enable dart2js for that entryoint."

It will then enable dart2js to run on just that entrypoint. After that, it will keep it enabled. So when you make changes to Dart code, it will immediately start re-compiling it to JS for you.

Making the above work correctly in the general case is tricky, but I've got some ideas here that I think will work.

What do you think?

pub build *

Will this copy stuff like pubspec.yaml and lib/ into the output directory as well? If not, what happens when you just write e.g. pub build lib/? Will it silently do nothing?

localhost:8080/myapp.html

I think you mean localhost:8080/index.html (or else you meant myapp.html above).

Pub serve waits until everything has been built once before it serves any assets

This isn't strictly true; it can serve files that are finished being transformed while other transformers are still running, it just won't consistently transform files in any particular order.

Will this copy stuff like pubspec.yaml and lib/ into the output directory as well?

No, it just means "build all of the whitelisted directories": asset/, benchmark/, bin/, example/, test/, and web/.

If not, what happens when you just write e.g. pub build lib/? Will it silently do nothing?

Probably an error?

Can I get pub serve to emulate the path structure of my destination web server? e.g. I often host apps on example.com/ and I'd like to develop with my app running from / (not web/)

Or, I might have to deploy to example.com/deep/nested/dir so I structure the contents of my web/ dir to contain deep/nested/dir (but I don't want pub serve to look like /web/deep/nested/dir)

Different request:
can we also support

pub build web/example.html

or

pub build example/foo/a.html

The idea is similar to what you have with pub build example/foo. What I'm thinking, in terms of how pub works, is the following:

• all assets of the package are still available to transformers
• a special option is available to all transformers so they can know what was requested in pub build/pub serve.

For example, this means that polymer might be called to process all assets in the same way as if you called pub build *, except, that it can act differently and treat example/foo/a.html as an entrypoint only when it was listed explicitly.

Another example, for dart2js calling pub build web/file.dart would practically just compile that file, but ignore other .dart files even if they have a main.

@sethladd, that's not a bad idea although I don't know how far we want to go down that road. The goal with pub serve is to be a good dev server, but not to totally emulate your app's web server. You're probably better off just using your app's actual server and then proxying to pub serve in some way in that case.

Either way, file a bug?

@sigmundch, I can see us extending it to that, but I'd want to think about the semantics some more. I can see what you propose falling down if you have multiple transformers in play and they all think that path is for them and not for the other transformers. If you're keen on this, file a bug and we can hash it out.

FWIW I just ran into this case. I needed to create a link back to the root of my app. I created <a href="/"> which won't work with pub serve. I think we had another case where we would need to proxy back over to pub serve: for example when you are serving your content via app engine. I remember now we were talking about wiring up pub build or something like it to a file watcher. I basically want to live rebuilding of pub serve without the web serving bits. BYOHTTPServer in a sense.

I'm not very comfortable with this one:

When you run this, it will still completely clean out your build directory. If you run:

pub build test
and then run:

pub build example
Your build/ directory will only have example in it. You could have made changes to stuff between those two pub > build runs. If we left the previous build in there, you'd end up with a mixture of inconsistent outputs from different > points in time.

I think this output directories should be treated as independent.
Pub build should only delete what it currently intends to build.

pub build example
should only delete and rebuild build/example
There are no dependencies between build/example, build/test, build/web, ...
Why delete these folders prematurely?

There are no dependencies between build/example, build/test, build/web, ...

There are. They can all be importing code from lib or other packages. So if you aren't careful, you could end up with a single build directory that contains a example directory built using some code in your lib directory and a web directory that was built with different, changed code from lib.

I suppose if you know that that's true, you can get used to it, but I worry that it will just confuse people.

I created which won't work with pub serve.

That should work now, but now that you mention it, this change would break that. Is that OK?

I basically want to live rebuilding of pub serve without the web serving bits. BYOHTTPServer in a sense.

Sure, but that's tangential to this proposal. :)

@munificent I missed your response. It seems I still don't get it. Isn't the generated output (e. g. build/example or build/web) completely self-contained? What lib directory change would matter after the build is finished?

Yes, they are self-contained.

I think what I'm trying to say is that the contents of your entire build directory should represent the state of your package at a single moment in time (when you ran pub build). If we don't wipe out the directory on each build, then you could have a web directory in there from an hour ago and a test directory in there from a few days ago, etc.

I worry that will really confuse people. Maybe that's not such a big deal?

Ok, then we are talking of the same thing. The situation I think of is that I get some error in example or web that needs some fix in lib. While working with lib and test I need to again take a closer look at the misbehavior or reproduce the error message but this isn't possible anymore because I alredy changed something and rebuilt e. g. tests.

When I always rebuild all then I would expect consistency. When I build web I would find it surprising when other parts are affected not to talk about deleted.
I hope you rethink this topic.