Skip to content

Instantly share code, notes, and snippets.

@donmccurdy
Last active September 13, 2023 20:50
Show Gist options
  • Save donmccurdy/db75b86a28a775135e33317ce2694a32 to your computer and use it in GitHub Desktop.
Save donmccurdy/db75b86a28a775135e33317ce2694a32 to your computer and use it in GitHub Desktop.
RFC: Adopting JSDoc in three.js

RFC: Adopting JSDoc in three.js

Summary

Request for Comment (RFC) and a proposal for adopting JSDoc within the three.js project. The proposal intends to mitigate or solve two longstanding problems:

  1. Unofficial type definitions: Increasingly, many web projects and IDEs rely on type definitions in libraries, which three.js does not natively provide. The community maintains type definitions in a separate repository, with some overhead in cost and potential for mistakes or delays.
  2. Cost of maintaining localized documentation: API documentation for each class is maintained in hand-written HTML files, one file for each {class, language} pair. Maintainers contribute to the English-language documentation, while volunteers are largely responsible for identifying where English translations have changed and updating the localized versions.

Milestone 1: JSDoc Type Definitions

Required if RFC is accepted.

JavaScript source files will be annotated with comments in JSDoc format, describing only those type definitions required for public-facing APIs. Human-readable API descriptions are not present at this milestone. From the JSDoc comments, release builds will generate TypeScript type definitions published to npm within the library.

Example: `src/core/Object.js`
/**
 * @property {true} isObject3D
 * @property {string} uuid
 * @property {number} id
 * @property {string} name
 * @property {string} type
 * @property {Object3D|null} parent
 * @property {Object3D[]} children
 * @property {Vector3} up
 * @property {Vector3} position
 * @property {Euler} rotation
 * @property {Vector3} scale
 * ...
 */
class Object3D extends EventDispatcher {

	constructor() { ... }
	
	/**
	 * @param {WebGLRenderer} renderer
	 * @param {Scene} scene
	 * @param {Camera} camera
	 * @param {BufferGeometry} geometry
	 * @param {Material} material
	 * @param {any} group
	 */
	onBeforeRender() { ... }
	
	/**
	 * @param {WebGLRenderer} renderer
	 * @param {Scene} scene
	 * @param {Camera} camera
	 * @param {BufferGeometry} geometry
	 * @param {Material} material
	 * @param {any} group
	 */
	onAfterRender() { ... }
	
	/**
	 * @param {Matrix4} matrix
	 */
	applyMatrix4(matrix) { ... }
 
  ...
}

We'll add typescript to package.json#devDependencies, and create a tsconfig.json file with checkJS and allowJS options enabled. Editors like VSCode should then automatically provide hints when types are invalid. The code will run as expected in a browser, regardless of whether types are updated or compiled. At build time, TypeScript compiles the JSDoc comments to build/three.d.ts, including API definitions for the entire library — including three and three/addons/*. See dts-buddy for details on how this should work.

Milestone 2: Generate English-language API Documentation

Optional, and depends on Milestone 1.

In addition to type annotations, JSDoc comments will now include human-readable descriptions, in English, for each public-facing API item. In a build step, tooling will process the JSDoc comments or the d.ts files, and generate corresponding HTML pages identical to our current pages.

What about inline examples?

If desired, examples may be specified in one of two ways:

/**
 * Description of Mesh class here.
 *
 * @example link_to_example.html
 *
 * @example
 * Inline example below:
 * ```javascript
 * import { Mesh } from 'three';
 * const mesh = new Mesh( geometry, material );
 * ```
 */
class Mesh {
}

The former requires external files, and the latter is more verbose. Either will require some work on the HTML generator, which I believe is manageable. For simplicity, it may be preferable to leave these examples out of the API documentation and link to other examples or the manual instead.

Milestone 3: Generate Translated API Documentation

Optional, and depends on Milestones 2–3.

JSDoc or type definitions are processed in a build step, assigning a unique key to each English-language comment (e.g. Mesh#geometry). For each target output language, a human readable file (likely .csv or .yaml) is generated with content equivalent to:

Example: lang_it.csv

key enUpdated itUpdated en it
Mesh#geometry 2021-05-04 2021-10-15 An instance of BufferGeometry... Un'istanza di BufferGeometry...
Mesh#material 2021-05-04 2021-10-15 An instance of Material... Un'istanza di Material...

The lastUpdated column will be maintained by the generator, allowing translators to more easily find sections where the English comments have changed since the last translation. Optionally, translations may be modified in external software like Microsoft Excel or Google Sheets.

In a final build step, these translation files are processed, and translated HTML pages for API documentation are generated alongside the English-language versions.

Diagram

Flow diagram of the process described by this RFC

Source: https://www.tldraw.com/v/2dwr_7vxMpiVC0TedXNoNZGe?viewport=-314%2C-339%2C1158%2C869&page=page%3AaiaUOhQsX2ckmNRcLpS4R

Disadvantages

  • By generating API documentation automatically, the process for maintaining API docs diverges from the process for maintaining the manual and other written resources on the website.
  • JSDoc type annotations are admittedly verbose, and less ergonomic to write than TypeScript annotations. This problem could be mitigated if proposals for JavaScript Type Annotations are approved and implemented.

Alternatives considered

  • Rewrite three.js in TypeScript. None of the primary maintainers of three.js are currently interested in a change toward writing and maintaining TypeScript code.

  • Maintain d.ts type definitions manually. Manually updating d.ts files was attempted as a trial for several three.js releases. Cost of maintenance was prohibitive, and inability to validate hand-written d.ts files against source files created unacceptable problems.

Possible blockers

@DefinitelyMaybe
Copy link

Throwing spaghetti around... for Milestone 3, maybe there's a translation API that en updates could be feed into? AI's getting much better at this these days.

Liking efforts being made in this area. Sending best wishes 👍

@donmccurdy
Copy link
Author

@onsummer thank you! I hadn't run into tsd-jsdoc before, and will give that a closer look.


JSDoc types might not be accurate (like the use of any and object).

@AndrewRayCode JSDoc expresses types with similar precision to TypeScript definitions, and may be as strict or as loose as its authors require. Both support any, it's not unique to JSDoc. And we'll be using JSDoc to generate TypeScript type definitions, as explained above.

Optionally, you can have the TypeScript compiler check JSDoc as well. Currently three.js type definitions are maintained by the community (https://github.com/three-types/three-ts-types), by hand, separately from the source code, and it's difficult for volunteers to keep that in sync. Generating typescript type definitions from JSDoc improves on that process.

Also jsdoc auto generated documentation (or readthedocs, or anything similar) is only usually useful if you know exactly what you're looking for, it's not good for general documentation.

Have you read through the proposal above? We would not be generating documentation from JSDoc — the JSDoc is used to generate TypeScript type definitions, which are then parsed into machine-readable CSV or YAML files. The system will then "generate corresponding HTML pages identical to our current pages." The user-facing API documentation (e.g. https://threejs.org/docs/#api/en/cameras/Camera) will not change at all.

For user friendly documentation you probably need hand written docs either way.

API docs (like https://threejs.org/docs/#api/en/cameras/Camera) are best generated automatically. Tutorials (like https://threejs.org/docs/index.html#manual/en/introduction/Creating-a-scene) are best hand-written. If you prefer to use tutorials that is fine, of course, but we're not trying to get rid of either.


@DefinitelyMaybe thank you!

@ddnn55
Copy link

ddnn55 commented Jul 27, 2023

Support.

I was recently shocked to learn, via a docs tooltip in VS Code, that Three’s Group constructor() creates a new Bone!

@Methuselah96
Copy link

Methuselah96 commented Jul 28, 2023

Let me know if I can be of any assistance. As the current maintainer of @types/three, I'd be very glad to help with an automated solution. I had thought that the approach presented above was off the table based on previous discussions, but would definitely advocate for a more automated solution like the one presented above.

We had a contributor that was writing a tool to convert the docs pages to JSDoc comments for the definition files, which was time-consuming, error-prone, and not maintainable. Going from JSDoc to HTML docs and then adding the ability to translate that seems like a much better use of the community's time.

My main remaining question is what to do about "TS related conversations have been too noisy and distracting"? I would be happy to help moderate the types if that would be any help. I have been consistently been involved with maintaining the three.js types since 2018 and would love to help out in a way that mutually benefits the maintainers and community.

@BlueMagnificent
Copy link

Supported!

@Methuselah96
Copy link

Methuselah96 commented Jul 28, 2023

Frankly I worry about JSDoc not being as well supported by TypeScript with weird issues that don't occur in normal TypeScript (some personal examples: 1, 2). If I were making a proposal, I think adding back the .d.ts files and adding the JSDocs to the declaration types will avoid some of the headache that will come with hand-holding the TypeScript definitions that are generated from the JSDocs and make the more complicated types easier to express the way we want to. It also prevents the JS from getting cluttered with JSDoc comments and allows more fine-grained control about whether a method or type is exposed or not.

@Methuselah96
Copy link

If the concern about TS conversations being too noisy remains and the idea of a build step for the HTML docs is acceptable, then maybe a separate repo (like three-ts-types) that hosts the .d.ts+JSDocs from which the HTML docs are generated from might be a worthwhile compromise.

@donmccurdy
Copy link
Author

donmccurdy commented Jul 28, 2023

@Methuselah96 Thanks for these comments! three.js maintainers have decided against having hand-written TypeScript files (.ts or .d.ts) in the project. But hand-written JSDoc within the source code may be accepted if it solves some or all problems described here. Generated types, generated API docs, and easier-to-manage translations via JSDoc would be very compelling.

As further context – the Svelte library is maintained in exactly this way. They've decided against writing TypeScript code, but still use JSDoc and the TypeScript compiler to check types and output type definitions. I don't know how/if Svelte does API documentation. But since generating .d.ts files is currently the hard part of this proposal, I'm not as worried about generating docs.

Limited support for JSDoc in TypeScript is a concern, yes. See "possible blockers" above, on this topic. I'm hoping we can work within or around that. If the TypeScript compiler cannot generate .d.ts files from JSDoc with acceptable quality, other tools may be able to do a better job. tsd-jsdoc may work, or ts-morph, or we could keep an eye on Svelte's in-progress dts-buddy library.

@Methuselah96
Copy link

Methuselah96 commented Jul 28, 2023

Thanks for these comments! three.js maintainers have decided against having hand-written TypeScript files (.ts or .d.ts) in the project. But hand-written JSDoc within the source code may be accepted if it solves some or all problems described here. Generated types, generated API docs, and easier-to-manage translations via JSDoc would be very compelling.

Makes sense, my understanding was that in-source JSDoc was also decided against as well (1, 2), so as long as we're reconsidering things, just wanted to find the solution that has the best chance of being adopted and is best for everyone involved.

@pailhead
Copy link

Can you elaborate on:

and inability to validate hand-written d.ts files against source files created unacceptable problems.

@pailhead
Copy link

My main remaining question is what to do about mrdoob/three.js#21174 (comment)?

I'm sure you could turn a rock and find a person willing to commit to this. I too would volunteer looking at TS only issues, and triaging the comments. I don't think though that these would go in perpetuity.

@Methuselah96
Copy link

Methuselah96 commented Sep 11, 2023

I'm sure you could turn a rock and find a person willing to commit to this. I too would volunteer looking at TS only issues, and triaging the comments. I don't think though that these would go in perpetuity.

No need to turn a rock, Josh and I volunteered at the time and the response was:

I think I'd prefer if the types are on DefinitelyTyped for now though. By the way, one benefit of having it outside is that you can update the types any time instead of just once a month.

@donmccurdy
Copy link
Author

donmccurdy commented Sep 11, 2023

I'd rather not discuss why the three.js maintainers don't want to write TypeScript by hand; I've had too much of that sorry. I'll gladly respond to questions or comments about generating .d.ts from JSDoc, though.

@pailhead
Copy link

right i might not be understanding what this is about.

  1. This would essentially then invalidate all the work on definitively typed?
  2. Instead of maintaining a .d.ts file, one would have to maintain a specific type of comment inside of code? And this would be done by three.js not definitively/js-docd or something like that?

If JSdoc is somehow more appealing than typescript, this seems like an excellent way to sneak typescript back into three.js. Ie whatever it takes to have a better experience, the semantics or implementation details don't really matter.

However, i fail to see why one would accept the burden of maintaining these enhanced comments, which seem more or less like typescript just even more verbose, if the less verbose version was already rejected. The issue of being able to update it just once a month would still stand?

@Methuselah96
I was just thinking along the lines - a few years ago it was a different place, in silicon valley i was under the impression that no one writes JS any more. Around 2018, flow was still an actual thing.

@pailhead
Copy link

Yeah TL:DR; you're going to run into this so why bother:
image

@donmccurdy
Copy link
Author

donmccurdy commented Sep 11, 2023

  1. Yes, this would replace @types/three.
  2. Yes, types would be represented as comments in .js files, used to generate .d.ts files, published in the three package.

This specific type of comment has been around since 1999, and was the preferred way to define types in Closure Compiler beginning in 2009, before TypeScript. The Svelte maintainers use JSDoc in the same way. This isn't fancy new tech, it's old boring tech — hopefully in a good way.

Having used both TypeScript and JSDoc types for around a decade, I think it's fair to say that JSDoc types are less ergonomic to write than TypeScript code, but still acceptable, and (unlike TypeScript) only needs to be written for public APIs. Ultimately that's a question for the maintainers to decide, so I'm not going into detail about syntax preferences here.

The technical obstacles to this proposal — namely, limits in TypeScript's understanding of JSDoc syntax — remain. If the generated .d.ts files will be missing properties defined by Object.defineProperty or getters/setters, that's a major issue.

@Methuselah96
Copy link

Just to clarify, which of these are you considering?

  1. Only using the standard JSDoc tags therefore "simplifying" the generated types (i.e., no generics, etc.)
  2. Using TypeScript-flavored JSDoc which allows for more complex features like generics

If only going with (1) standard JSDoc tags, the TypeScript type definitions will suffer.

If going with (2) TypeScript-flavored JSDoc, it still seems like:

  1. TypeScript concerns will be brought back into three.js which has been controversial in the past.
  2. To some extent, JS contributors may have to deal with complicated JSDoc/TypeScript issues, which has been frowned upon in the past.
  3. I continue to discover more limitations when writing TypeScript-flavored JSDoc. For example, I wanted to improve lil-gui's TS definitions, but noticed that there are limitations to TypeScript's JSDoc function overloading support (just added in 5.0) where different function overloads can't have different generic constraints. I ended up giving up and using hand-written TS definitions in three-types/three-ts-types#574 as part of @types/three to overcome these limitations. It seems likely that three would hit these limitations fairly quickly since I've run into them on every library that uses JSDoc to generate TypeScript definitions.

I know these discussions can get tiresome and repetitive, so feel free to let me know if I'm being too negative or repeating myself too much.

@donmccurdy
Copy link
Author

donmccurdy commented Sep 11, 2023

I'm not aware of a firm division between paths (1) and (2), but given that we would most likely use the official TypeScript library to compile JSDoc to .d.ts files, probably closer to path (2).

  1. TypeScript concerns will be brought back into three.js which has been controversial in the past.

I believe the proposal here has a good chance of overcoming maintainers' concerns in this area.

  1. To some extent, JS contributors may have to deal with complicated JSDoc/TypeScript issues, which has been frowned upon in the past.

There's a long history of plain-JS projects using JSDoc to generate API documentation, and I've never found that this is a deterrent to contributions from new JS developers.

  1. I continue to discover more limitations when writing TypeScript-flavored JSDoc...

Yes, JSDoc is less expressive, at least within the toolchains available today. We may not have things like overloads with different generic constraints.

In my view issue (3) is not entirely a bad thing, and having these limitations may help to reduce issues (1) and (2). The types may be looser in some areas than @types/three, to be sure. How much that matters is something we can test, and which may evolve over time.

@Methuselah96
Copy link

Makes sense, as someone who appreciates more expressive/stricter types and has worked on improving those aspects of the types for the past 5 years, I will be sad to see it go in that direction if it comes to it.

@pailhead
Copy link

I urge you to reconsider pushing for this. It feels like you will cut off people who are already committed to solving this problem with a passion, and force it upon someone who has already pushed back against what is essentially the same thing.

@pailhead
Copy link

pailhead commented Sep 11, 2023

The Svelte maintainers use JSDoc in the same way.

It may be worth pointing out what the same maintainers feel when types randomly get removed in a patch. However it has been clarified that three.js is exempt from this due to the personal relationship between the involved parties.

image

I'm one of these users that are affected by these types of changes. I volunteered my time to improve some types. One blocker seems to have been that for some reason, at the time, it had to be reviewed by you @donmccurdy when you yourself said that you didn't know TS as well at the time and were unfamiliar with some advanced usage (eg infer, i also never found an application for this).

But others have volunteered to fill that void and bring TS experience to the table.
image

The very next comment is this one way announcement that all types have been removed from three.js

If you can some how guarantee that it wont come to this:
image

this proposal would make sense. Otherwise, i have to agree with the Svelte maintainers, this whole thing is what they say it is - user hostile.

You will hurt users in the long run, DT works, it suffers from the same issues as every other package depending on three.js - the library breaks everyones project once per month.

You are solving for an issue that does not exist.

Consider proposing semver, and a stable three.js version. That would give a fighting chance to people to actually make some sane stable types. This is hard to do in general, let alone on a moving target.

Svelte maintainers have released Svelte 1,2,3 and 4. This type of contract would make it orders of magnitude easier to write powerful types. This is next to impossible to do if these contracts change once per month. Listen to svelte maintainers, don't be user hostile, feed the poor.

@Methuselah96
Copy link

Methuselah96 commented Sep 12, 2023

While it would sadden me to see the types become more loose, there are numerous trade-offs and nuances to consider here. I personally don't think that JSDoc comments are expressive enough to create the quality of types that are generally preferred by the TS community and that libraries that don't care for strict typing should leave type definitions up the TS community since there's an established pattern for doing so. However it's not my library, so I wouldn't hold any grudge against the maintainers if they decide differently.

@pailhead Please try to stay on topic. Discussions regarding three.js versioning are out-of-scope of this RFC (i.e., changing how three.js is versioned is not under consideration). The problems this is trying to solve are clearly outlined in the initial post and this thread is already getting long enough.

@pailhead
Copy link

pailhead commented Sep 12, 2023

Yes sorry. Please don’t do this. Please.

For all the reasons you mention. It would sadden me if the types were taken from you at random, like this. probably others too.

@donmccurdy people just realized that you are on mastodon, but you are being summoned on Twitter for this topic.

@donmccurdy
Copy link
Author

I have no interest in taking types away from anyone, or making the types worse. As mentioned in the first section, the goals of this proposal are to (1) improve the experience of three.js users writing TypeScript, and (2) improve quality and maintainability of the API documentation.

For now, these generated .d.ts files don't even exist, so I'd like to put a pause on speculation about their quality. If types generated by JSDoc are indeed worse, and three.js users writing TypeScript including @Methuselah96 would prefer that types remain in @types/three, I'll support that decision.


In any case, we have a major problem with maintaining our translated API documentation, and insufficient volunteers for the current approach. This is discussed at greater length in mrdoob/three.js#24984, and was the original motivation for this proposal.

JSDoc is the universal, standard way to write API documentation for JavaScript classes, methods, and other API interfaces. Same thing as TSDoc in TypeScript, JavaDoc in Java, or Docstring in Python. No meaningful learning curve is imposed by adding comments like the one below:

/**
 * Base class for most objects in three.js. Provides a set of properties and
 * methods for manipulating objects in 3D space.
 *
 * @property {true} isObject3D
 * @property {string} uuid
 * ...
 */
class Object3D {
  // ...
}

If it turns out that JSDoc is not an acceptable way to generate type definitions, I think we'd still want to consider using JSDoc comments for generating API documentation. I'd prefer to kill two birds with one stone here, but if only one is possible, we'll weigh the cost/benefit accordingly.

@pailhead
Copy link

pailhead commented Sep 12, 2023

Also based on the twitter discussion, the fact that this is in the same file may make it slightly easier to push.

  1. What happens when you yarn @types/three while having some JSDoc embedded?
  2. Does one have to include props, params and types in JSDoc?
  3. What happens when someone makes a PR and does not touch the relevant JSDoc? If possible, this feels like it would cause the two to be out of sync.

If not on 2, then it seems like the issues of documentation/translation can be solved agnostic of the types?

@donmccurdy
Copy link
Author

donmccurdy commented Sep 12, 2023

The answers to 1–3 are not complicated, but they are frankly a distraction from the issues here. If we get to the point where this proposal is no longer blocked by known issues, and we have actual PRs or JSDoc changes to discuss, I'd be happy to answer these questions.

@pailhead In the interest of clarity, I'll be direct. It's my opinion that you have a history of taking up more of the available oxygen in three.js discussions than is professionally acceptable, or commensurate with your expertise and contributions. You appear to be repeating the pattern now in this thread. I've noted your concerns, and would ask that you step back and leave room for others to give feedback.

@pailhead
Copy link

Noted.

@Methuselah96
Copy link

Methuselah96 commented Sep 12, 2023

@donmccurdy Thanks for your thoughtful and patient responses. As noted above I'd be interested in helping in any way I can. Is there a good place we can chat without taking up space here? My Discord username is the same as my GitHub username.

I'm interested in discussing what next steps might look like (e.g., creating some draft PRs to explore different options, working on tooling, waiting on comment from other maintainers, etc.). as well as whether you're okay with me proposing a slightly modified RFC that details how .d.ts+JSDoc could solve the same root issues with different trade-offs. I'm still hopeful that a .d.ts+JSDoc solution might be appealing despite it being decided against in the past if it solves the problems you're trying to solve, in the same way you're hopeful about a JSDoc solution being appealing despite it also being decided against in the past.

@Methuselah96
Copy link

Methuselah96 commented Sep 13, 2023

I now remember you mentioning a desire to not discuss writing .d.ts by hand, sorry for bringing that up again. There are definitely advantages to the JSDoc being in the source file anyway, mainly it being more likely that it will get updated with source code changes.

The first example that comes to mind of types that we probably couldn't express in the same way in JSDoc are some of the EventDispatcher methods because they has overloads with different generic constraints. You mentioned tsd-jsdoc (looks potentially unmaintained), ts-morph, or dts-buddy as some potential tools to investigate. I'm interested in helping doing some investigation, but if it's too soon or you'd rather do it yourself, that's fine too.

@donmccurdy
Copy link
Author

Thanks @Methuselah96! Let's continue offline, I'll send over a message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment