a guide on working with conda & RTD. Noting things like how to use conda-forge, and the best practices for using conda and pip together.
- Who are they?
Data Scientists, Data Engineers and Developers. Heterogeneous level of programming skills.
- What do they need?
To deploy the documentation of their software package using conda and Sphinx on Read the Docs.
- Where will they be reading
From their desktop computers or laptops.
- When will they be reading?
Any time. During working hours for those who get paid to write the docs, outside working hours for those who maintain a volunteer open source project.
- Why will they be reading?
Either they have found some errors while trying to use conda on Read the Docs, or they haven't tried yet and want to figure out how to do it, or they don't know where to start.
- How will they be reading?
Some of them will skim and Ctrl+F for the information they need, some others will read from top to bottom.
- How to configure the build to use conda
- "If you are using a Conda environment to manage the build, this setting will not have any effect, as the Python version is managed by Conda." https://docs.readthedocs.io/en/stable/config-file/v2.html#python-version
- Does this apply to
python
or only topython.version
? - Also mentioned in
python.install
conda
andconda.environment
properties
- How to create your
environment.yml
file- Understand how the export works, and whether it should be written manually!
- The active voice is easier to read and understand than the passive voice. Whenever possible, choose the active voice in your sentences.
- Choose your words thoughtfully. Use the best word for the context. Include necessary details that make the text understandable and precise. Avoid overusing pronouns such as ‘it’ and ‘this’ as the reader may have difficulty identifying the antecedent.
- Many technical documents provide instructions to the reader. Therefore, a task-based approach makes the content easier to understand
- Always put the most important information in the main clause. The reader will better digest the priority information.
- Refrain from writing in the first or second person. This means avoiding the use of the words “I”, “You”, or “We” in your writing.
- "Write for translation"
- Always write a document that is useful to the user.
- Use mind maps to organize the information
- As you prepare, continuously step back and view the document as the reader. Ask yourself: Is it accessible? How would they be using it? When will they be using it? Is it easy to navigate?
"Documenting projects is hard, hosting them shouldn’t be. Read the Docs was created to make hosting documentation simple." https://docs.readthedocs.io/en/stable/story.html
- Sign up
- Confirm email 3a. "Not sure how to start documenting your project? Check out the Getting Started Guide to learn how." https://docs.readthedocs.io/en/stable/intro/getting-started-with-sphinx.html 3b. "Want to try a demo? Import our own demo project." https://readthedocs.org/dashboard/import/manual/demo/
- "Your documentation is building" Easy peasy!
- "Your documentation has been built. Ensure your documentation is kept up to date with every commit to your repository, by setting up a webhook." https://docs.readthedocs.io/page/webhooks.html
- "Configure your documentation builds! Adding a .readthedocs.yml file..." https://docs.readthedocs.io/page/config-file/v2.html
- conda is mentioned, but not explained 3c. "Import a project"
- "Connect your account" or "Import manually" 3d. "Have a GitHub account? Connect your account and import your existing projects automatically."
And now, if I want to use conda, I'm on my own.
- 1h gathering the information above
- 20m setting up the development environment
- 20m setting a test account, importing the demo project, and taking notes of the customer journey
- 10m reading the configuration documentation
- 10m exploring the rest of the RTD docs
- 10m setting up the development environment for real after I found the install docs
- And then realizing that it won't be necessary to see my local changes
- 5m focusing on the existing conda docs
- Basically how to enable it https://docs.readthedocs.io/en/stable/guides/conda.html not much else!
- 1h reading conda docs
= 3h15m
- 1h45m reading old issues to look for common problems, tips, tricks, and some context about the current status and the history of the decisions
= 5h
- Tabs! https://docs.readthedocs.io/en/stable/config-file/v2.html
- https://www.ericholscher.com/blog/2016/jul/1/sphinx-and-rtd-for-writers/
- https://www.anaconda.com/blog/understanding-and-improving-condas-performance
- Introduction to Conda for (Data) Scientists! https://carpentries-incubator.github.io/introduction-to-conda-for-data-scientists/
- (comes from a broken link here https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/data-science.html)
- How to declare conda dependencies! https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/pkg-specs.html#package-match-specifications
- What is a conda channel https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/channels.html#what-is-a-conda-channel
- Is the
pip_interop_enabled
flag enabled in RTD? What are its implications? https://docs.conda.io/projects/conda/en/latest/user-guide/configuration/pip-interoperability.html- Notice that it's still experimental
- Old, but perhaps still applicable, advice https://www.anaconda.com/blog/using-pip-in-a-conda-environment
- Sharing an environment https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#sharing-an-environment
- "If you want to make your environment file work across platforms, you can use the conda env export --from-history flag. This will only include packages that you’ve explicitly asked for, as opposed to including every package in your environment." https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#exporting-an-environment-file-across-platforms
- Beware of micro versions! https://twitter.com/esc___/status/1356946396746838016 how does this interact with the package spec above?
- Also,
nodefaults
to remove defaults channels
- Troubleshooting
- Unsatisfiable spec https://docs.conda.io/projects/conda/en/latest/user-guide/troubleshooting.html#unsatisfiablespecifications-error
- "It is important to have 1 space before the == operator and no space after." oh wow https://docs.conda.io/projects/conda/en/latest/user-guide/troubleshooting.html#conda-automatically-upgrades-to-unwanted-version
- What is conda-forge https://conda-forge.org/docs/user/introduction.html
- Compatibility between conda-forge and default channels https://conda-forge.org/docs/user/tipsandtricks.html#using-multiple-channels
- Anaconda change in ToS https://www.anaconda.com/blog/sustaining-our-stewardship-of-the-open-source-data-science-community and FAQ https://www.anaconda.com/blog/anaconda-commercial-edition-faq
- "The new TOS changes apply to the defaults Repo and not to conda forge packages on anaconda.org" https://twitter.com/pwang/status/1325897465107771392
- "This is because we use Miniconda, which its release process is a little more slow than conda itself" https://docs.readthedocs.io/en/latest/guides/feature-flags.html
- Relevant feature flags https://docs.readthedocs.io/en/latest/guides/feature-flags.html:
UPDATE_CONDA_STARTUP
CONDA_APPEND_CORE_REQUIREMENTS
CONDA_USES_MAMBA
- "Seem like adding - nodefaults to the channels list in the environment file helps conda come within limits again." readthedocs/readthedocs.org#5546 (comment)
- "If you don't specify a version, you get the default (1.8.5)" readthedocs/readthedocs.org#5810 (comment)
- "We run 3 steps when a project depends on conda" (2019-07) readthedocs/readthedocs.org#5956 (comment)
- Is this still the case? -- Yep
- Some dependencies kill the RTD worker readthedocs/readthedocs.org#6742
- And it looks like pip has a huge memory spike when installing large wheels readthedocs/readthedocs.org#6742 (comment)
- You can use any version of sphinx you want https://docs.readthedocs.io/en/stable/guides/specifying-dependencies.html
- (2017-04) conda and pip, "it would be nice to have an actual working example of this" readthedocs/readthedocs.org#2776 (comment)
- (2019-04) "looks like we shouldn't use pip at all when using conda" readthedocs/readthedocs.org#2776 (comment)
- (2018-03) "Ability to pin Sphinx and friends in a conda environment" readthedocs/readthedocs.org#3829
- (2021-02) "Looks like
defaults
has really old versions of Sphinx and rtd-theme" readthedocs/readthedocs.org#3829 (comment) - (2021-02) My quick summary readthedocs/readthedocs.org#3829 (comment)
- Looks like this issue is the source of a lot of problems, although the feature flag
CONDA_APPEND_CORE_REQUIREMENTS
seems to solve it
- (2021-02) "Looks like
- (2019-03)
install_core_requirements
? readthedocs/readthedocs.org#5546 (comment)
- Where should I include myself? https://docs.readthedocs.io/en/stable/team.html
- We should separate the "project build" step from the "documentation build" step
- The API docs are also in the build artifacts!
- Creating wheels for development is something NumPy, Tensorflow, PyTorch and others are already doing
- Perhaps abusing test.pypi.org? Read again on the topic
- Idea: temporary wheel hosting for projects?
- Is there a schema for conda environment files?
- Anaconda probably supports Python 3.8 these days already https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-python.html