Created
November 20, 2014 16:41
-
-
Save derwolfe/ca5f7e86790e1c2df6ca to your computer and use it in GitHub Desktop.
irc notes about twisted packaging/docs - ticket #3696
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
09:41 <herrwolfe45> I'm working on ticket #3696 and am a bit stuck on what type of documentation I should write - I'm thinking I should write a how-to describing how one should install twisted and it's various optional dependencies. This how-to would be in the narrative docs for twisted core. Does this sound like a decent idea? | |
09:41 <glyph> herrwolfe45: That sounds fantastic. | |
09:42 <herrwolfe45> glyph: excellent, I'll start with that - thanks | |
09:45 <herrwolfe45> ergh its | |
09:48 <exarkun> When writing that, don't forget that `pip install Twisted` is the least preferred installation method. When possible people should prefer an OS-supplied package. (But it's certainly not always possible.) | |
09:49 <exarkun> (Or possibly tailoring different sections of the document to different audiences makes sense - developers working on Twisted have slightly different requirements from developers using Twisted have very different requirements from end-users using an application that depends on Twisted.) | |
09:50 <glyph> exarkun: Least preferred by whom? | |
09:51 <glyph> exarkun: I would actually say "pip install Twisted" is the _most_ preferred installation method at this point. OS-provided packages have kinda dropped the ball at this point as far as keeping up with releases and providing isolated, repeatable installations. | |
09:52 <glyph> The major reasons to prefer OS-vendor-provided packages are integrations with things like GTK and pyObjC, which are definitely not the most popular use-case :-\ | |
09:52 <exarkun> It's most preferred if it's most preferred, I agree - but that's also sort of a useless statement. ;) | |
09:52 <exarkun> If your OS vendor has packaged the version of Twisted you need, you should use that package. | |
09:52 <exarkun> That's why I said "when possible". | |
09:53 <glyph> exarkun: Why? | |
09:53 <exarkun> Because it's going to get security updates, for example. | |
09:53 <glyph> exarkun: We ship security updates via pip as well, though. | |
09:54 <exarkun> Yes, but the upgrade experience for pip isn't as good. | |
09:54 <exarkun> You want unattended-upgrades to just install the security update - if possible. | |
09:54 <exarkun> You want as few installations on a system as possible. You don't want to have to remember to pip upgrade all of your deployments. | |
09:54 <glyph> exarkun: Except unattended-upgrades is insufficient, because you need something that is going to restart your Twisted services for you when the updates are available; and your OS isn't going to help with that anyway. | |
09:55 <exarkun> And (not security related) it's not going to screw with other OS-provided packages that have a Twisted dependency. Perhaps those apps break when you install the newer Twisted into your environment (so, yay, virtualenv or whatever, but `pip` doesn't /require/ virtualenv). | |
09:55 <glyph> exarkun: also, like, Docker completely breaks unattended-upgrades anyway, which I am still trying to figure out how to deal with :-) | |
09:55 <glyph> Aah. | |
09:55 <exarkun> glyph: You're basically saying "OS packages aren't 100% perfect so use pip" | |
09:55 <glyph> Right. | |
09:55 <exarkun> ignoring the fact that pip is also not 100% perfect | |
09:56 <glyph> exarkun: Virtualenv is the thing, really, not pip. | |
09:56 <exarkun> There are advantages and disadvantages to both. | |
09:57 <exarkun> Maybe you're even right that on balance virtualenv & pip gives you the best trade-offs - but only if you understand how to manage it. I'd argue that someone who understands those things deeply isn't going to read our installation documentation anyway: they already know how to do this. | |
09:57 <exarkun> For the folks most likely to need this documentation (again, *if possible*) their OS packages are going to provide a better experience. | |
09:58 <exarkun> And I'm not saying don't provide docs about both approaches, I'm just saying don't provide docs that unconditionally say "install Twisted with pip no matter what". | |
09:58 <glyph> exarkun: OK, let's talk about the folks most likely to need this documentation. | |
09:58 <glyph> My understanding of the target audience is that this is people with a cursory understanding of Python but no real understanding of packaging or the tradeoffs involved. | |
09:59 <exarkun> I gave three example target audiences above | |
09:59 <exarkun> I think your description is pretty close to the third example and it's probably the group most in need of any kind of installation docs, yes. | |
10:00 <glyph> exarkun: My experience of that audience is that they know what "pip" is because they have 2 other dependencies they've already installed, both of which had installation documentation that told them to do "sudo pip install". | |
10:00 <exarkun> (But the other groups are interested in the particulars of how they might get all of Twisted's dependencies since Python's packaging ecosystem is still immature that you can't *totally* rely on that going smoothly) | |
10:01 <glyph> exarkun: Frequently they've also installed 2 or 3 versions of Python by downloading source code or installers from python.org. | |
10:01 <exarkun> You're narrowing in on the *really* screwed end of the spectrum. | |
10:02 <exarkun> Our docs can't be a comprehensive course in how not to screw up your computer, either. | |
10:02 → iffy (was magmatt) joined | |
10:02 <glyph> exarkun: Yes, and I don't want them to be. | |
10:03 <glyph> exarkun: But, if we say "use your operating-system provided packages", this presupposes a level of knowledge of their operating environment that I think literally zero of the people in this audience (people who need some kind of set-up/installation documentation) actually have. | |
10:03 <exarkun> Again, I don't think that's *all* we should say. | |
10:04 <glyph> exarkun: I'm not arguing that we should just say "use pip, hth hand" either. | |
10:05 <glyph> In fact I'm not sure that I'm arguing *for* anything in particular. I think our preference for saying "use your OS installed packages" produces undesirable outcomes and I would like to explore what we should be saying instead. | |
10:06 <exarkun> Okay | |
10:06 <glyph> Maybe we should have an entire document that says "using the Twisted packages from your operating system" | |
10:07 <glyph> The chief undesirable outcome I have noticed this guidance producing is people having really no idea what "operating system packages" are, and having to deal with multiple environments (almost always: OS X for dev, Linux for prod) | |
10:07 <glyph> so they build some knowledge in one environment that completely fails to transfer to the other | |
10:08 ⇐ itamarst quit (~itamarst@pool-108-20-253-103.bstnma.east.verizon.net) Quit: Leaving. | |
10:08 <exarkun> That seems like the kind of basic misunderstanding that it might be very difficult for us to correct. | |
10:08 <glyph> exarkun: well, this is the advantage of suggesting a virtualenv-based approach to installations. | |
10:08 <exarkun> For one thing, someone who thinks they know about installing software because they've done it on OS X might not even think to look at our documentation to learn how to install things on Linux. | |
10:09 <exarkun> glyph: Calling it an advantage seems a little tricky to me. | |
10:09 <exarkun> It's definitely a trait. | |
10:10 <exarkun> Offering one lowest-common-denominator solution is good because once someone learns it they don't have to learn something else but it's bad because maybe in learning something else they could produce a better result. | |
10:12 <glyph> I'm not really sold on the OS-pavkages route being "a better result" | |
10:13 <glyph> Thus far, the advantages we've discussed are: half of an automated security updates solution if you're using Debian, and reduced build interdependencies for desktop / GUI integration stuff | |
10:15 <glyph> The advantages of virtualenv+pip are: portability, a greater degree of environment isolation and therefore predictability, and more knowledge sharing with other projects (since most other python projects recommend that deployment strategy) | |
10:15 ⇐ khorn quit (~funsize@pool-72-64-101-86.dllstx.fios.verizon.net) Read error: Connection reset by peer | |
10:16 <exarkun> Another factor worth mentioning, probably, is chances of actually producing a working installation | |
10:17 <exarkun> virtualenv & pip have extra dependencies - eg, a C compiler and various headers | |
10:17 <exarkun> If you actually want `pip install Twisted` to work in a virtualenv, better install libssl-dev (or is it libssl-devel) and other such things first | |
10:18 <exarkun> (using your OS's package manager!) | |
10:19 <glyph> Good point, the portability argument is weakened considerably by the totally non-portable nature of development dependencies | |
10:20 <Alex_Gaynor> Isn't it a goal that twisted be installable without a C compiler? | |
10:20 <glyph> Not just libssl-dev, of course: don't forget libffi-dev ;) | |
10:20 <exarkun> Alex_Gaynor: Yes - but that's just Twisted. If you're installing Twisted and all of its dependencies, which you are if you use pip in a virtualenv, life is a lot harder. | |
10:20 <glyph> Alex_Gaynor: Sure, but then you won't get SSL support (unless somehow cryptography is going to be installable without a C compiler too)? | |
10:21 <exarkun> Right, so even in the best possible case, you end up with a crippled Twisted install - which may not be what you wanted. | |
10:21 → khorn joined (~funsize@pool-72-64-101-86.dllstx.fios.verizon.net) | |
10:24 <exarkun> So, getting practical, would our virtualenv & pip installation documentation include a section on how to use various OS's native package managers to install those dependencies? | |
10:25 <glyph> exarkun: I think so. | |
10:25 <hynek> that’s three lines to cover Linux & FreeBSE. OS X is the real issue because it’s a) unstandardized and b) used by a lot of junior peole | |
10:26 <glyph> hynek: "type cc in a terminal and follow the on-screen instructions" | |
10:26 <glyph> hynek: we _don't_ need to help you install PyGame and GTK development libraries, remember: libffi and openssl are both bundled with the platform | |
10:27 <exarkun> hynek: But once you've got those three lines, you've brought into the extra conceptual overhead - and putting a "python-twisted" package name somewhere on those lines doesn't increase that burden (and you get to stop there instead of talking about virtualenv & pip if you want) | |
10:27 <exarkun> So *some* of the portability advantage has been lost | |
10:27 <glyph> exarkun: except you _might_ not get to stop there, because your *other* dependencies, which you want to integrate with Twisted, may not install cleanly in that environment (or may require the use of 'sudo pip') | |
10:28 <exarkun> glyph: Sure, but some people will get to stop there. | |
10:28 <exarkun> This suggests to me the docs should be something like | |
10:29 <hynek> once people start using python-twisted, they start installing crap into their site-packages and then they come crying | |
10:29 <glyph> hynek: let's be specific :-). What does "come crying" mean? What breaks? | |
10:30 <hynek> glyph: https://hynek.me/articles/virtualenv-lives/ the examples are extra-twistedish but the points remain | |
10:30 <exarkun> "Maybe this OS-specific command is sufficient for you: <dpkg / yum / whatever to install Twisted>. If you need a newer version, start with this OS-specific command: <dpkg / yum / whatever to install Twisted's dependencies> and then <virtualenv & pip instructions instructions>" | |
10:30 → itamarst joined (~itamarst@pool-108-20-253-103.bstnma.east.verizon.net) | |
10:30 <exarkun> hynek: People will /have/ to install some native packages. | |
10:30 <hynek> it’s not just about newer versions. it’s also about klein, treq etc pp | |
10:30 ⇐ trenton42 quit (~trenton@199.116.53.69) Ping timeout: 240 seconds | |
10:31 <hynek> exarkun: that statement has absolutely nothing to do with my point. not sure whether it’s my fault or yours though. :) | |
10:31 <exarkun> hynek: Given that, I don't think you can get away from your concern about people incorrectly installing unpackaged software in system paths. | |
10:32 <glyph> exarkun: I think there's an unspoken assumption here: the number of native packages required from the platform is small (literally 2: openssl-dev and libffi-dev); the number of python packages required by application developers is large (klein treq characteristic txamqp txkazoo ampoule service_identity pyasn1_modules and so on and so on) | |
10:33 <exarkun> Or maybe not "If you need a newer version" but "If you'd prefer to have an isolated installation (for example, so you can have multiple versions of Twisted) ..." | |
10:33 <exarkun> glyph: It's larger than that. | |
10:33 <exarkun> glyph: And even if it's just one thing, it's the conceptual overhead of switching to another tool. | |
10:33 <glyph> exarkun: Perhaps I should have said "typically"; for some developers it's huge. What am I forgetting, though? | |
10:33 <exarkun> The number of arguments you copy/paste along with that tool's invocation command is sort of irrelevant. | |
New messages since you tabbed out | |
10:34 <glyph> exarkun: On OS X and Windows, you don't actually need to switch to another tool though | |
10:34 <glyph> exarkun: on Windows, cryptography wheels address the need for openssl and cffi wheels address the need for libffi | |
10:34 ⇐ itamarst quit (~itamarst@pool-108-20-253-103.bstnma.east.verizon.net) Client Quit | |
10:34 <dstufft> glyph: also python-dev | |
10:34 <glyph> exarkun: on OS X, "install the developer tools" is a single operation you do once and almost _certainly_ need to do to get a lot of things done | |
10:35 <exarkun> Are there binary wheels for cryptography now? | |
10:35 <hynek> I think jp’s point is basically that there should be as little hurdles as possible to try Twisted which I completely agree with. Although if documented, it should be pointed out that it’s just a sneak-peak and if pursuited for anything else than writing distribution-specific software, it will end in tears. | |
10:35 <glyph> exarkun: for windows, yes! | |
10:35 <glyph> exarkun: it is ~~~amazing~~~ | |
10:36 <exarkun> Okay, cool. I didn't notice when that happened. | |
10:36 <glyph> exarkun: they statically link openssl, you don't need to go to that shitty website any more | |
10:36 <hynek> it’s still impossible to build linux binary wheels :( | |
10:36 <exarkun> Right. But you do need to pip upgrade every time an OpenSSL security update comes out (~ once a month) | |
10:36 <glyph> exarkun: Yup. | |
10:36 <exarkun> But it's Windows. So too bad, that's what you have to do. We don't have to argue about that. | |
10:37 <glyph> exarkun: Or it's Docker and that's what you have to do too ;-) (seriously somewhere else we should have a conversation about container security updates because I do not know what the _heck_ is going on with that) | |
10:37 → itamarst joined (~itamarst@pool-108-20-253-103.bstnma.east.verizon.net) | |
10:38 <exarkun> Okay I gotta do some work. | |
10:38 <hynek> glyph: there is none. my tour into docker made me drink. | |
10:38 <glyph> exarkun: I probably should too. | |
10:38 — glyph wonders if he can capture some notes from this conversation in a useful way |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment