Jim Klimov on the Open Source way

JimKlimov-opensource.txt

Originally posted at https://github.com/jenkins-infra/jenkins.io/pull/5537#issuecomment-1272498645

Cheers @jmMeessen - I suppose we've met on some of the FOSDEMs so you might have some answers already, but I understand this is a neat questionnaire anyway :) Feel free to arrange and share:

> This is why I am reaching out to you to better get to know you as a community member and a contributor. Would you mind to answer the following questions?

Sure, but in a bit of a different order for a better narrative :)

>     * Who are you? Where are you located? What do you do for a living?

I am Jim Klimov, and my studies and further career took me through academia (minoring in "Open Information Systems", no less) through systems administration in a college campus, then in a company spun off from an university lab to deal with various enterprise document workflow and messaging systems, raising to CTO of that small collective, later moving to Czech Republic and working in a couple of corporate settings.

During my path I discovered a few things about myself and the world:
* that no tool made by others meets all of our needs perfectly - so there indeed is a need to keep forging them, and preferably get someone else to maintain the fixes long-term (hello Open Source and contributions);
* that I like "to forge the tools of my trade" (mostly by scripting);
* that even though I am not too proficient as a programmer, I can get things done in any language (people did say my Java looked like shell script... but hey, it works);
* and that I like to help "real programmers" get their job done well so that a project can ultimately reach its users.

Currently I am a "DevOps architect", working closely with product architects and QA leaders, helping a company design and implement how the product code they write and run on several platforms would become a well-tested, upgradable and reliable deliverable for end-users (as on-premise or managed-cloud service), as they crush a legacy monolithic application into a swarm of services.

>     * Did you already contribute to Open Source before?

Yes, starting last millennium (in the university network administration), posting some patches to Samba and exchanging ideas on various Linux mailing lists. Later on I collaborated on several milters (sendmail filters) for anti-spam, SDIG (Switch Digger) and NUT to monitor our infra hardware, OpenVPN, ClamAV, sort of maintained a campus baseline Linux distro recipes built with optimization for each ARCH of very different computers posing as servers, etc. On the professional side we worked with Sun Microsystems projects a lot (OS, terminal services, VMs for workspaces, messaging suite, Java app servers, ZFS and NFS based file services being a tip of the iceberg), so learned a lot from their engineering blogs and I frequented the forums and mailing lists to share back tips and tricks that I discovered regarding their products - in a way, this was Open Source knowledge about a lot of proprietary products.

Some of these projects originated in Netscape, ISC or similar shops, and their derivatives were open-sourced elsewhere; also later on many of the Sun projects got opened up officially and often forked as OpenSolaris (now illumos and many distros based on it), Hudson (now Jenkins), Sun Application Server, later Glassfish (now Payara), OpenLDAP and OpenIDM (picked up by ForgeRock for a while), NetBeans, VirtualBox, etc. This grand opening in early 2000's let us systems engineers see the code running in production builds (or its close relative) to understand whether something not working is our mistake (e.g. misunderstanding of configuration) or we found an issue or corner case in their product - to ultimately work around it for the current build while proposing a bug report or fix ideas to the upstream. Reading the sources also helped understand the thought flow and reasoning of original developers, both to learn and become better developers or contributors ourselves, and to predict how to understand the behaviors seen in the field (built from probably different revisions/branches of that code we saw). This went on after the "Sun-set" as the projects moved on, forked often by same teams that created them and communities that used them - who could now become true collaborators and co-authors.

One thing led to another, professionally and enthusiastically, and I've participated a lot in packaging maintenance for Solaris, OpenIndiana and OmniOS (all with different recipe techs and approaches), Debian-ish and RPM-ish systems, and later got introduced to OBS (OpenSUSE Build System) to automate much of that for Linux at least - and surely posted PRs to scratch some itches there too.

I have maintained Jenkins servers for the past decade in several companies, introducing Pipelines as they appeared (so redoing a lot as the idea grew and early-release APIs changed), culminating in contributions to libzfs4j (so it accepts more native libzfs APIs - which are internal and incompatible over the years - and Jenkins does not crash upon startup) on one of the FOSDEM meet-ups, and ending up with a take-over (and releasing the reins) of some Jenkins plugins where functionality lacked for us, and current maintainers stretched thin could not merge the fixes in a timely fashion :)

Another important part in my recent years' growth was introduction to ZeroMQ community and more so to their process and ideology than to particular products, including the https://rfc.zeromq.org/spec/44/ "C4 process" (Collective Code Construction Contract) for maintainable communities and projects that survive their creator, as well as meeting Pieter Hintjens and reading his blogs and books. One eye-opener was to not strive for perfection unless necessary for your goals today - but to leave a few rough edges here and there, so someone else would get an itch to scratch and eventually possibly pull into the community. Like with other processes (e.g. Agile) it is important however that teams use the whole rule book, with its checks and balances, and not only the easy part of the book - otherwise it does spiral out of control.

As one single largest endeavour of several past years, I became a team-member and later maintainer of NUT (Network UPS Tools) where Jenkins is indispensable to arrange true multi-platform testing - along with a shared library to enable a build matrix driven by current population of build agents and their declared capabilities. This is detailed at https://stories.jenkins.io/user-story/jenkins-is-the-way-for-networkupstools now :)

>     * Why is Open Source important to you (as you are contributing)?

I learned a lot from people before me, and made interesting projects standing on the shoulders of giants. Open Source allows to pass such benefits on to the next generations.

Beside contributing to others' work, or maintaining some, I have also published a number of projects that I've started alone - either on my spare time, or as FOSS work for employers. This sort of builds a brand for me (and employers), helps get feedback and contributions from other bright minds, and lets me use anywhere the solutions I made somewhere some time ago.

>     * What are your motivations?

As a consumer of the sources, even the half-way access to the opened Sun products sources was very helpful to set up systems in the field :)

As a contributor, it is mixed:
* a way to learn from PR reviews and improve the proposal - and also the original in-house requested solution;
* quite often (annoying but necessary) a nudge to introduce or add test cases for the feature, spend an order of magnitude more time than the original effort, usually find bugs along the way, and improve the proposal - and also the original in-house requested solution;
* gotta do what is needed to upstream the proposed change, and so hand off the maintenance to someone else: both "egoistically" to not keep re-merging and re-building the in-house solution as the upstream evolves - and for the in-house solution to remain available via upgrades even as you move on from the project, and "practically" to benefit from bug-fixes someone inevitably applies to the feature making it better and more reliable (or replaces the implementation keeping the idea compatible).

In some cases, publishing new work and upstreaming improvements to the tools we use was literally part of the job description (and I am picky about jobs to directly include or at least not forbid such collaboration).

>     * Is it the first time you participate to Hacktoberfest?

Officially (swag-proven) it is my second time, I guess. That said, I contribute all year round so probably had many other Octobers without the stamp :)

>     * What did you contribute to during this hacktoberfest?

I have posted a "Jenkins is the way" story about how it helps NUT remain multi-platform across decades worth of toolkit, platform and implementation vendor differences.

I've also persisted on several Jenkins PRs that I have proposed earlier but still did not get merged, including a change for `@Library` annotation, a much older proposal for the git-client-plugin to work with reference repositories more efficiently, and hope to get to finally review older proposed and stalled efforts on lockable-resources-plugin, to name a few.

>     * What do you experience as most difficult when contributing to open source?

Meeting maintainability expectations of the team to get something merged. I was (and regularly am) on both sides of the equation :)

The spectrum is very wide, yet surprisingly they all lead to stable releases converging over time; to highlight a couple of the extremes:
* for ZeroMQ C4 process, if CI passes and nobody vetoes/delays the change in discussion, it is merged - maybe 5 minutes after posting (it may be reverted or supplanted by another fix in 5 minutes more however);
* for Solaris process, a PR must include everything and a kitchen sink (at least in early years after the fork, when people were used to a team behind each architected change and not a single person scratching an itch) - reasoning, docs (including code comments), actual change, tests... getting a PR accepted could take from months to a couple of years, but each revision of master is "perfect" as a baseline for others to branch off for their features, or to ship a build to commercial end-users.

For most communities and processes, difficulty lies in designing and implementing tests: in my experience, this is at least 10x the effort as the original implementation that "worked for us". Tests do serve as documentation and automation of your change's intentions, and at least serve as a non-regression baseline for future work that might impact the feature (what you expect to work today, remains working in a year or several). Though not usually required explicitly, work on tests tends to dig into "negative cases" (what is expected to fail), find corner cases and discover limitations of your idea (or outright wrong presumptions), so you iterate on the proposed change including its documentation - and this certainly makes the proposal better and more reliable even for the author. You no longer "guess", "hope" or "expect" the solution to work - you _know_ it does (and cases where it does not).

Another difficult part is a mismatch of architectural expectations - you propose a change where you saw it is possible to chisel it into, and it works for you, yet people better aware about project internals say it belonged completely elsewhere, that you had to start a new plugin from scratch yet have no idea about making a first step, design a dozen new classes instead of changing one function, etc. This is something I often face in Jenkins contributions, and not really being a Java designer or developer, get inclined to abandon the upstreaming effort (luckily here it is also quite possible to try and convince someone else to do the porting to proper design - since at least the idea/goal is developed and proven by practice).

And for many coders, communications. It is one thing to create a shiny new feature, gift it to the world and let everyone revel in its beauty, and another to spend hours or days convincing people why it is useful, good, functional, made this way and not that way, and that they should accept it to mainstream to maintain as their own. For good or bad, that alone is often behind stalled and abandoned PRs. Sometimes someone else picks it up to complete the merge or implement the idea differently, other times the effort gets lost in the queue forever as authors move to new tasks and lower-hanging fruit in other projects. Remember however, that by contributing you share (or hand away) not only a new feature or bug-fix, but also the burden of maintenance and possibly bug-hunt due to regressions introduced for some situation you never stumbled upon in development, testing or thought experiments - hence your changes should be easy to understand, and code (including style, dependencies, data-types used, tests again, etc.) easy to maintain, cherry-pick, etc. without your direct involvement anymore.

To add on this area, as both a maintainer and contributor one most helpful thing I find is presence of automated testing for the PRs (e.g. public Jenkins servers, GitHub Actions, CircleCI, AppVeyor, LGTM.com, OBS... across as many platforms relevant for the project as possible), beside an ability to run a test locally (ideally CI logs would hint at how to reproduce the failures). If the project has expectations for spell-checking, coding style, building with toolkit X on OS Y - please do code it and automate it and make the results (especially failures) visible to contributors. If you have PR guidelines like "update docs" or "post HW test results" - make PR templates with a check-list. If you require tests for new features - track the test coverage and require the number to not reduce from build to build.

Keep in mind mantras like "Something done manually once is exploration, twice - confirmation, thrice - sabotage!" and "If it is not in Git, it never existed!"

This ends up in less work for maintainers to repeat same requirements in PR comments, less frustration for people getting their PR initially rejected (rage against the objective machine, rather than frustration against a subjective peer), and better PRs (issues that would block merge are resolved before maintainers get to look at the proposal). Largely this becomes an exercise in test-driven development for reference conditions a developer has no control over, yet the developer can iterate alone.

While not many people run dozens of unrelated OSes to build in, nor do hundreds of builds on their laptop for each commit to make sure it is okay across the board, for those who do -- a local VM/container farm (VirtualBox etc.) and local Jenkins can help do just that, using their actual Git workspaces as SCM sources for independent re-builds, in the confines of a beefy workstation.

>     * What would you say to others that are hesitating to contribute (because of lack of skills/fear/...)

Find a project (or many) that interests you, with technologies or approaches you want to learn, go tinker, and post PRs. Maybe your proposal would be accepted, maybe not, maybe improvements would be suggested or a different way to solve the issue. In any case, you would learn more about the world, project, yourself, interactions and patience :)

Documentation is one area almost everyone can do better, and almost anyone can help improve. It is easy to overlook something as "apparent" after a decade of experience with a project when it is really non-trivial for a newcomer. Keep track of your failures to set something up (difficult deployment, complicated testing, guessing how to start a new component, hiccups you had in PR discussions) and propose documentation changes. This might lead into recipe/automation changes, learning more about the code and project structure, and eventually solid feature proposals.

In case of Jenkins core, plugin or shared library contributions, keep in mind that Java IDEs like NetBeans or IDEA can be very helpful to step through the server sources with a debugger. Peppering code with temporary `println`s only goes this far, sometimes you will need real tools :)

// Hope this helps,
// Jim Klimov