koolfy/gist:7503345

## gistfile1.txt
Relating to https://github.com/cryptocat/cryptocat/issues/502 , I think it is actually a very good opportunity to work on a development/release process that would strenghten the security and reliability of cryptocat software.

------------

The mains goals of these suggestions are:
- More stability of the security-critical code during the development phases
- Better beta testing and less bugs/flaws in stable releases
- Overall better documentation and making it easier for people to review our choices

------------
Ongoing development
==================

**Files and pieces of code relating (even indirectly) to the resulting security of the software must be identified and flagged in some way.**

- None of these files is allowed to be modified without a rigid procedure consisting of, in this order:

> 1. **An explanation** of what the modification intends to fix (implied: "How is it now, how will it be then.")
> 2. A **complete specification** of the proposal, with as much concrete examples as necessary. You know the proposal is correctly described if *anyone* in/outside the project can pick it up and implement it with the proposal as sole "cookbook"
> 3. **A fully working patch/pull request implementing that proposal** (and nothing else, no group merge requests for multiple security-related fixes. They should be atomic.)
> 4. The pull request/patch must be **reviewed and tested by the "Cryptocat Team"**
> 5. This code must then be **left frozen for a period of time** to allow for public, open exterior review
> 6. Only when points **2**, **3**, **4** and **5** are considered sufficiently "correct", can the pull request/patch be applied

Files and portions of code which have not been tagged as "security critical" are not subjected to this procedure. However, even one-character commits must go though it, even if quickly, as we might break the entire security of the project with a single misplaces character.

- On important and meaningful changes to cryptographic or security strategies, **an announcement must be publicly** made after step **5** to make sure enough peer review has been carried out and the peer community approves of these changes. The associated freeze period shall be prolonged to as long as necessary for enough peers to find the time to have a look at the changes and their implications.

- For critical security fixes which need to be pushed ASAP for the security of users, **a threat assessment must be published online immediately**, and steps **1** and **3** can be ignored, while **steps 2 and 5 must still happen after the patch has been provided, applied and released**. This is to find a balance between urgency and making sure we're not making things worse in the haste.

- **Critical security fixes will be rooted in the last stable version**'s code tag and will consist of a new signed tag altogether. (no applying quick fixes to an untested git master after 6 months of modifying the codebase since the last stable release, and releasing that out as stable)

- **A patch or pull request will not be accepted, reviewed or even considered until a full specification has been formulated**. If the idea is great someone else can write a specification around it, but the acceptation process will not go further than **3** until **2** is completed.


-----------------------
Release process
==============

1. **One month before any release**, we apply a **complete feature freeze** of the project. Only bugfixes and security fixes are worked on.

2. **Two weeks before any release**, a **complete code freeze** is applied, packages are generated and signed. They will be pushed publicly as "Beta" versions, and will only be sent to people specifically willing to use bleeding edge software at the expense of security.

3. **During these two weeks**, the only issues and bugs that are worked on are those coming directly from these beta testings. They will be merged to the git repository for testing and deployment to developpers but no new beta build will be pushed to the public.

4. **At the end of the two weeks**, it will be **up to the "Cryptocat Team" to decide** whether there has been enough fixes necessary to **grant a new beta build** incorporating all the fixes **which will in turn be tested for two weeks**. Repeat this cycle until a beta phase happens with as little problems as necessary to be confident that important regressions would have been spotted by then. (Linux kernel style)

5. If some of the bugs found during the beta phase need important refactoring of the code, **lift the code freeze and delay the release** as much as necessary, considering these new taks as "blocking" before any release happens. This means going back to setp **1** when the refactoring is completed.

6. **When a codebase has been approved for release**, the protocol documentation associated with modified portions of the code must be updated. **No release will be carried out until the documentation matches the code**.

7. **Simultaneously**, unit tests are applied to the resulting codebase. If they all pass, go to **8**, if not, file a regression bug and go back to Beta phase.

8. Tag the version of the git source code, sign the tag with a PGP key, build signed packages, publish their checksums and provide build instructions which would result in the exact same build with the exact same checksum as the project's build process produced. (a.k.a deterministic builds. It shouldn't be too hard for browser extensions.)

9. Publish a **FULL changelog** with links to associated github issues if there are any, along with a blog post explaining what important changes happened. We cannot expect everyone to open the code itself to review our choices, so a link to the up-do-date documentation is provided along. (this is where having a proper documentation is crucial.)

10. New development cycle.

----------
Side notes
=========

- Obviously, we already do some of this in one way or another, but formalizing won't hurt.
- The beta packages distribution will be a tricky problem. The ideal solution is making a separate browser extension with "Experminetal" in the name, that will get updated at its own pace, but some distirbutions platforms have too much overhead to allow for this strategy (hello Mozilla.)
- signing keys must be publicly advertised, accessible and verifiable
- **For each documented important security bug, a unit test must exist** for **all subsequent releases**. Not passing any one of these is a red flag and a hard blocker. If no unit test can be automated for a specific security bug, create a CheckList of things to manually inspect in the code. (this last part will be costly and exausting, which is a good incentive to automate it whenever possible :) )
- Make full use of media (twitter, facebook etc) to ask for external review each time

> A. **an important proposal is formulated**
> B. **an important pull request/patch has been merged/applied**
> C. **a release is imminent** (for example, beta testers are needed) **and the documentation is up-to-date**
> D. **the documentation seems in incoherent state and a joint effort is needed to get it to an acceptable level.** (<<- This is where we stand right now, IMHO, but that's not our focus here.)

- This procedure must be the object of a public debate and review itself as if it was part of the codebase itself. (this includes the proposal step as a requirement for any procedure change.)
- This procedure must be clearly defined, documented, and easily accessible from the projet's website and the code repository in a format that's as readable as possible.
- This document looks great and we should link it along with out protocol documentation, so people have a nice list of things to check for when reviewing : https://github.com/iSECPartners/LibTech-Auditing-Cheatsheet

-----------------------

This is nothing but a suggestion for a draft to spark a debate. all this bulletpoints can and should be debate until we draft a complete process we all agree on.

So this is not a "yes" or "no" question. It's a "what do you think" one :)

Keep in mind that if we get this right, it could be a strong common basis for other security-related projets facing the same difficulties as we do. So it's not only cryptocat-centric, it could help a lot of other projects !
	Relating to https://github.com/cryptocat/cryptocat/issues/502 , I think it is actually a very good opportunity to work on a development/release process that would strenghten the security and reliability of cryptocat software.

	------------

	The mains goals of these suggestions are:
	- More stability of the security-critical code during the development phases
	- Better beta testing and less bugs/flaws in stable releases
	- Overall better documentation and making it easier for people to review our choices

	------------
	Ongoing development
	==================

	Files and pieces of code relating (even indirectly) to the resulting security of the software must be identified and flagged in some way.

	- None of these files is allowed to be modified without a rigid procedure consisting of, in this order:

	> 1. An explanation of what the modification intends to fix (implied: "How is it now, how will it be then.")
	> 2. A complete specification of the proposal, with as much concrete examples as necessary. You know the proposal is correctly described if anyone in/outside the project can pick it up and implement it with the proposal as sole "cookbook"
	> 3. A fully working patch/pull request implementing that proposal (and nothing else, no group merge requests for multiple security-related fixes. They should be atomic.)
	> 4. The pull request/patch must be reviewed and tested by the "Cryptocat Team"
	> 5. This code must then be left frozen for a period of time to allow for public, open exterior review
	> 6. Only when points 2, 3, 4 and 5 are considered sufficiently "correct", can the pull request/patch be applied

	Files and portions of code which have not been tagged as "security critical" are not subjected to this procedure. However, even one-character commits must go though it, even if quickly, as we might break the entire security of the project with a single misplaces character.

	- On important and meaningful changes to cryptographic or security strategies, an announcement must be publicly made after step 5 to make sure enough peer review has been carried out and the peer community approves of these changes. The associated freeze period shall be prolonged to as long as necessary for enough peers to find the time to have a look at the changes and their implications.

	- For critical security fixes which need to be pushed ASAP for the security of users, a threat assessment must be published online immediately, and steps 1 and 3 can be ignored, while steps 2 and 5 must still happen after the patch has been provided, applied and released. This is to find a balance between urgency and making sure we're not making things worse in the haste.

	- Critical security fixes will be rooted in the last stable version's code tag and will consist of a new signed tag altogether. (no applying quick fixes to an untested git master after 6 months of modifying the codebase since the last stable release, and releasing that out as stable)

	- A patch or pull request will not be accepted, reviewed or even considered until a full specification has been formulated. If the idea is great someone else can write a specification around it, but the acceptation process will not go further than 3 until 2 is completed.


	-----------------------
	Release process
	==============

	1. One month before any release, we apply a complete feature freeze of the project. Only bugfixes and security fixes are worked on.

	2. Two weeks before any release, a complete code freeze is applied, packages are generated and signed. They will be pushed publicly as "Beta" versions, and will only be sent to people specifically willing to use bleeding edge software at the expense of security.

	3. During these two weeks, the only issues and bugs that are worked on are those coming directly from these beta testings. They will be merged to the git repository for testing and deployment to developpers but no new beta build will be pushed to the public.

	4. At the end of the two weeks, it will be up to the "Cryptocat Team" to decide whether there has been enough fixes necessary to grant a new beta build incorporating all the fixes which will in turn be tested for two weeks. Repeat this cycle until a beta phase happens with as little problems as necessary to be confident that important regressions would have been spotted by then. (Linux kernel style)

	5. If some of the bugs found during the beta phase need important refactoring of the code, lift the code freeze and delay the release as much as necessary, considering these new taks as "blocking" before any release happens. This means going back to setp 1 when the refactoring is completed.

	6. When a codebase has been approved for release, the protocol documentation associated with modified portions of the code must be updated. No release will be carried out until the documentation matches the code.

	7. Simultaneously, unit tests are applied to the resulting codebase. If they all pass, go to 8, if not, file a regression bug and go back to Beta phase.

	8. Tag the version of the git source code, sign the tag with a PGP key, build signed packages, publish their checksums and provide build instructions which would result in the exact same build with the exact same checksum as the project's build process produced. (a.k.a deterministic builds. It shouldn't be too hard for browser extensions.)

	9. Publish a FULL changelog with links to associated github issues if there are any, along with a blog post explaining what important changes happened. We cannot expect everyone to open the code itself to review our choices, so a link to the up-do-date documentation is provided along. (this is where having a proper documentation is crucial.)

	10. New development cycle.

	----------
	Side notes
	=========

	- Obviously, we already do some of this in one way or another, but formalizing won't hurt.
	- The beta packages distribution will be a tricky problem. The ideal solution is making a separate browser extension with "Experminetal" in the name, that will get updated at its own pace, but some distirbutions platforms have too much overhead to allow for this strategy (hello Mozilla.)
	- signing keys must be publicly advertised, accessible and verifiable
	- For each documented important security bug, a unit test must exist for all subsequent releases. Not passing any one of these is a red flag and a hard blocker. If no unit test can be automated for a specific security bug, create a CheckList of things to manually inspect in the code. (this last part will be costly and exausting, which is a good incentive to automate it whenever possible :) )
	- Make full use of media (twitter, facebook etc) to ask for external review each time

	> A. an important proposal is formulated
	> B. an important pull request/patch has been merged/applied
	> C. a release is imminent (for example, beta testers are needed) and the documentation is up-to-date
	> D. the documentation seems in incoherent state and a joint effort is needed to get it to an acceptable level. (<<- This is where we stand right now, IMHO, but that's not our focus here.)

	- This procedure must be the object of a public debate and review itself as if it was part of the codebase itself. (this includes the proposal step as a requirement for any procedure change.)
	- This procedure must be clearly defined, documented, and easily accessible from the projet's website and the code repository in a format that's as readable as possible.
	- This document looks great and we should link it along with out protocol documentation, so people have a nice list of things to check for when reviewing : https://github.com/iSECPartners/LibTech-Auditing-Cheatsheet

	-----------------------

	This is nothing but a suggestion for a draft to spark a debate. all this bulletpoints can and should be debate until we draft a complete process we all agree on.

	So this is not a "yes" or "no" question. It's a "what do you think" one :)

	Keep in mind that if we get this right, it could be a strong common basis for other security-related projets facing the same difficulties as we do. So it's not only cryptocat-centric, it could help a lot of other projects !