ArneBab/build0014-draft-changelog.txt Secret

## build0014-draft-changelog.txt
This build concludes the work done in the past 6 months. Along with heavy restructuring there is an insane amount of code quality improvements and bugfixes. This will pay out a lot during future development, as the huge maintenance backlog which built up during the years I wasn't working on WOT has now started to go away.

The main feature is the introduction of the "event-notifications" FCP API which enables applications to request notification of changes without always having to request all known identities.

This is a framework which applications built on top of WOT such as Sone etc. need to first start using before it reduces their performance impact. A reallife example to understand why this is the case: If your mail service delivers letters by horse, they won't arrive any faster because the government has just build a railway system. The mail company needs to first start using trains instead of horses.
If you are a developer of a WOT applicationn, please read the "Changelog for developers" section for instructions how to use this.

There are at least 5 further strong algorithmic performance optimizations possible within WOT itself which I shall work on for the following releases.

Those should speed up WOT without requiring adaptions from client application developers. It was chosen to implement event-notifications before these so we can have a parallelization of development: Client app developers can start integrating event-notifications now, while I can work on the WOT core optimizations.

Sorry that speeding up WOT takes so long. WOT is a major project - 18 000 lines of code already, and as such just takes its time to optimize. I am as annoyed as anyone else by the performance issues, and trying to do my best to get them fixed ASAP. Please stay patient :)


What follows now are two changelogs of this build: One of the new features which are primarily relevant to the users; and one for new stuff which can be used by developers of applications which use WOT. The changelogs are briefly sorted by importance, most important stuff first.
The number left to each change is the issue number in the bugtracker.

Notice: The changelogs exclude a LOT of the actual changes. This is because the development of this build took a very long time, and thus to speed up the release, I produced the changelog from the bugtracker instead of reviewing the insane amount of the FIXME commits. So what had no bugtracker entry is likely not included. Also, I've removed anything which isn't "visible" to users or developers, i.e. anything which is not a new feature or bugfix. And bugfixes are also only listed if they are easy to explain/understand. As a conclusion, the very very many code quality improvements and core reliability bugfixes are not in the changelog. If you're interested in the unfiltered list of the 48 resolved bugtracker entries in this release, go to https://bugs.freenetproject.org/changelog_page.php?version_id=63.
Sorry about the lack of a complete changelog due to the time pressure to release soon. I will aim for a MUCH shorter release cycle with following builds; and then you'll get full changelogs again.


Changelog for users:
- 0006039: [Web Interface] WebInterface "Log in" refactoring TODOs (xor)
	Operhiem1 helped a lot with dealing with the job of refactoring the general style of the web interface to allow users to "Log in" with a certain identity instead of having to chose which identity to work with at every page. This should greatily increase usability as it matches the general convention of how websites work.
	Also, this functionality can be used by authors of other plugins so there is one common WOT login for anything which uses WOT.

- 0006027: [Web Interface] Backport LogInPage features from Freetalk to WOT (xor)
	The new "Log in" page features a wizard for creating and restoring identities. This wizard explains what WOT is, what an identity is, what trust values are, etc. It allows the user to chose preferences. Also, it fixes the issue of people mixing up the features for restoring an existing identity and creating a new one from a SSK, which could have caused their identity to be permanently broken. This is fixed by explaining the difference very clearly.
	Same as with the "Log in" page, the wizard can be used by other plugins.

- 0006225: [Web Interface] Make the KnownIdentitiesPage split the list into pages (xor)
	Previously, the "Known identities" page would list ALL identites in a single page. This was obviously not acceptable anymore since we have > 10 000 identities nowadays.
	Now the list is split into pages.

- [Not in bugtracker] [Web Interface]: Translation updates
	Add French and Simplified Chinese translations, and update Italian, Dutch, and Russian.
	Thanks to the volunteers on Transifex, and thanks to operhiem1 for putting this into a pull request.
	Notice: This is probably incomplete already because it was done before adding the "Create identity" wizard. AFAIK, the new stuff is up on Transifex already, so if you want to contribute with translating it, please contact me or operhiem1. xor@freenetproject.org, or even better the #freenet channel on Freenode IRC.

- [No bugtracker entry] [Web Interface]: Tell the user whats invalid about a chosen nickname
	When the user enters a nick which contains non-allowed characters, we now tell him which those characters are.

- 0006136: [Web Interface] "Restore identity" is only accessible if the user creates an identity first (xor)
	Previously, the menu for restoring an old identity was only accessible if you actually had already created least one identity. The "Restore identity" feature can now be used through the "Create identity" wizard without creating a completely new identity first.

- 0006148: [Web Interface] Randomly generated nicknames may exceed 30 character limit (xor)
	The nicknames now always fit the 30 character limit.

- 0005354: [Web Interface] Usability of the web interface (xor)
	When clicking on "Community" or "Community -> Home", the "Own anonymous identities" page is now shown instead of the "Statistics" page.

- 0006458: [Performance] Parallelize shutdown (xor)
	During shutdown, instead of waiting for each subsystem of WOT to have terminated before termining the next one, we now terminate them all in parallel using threads.
	This should speed up shutdown a lot (but please don't expect wonders, there are further issues with shutdown which the next release will improve).

- 0006447: [Performance] IdentityInserter thread period constant is too low and causes inserts to never succeed (xor)
	This fixed bug would prevent your identities from ever being inserted unless you had very high uptime.

- 0006001: [Core] Restore WebOfTrust.db4o.backup if WebOfTrust.db4o does not exist (xor)
	This fixed bug would happen if your computer crashed while WOT was starting up. The symptom would be loss of your database, i.e. all your identities would be gone.

- 0004854: [Web Interface] Stacktrace when clicking 'switch to simple mode' while viewing an identity (xor)
	Clicking this web interface link of Freenet did cause WOT to display an error, which is now fixed.

- [No bugtracker entry] [Performancde] New seed identity, thanks to operhiem1
	Having another seed identity should speed up the bootstrapping for new users.


Changelog for developers:
- [Various bugtracker entries] Event-notifications:
	This new FCP API should provide a HUGE reduction of the load which client apps put onto WOT.
	What this changes is as follows: To get an up-to-date list of all known identities (or trusts/scores), you previously had to download ALL of them by FCP every X minutes. You would do this even if NONE of them changed - a huge CPU/IO load for nothing. Now you just "Subscribe" to the list of identities. As a result, WOT will send you all of them once as you subscribe, and then keep you up to date on its own by sending changed identities only when a change happens.
	Thus, instead of a constant load of O(N = number of identites) every X minutes, independent of the amount of changes, you now get a load of O(1) for each change as it happens.

	If you are a client application developer, and want to integrate event-notifications in your app, I strongly recommend you to
	1) Use the new classes DebugFCPClient and FCPClientReferenceImplementation. The FCPClientReferenceImplementation aims at being something which you can copy-paste to your client application plugin so you have a fully working FCP client which you can use to connect to WOT and obtain event-notifications. The DebugFCPClient uses the aforementioned reference FCP client to have WOT connect to itself and test whether event-notifications ships correct information (test runs of this have been done for several nights). For your purposes, it can also be considered as an example of how to actually use the FCPClientReferenceImplementation.
	2) Contact me in case you need help in understanding the whole stuff! :) xor@freenetproject.org, or even better the #freenet channel on Freenode IRC.

- 0006039: [Web Interface] WebInterface "Log in" refactoring TODOs (xor)
	(Even though this seems something for users, it IS interesting for developers, please continue reading)
	Operhiem1 helped a lot with dealing with the job of refactoring the general style of the web interface to allow users to "Log in" with a certain identity instead of having to chose which identity to work with at every page.

	And this functionality can be used by your plugins! You can redirect users of your plugin to the log in page of WOT, and have WOT redirect the users back to your plugin after they have logged in. For an example of how to use this, see FIXME. Please contact me or operhiem1 for further help. xor@freenetproject.org, or even better the #freenet channel on Freenode IRC.

- 0006027: [Web Interface] Backport LogInPage features from Freetalk to WOT (xor)
	The new "Log in" page features a wizard for creating and restoring identities.
	Same as with the "Log in" page, the wizard can be used by other plugins. For an example of how to use this, see FIXME. Please contact me or operhiem1 for further help. xor@freenetproject.org, or even better the #freenet channel on Freenode IRC.

- 0006231: [Core] Finish synchronous FCP API at fred (xor)
	The implementation of event-notification spawned a huge 569-commit side-project at Freenet itself aka "plugin-fcp-rewrite": The rewrite of the FCP API for communication with plugins, and especially for plugin-to-plugin FCP communication.
	This adds a lot of new features to replace the code duplication in existing plugins, as they had usually implemented their own extensions of the old API to fix its shortcomings. It also provides new stuff which was completely impossible with the previous API. Notably for example the ability for servers to send messages to the client on their own, without any recent message from the client. This allows plugin FCP server authors to deploy events to clients.
	Also it improves the code quality of the FCP plugin API a lot - the old API was completely undocumented, the new one has full JavaDoc.
	There is a much longer and more complete description of whats new in the new API at the pull request. Anyone who has written a plugin which offers an FCP server or uses FCP to connect to another plugin such as WOT, please read it: https://github.com/freenet/fred/pull/319

	Notice: WOT will for a while stay backwards-compatible to users of the old API. However, to use event-notifications, you must use the new API. I would strongly recommend you to just copypaste the WOT class FCPClientReferenceImplementation for communicating with WOT by FCP, it does use the new API alredy.

- [Pull request 26] Background jobs
	Bertm did the great job of implementing a framework for replacing dangerous usage of the fred implementations of interface Ticker such as especially TrivialTicker. This might be interesting for fred code and other plugins which use those fred classes. The issue is that the existing fred classes provide a "no duplicates" flag which sounds like it ensures that a scheduled job will only ever be executed once. But under certain conditions, the job can actually be executed multiple times in parallel. The new interface "BackgroundJob" and its implementations provide a way to run a thread with a time delay and the guarantee that it will execute only once. This is a candidate for being moved to fred or a common library.
	Bertm saved me probably weeks of work with that, thanks a lot!

- [No bugtracker entry]: Full-node unit tests
	WOT now contains class "AbstractFullNodeTest" which can be used as a base class for unit tests. This class starts a regular Freenet node, and loads the WOT plugin into it just as it would be loaded during normal operation of Freenet. From a quality point of view, this is a lot better since it provides a more real environment than the previous approach of manually creating certain WOT classes in a synthetic environment. Also, it allows unit tests to use the full capabilities of Freenet. This is a foundation for future tests which could do fancy stuff such as testing the Identity introduction code by doing real uploads / downloads of puzzles and their solutions.

	Notice: This might also be interesting to authors of other plugins. The code is pretty generic and should be easy to re-use there.

- [No bugtracker entry]: Randomized unit tests which aim to cover all code paths of WOT
	Function doRandomChangesToWOT() in the new class AbstractJUnit4BaseTest is a framework for thorough, randomized unit tests. It simulates real usage by doing a random amount of changes chosen among all possible user actions:
		- WebOfTrust.createOwnIdentity()
		- WebOfTrust.deleteOwnIdentity()
		- WebOfTrust.restoreOwnIdentity()
		- WebOfTrust.restoreOwnIdentity() with existing colliding non-own Identity
		- WebOfTrust.addIdentity()
		- WebOfTrust.addContext() (adds context to identity)
		- WebOfTrust.setProperty (adds property to identity)
		- Add/change/Remove trust values
	This can help with testing stuff for which it is difficult to say what might cause problems.
	It is already used for testing-event notifications: Any of the mentioned user actions are sources of events, so it is a good idea to test a random list of events for whether event-notifications ships them correctly.

Thanks to:
	- operhiem1 for various pull requests, the great amount of web interface work, and insane amount of code review
	- bertm for the giant background-job pull request which probably saved me weeks worth of work
	- ArneBab for various pull requests and code review
	- TheSeeker for testing
	- anyone who I forgot about
	- and basically EVERYONE in the project for having the great great patience of awating this release for over a year. Sorry again that it took so long! The only explanation I can offer of why this happend is because I wanted to not release half-finished stuff anymore given that the whole WOT peformance disaster was a result of deploying half-finished code as an official plugin. Nevertheless, I hope to have learned something from all the complaints about the huge delay of the release, and will try to have a MUCH shorter release cycle in the future.
	This build concludes the work done in the past 6 months. Along with heavy restructuring there is an insane amount of code quality improvements and bugfixes. This will pay out a lot during future development, as the huge maintenance backlog which built up during the years I wasn't working on WOT has now started to go away.

	The main feature is the introduction of the "event-notifications" FCP API which enables applications to request notification of changes without always having to request all known identities.

	This is a framework which applications built on top of WOT such as Sone etc. need to first start using before it reduces their performance impact. A reallife example to understand why this is the case: If your mail service delivers letters by horse, they won't arrive any faster because the government has just build a railway system. The mail company needs to first start using trains instead of horses.
	If you are a developer of a WOT applicationn, please read the "Changelog for developers" section for instructions how to use this.

	There are at least 5 further strong algorithmic performance optimizations possible within WOT itself which I shall work on for the following releases.

	Those should speed up WOT without requiring adaptions from client application developers. It was chosen to implement event-notifications before these so we can have a parallelization of development: Client app developers can start integrating event-notifications now, while I can work on the WOT core optimizations.

	Sorry that speeding up WOT takes so long. WOT is a major project - 18 000 lines of code already, and as such just takes its time to optimize. I am as annoyed as anyone else by the performance issues, and trying to do my best to get them fixed ASAP. Please stay patient :)


	What follows now are two changelogs of this build: One of the new features which are primarily relevant to the users; and one for new stuff which can be used by developers of applications which use WOT. The changelogs are briefly sorted by importance, most important stuff first.
	The number left to each change is the issue number in the bugtracker.

	Notice: The changelogs exclude a LOT of the actual changes. This is because the development of this build took a very long time, and thus to speed up the release, I produced the changelog from the bugtracker instead of reviewing the insane amount of the FIXME commits. So what had no bugtracker entry is likely not included. Also, I've removed anything which isn't "visible" to users or developers, i.e. anything which is not a new feature or bugfix. And bugfixes are also only listed if they are easy to explain/understand. As a conclusion, the very very many code quality improvements and core reliability bugfixes are not in the changelog. If you're interested in the unfiltered list of the 48 resolved bugtracker entries in this release, go to https://bugs.freenetproject.org/changelog_page.php?version_id=63.
	Sorry about the lack of a complete changelog due to the time pressure to release soon. I will aim for a MUCH shorter release cycle with following builds; and then you'll get full changelogs again.


	Changelog for users:
	- 0006039: [Web Interface] WebInterface "Log in" refactoring TODOs (xor)
	Operhiem1 helped a lot with dealing with the job of refactoring the general style of the web interface to allow users to "Log in" with a certain identity instead of having to chose which identity to work with at every page. This should greatily increase usability as it matches the general convention of how websites work.
	Also, this functionality can be used by authors of other plugins so there is one common WOT login for anything which uses WOT.

	- 0006027: [Web Interface] Backport LogInPage features from Freetalk to WOT (xor)
	The new "Log in" page features a wizard for creating and restoring identities. This wizard explains what WOT is, what an identity is, what trust values are, etc. It allows the user to chose preferences. Also, it fixes the issue of people mixing up the features for restoring an existing identity and creating a new one from a SSK, which could have caused their identity to be permanently broken. This is fixed by explaining the difference very clearly.
	Same as with the "Log in" page, the wizard can be used by other plugins.

	- 0006225: [Web Interface] Make the KnownIdentitiesPage split the list into pages (xor)
	Previously, the "Known identities" page would list ALL identites in a single page. This was obviously not acceptable anymore since we have > 10 000 identities nowadays.
	Now the list is split into pages.

	- [Not in bugtracker] [Web Interface]: Translation updates
	Add French and Simplified Chinese translations, and update Italian, Dutch, and Russian.
	Thanks to the volunteers on Transifex, and thanks to operhiem1 for putting this into a pull request.
	Notice: This is probably incomplete already because it was done before adding the "Create identity" wizard. AFAIK, the new stuff is up on Transifex already, so if you want to contribute with translating it, please contact me or operhiem1. xor@freenetproject.org, or even better the #freenet channel on Freenode IRC.

	- [No bugtracker entry] [Web Interface]: Tell the user whats invalid about a chosen nickname
	When the user enters a nick which contains non-allowed characters, we now tell him which those characters are.

	- 0006136: [Web Interface] "Restore identity" is only accessible if the user creates an identity first (xor)
	Previously, the menu for restoring an old identity was only accessible if you actually had already created least one identity. The "Restore identity" feature can now be used through the "Create identity" wizard without creating a completely new identity first.

	- 0006148: [Web Interface] Randomly generated nicknames may exceed 30 character limit (xor)
	The nicknames now always fit the 30 character limit.

	- 0005354: [Web Interface] Usability of the web interface (xor)
	When clicking on "Community" or "Community -> Home", the "Own anonymous identities" page is now shown instead of the "Statistics" page.

	- 0006458: [Performance] Parallelize shutdown (xor)
	During shutdown, instead of waiting for each subsystem of WOT to have terminated before termining the next one, we now terminate them all in parallel using threads.
	This should speed up shutdown a lot (but please don't expect wonders, there are further issues with shutdown which the next release will improve).

	- 0006447: [Performance] IdentityInserter thread period constant is too low and causes inserts to never succeed (xor)
	This fixed bug would prevent your identities from ever being inserted unless you had very high uptime.

	- 0006001: [Core] Restore WebOfTrust.db4o.backup if WebOfTrust.db4o does not exist (xor)
	This fixed bug would happen if your computer crashed while WOT was starting up. The symptom would be loss of your database, i.e. all your identities would be gone.

	- 0004854: [Web Interface] Stacktrace when clicking 'switch to simple mode' while viewing an identity (xor)
	Clicking this web interface link of Freenet did cause WOT to display an error, which is now fixed.

	- [No bugtracker entry] [Performancde] New seed identity, thanks to operhiem1
	Having another seed identity should speed up the bootstrapping for new users.


	Changelog for developers:
	- [Various bugtracker entries] Event-notifications:
	This new FCP API should provide a HUGE reduction of the load which client apps put onto WOT.
	What this changes is as follows: To get an up-to-date list of all known identities (or trusts/scores), you previously had to download ALL of them by FCP every X minutes. You would do this even if NONE of them changed - a huge CPU/IO load for nothing. Now you just "Subscribe" to the list of identities. As a result, WOT will send you all of them once as you subscribe, and then keep you up to date on its own by sending changed identities only when a change happens.
	Thus, instead of a constant load of O(N = number of identites) every X minutes, independent of the amount of changes, you now get a load of O(1) for each change as it happens.

	If you are a client application developer, and want to integrate event-notifications in your app, I strongly recommend you to
	1) Use the new classes DebugFCPClient and FCPClientReferenceImplementation. The FCPClientReferenceImplementation aims at being something which you can copy-paste to your client application plugin so you have a fully working FCP client which you can use to connect to WOT and obtain event-notifications. The DebugFCPClient uses the aforementioned reference FCP client to have WOT connect to itself and test whether event-notifications ships correct information (test runs of this have been done for several nights). For your purposes, it can also be considered as an example of how to actually use the FCPClientReferenceImplementation.
	2) Contact me in case you need help in understanding the whole stuff! :) xor@freenetproject.org, or even better the #freenet channel on Freenode IRC.

	- 0006039: [Web Interface] WebInterface "Log in" refactoring TODOs (xor)
	(Even though this seems something for users, it IS interesting for developers, please continue reading)
	Operhiem1 helped a lot with dealing with the job of refactoring the general style of the web interface to allow users to "Log in" with a certain identity instead of having to chose which identity to work with at every page.

	And this functionality can be used by your plugins! You can redirect users of your plugin to the log in page of WOT, and have WOT redirect the users back to your plugin after they have logged in. For an example of how to use this, see FIXME. Please contact me or operhiem1 for further help. xor@freenetproject.org, or even better the #freenet channel on Freenode IRC.

	- 0006027: [Web Interface] Backport LogInPage features from Freetalk to WOT (xor)
	The new "Log in" page features a wizard for creating and restoring identities.
	Same as with the "Log in" page, the wizard can be used by other plugins. For an example of how to use this, see FIXME. Please contact me or operhiem1 for further help. xor@freenetproject.org, or even better the #freenet channel on Freenode IRC.

	- 0006231: [Core] Finish synchronous FCP API at fred (xor)
	The implementation of event-notification spawned a huge 569-commit side-project at Freenet itself aka "plugin-fcp-rewrite": The rewrite of the FCP API for communication with plugins, and especially for plugin-to-plugin FCP communication.
	This adds a lot of new features to replace the code duplication in existing plugins, as they had usually implemented their own extensions of the old API to fix its shortcomings. It also provides new stuff which was completely impossible with the previous API. Notably for example the ability for servers to send messages to the client on their own, without any recent message from the client. This allows plugin FCP server authors to deploy events to clients.
	Also it improves the code quality of the FCP plugin API a lot - the old API was completely undocumented, the new one has full JavaDoc.
	There is a much longer and more complete description of whats new in the new API at the pull request. Anyone who has written a plugin which offers an FCP server or uses FCP to connect to another plugin such as WOT, please read it: https://github.com/freenet/fred/pull/319

	Notice: WOT will for a while stay backwards-compatible to users of the old API. However, to use event-notifications, you must use the new API. I would strongly recommend you to just copypaste the WOT class FCPClientReferenceImplementation for communicating with WOT by FCP, it does use the new API alredy.

	- [Pull request 26] Background jobs
	Bertm did the great job of implementing a framework for replacing dangerous usage of the fred implementations of interface Ticker such as especially TrivialTicker. This might be interesting for fred code and other plugins which use those fred classes. The issue is that the existing fred classes provide a "no duplicates" flag which sounds like it ensures that a scheduled job will only ever be executed once. But under certain conditions, the job can actually be executed multiple times in parallel. The new interface "BackgroundJob" and its implementations provide a way to run a thread with a time delay and the guarantee that it will execute only once. This is a candidate for being moved to fred or a common library.
	Bertm saved me probably weeks of work with that, thanks a lot!

	- [No bugtracker entry]: Full-node unit tests
	WOT now contains class "AbstractFullNodeTest" which can be used as a base class for unit tests. This class starts a regular Freenet node, and loads the WOT plugin into it just as it would be loaded during normal operation of Freenet. From a quality point of view, this is a lot better since it provides a more real environment than the previous approach of manually creating certain WOT classes in a synthetic environment. Also, it allows unit tests to use the full capabilities of Freenet. This is a foundation for future tests which could do fancy stuff such as testing the Identity introduction code by doing real uploads / downloads of puzzles and their solutions.

	Notice: This might also be interesting to authors of other plugins. The code is pretty generic and should be easy to re-use there.

	- [No bugtracker entry]: Randomized unit tests which aim to cover all code paths of WOT
	Function doRandomChangesToWOT() in the new class AbstractJUnit4BaseTest is a framework for thorough, randomized unit tests. It simulates real usage by doing a random amount of changes chosen among all possible user actions:
	- WebOfTrust.createOwnIdentity()
	- WebOfTrust.deleteOwnIdentity()
	- WebOfTrust.restoreOwnIdentity()
	- WebOfTrust.restoreOwnIdentity() with existing colliding non-own Identity
	- WebOfTrust.addIdentity()
	- WebOfTrust.addContext() (adds context to identity)
	- WebOfTrust.setProperty (adds property to identity)
	- Add/change/Remove trust values
	This can help with testing stuff for which it is difficult to say what might cause problems.
	It is already used for testing-event notifications: Any of the mentioned user actions are sources of events, so it is a good idea to test a random list of events for whether event-notifications ships them correctly.

	Thanks to:
	- operhiem1 for various pull requests, the great amount of web interface work, and insane amount of code review
	- bertm for the giant background-job pull request which probably saved me weeks worth of work
	- ArneBab for various pull requests and code review
	- TheSeeker for testing
	- anyone who I forgot about
	- and basically EVERYONE in the project for having the great great patience of awating this release for over a year. Sorry again that it took so long! The only explanation I can offer of why this happend is because I wanted to not release half-finished stuff anymore given that the whole WOT peformance disaster was a result of deploying half-finished code as an official plugin. Nevertheless, I hope to have learned something from all the complaints about the huge delay of the release, and will try to have a MUCH shorter release cycle in the future.