build0014-rc02-changelog.txt

With >1800 commits, this build includes more new commits than WOT ever
consisted of as of the previous build which was 1461 commits.
There is a very large amount of code quality improvements and bugfixes.
This will pay out a lot during future development hopefully; for example
the improved unit testing architecture may speed up debugging a lot.

Even though the primary purpose of this build is to alleviate the
performance issues, it will still be very slow until applications built
on top of WOT have been adapted to use the new performance features.
This has a technical reason: 
The main new performance improvement is the so-called
"event-notifications" FCP API which enables apps to request notification
of changes without having to download the whole WOT database every few
minutes. This cannot be put to use without applications being changed to
use it. A reallife example to understand why this is the case: If your
mail service delivers letters by horse, they won't arrive any faster
because the government has just build a railway system. The mail company
needs to first start using trains instead of horses.
If you are a developer of a WOT application, please read the "Changelog
for developers" section for instructions how to wire it in to your app.

There are at least 5 further strong algorithmic performance
optimizations possible within WOT itself which I shall work on for the
following releases. Those then should speed up WOT without requiring
adaptions from client application developers. It was chosen to implement
event-notifications before these so we can have a parallelization of
development: Client app developers can start integrating
event-notifications now, while I can work on the WOT core optimizations.

Sorry that speeding up WOT takes so long. WOT is a major project of
30 000 lines of code already, and as such just takes its time to
optimize. I am as annoyed as anyone else by the performance issues, and
trying to do my best to get them fixed ASAP. Please stay patient :)


What follows now are two changelogs of this build:
- One of the new features which are primarily relevant to the users
- One for new stuff which can be used by developers of applications
  which use WOT.
The changelogs are briefly sorted by importance, most important stuff
first. The number left to each change is the issue number in the
bugtracker.

Notice: The changelogs exclude a LOT of the actual changes. This is
because the development of this build took a very long time, and thus to
speed up the release, I produced the changelog from the bugtracker
instead of reviewing the whole amount of the 1800 commits. So what had
no bugtracker entry is likely not included. Also, I've removed anything
which isn't "visible" to users or developers, i.e. anything which is not
a new feature or bugfix. And bugfixes are also only listed if they are
easy to explain/understand. As a conclusion, the very very many code
quality improvements and core reliability bugfixes are not in the
changelog. If you're interested in the unfiltered list of the 48
resolved bugtracker entries in this release, go to
https://bugs.freenetproject.org/changelog_page.php?version_id=63.
Sorry about the lack of a complete changelog due to the time pressure to
release soon. I will aim for a MUCH shorter release cycle with following
builds; and then you'll get full changelogs again.


Changelog for users:

- 0006039: [Web Interface] WebInterface "Log in" refactoring TODOs (xor)
  Operhiem1 helped a lot with dealing with the job of refactoring the
  general style of the web interface to allow users to "Log in" with a
  certain identity instead of having to chose which identity to work
  with at every page. This should greatly increase usability as it
  matches the general convention of how websites work.
  Also, this functionality can be used by authors of other plugins so
  there is one common WOT login for anything which uses WOT.

- 0006027: [Web Interface] Backport LogInPage features from Freetalk to
  WOT (xor)
  The new "Log in" page features a wizard for creating and restoring
  identities. This wizard explains what WOT is, what an identity is,
  what trust values are, etc. It allows the user to chose preferences.
  Also, it fixes the issue of people mixing up the features for
  restoring an existing identity and creating a new one from a SSK,
  which could have caused their identity to be permanently broken. This
  is fixed by explaining the difference very clearly.
  Same as with the "Log in" page, the wizard can be used by other
  plugins.

- 0006225: [Web Interface] Make the KnownIdentitiesPage split the list
  into pages (xor)
  Previously, the "Known identities" page would list ALL identites in a
  single page. This was obviously not acceptable anymore since we have
  > 11 000 identities nowadays.
  Now the list is split into pages.

- [Not in bugtracker] [Web Interface]: Translation updates
  Add French and Simplified Chinese translations, and update Italian,
  Dutch, and Russian.
  Thanks to the volunteers on Transifex, and thanks to operhiem1 for
  putting this into a pull request.
  Notice: This is probably incomplete already because it was done before
  adding the "Create identity" wizard. AFAIK, the new stuff is up on
  Transifex already, so if you want to contribute with translating it,
  please contact me or operhiem1. Mail xor@freenetproject.org, or even
  better join the #freenet channel on Freenode IRC.

- [No bugtracker entry] [Web Interface]: Tell the user whats invalid 
  bout a chosen nickname
  When the user enters a nick which contains non-allowed characters, we
  now tell him which those characters are.

- 0006136: [Web Interface] "Restore identity" is only accessible if the
  user creates an identity first (xor)
  Previously, the menu for restoring an old identity was only accessible
  if you actually had already created least one identity. The
  "Restore identity" feature can now be used through the
  "Create identity" wizard without creating a completely new identity
  first.

- 0006148: [Web Interface] Randomly generated nicknames may exceed 30
  character limit (xor)
  The nicknames now always fit the 30 character limit.

- 0005354: [Web Interface] Usability of the web interface (xor)
  When clicking on "Community" or "Community -> Home", the "Own
  anonymous identities" page is now shown instead of the "Statistics"
  page.

- 0006458: [Performance] Parallelize shutdown (xor)
  During shutdown, instead of waiting for each subsystem of WOT to have
  terminated before termining the next one, we now terminate them all in
  parallel using threads.
  This should speed up shutdown a lot (but please don't expect wonders,
  there are further issues with shutdown which the next release will
  improve).

- 0006447: [Performance] IdentityInserter thread period constant is too
  low and causes inserts to never succeed (xor)
  This fixed bug would prevent your identities from ever being inserted
  unless you had very high uptime.

- 0006001: [Core] Restore WebOfTrust.db4o.backup if WebOfTrust.db4o does
  not exist (xor)
  This fixed bug would happen if your computer crashed while WOT was
  starting up. The symptom would be loss of your database, i.e. all your
  identities would be gone.

- 0004854: [Web Interface] Stacktrace when clicking 'switch to simple
  mode' while viewing an identity (xor)
  Clicking this web interface link of Freenet did cause WOT to display
  an error, which is now fixed.

- [No bugtracker entry] [Performancde] New seed identity, thanks to
  operhiem1
  Having another seed identity should speed up the bootstrapping for new
  users.


Changelog for developers:
- [Various bugtracker entries] Event-notifications:
  This new FCP API should provide a HUGE reduction of the load which
  client apps put onto WOT.
  What this changes is as follows: To get an up-to-date list of all
  known identities (or trusts/scores), you previously had to download
  ALL of them by FCP every X minutes. You would do this even if NONE of
  them changed - a huge CPU/IO load for nothing. Now you just
  "Subscribe" to the list of identities. As a result, WOT will send you
  all of them once as you subscribe, and then keep you up to date on its
  own by sending changed identities only when a change happens.
  Thus, instead of a constant load of O(N = number of identites) every X
  minutes, independent of the amount of changes, you now get a load of
  O(1) for each change as it happens.

  If you are a client application developer, and want to integrate
  event-notifications in your app, I strongly recommend you to
  1) Use the new classes DebugFCPClient and
     FCPClientReferenceImplementation. The latter aims at being
     something which you can copy-paste to your client application
     plugin so you have a fully working FCP client which you can use to
     connect to WOT and obtain event-notifications. The DebugFCPClient
     uses the aforementioned reference FCP client to have WOT connect
     to itself and test whether event-notifications ships correct
     information (test runs of this have been done for several nights).
     For your purposes, it can also be considered as an example of how
     to actually use the FCPClientReferenceImplementation.
  2) Contact me in case you need help in understanding how to use it! :)
     Mail xor@freenetproject.org, or even better join the #freenet
     channel on Freenode IRC.

- 0006039: [Web Interface] WebInterface "Log in" refactoring TODOs (xor)
  (Even though this seems something for users, it IS interesting for
  developers, please continue reading)
  Operhiem1 helped a lot with dealing with the job of refactoring the
  general style of the web interface to allow users to "Log in" with a
  certain identity instead of having to chose which identity to work
  with at every page.

  And this functionality can be used by your plugins! You can redirect
  users of your plugin to the log in page of WOT, and have WOT redirect
  the users back to your plugin after they have logged in. For an
  example of how to use this, see
  https://github.com/Thynix/plugin-DVCS-WebUI/blob/4893cb22d919c0d6b9178711dbb4990f66a52232/src/main/java/org/freenetproject/plugin/dvcs_webui/ui/web/VelocityToadlet.java
  Please contact me or operhiem1 for further help. Mail
  xor@freenetproject.org, or even better join the #freenet channel on
  Freenode IRC.

- 0006027: [Web Interface] Backport LogInPage features from Freetalk to
  WOT (xor)
  The new "Log in" page features a wizard for creating and restoring
  identities.
  Same as with the "Log in" page, the wizard can be used by other
  plugins. For an example of how to use this, contact me. Mail
  xor@freenetproject.org, or even better join the #freenet channel on 
  Freenode IRC.

- 0006231: [Core] Finish synchronous FCP API at fred (xor)
  The implementation of event-notification spawned a huge 569-commit
  side-project at Freenet itself aka "plugin-fcp-rewrite": The rewrite
  of the FCP API for communication with plugins, and especially for
  plugin-to-plugin FCP communication. 
  This adds a lot of new features to replace the code duplication in
  existing plugins, as they had usually implemented their own extensions
  of the old API to fix its shortcomings. It also provides new stuff
  which was completely impossible with the previous API. Notably for
  example the ability for servers to send messages to the client on
  their own, without any recent message from the client. This allows
  plugin FCP server authors to deploy events to clients.
  Also it improves the code quality of the FCP plugin API a lot - the
  old API was completely undocumented, the new one has full JavaDoc.
  There is a much longer and more complete description of whats new in
  the new API at the pull request. Anyone who has written a plugin which
  offers an FCP server or uses FCP to connect to another plugin such as
  WOT, please read it: https://github.com/freenet/fred/pull/319

  Notice: WOT will for a while stay backwards-compatible to users of the
  old API. However, to use event-notifications, you must use the new
  API. I would strongly recommend you to just copypaste the WOT class
  FCPClientReferenceImplementation for communicating with WOT by FCP, it
  does use the new API alredy.

- [Pull request 26] Background jobs
  Bertm did a great job with implementing a framework for replacing
  dangerous usage of the fred implementations of interface Ticker such
  as especially TrivialTicker. This might be interesting for fred code
  and other plugins which use those fred classes. The issue was that the
  existing fred classes provide a "no duplicates" flag which sounds like
  it ensures that a scheduled job will only ever be executed once. But
  under certain conditions, the job can actually be executed multiple
  times in parallel. The new interface "BackgroundJob" and its
  implementations provide a way to run a thread with a time delay and
  the guarantee that it will execute only once in parallel. This is a
  candidate for being moved to fred or a common library.
  Bertm saved me probably weeks of work with that, thanks a lot!

- [No bugtracker entry]: Full-node unit tests
  WOT now contains class "AbstractFullNodeTest" which can be used as a
  base class for unit tests. This class starts a regular Freenet node,
  and loads the WOT plugin into it just as it would be loaded during
  normal operation of Freenet. From a quality point of view, this is a
  lot better since it provides a more real environment than the previous
  approach of manually creating certain WOT classes in a synthetic
  environment. Also, it allows unit tests to use the full capabilities
  of Freenet. This is a foundation for future tests which could do fancy
  stuff such as testing the Identity introduction code by doing real
  uploads / downloads of puzzles and their solutions.

  Notice: This might also be interesting to authors of other plugins.
  The code is pretty generic and should be easy to re-use there.

- [No bugtracker entry]: Randomized unit tests which aim to cover all
  code paths of WOT
  Function doRandomChangesToWOT() in the new class
  AbstractJUnit4BaseTest is a framework for thorough, randomized unit
  tests. It simulates real usage by doing a random amount of changes
  chosen among many of the possible UI action:
    - WebOfTrust.createOwnIdentity()
    - WebOfTrust.deleteOwnIdentity()
    - WebOfTrust.restoreOwnIdentity()
    - WebOfTrust.restoreOwnIdentity() with existing colliding non-own
      Identity
    - WebOfTrust.addIdentity()
    - WebOfTrust.addContext() (adds context to identity)
    - WebOfTrust.setProperty (adds property to identity)
    - Add/change/Remove trust values
  This can help with testing stuff for which it is difficult to say what
  might cause problems.
  It is already used for testing-event notifications: Any of the
  mentioned user actions are sources of events, so it is a good idea to
  test a random permutation of them for whether event-notifications
  ships their events correctly.

Thanks to:
  - operhiem1 for various pull requests, the great amount of web
    interface work, and insane amount of code review.
  - bertm for the giant background-job pull request which probably saved
    me weeks worth of work
  - ArneBab for various pull requests and code review
  - TheSeeker for testing
  - anyone who I forgot about
  - and basically EVERYONE in the project for having the great great
    patience of awating this release for over a year. Sorry again that
    it took so long! The only explanation I can offer of why this
    happend is because I wanted to not release half-finished stuff
    anymore given that the whole WOT peformance disaster was a result of
    deploying half-finished code as an official plugin. Nevertheless, I
    hope to have learned something from all the complaints about the
    huge delay of the release, and will try to have a MUCH shorter
    release cycle in the future.