Camunda 7 search `noindex` experiment results

experiment-results.md

(Contents copied with permission from a private GitHub issue.)

Ok, I'm ready to make an official decision on this experiment.

Data

1. "Google chose different canonical than user" pages are dropping significantly! This is a huge win:
2. A non-trivial number of pages in that dataset have canonicals in version 7.15. I think it's around 6 or 7 of the 33 pages remaining. This is definitely not a decrease from the pre-experiment state, and might even be an increase.
3. Seven 7.15 pages now appear in Google's index. It may have been unrealistic, but I was hoping for zero of these:
4. Weirdly, the pages that appear in the Google search results are not the same pages that appear in the "Google chose a different canonical" dataset. There is some overlap, but they are mostly different pages. I think this is an "eventual consistency" thing with Google, and not something we did wrong, but it is also very confusing.

Interpretation

1. I think this experiment is unsuccessful. The original success criteria were:

   • 7.15 canonicals have declined by at least 50% in the "Duplicate, Google chose different canonical than user" dataset
   • No new 7.15 results show i.e. The Google search query site:docs.camunda.org inurl:"15" -inurl:"javadoc" still yields 0 results.

   Neither of those criteria were met.

2. I think the original success criteria were maybe slightly misguided, but I don't think they were completely invalid. I do think my goal of "zero 7.15 search results" is not as important as I originally thought when I wrote those goals. It's likely better to have any version of a page in the search results than no version in the search results. Though I will note that even if a non-7.18 page appeared in Google's search results, on google.com, it still would not appear in the c7 in-site search, because it's programmed to only search version 7.18.

   Having said all that, I am nervous about the quantity of 7.15 pages that are considered canonical by Google. The fact that 7.15 pages appearing in the search results have been increasing throughout this experiment suggests that Google saw these pages as no longer being marked noindex, and did something....but still chose not to use our suggested 7.18 page as canonical.

   This gives me the feeling that if we removed noindex from all old versions, we would end up with content from a ton of different versions being canonical.

   As I said earlier, I do think there's value in having an old version be canonical+indexed, when the current version is not indexed because it's not canonical. In a vacuum, this would make me feel okay about removing noindex from all old versions; however....

3. The consistent and significant decline of "Google chose a different canonical than user" pages, despite the canonical+indexed 7.15 pages not decreasing and possibly increasing, suggests to me that good things are happening due to 7.18 canonical declarations, not because of the 7.15 removal of noindex. My opinion is that things are getting better for overall search results despite 7.15 absorbing more canonicals.

Recommendation

1. Roll back the 7.15 experiment
2. Roll back the 7.17 experiment
3. Keep an eye on two graphs, to ensure that things continue to get better over time:
   • The number of pages indexed from the 7.18 sitemap should continue to increase over time. (Currently, 388 pages are indexed, and 576 are not indexed.)
   • The number of pages in the "Duplicate, Google chose different canonical than user" dataset should not increase significantly over time. (Currently at 33 pages.)