LessWrong analytics post

.gitignore

post.html
rolling-pageviews.png
rolling-sessions.png

LICENSE

The work by Issa Rice is released under the CC0 1.0 Universal Public
Domain Dedication, reproduced below.  The data files, lw_audience.csv
and lw_pages.csv, are not by me and I don't know who owns the copyright
on those.

Creative Commons Legal Code

CC0 1.0 Universal

    CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
    LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
    ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
    INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
    REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
    PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
    THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
    HEREUNDER.

Statement of Purpose

The laws of most jurisdictions throughout the world automatically confer
exclusive Copyright and Related Rights (defined below) upon the creator
and subsequent owner(s) (each and all, an "owner") of an original work of
authorship and/or a database (each, a "Work").

Certain owners wish to permanently relinquish those rights to a Work for
the purpose of contributing to a commons of creative, cultural and
scientific works ("Commons") that the public can reliably and without fear
of later claims of infringement build upon, modify, incorporate in other
works, reuse and redistribute as freely as possible in any form whatsoever
and for any purposes, including without limitation commercial purposes.
These owners may contribute to the Commons to promote the ideal of a free
culture and the further production of creative, cultural and scientific
works, or to gain reputation or greater distribution for their Work in
part through the use and efforts of others.

For these and/or other purposes and motivations, and without any
expectation of additional consideration or compensation, the person
associating CC0 with a Work (the "Affirmer"), to the extent that he or she
is an owner of Copyright and Related Rights in the Work, voluntarily
elects to apply CC0 to the Work and publicly distribute the Work under its
terms, with knowledge of his or her Copyright and Related Rights in the
Work and the meaning and intended legal effect of CC0 on those rights.

1. Copyright and Related Rights. A Work made available under CC0 may be
protected by copyright and related or neighboring rights ("Copyright and
Related Rights"). Copyright and Related Rights include, but are not
limited to, the following:

  i. the right to reproduce, adapt, distribute, perform, display,
     communicate, and translate a Work;
 ii. moral rights retained by the original author(s) and/or performer(s);
iii. publicity and privacy rights pertaining to a person's image or
     likeness depicted in a Work;
 iv. rights protecting against unfair competition in regards to a Work,
     subject to the limitations in paragraph 4(a), below;
  v. rights protecting the extraction, dissemination, use and reuse of data
     in a Work;
 vi. database rights (such as those arising under Directive 96/9/EC of the
     European Parliament and of the Council of 11 March 1996 on the legal
     protection of databases, and under any national implementation
     thereof, including any amended or successor version of such
     directive); and
vii. other similar, equivalent or corresponding rights throughout the
     world based on applicable law or treaty, and any national
     implementations thereof.

2. Waiver. To the greatest extent permitted by, but not in contravention
of, applicable law, Affirmer hereby overtly, fully, permanently,
irrevocably and unconditionally waives, abandons, and surrenders all of
Affirmer's Copyright and Related Rights and associated claims and causes
of action, whether now known or unknown (including existing as well as
future claims and causes of action), in the Work (i) in all territories
worldwide, (ii) for the maximum duration provided by applicable law or
treaty (including future time extensions), (iii) in any current or future
medium and for any number of copies, and (iv) for any purpose whatsoever,
including without limitation commercial, advertising or promotional
purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
member of the public at large and to the detriment of Affirmer's heirs and
successors, fully intending that such Waiver shall not be subject to
revocation, rescission, cancellation, termination, or any other legal or
equitable action to disrupt the quiet enjoyment of the Work by the public
as contemplated by Affirmer's express Statement of Purpose.

3. Public License Fallback. Should any part of the Waiver for any reason
be judged legally invalid or ineffective under applicable law, then the
Waiver shall be preserved to the maximum extent permitted taking into
account Affirmer's express Statement of Purpose. In addition, to the
extent the Waiver is so judged Affirmer hereby grants to each affected
person a royalty-free, non transferable, non sublicensable, non exclusive,
irrevocable and unconditional license to exercise Affirmer's Copyright and
Related Rights in the Work (i) in all territories worldwide, (ii) for the
maximum duration provided by applicable law or treaty (including future
time extensions), (iii) in any current or future medium and for any number
of copies, and (iv) for any purpose whatsoever, including without
limitation commercial, advertising or promotional purposes (the
"License"). The License shall be deemed effective as of the date CC0 was
applied by Affirmer to the Work. Should any part of the License for any
reason be judged legally invalid or ineffective under applicable law, such
partial invalidity or ineffectiveness shall not invalidate the remainder
of the License, and in such case Affirmer hereby affirms that he or she
will not (i) exercise any of his or her remaining Copyright and Related
Rights in the Work or (ii) assert any associated claims and causes of
action with respect to the Work, in either case contrary to Affirmer's
express Statement of Purpose.

4. Limitations and Disclaimers.

 a. No trademark or patent rights held by Affirmer are waived, abandoned,
    surrendered, licensed or otherwise affected by this document.
 b. Affirmer offers the Work as-is and makes no representations or
    warranties of any kind concerning the Work, express, implied,
    statutory or otherwise, including without limitation warranties of
    title, merchantability, fitness for a particular purpose, non
    infringement, or the absence of latent or other defects, accuracy, or
    the present or absence of errors, whether or not discoverable, all to
    the greatest extent permissible under applicable law.
 c. Affirmer disclaims responsibility for clearing rights of other persons
    that may apply to the Work or any use thereof, including without
    limitation any person's Copyright and Related Rights in the Work.
    Further, Affirmer disclaims responsibility for obtaining any necessary
    consents, permissions or other rights required for any use of the
    Work.
 d. Affirmer understands and acknowledges that Creative Commons is not a
    party to this document and has no duty or obligation with respect to
    this CC0 or use of the Work.

lw_audience.csv

lw_pages.csv

Makefile

# License: CC0

# Obtain GitHub's cmark at https://github.com/github/cmark
post.html: post.md
	cmark-gfm -e table --smart "$<" > "$@"

# Note that post.md uses the plots hosted on ram.issarice.com, so regenerating
# plots will not update the plots in the post when viewing locally
.PHONY: generate_plots
generate_plots: plot.py
	./plot.py

.PHONY: clean
clean:
	rm -f post.html rolling-pageviews.png rolling-sessions.png

plot.py

#!/usr/bin/env python3
# License: CC0

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv("lw_audience.csv", sep=",")
df = df[:-1]  # Drop the totals line at bottom
df = df.set_index('Day Index')
df.index = pd.to_datetime(df.index)

df.Sessions.rolling(window=30).sum().plot()
plt.xlabel("Date")
plt.ylabel("Sessions")
plt.title("30-day rolling sum of Sessions")
plt.tight_layout()
plt.savefig("rolling-sessions.png")
plt.clf()

df.Pageviews.rolling(window=30).sum().plot()
plt.xlabel("Date")
plt.ylabel("Pageviews")
plt.title("30-day rolling sum of Pageviews")
plt.tight_layout()
plt.savefig("rolling-pageviews.png")
plt.clf()

post.md

Table of contents

• Introduction
• Pageviews and sessions
• Top posts
• Source code
• Further reading
• Acknowledgments

Introduction

In January 2017, Vipul Naik obtained Google Analytics daily sessions and pageviews data for LessWrong from Kaj Sotala. Vipul asked me to write a short post giving an overview of the data, so here it is.

This post covers just the basics. Vipul and I are eager to hear thoughts on what sort of deeper analysis people are interested in; we may incorporate these ideas in future posts.

Pageviews and sessions

The data for both sessions and pageviews span from February 26, 2009 to January 3, 2017. LessWrong seems to have launched in February 2009, so this is close to the full duration for which LessWrong has existed.

Pageviews plot:

Total pageviews recorded by Google Analytics for this period is 52.2 million.

Sessions plot:

Total sessions recorded by Google Analytics for this period is 19.7 million.

Both plots end with an upward swing, coinciding with the effort to revive LessWrong that began in late November 2016. However, as of early January 2017 (the latest period for which we have data) the scale of any recent increase in LessWrong usage is small in the context of the general decline starting in early 2012.

Top posts

The top 20 posts of all time (by total pageviews), with pageviews and unique pageviews rounded to the nearest thousand, are as follows:

Title	Pageviews (thousands)	Unique Pageviews (thousands)
Don't Get Offended	681	128
How to Be Happy	551	482
How to Beat Procrastination	378	342
The Best Textbooks on Every Subject	266	233
Do you have High-Functioning Asperger's Syndrome?	188	168
Superhero Bias	169	154
The Quantum Physics Sequence	157	130
Bayesian Judo	140	126
An Alien God	125	113
An Intuitive Explanation of Quantum Mechanics	123	106
Three Worlds Collide (0/8)	121	93
Bayes' Theorem Illustrated (My Way)	121	112
9/26 is Petrov Day	121	115
The Baby-Eating Aliens (1/8)	109	98
The noncentral fallacy - the worst argument in the world?	107	99
Advanced Placement exam cutoffs and superficial knowledge over deep knowledge	107	94
Guessing the Teacher's Password	102	96
The Fun Theory Sequence	102	90
Optimal Employment	102	97
Ugh fields	95	86

Note that Google Analytics reports are subject to sampling when the number of sessions is large (as it is here) so the input numbers are not exact. More details can be found in a post at LunaMetrics. This doesn't affect the estimates for the top posts, but those wishing to work with the exported data should be aware of this.

Each post on LessWrong can have numerous URLs. In the case of posts that were renamed, a significant number of pageviews could be recorded at both the old and new URL. To take an example, the following URLs all point to lukeprog's post "How to Be Happy":

• http://lesswrong.com/lw/4su/the_science_of_happiness/
• http://lesswrong.com/lw/4su/how_to_be_happy/
• http://lesswrong.com/lw/4su
• http://lesswrong.com/lw/4su/foo
• http://lesswrong.com/r/lesswrong/lw/4su/how_to_be_happy/
• http://lesswrong.com/r/lesswrong/lw/4su/foo/
• http://lesswrong.com/r/discussion/lw/4su/foo/

All that matters for identifying this particular post is that we have the substring "/lw/4su" in the URL. In the above table, I have grouped the URLs by this identifying substring and summed to get the pageview counts.

In addition, each post has two "canonical" URLs that can be obtained by clicking on the post titles: one that begins with either "/r/lesswrong/lw" or "/r/discussion/lw" and one that begins with just "/lw". I have used the latter in linking to the posts from my table.

Source code

The data, source code used to generate the plots, as well as the Markdown source of this post are available in a GitHub Gist.

Clone the Git repository with:

git clone https://gist.github.com/cbdd400180417c689b2befbfbe2158fc.git

Further reading

• Alexa profile for lesswrong.com
• SimilarWeb profile for lesswrong.com
• "Effective Altruism Forum web traffic from Google Analytics", a post by Vipul
• gwern.net analytics by gwern

Here are a few related PredictionBook predictions:

• Total LessWrong Google Analytics Sessions for 2017 will be higher than for 2016
• Total LessWrong Google Analytics Pageviews for 2017 will be higher than for 2016
• A LessWrong post published during 2017 will become a top-10 post by pageviews on LW by 2018-01-01

Acknowledgments

Thanks to Kaj for providing the data used in this post. Thanks to Vipul for asking around for the data, for the idea of this post, and for sponsoring my work on this post.

top_posts.py

#!/usr/bin/env python3
# License: CC0

import pandas as pd
import numpy as np

df = pd.read_csv("lw_pages.csv", sep=',')
df = df[:-1]  # Remove final row with totals

# Get just the posts, removing things like user profiles
posts = df[df.Page.str.contains("/lw/", regex=False)].reset_index()
# The "slug" is the thing that acts as the post's ID (e.g. "4su"), which
# appears right after the "/lw/" in the URL
posts['slug'] = posts.Page.apply(lambda x:
                                 x.split("/")[x.split("/").index("lw") + 1])

top = posts.groupby('slug')["Pageviews", "Unique Pageviews"].sum() \
        .sort_values(by="Pageviews", ascending=False)[:20]

for index, row in top.iterrows():
    print("|[](http://lesswrong.com/lw/{})|{}|{}|".format(
        index,
        (row['Pageviews']/1000).round().astype(np.int64),
        (row['Unique Pageviews']/1000).round().astype(np.int64)
    ))