Mastodon.at Changelog

mastodon-changelog.md

Changelog

This is a list of major changes made to the Mastodon instance hosted at mastodon.at, including software updates, customizations and infrastructure updates, starting from September 17th, 2017. All times are in UTC.

Other notable events such as outages are also recorded here.

2019-04-27:

• 21:45: Upgraded ImageMagick to 7.0.8-42
• 21:50: Upgraded FFmpeg to releases/4.1@7c2dd1f
• 22:00: Upgraded Ruby to 2.6.3
• 22:05: Upgraded Mastodon to 2.8.0

2019-03-31:

• 18:00: Completed migration to new infrastructure provider

2019-03-19:

• 20:30: Upgraded redis to 5.0.4
• 20:40 - 21:05: Upgraded Mastodon to 2.7.4

2019-03-02:

• 19:20: Upgraded nginx to 1.15.9

2019-02-24:

• 13:55 - 15:10: Upgraded Mastodon to 2.7.3

2019-02-17:

• 19:20 - 19:40: Upgraded Mastodon to 2.7.2

2019-02-10:

• 16:00: Upgraded FFmpeg to releases/4.1@74700e5
• 16:20: Upgraded ImageMagick to 7.0.8-27

2019-02-08:

• 18:00: Upgraded Elasticsearch to 6.6.0

2019-01-31:

• 19:05: Upgraded Ruby to 2.6.1

2019-01-29:

• 04:25 - 04:40: Upgraded Mastodon to 2.7.1
• 05:00: Upgraded ImageMagick to 7.0.8-25

2019-01-25:

• 20:45: Upgraded ImageMagick to 7.0.8-24
• 21:10: Upgraded FFmpeg to releases/4.1@bcfd82b
• 21:15: Upgraded nodejs to 10.15.0
• 21:15 - 21:20: Upgraded Ruby to 2.6.0
• 21:20 - 21:45: Upgraded Mastodon to 2.7.0

2018-12-26:

• 18:00: Upgraded Elasticsearch to 6.5.4
• 18:15: Upgraded nginx to 1.15.8
• 18:15: Upgraded LibreSSL to 2.9.0
• 19:35: Upgraded ImageMagick to 7.0.8-20

2018-12-15:

• 18:15: Upgraded ImageMagick to 7.0.8-16

2018-12-07:

• 19:15 - 19:30: Upgraded Mastodon to 2.6.5
• 19:45: Upgraded ImageMagick to 7.0.8-15

2018-11-24:

• 18:15 - 18:20: Upgraded redis to 5.0.2
• 18:20 - 18:25: Upgraded OpenJDK to jdk-11.0.1+13
• 18:25 - 20:00: Upgraded PostgreSQL to 11.1
• 20:05 - 20:20: Upgraded Mastodon to 2.6.2

2018-11-09:

• 15:10: Upgraded FFmpeg to releases/4.1@10506de

2018-11-03:

• 20:05 - 20:45: Upgraded Mastodon to 2.6.1
• 21:20: Upgraded ImageMagick to 7.0.8-14
• 21:30: Upgraded FFmpeg to releases/4.0@dee8f4b

2018-10-31:

• 12:03 - 22:42: Database server suffered a partial network outage starting at 12:03 and became fully unresponsive at 14:21. Infrastructure provider was unable to restore the server within an acceptable timeframe. Data was restored from the last successfull backup at 12:05. Services were restored at 22:42. 2.5 hours worth of data were unrecoverable.

2018-10-12:

• 05:40 - 05:50: Upgraded Mastodon to 2.5.2

2018-10-07:

• 22:10 - 22:15: Upgraded Mastodon to 2.5.1

2018-10-02:

• 18:40: Upgraded Elasticsearch to 6.4.2
• 18:45: Upgraded nginx to 1.15.5

2018-09-26:

• 17:40: Upgraded ImageMagick to 7.0.8-12
• 17:50: Upgraded nginx to 1.15.4
• 17:50: Upgraded LibreSSL to 2.8.1

2018-09-15:

• 18:30 - 19:00: Upgraded Mastodon to 2.5.0

2018-08-31:

• 18:15: Upgraded nginx to 1.15.3
• 18:15: Upgraded LibreSSL to 2.8.0
• 19:10: Upgraded ImageMagick to 7.0.8-11

2018-08-25:

• 14:00: Upgraded Mastodon to 2.4.5
• 14:25: Upgraded ImageMagick to 7.0.8-10

2018-08-23:

• 00:50 - 01:00: Upgraded Mastodon to 2.4.4

2018-08-22:

• 19:30: Upgraded OpenJDK to 10.0.2

2018-07-29:

• 17:30: Upgraded nginx to 1.15.2
• 18:30: Upgraded ImageMagick to 7.0.8-8

2018-07-24:

• 16:40: Upgraded Elasticsearch to 6.3.2

2018-07-23:

• 20:10: Upgraded ImageMagick to 7.0.8-7

2018-07-17:

• 22:50: Upgraded FFmpeg to releases/4.0@3ef38c4

2018-07-12:

• 17:55: Upgraded ImageMagick to 7.0.8-6

2018-07-11:

• 21:05 - 21:25: Upgraded Mastodon to 2.4.3

2018-07-05:

• 19:45: Upgraded Elasticsearch to 6.3.1
• 20:15: Upgraded nginx to 1.15.1

2018-06-29:

• 19:00: Upgraded ImageMagick to 7.0.8-3

2018-06-19:

• 20:00 - 20:05: Upgraded Mastodon to 2.4.2
• 20:20: Upgraded Sprockets to 3.7.2

2018-06-16:

• 12:10: Upgraded FFmpeg to releases/4.0@e049f7c

2018-06-13:

• 18:15: Upgraded redis to 4.0.10
• 18:50: Upgraded LibreSSL to 2.7.4

2018-06-12:

• 18:25 - 18:50: Upgraded Mastodon to 2.4.1
• 19:15: Upgraded ImageMagick to 7.0.7-39
• 19:30: Upgraded nginx to 1.15.0

2018-05-25:

• 16:00 - 16:20: Upgraded Mastodon to 2.4.0
• 16:30: Upgraded FFmpeg to releases/4.0@da3999
• 16:40: Upgraded ImageMagick to 7.0.7-35

2018-05-18:

• 01:50: Upgraded ImageMagick to 7.0.7-33

2018-05-06:

• 03:50: Upgraded LibreSSL to 2.7.3

2018-05-02:

• 17:00: Upgraded ImageMagick to 7.0.7-29

2018-04-18:

• 12:15: Upgraded nginx to 1.14.0

2018-04-17:

• 23:05: Upgraded Elasticsearch to 6.2.4

2018-04-13:

• 19:45: Upgraded nginx to 1.13.12

2018-04-04:

• 17:15: Upgraded nginx to 1.13.11

2018-04-02:

• 20:30: Upgraded LibreSSL to 2.7.2
• 21:00: Upgraded redis to 4.0.9

2018-03-27:

• 02:35 - 02:40: Upgraded Mastodon to 2.3.3

2018-03-22:

• 15:45 - 16:00: Upgraded Mastodon to 2.3.2
• 16:50: Upgraded LibreSSL to 2.7.0

2018-03-21:

• 14:45: Upgraded Elasticsearch to 6.2.3
• 15:00: Upgraded ImageMagick to 7.0.7-27
• 16:00: Upgraded nginx to 1.13.10

2018-03-14:

• 06:55 - 07:25: Upgraded Mastodon to 2.3.1

2018-03-09:

• 19:05 - 19:30: Upgraded Mastodon to 2.3.0

2018-03-04:

• 15:45: Upgraded FFmpeg to releases/3.4@e4b95f7

2018-03-01:

• 13:30: Upgraded ImageMagick to 7.0.7-24
• 19:45: Upgraded PostgreSQL to 10.3

2018-02-20:

• 18:35: Upgraded Elasticsearch to 6.2.2

2018-02-19:

• 22:45: Upgraded ImageMagick to 7.0.7-23

2018-02-12:

• 04:10: Upgraded FFmpeg to releases/3.4@6a97ba5

2018-02-08:

• 22:00 - 22:05: Upgraded Elasticsearch to 6.2.1
• 22:05 - 22:10: Upgraded PostgreSQL to 10.2

2018-02-06:

• 19:00 - 19:05: Upgraded Elasticsearch to 6.2.0

2018-02-05:

• 14:40 - 14:45: Deployed full-text search for authorized statuses using Elasticsearch

2018-02-04:

• 00:00 - 18:30: Connectivity issues for users connecting from South America due to a failing configuration deployment process that caused a TLS certificates to not be renewed.

2018-01-29:

• 03:15 - 04:20: Upgraded Mastodon to 2.2.0

2018-01-24:

• 15:00: Upgraded CoreDNS to 1.0.4
• 15:10: Upgraded ImageMagick to 7.0.7-22
• 15:10: Upgraded FFmpeg to releases/3.4@4a53ecb
• 15:45: Upgraded redis to 4.0.7

2018-01-10:

• 00:25 - 00:40: Upgraded Mastodon to 2.1.3

2018-01-07:

• 16:50: Upgraded ImageMagick to 7.0.7-21

2018-01-06:

• 00:00 - 00:15: Upgraded Mastodon to 2.1.2

2018-01-04:

• 18:55 - 19:10: Upgraded Mastodon to 2.1.1

2017-12-28:

• 02:30: Upgraded FFmpeg to releases/3.4@d810497
• 02:35: Upgraded nsjail to 2.3
• 02:50: Upgraded ImageMagick to 7.0.7-17
• 05:45: Upgraded nginx to 1.13.8
• 05:45: Upgraded LibreSSL to 2.6.4

2017-12-21:

• 02:45: Upgraded CoreDNS to 1.0.1

2017-12-18:

• 09:00 - 09:35: Enabled data checksums in PostgreSQL to detect silent corruption

2017-12-15:

• 20:00: Merged app server with reverse proxy
• 21:00: Added edge cache in São Paulo, Brazil
• 21:45 - 22:15: Upgraded Mastodon to 2.1.0
• 22:30: Upgraded FFmpeg to releases/3.4@bc839fb

2017-12-08:

• 09:40: Deployed nsjail sandbox for file(1)

2017-12-07:

• 09:45: Upgraded ImageMagick to 7.0.7-14
• 13:15: Deployed nsjail sandbox for FFmpeg

2017-12-06:

• 02:30: Recompiled Ruby with the jemalloc memory allocator

2017-12-04:

• 20:20: Upgraded redis to 4.0.6

2017-12-03:

• 03:20: Deployed nsjail to sandbox ImageMagick

2017-12-02:

• 01:30: Upgraded CoreDNS to 1.0.0

2017-12-01:

• 16:30: Upgraded redis to 4.0.5
• 19:40: Upgraded ImageMagick to 7.0.7-13

2017-11-30:

• 15:45: Upgraded redis to 4.0.3
• 16:00: Upgraded ImageMagick to 7.0.7-12
• 18:30: Upgraded redis to 4.0.4

2017-11-27:

• 23:15: Upgraded FFmpeg to releases/3.4@752659f

2017-11-22:

• 00:00: Upgraded nginx to 1.13.7

2017-11-17:

• 00:30: Upgraded ImageMagick to 7.0.7-11

2017-11-15:

• 14:00 - 2017-11-16 19:00: Performed maintenance to improve scalability and ability to reroute traffic to a secondary data center in the event of an outage.

2017-11-14:

• 01:00: Completed deployment of on-demand application servers and background workers in a secondary data center on Google Compute Platform (GCP).

2017-11-09:

• 06:12 - 12:30: Infrastructure provider suffered a power outage in our data center. Servers started coming back online at around 11:30. mastodon.at partially recovered at 12:00. Full service was restored by 12:30.

2017-11-08:

• 02:00 - 03:15: Replaced ImageMagick from Debian repositories with ImageMagick 7.0.7-10

2017-11-06:

• 15:45 - 16:00: Upgraded LibreSSL to 2.6.3
• 16:35 - 17:35: Replaced FFmpeg from Debian repository with FFmpeg 3.4

2017-10-29:

• 11:15: Decommissioned app server app002-sbg due to overprovisioning

2017-10-22:

• 01:00: Upgraded CoreDNS to 0.9.9

2017-10-18:

• 17:35 - 18:25: Upgraded Mastodon to 2.0.0

2017-10-17:

• 12:30 - 12:50: Upgraded PostgreSQL to 10.0

2017-10-16:

• 07:35 - 07:41: Requests from regions other than NA and SA were dropped due to a frozen VM
• 08:45 - 09:50: New statuses are not delivered to local users

2017-10-12:

• 15:15: Upgraded nginx to 1.13.6

2017-10-08:

• 05:30 - 06:25: Replaced dnsmasq with CoreDNS

2017-10-04:

• 10:36 - 10:50: Requests from regions other than NA and SA were dropped due to network connectivity issues

2017-09-26:

• 17:10: Upgraded LibreSSL to 2.6.2

2017-09-21:

• 16:10: Upgraded redis to 4.0.2
• 22:14 - 00:00: High load and slow status delivery due to a large number of federated status deletes from one remote account

2017-09-20:

• 11:50: Deployed update to librato-sidekiq with retry tracking
• 16:30: Deployed database row creation metrics
• 18:30: Updated Content Security Policy to allow embedded YouTube and fix an issue on the profile image update page

2017-09-19:

• 09:30 - 09:50: Migration of database server to OVH Public Cloud
• 18:10: Deployed nokogiri 1.8.1 to fix libxml2 vulnerabilities

2017-09-17

• 21:00: Upgraded Mastodon to 1.6.1