Changelog
This is a list of major changes made to the Mastodon instance hosted at mastodon.at, including software updates, customizations and infrastructure updates, starting from September 17th, 2017. All times are in UTC.
Other notable events such as outages are also recorded here.
2019-02-17:
- 19:20 - 19:40: Upgraded Mastodon to 2.7.2
2019-02-10:
- 16:00: Upgraded FFmpeg to releases/4.1@74700e5
- 16:20: Upgraded ImageMagick to 7.0.8-27
2019-02-08:
- 18:00: Upgraded Elasticsearch to 6.6.0
2019-01-31:
- 19:05: Upgraded Ruby to 2.6.1
2019-01-29:
- 04:25 - 04:40: Upgraded Mastodon to 2.7.1
- 05:00: Upgraded ImageMagick to 7.0.8-25
2019-01-25:
- 20:45: Upgraded ImageMagick to 7.0.8-24
- 21:10: Upgraded FFmpeg to releases/4.1@bcfd82b
- 21:15: Upgraded nodejs to 10.15.0
- 21:15 - 21:20: Upgraded Ruby to 2.6.0
- 21:20 - 21:45: Upgraded Mastodon to 2.7.0
2018-12-26:
- 18:00: Upgraded Elasticsearch to 6.5.4
- 18:15: Upgraded nginx to 1.15.8
- 18:15: Upgraded LibreSSL to 2.9.0
- 19:35: Upgraded ImageMagick to 7.0.8-20
2018-12-15:
- 18:15: Upgraded ImageMagick to 7.0.8-16
2018-12-07:
- 19:15 - 19:30: Upgraded Mastodon to 2.6.5
- 19:45: Upgraded ImageMagick to 7.0.8-15
2018-11-24:
- 18:15 - 18:20: Upgraded redis to 5.0.2
- 18:20 - 18:25: Upgraded OpenJDK to jdk-11.0.1+13
- 18:25 - 20:00: Upgraded PostgreSQL to 11.1
- 20:05 - 20:20: Upgraded Mastodon to 2.6.2
2018-11-09:
- 15:10: Upgraded FFmpeg to releases/4.1@10506de
2018-11-03:
- 20:05 - 20:45: Upgraded Mastodon to 2.6.1
- 21:20: Upgraded ImageMagick to 7.0.8-14
- 21:30: Upgraded FFmpeg to releases/4.0@dee8f4b
2018-10-31:
- 12:03 - 22:42: Database server suffered a partial network outage starting at 12:03 and became fully unresponsive at 14:21. Infrastructure provider was unable to restore the server within an acceptable timeframe. Data was restored from the last successfull backup at 12:05. Services were restored at 22:42. 2.5 hours worth of data were unrecoverable.
2018-10-12:
- 05:40 - 05:50: Upgraded Mastodon to 2.5.2
2018-10-07:
- 22:10 - 22:15: Upgraded Mastodon to 2.5.1
2018-10-02:
- 18:40: Upgraded Elasticsearch to 6.4.2
- 18:45: Upgraded nginx to 1.15.5
2018-09-26:
- 17:40: Upgraded ImageMagick to 7.0.8-12
- 17:50: Upgraded nginx to 1.15.4
- 17:50: Upgraded LibreSSL to 2.8.1
2018-09-15:
- 18:30 - 19:00: Upgraded Mastodon to 2.5.0
2018-08-31:
- 18:15: Upgraded nginx to 1.15.3
- 18:15: Upgraded LibreSSL to 2.8.0
- 19:10: Upgraded ImageMagick to 7.0.8-11
2018-08-25:
- 14:00: Upgraded Mastodon to 2.4.5
- 14:25: Upgraded ImageMagick to 7.0.8-10
2018-08-23:
- 00:50 - 01:00: Upgraded Mastodon to 2.4.4
2018-08-22:
- 19:30: Upgraded OpenJDK to 10.0.2
2018-07-29:
- 17:30: Upgraded nginx to 1.15.2
- 18:30: Upgraded ImageMagick to 7.0.8-8
2018-07-24:
- 16:40: Upgraded Elasticsearch to 6.3.2
2018-07-23:
- 20:10: Upgraded ImageMagick to 7.0.8-7
2018-07-17:
- 22:50: Upgraded FFmpeg to releases/4.0@3ef38c4
2018-07-12:
- 17:55: Upgraded ImageMagick to 7.0.8-6
2018-07-11:
- 21:05 - 21:25: Upgraded Mastodon to 2.4.3
2018-07-05:
- 19:45: Upgraded Elasticsearch to 6.3.1
- 20:15: Upgraded nginx to 1.15.1
2018-06-29:
- 19:00: Upgraded ImageMagick to 7.0.8-3
2018-06-19:
- 20:00 - 20:05: Upgraded Mastodon to 2.4.2
- 20:20: Upgraded Sprockets to 3.7.2
2018-06-16:
- 12:10: Upgraded FFmpeg to releases/4.0@e049f7c
2018-06-13:
- 18:15: Upgraded redis to 4.0.10
- 18:50: Upgraded LibreSSL to 2.7.4
2018-06-12:
- 18:25 - 18:50: Upgraded Mastodon to 2.4.1
- 19:15: Upgraded ImageMagick to 7.0.7-39
- 19:30: Upgraded nginx to 1.15.0
2018-05-25:
- 16:00 - 16:20: Upgraded Mastodon to 2.4.0
- 16:30: Upgraded FFmpeg to releases/4.0@da3999
- 16:40: Upgraded ImageMagick to 7.0.7-35
2018-05-18:
- 01:50: Upgraded ImageMagick to 7.0.7-33
2018-05-06:
- 03:50: Upgraded LibreSSL to 2.7.3
2018-05-02:
- 17:00: Upgraded ImageMagick to 7.0.7-29
2018-04-18:
- 12:15: Upgraded nginx to 1.14.0
2018-04-17:
- 23:05: Upgraded Elasticsearch to 6.2.4
2018-04-13:
- 19:45: Upgraded nginx to 1.13.12
2018-04-04:
- 17:15: Upgraded nginx to 1.13.11
2018-04-02:
- 20:30: Upgraded LibreSSL to 2.7.2
- 21:00: Upgraded redis to 4.0.9
2018-03-27:
- 02:35 - 02:40: Upgraded Mastodon to 2.3.3
2018-03-22:
- 15:45 - 16:00: Upgraded Mastodon to 2.3.2
- 16:50: Upgraded LibreSSL to 2.7.0
2018-03-21:
- 14:45: Upgraded Elasticsearch to 6.2.3
- 15:00: Upgraded ImageMagick to 7.0.7-27
- 16:00: Upgraded nginx to 1.13.10
2018-03-14:
- 06:55 - 07:25: Upgraded Mastodon to 2.3.1
2018-03-09:
- 19:05 - 19:30: Upgraded Mastodon to 2.3.0
2018-03-04:
- 15:45: Upgraded FFmpeg to releases/3.4@e4b95f7
2018-03-01:
- 13:30: Upgraded ImageMagick to 7.0.7-24
- 19:45: Upgraded PostgreSQL to 10.3
2018-02-20:
- 18:35: Upgraded Elasticsearch to 6.2.2
2018-02-19:
- 22:45: Upgraded ImageMagick to 7.0.7-23
2018-02-12:
- 04:10: Upgraded FFmpeg to releases/3.4@6a97ba5
2018-02-08:
- 22:00 - 22:05: Upgraded Elasticsearch to 6.2.1
- 22:05 - 22:10: Upgraded PostgreSQL to 10.2
2018-02-06:
- 19:00 - 19:05: Upgraded Elasticsearch to 6.2.0
2018-02-05:
- 14:40 - 14:45: Deployed full-text search for authorized statuses using Elasticsearch
2018-02-04:
- 00:00 - 18:30: Connectivity issues for users connecting from South America due to a failing configuration deployment process that caused a TLS certificates to not be renewed.
2018-01-29:
- 03:15 - 04:20: Upgraded Mastodon to 2.2.0
2018-01-24:
- 15:00: Upgraded CoreDNS to 1.0.4
- 15:10: Upgraded ImageMagick to 7.0.7-22
- 15:10: Upgraded FFmpeg to releases/3.4@4a53ecb
- 15:45: Upgraded redis to 4.0.7
2018-01-10:
- 00:25 - 00:40: Upgraded Mastodon to 2.1.3
2018-01-07:
- 16:50: Upgraded ImageMagick to 7.0.7-21
2018-01-06:
- 00:00 - 00:15: Upgraded Mastodon to 2.1.2
2018-01-04:
- 18:55 - 19:10: Upgraded Mastodon to 2.1.1
2017-12-28:
- 02:30: Upgraded FFmpeg to releases/3.4@d810497
- 02:35: Upgraded nsjail to 2.3
- 02:50: Upgraded ImageMagick to 7.0.7-17
- 05:45: Upgraded nginx to 1.13.8
- 05:45: Upgraded LibreSSL to 2.6.4
2017-12-21:
- 02:45: Upgraded CoreDNS to 1.0.1
2017-12-18:
- 09:00 - 09:35: Enabled data checksums in PostgreSQL to detect silent corruption
2017-12-15:
- 20:00: Merged app server with reverse proxy
- 21:00: Added edge cache in São Paulo, Brazil
- 21:45 - 22:15: Upgraded Mastodon to 2.1.0
- 22:30: Upgraded FFmpeg to releases/3.4@bc839fb
2017-12-08:
- 09:40: Deployed nsjail sandbox for file(1)
2017-12-07:
- 09:45: Upgraded ImageMagick to 7.0.7-14
- 13:15: Deployed nsjail sandbox for FFmpeg
2017-12-06:
- 02:30: Recompiled Ruby with the jemalloc memory allocator
2017-12-04:
- 20:20: Upgraded redis to 4.0.6
2017-12-03:
- 03:20: Deployed nsjail to sandbox ImageMagick
2017-12-02:
- 01:30: Upgraded CoreDNS to 1.0.0
2017-12-01:
- 16:30: Upgraded redis to 4.0.5
- 19:40: Upgraded ImageMagick to 7.0.7-13
2017-11-30:
- 15:45: Upgraded redis to 4.0.3
- 16:00: Upgraded ImageMagick to 7.0.7-12
- 18:30: Upgraded redis to 4.0.4
2017-11-27:
- 23:15: Upgraded FFmpeg to releases/3.4@752659f
2017-11-22:
- 00:00: Upgraded nginx to 1.13.7
2017-11-17:
- 00:30: Upgraded ImageMagick to 7.0.7-11
2017-11-15:
- 14:00 - 2017-11-16 19:00: Performed maintenance to improve scalability and ability to reroute traffic to a secondary data center in the event of an outage.
2017-11-14:
- 01:00: Completed deployment of on-demand application servers and background workers in a secondary data center on Google Compute Platform (GCP).
2017-11-09:
- 06:12 - 12:30: Infrastructure provider suffered a power outage in our data center. Servers started coming back online at around 11:30. mastodon.at partially recovered at 12:00. Full service was restored by 12:30.
2017-11-08:
- 02:00 - 03:15: Replaced ImageMagick from Debian repositories with ImageMagick 7.0.7-10
2017-11-06:
- 15:45 - 16:00: Upgraded LibreSSL to 2.6.3
- 16:35 - 17:35: Replaced FFmpeg from Debian repository with FFmpeg 3.4
2017-10-29:
- 11:15: Decommissioned app server
app002-sbg
due to overprovisioning
2017-10-22:
- 01:00: Upgraded CoreDNS to 0.9.9
2017-10-18:
- 17:35 - 18:25: Upgraded Mastodon to 2.0.0
2017-10-17:
- 12:30 - 12:50: Upgraded PostgreSQL to 10.0
2017-10-16:
- 07:35 - 07:41: Requests from regions other than NA and SA were dropped due to a frozen VM
- 08:45 - 09:50: New statuses are not delivered to local users
2017-10-12:
- 15:15: Upgraded nginx to 1.13.6
2017-10-08:
- 05:30 - 06:25: Replaced dnsmasq with CoreDNS
2017-10-04:
- 10:36 - 10:50: Requests from regions other than NA and SA were dropped due to network connectivity issues
2017-09-26:
- 17:10: Upgraded LibreSSL to 2.6.2
2017-09-21:
- 16:10: Upgraded redis to 4.0.2
- 22:14 - 00:00: High load and slow status delivery due to a large number of federated status deletes from one remote account
2017-09-20:
- 11:50: Deployed update to librato-sidekiq with retry tracking
- 16:30: Deployed database row creation metrics
- 18:30: Updated Content Security Policy to allow embedded YouTube and fix an issue on the profile image update page
2017-09-19:
- 09:30 - 09:50: Migration of database server to OVH Public Cloud
- 18:10: Deployed nokogiri 1.8.1 to fix libxml2 vulnerabilities
2017-09-17
- 21:00: Upgraded Mastodon to 1.6.1