This is a list of major changes made to the Mastodon instance hosted at mastodon.at, including software updates, customizations and infrastructure updates, starting from September 17th, 2017. All times are in UTC.
Other notable events such as outages are also recorded here.
- 21:45: Upgraded ImageMagick to 7.0.8-42
- 21:50: Upgraded FFmpeg to releases/4.1@7c2dd1f
- 22:00: Upgraded Ruby to 2.6.3
- 22:05: Upgraded Mastodon to 2.8.0
- 18:00: Completed migration to new infrastructure provider
- 20:30: Upgraded redis to 5.0.4
- 20:40 - 21:05: Upgraded Mastodon to 2.7.4
- 19:20: Upgraded nginx to 1.15.9
- 13:55 - 15:10: Upgraded Mastodon to 2.7.3
- 19:20 - 19:40: Upgraded Mastodon to 2.7.2
- 16:00: Upgraded FFmpeg to releases/4.1@74700e5
- 16:20: Upgraded ImageMagick to 7.0.8-27
- 18:00: Upgraded Elasticsearch to 6.6.0
- 19:05: Upgraded Ruby to 2.6.1
- 04:25 - 04:40: Upgraded Mastodon to 2.7.1
- 05:00: Upgraded ImageMagick to 7.0.8-25
- 20:45: Upgraded ImageMagick to 7.0.8-24
- 21:10: Upgraded FFmpeg to releases/4.1@bcfd82b
- 21:15: Upgraded nodejs to 10.15.0
- 21:15 - 21:20: Upgraded Ruby to 2.6.0
- 21:20 - 21:45: Upgraded Mastodon to 2.7.0
- 18:00: Upgraded Elasticsearch to 6.5.4
- 18:15: Upgraded nginx to 1.15.8
- 18:15: Upgraded LibreSSL to 2.9.0
- 19:35: Upgraded ImageMagick to 7.0.8-20
- 18:15: Upgraded ImageMagick to 7.0.8-16
- 19:15 - 19:30: Upgraded Mastodon to 2.6.5
- 19:45: Upgraded ImageMagick to 7.0.8-15
- 18:15 - 18:20: Upgraded redis to 5.0.2
- 18:20 - 18:25: Upgraded OpenJDK to jdk-11.0.1+13
- 18:25 - 20:00: Upgraded PostgreSQL to 11.1
- 20:05 - 20:20: Upgraded Mastodon to 2.6.2
- 15:10: Upgraded FFmpeg to releases/4.1@10506de
- 20:05 - 20:45: Upgraded Mastodon to 2.6.1
- 21:20: Upgraded ImageMagick to 7.0.8-14
- 21:30: Upgraded FFmpeg to releases/4.0@dee8f4b
- 12:03 - 22:42: Database server suffered a partial network outage starting at 12:03 and became fully unresponsive at 14:21. Infrastructure provider was unable to restore the server within an acceptable timeframe. Data was restored from the last successfull backup at 12:05. Services were restored at 22:42. 2.5 hours worth of data were unrecoverable.
- 05:40 - 05:50: Upgraded Mastodon to 2.5.2
- 22:10 - 22:15: Upgraded Mastodon to 2.5.1
- 18:40: Upgraded Elasticsearch to 6.4.2
- 18:45: Upgraded nginx to 1.15.5
- 17:40: Upgraded ImageMagick to 7.0.8-12
- 17:50: Upgraded nginx to 1.15.4
- 17:50: Upgraded LibreSSL to 2.8.1
- 18:30 - 19:00: Upgraded Mastodon to 2.5.0
- 18:15: Upgraded nginx to 1.15.3
- 18:15: Upgraded LibreSSL to 2.8.0
- 19:10: Upgraded ImageMagick to 7.0.8-11
- 14:00: Upgraded Mastodon to 2.4.5
- 14:25: Upgraded ImageMagick to 7.0.8-10
- 00:50 - 01:00: Upgraded Mastodon to 2.4.4
- 19:30: Upgraded OpenJDK to 10.0.2
- 17:30: Upgraded nginx to 1.15.2
- 18:30: Upgraded ImageMagick to 7.0.8-8
- 16:40: Upgraded Elasticsearch to 6.3.2
- 20:10: Upgraded ImageMagick to 7.0.8-7
- 22:50: Upgraded FFmpeg to releases/4.0@3ef38c4
- 17:55: Upgraded ImageMagick to 7.0.8-6
- 21:05 - 21:25: Upgraded Mastodon to 2.4.3
- 19:45: Upgraded Elasticsearch to 6.3.1
- 20:15: Upgraded nginx to 1.15.1
- 19:00: Upgraded ImageMagick to 7.0.8-3
- 20:00 - 20:05: Upgraded Mastodon to 2.4.2
- 20:20: Upgraded Sprockets to 3.7.2
- 12:10: Upgraded FFmpeg to releases/4.0@e049f7c
- 18:15: Upgraded redis to 4.0.10
- 18:50: Upgraded LibreSSL to 2.7.4
- 18:25 - 18:50: Upgraded Mastodon to 2.4.1
- 19:15: Upgraded ImageMagick to 7.0.7-39
- 19:30: Upgraded nginx to 1.15.0
- 16:00 - 16:20: Upgraded Mastodon to 2.4.0
- 16:30: Upgraded FFmpeg to releases/4.0@da3999
- 16:40: Upgraded ImageMagick to 7.0.7-35
- 01:50: Upgraded ImageMagick to 7.0.7-33
- 03:50: Upgraded LibreSSL to 2.7.3
- 17:00: Upgraded ImageMagick to 7.0.7-29
- 12:15: Upgraded nginx to 1.14.0
- 23:05: Upgraded Elasticsearch to 6.2.4
- 19:45: Upgraded nginx to 1.13.12
- 17:15: Upgraded nginx to 1.13.11
- 20:30: Upgraded LibreSSL to 2.7.2
- 21:00: Upgraded redis to 4.0.9
- 02:35 - 02:40: Upgraded Mastodon to 2.3.3
- 15:45 - 16:00: Upgraded Mastodon to 2.3.2
- 16:50: Upgraded LibreSSL to 2.7.0
- 14:45: Upgraded Elasticsearch to 6.2.3
- 15:00: Upgraded ImageMagick to 7.0.7-27
- 16:00: Upgraded nginx to 1.13.10
- 06:55 - 07:25: Upgraded Mastodon to 2.3.1
- 19:05 - 19:30: Upgraded Mastodon to 2.3.0
- 15:45: Upgraded FFmpeg to releases/3.4@e4b95f7
- 13:30: Upgraded ImageMagick to 7.0.7-24
- 19:45: Upgraded PostgreSQL to 10.3
- 18:35: Upgraded Elasticsearch to 6.2.2
- 22:45: Upgraded ImageMagick to 7.0.7-23
- 04:10: Upgraded FFmpeg to releases/3.4@6a97ba5
- 22:00 - 22:05: Upgraded Elasticsearch to 6.2.1
- 22:05 - 22:10: Upgraded PostgreSQL to 10.2
- 19:00 - 19:05: Upgraded Elasticsearch to 6.2.0
- 14:40 - 14:45: Deployed full-text search for authorized statuses using Elasticsearch
- 00:00 - 18:30: Connectivity issues for users connecting from South America due to a failing configuration deployment process that caused a TLS certificates to not be renewed.
- 03:15 - 04:20: Upgraded Mastodon to 2.2.0
- 15:00: Upgraded CoreDNS to 1.0.4
- 15:10: Upgraded ImageMagick to 7.0.7-22
- 15:10: Upgraded FFmpeg to releases/3.4@4a53ecb
- 15:45: Upgraded redis to 4.0.7
- 00:25 - 00:40: Upgraded Mastodon to 2.1.3
- 16:50: Upgraded ImageMagick to 7.0.7-21
- 00:00 - 00:15: Upgraded Mastodon to 2.1.2
- 18:55 - 19:10: Upgraded Mastodon to 2.1.1
- 02:30: Upgraded FFmpeg to releases/3.4@d810497
- 02:35: Upgraded nsjail to 2.3
- 02:50: Upgraded ImageMagick to 7.0.7-17
- 05:45: Upgraded nginx to 1.13.8
- 05:45: Upgraded LibreSSL to 2.6.4
- 02:45: Upgraded CoreDNS to 1.0.1
- 09:00 - 09:35: Enabled data checksums in PostgreSQL to detect silent corruption
- 20:00: Merged app server with reverse proxy
- 21:00: Added edge cache in São Paulo, Brazil
- 21:45 - 22:15: Upgraded Mastodon to 2.1.0
- 22:30: Upgraded FFmpeg to releases/3.4@bc839fb
- 09:40: Deployed nsjail sandbox for file(1)
- 09:45: Upgraded ImageMagick to 7.0.7-14
- 13:15: Deployed nsjail sandbox for FFmpeg
- 02:30: Recompiled Ruby with the jemalloc memory allocator
- 20:20: Upgraded redis to 4.0.6
- 03:20: Deployed nsjail to sandbox ImageMagick
- 01:30: Upgraded CoreDNS to 1.0.0
- 16:30: Upgraded redis to 4.0.5
- 19:40: Upgraded ImageMagick to 7.0.7-13
- 15:45: Upgraded redis to 4.0.3
- 16:00: Upgraded ImageMagick to 7.0.7-12
- 18:30: Upgraded redis to 4.0.4
- 23:15: Upgraded FFmpeg to releases/3.4@752659f
- 00:00: Upgraded nginx to 1.13.7
- 00:30: Upgraded ImageMagick to 7.0.7-11
- 14:00 - 2017-11-16 19:00: Performed maintenance to improve scalability and ability to reroute traffic to a secondary data center in the event of an outage.
- 01:00: Completed deployment of on-demand application servers and background workers in a secondary data center on Google Compute Platform (GCP).
- 06:12 - 12:30: Infrastructure provider suffered a power outage in our data center. Servers started coming back online at around 11:30. mastodon.at partially recovered at 12:00. Full service was restored by 12:30.
- 02:00 - 03:15: Replaced ImageMagick from Debian repositories with ImageMagick 7.0.7-10
- 15:45 - 16:00: Upgraded LibreSSL to 2.6.3
- 16:35 - 17:35: Replaced FFmpeg from Debian repository with FFmpeg 3.4
- 11:15: Decommissioned app server
app002-sbg
due to overprovisioning
- 01:00: Upgraded CoreDNS to 0.9.9
- 17:35 - 18:25: Upgraded Mastodon to 2.0.0
- 12:30 - 12:50: Upgraded PostgreSQL to 10.0
- 07:35 - 07:41: Requests from regions other than NA and SA were dropped due to a frozen VM
- 08:45 - 09:50: New statuses are not delivered to local users
- 15:15: Upgraded nginx to 1.13.6
- 05:30 - 06:25: Replaced dnsmasq with CoreDNS
- 10:36 - 10:50: Requests from regions other than NA and SA were dropped due to network connectivity issues
- 17:10: Upgraded LibreSSL to 2.6.2
- 16:10: Upgraded redis to 4.0.2
- 22:14 - 00:00: High load and slow status delivery due to a large number of federated status deletes from one remote account
- 11:50: Deployed update to librato-sidekiq with retry tracking
- 16:30: Deployed database row creation metrics
- 18:30: Updated Content Security Policy to allow embedded YouTube and fix an issue on the profile image update page
- 09:30 - 09:50: Migration of database server to OVH Public Cloud
- 18:10: Deployed nokogiri 1.8.1 to fix libxml2 vulnerabilities
- 21:00: Upgraded Mastodon to 1.6.1