2019/01/03

alert:CRITICAL mstdn.unigiri.net Filesystem % 90.01% > 90.0%

発生・解消時刻

概要

ディスク容量のうち約90%を使用していた。mastodon関連の容量が大きくなっていたため不要なファイルの削除を行った結果、約80%まで抑えられた。

対応概要

対応詳細

dfとduによりディスク容量を調査。/home/mastodon/live/public/system/以下が比較的肥大化していた。

$ sudo du /home/mastodon/live/public/system/ -h -d1
189M    /home/mastodon/live/public/system/preview_cards
55M     /home/mastodon/live/public/system/custom_emojis
987M    /home/mastodon/live/public/system/media_attachments
612M    /home/mastodon/live/public/system/accounts
1.8G    /home/mastodon/live/public/system/

/home/mastodon/live/public/system/media_attachments対応として古いメディアファイルの削除を実施。個人用インスタンスであり過去Tootsを頻繁に再読する使い方をしていないため、1日前より古いものは不要と判断した。

$ RAILS_ENV=production bin/tootctl media remove --days=2
............................................................................................................................................................................................................
............................................................................................................................................................................................................
............................................................................................................................................................................................................
............................................................................................................................................................................................................
............................................................................................................................................................................................................
Removed 1020 media attachments
$ RAILS_ENV=production bin/tootctl media remove --days=1
.............................................................................................................................
Removed 125 media attachments

アラート発生時は7日前より古いものを削除対象としていたが、上記の判断により1日前に変更した。

$ crontab -l
RAILS_ENV=production
@daily cd /home/mastodon/live && /home/mastodon/.rbenv/shims/bundle exec /home/mastodon/live/bin/tootctl media remove --days=7
$ crontab -e
crontab: installing new crontab
$ crontab -l
RAILS_ENV=production
@daily cd /home/mastodon/live && /home/mastodon/.rbenv/shims/bundle exec /home/mastodon/live/bin/tootctl media remove --days=1

/home/mastodon/live/public/system/accounts対応として存在しないユーザの削除を実施。

$ RAILS_ENV=production bin/tootctl accounts cull
............................................................................................................................................................................................................
............................................................................................................................................................................................................
............................................................................................................................................................................................................
............................................................................................................................................................................................................
............................................................................................................................................................................................................
............................................................................................................................................................................................................
............................................................................................................................................................................................................
............................................................................................................................................................................................................
............................................................................................................................................................................................................
.................................................................................................................................
Removed 58 accounts. 16 servers skipped
The following servers were not available during the check:
    zhenyabot.m.to
    mauve.m.to
    mastodon.home.js4.in
    bofa.lol
    dragon.garden
    toot.crystalyx.net
    m.parkgaedae.com
    smartdonkey.social
    raki.social
    windoos.click
    henshaw.org
    pennywise.m.to
    don.tacostea.net
    kowai.youkai.town
    newjack.city
    mstdn.tyamac.com
$ RAILS_ENV=production bin/tootctl domains purge zhenyabot.m.to
.
Removed 1 accounts
$ RAILS_ENV=production bin/tootctl domains purge zhenyabot.m.to

Removed 0 accounts
$ RAILS_ENV=production bin/tootctl domains purge mauve.m.to
.
Removed 1 accounts
$ RAILS_ENV=production bin/tootctl domains purge mastodon.home.js4.in
..
Removed 2 accounts
$ RAILS_ENV=production bin/tootctl domains purge bofa.lol
......................
Removed 22 accounts
$ RAILS_ENV=production bin/tootctl domains purge dragon.garden
.
Removed 1 accounts
$ RAILS_ENV=production bin/tootctl domains purge toot.crystalyx.net
.
Removed 1 accounts
$ RAILS_ENV=production bin/tootctl domains purge m.parkgaedae.com
.
Removed 1 accounts
$ RAILS_ENV=production bin/tootctl domains purge smartdonkey.social
.
Removed 1 accounts
$ RAILS_ENV=production bin/tootctl domains purge raki.social
..
Removed 2 accounts
$ RAILS_ENV=production bin/tootctl domains purge windoos.click
.
Removed 1 accounts
$ RAILS_ENV=production bin/tootctl domains purge henshaw.org
.
Removed 1 accounts
$ RAILS_ENV=production bin/tootctl domains purge pennywise.m.to
.
Removed 1 accounts
$ RAILS_ENV=production bin/tootctl domains purge don.tacostea.net
.
Removed 1 accounts
$ RAILS_ENV=production bin/tootctl domains purge kowai.youkai.town
.
Removed 1 accounts
$ RAILS_ENV=production bin/tootctl domains purge newjack.city
.
Removed 1 accounts
$ RAILS_ENV=production bin/tootctl domains purge mstdn.tyamac.com
.
Removed 1 accounts

結果、/home/mastodon/live/public/system/以下が1.8Gから1022Mまで削減された。

$ du /home/mastodon/live/public/system/ -h -d1
189M    /home/mastodon/live/public/system/preview_cards
55M     /home/mastodon/live/public/system/custom_emojis
184M    /home/mastodon/live/public/system/media_attachments # before:987M
596M    /home/mastodon/live/public/system/accounts # before:612M
1022M   /home/mastodon/live/public/system/

アラート発生時はWarning 80%, Critical 90%となっており、Warning 90%, Critical 95%に変更した。

アラート発生から解消までの13日間で90.04〜89.77%と急激な上昇が見られなかったため、80%は対応開始には早すぎると判断した。