Oh I got this. Real good fuck up. So, I run a bunch of services for me and family and friends, like Nextcloud, Immich, Mastodon, etc. All of their data is stored on a big RAID array built with mdadm. I built it before I knew about dm-integrity, so I wanted to rebuild it with dm-integrity running on all the disks. So I took out one disk (it’s a RAID6, so it can stand losing a disk just fine), and added dm-integrity on top of it. When I tried to add it back to the array, mdadm was like, “that’s not a big enough device”. Ok, yeah, makes sense, dm-integrity takes up some space. So I took all the services offline and started a resize2fs to reduce from 22TB to 18TB. Well that was taking forever, so I canceled and ran e2fsck, no worries. I started a new resize2fs reducing from 22TB to 21.8TB, which should be good enough. On pass 4, it’s looking like it’s gonna take 8 DAYS. I can’t have my services offline for that long, so I cancel and run e2fsck. Thousands of errors. Oops.
At least it’s looking like all the errors are just in Mastodon’s cache files.
So now I’m moving all the files to an external drive in order to migrate the array properly. In its degraded state though, the array is painfully slow. At least my services are online.
So basically I risked my files (though I have a backup from about three days before all this), and gave myself an extra maybe ten days(?) of work.
Moral of the story, don’t ever use resize2fs unless you have time to wait.
I’ve done something like this before, but thankfully it only impacted services that I host for my own use, didn’t affect any family and friends.
Btw, I’ve found the easiest (but not the cheapest) way to fix this is to simply buy bigger disks. Swap out each disk for a bigger one one-by-one, then resize the whole volume to fill the new disks. Resizing upwards is much faster than shrinking a volume.
I’ve never had a volume shrink operation work without errors, and yes it takes days if you have more than 4TB.
Oh I got this. Real good fuck up. So, I run a bunch of services for me and family and friends, like Nextcloud, Immich, Mastodon, etc. All of their data is stored on a big RAID array built with mdadm. I built it before I knew about dm-integrity, so I wanted to rebuild it with dm-integrity running on all the disks. So I took out one disk (it’s a RAID6, so it can stand losing a disk just fine), and added dm-integrity on top of it. When I tried to add it back to the array, mdadm was like, “that’s not a big enough device”. Ok, yeah, makes sense, dm-integrity takes up some space. So I took all the services offline and started a resize2fs to reduce from 22TB to 18TB. Well that was taking forever, so I canceled and ran e2fsck, no worries. I started a new resize2fs reducing from 22TB to 21.8TB, which should be good enough. On pass 4, it’s looking like it’s gonna take 8 DAYS. I can’t have my services offline for that long, so I cancel and run e2fsck. Thousands of errors. Oops.
At least it’s looking like all the errors are just in Mastodon’s cache files.
So now I’m moving all the files to an external drive in order to migrate the array properly. In its degraded state though, the array is painfully slow. At least my services are online.
So basically I risked my files (though I have a backup from about three days before all this), and gave myself an extra maybe ten days(?) of work.
Moral of the story, don’t ever use resize2fs unless you have time to wait.
I’ve done something like this before, but thankfully it only impacted services that I host for my own use, didn’t affect any family and friends.
Btw, I’ve found the easiest (but not the cheapest) way to fix this is to simply buy bigger disks. Swap out each disk for a bigger one one-by-one, then resize the whole volume to fill the new disks. Resizing upwards is much faster than shrinking a volume.
I’ve never had a volume shrink operation work without errors, and yes it takes days if you have more than 4TB.