We're currently scaling our disk space to twice the size, and will handle this in a more graceful manner in the future. Sorry for the inconvience.
Wed May 27 02:56:32 CEST 2009: Part of resizing a ext3 filesystem, is to run a full check on it. This check is taking a while to run, but should be done soonish. I hope to be up and running within the hour.
# fsck -f /dev/sdf fsck 1.41.3 (12-Oct-2008) e2fsck 1.41.3 (12-Oct-2008) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/sdf: 4587520/4587520 files (2.3% non-contiguous), 13919184/18350080 blocks # resize2fs -p /dev/sdf resize2fs 1.41.3 (12-Oct-2008) Resizing the filesystem on /dev/sdf to 52428800 (4k) blocks. Begin pass 1 (max = 1040) Extending the inode table XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX The filesystem on /dev/sdf is now 52428800 blocks long. # tune2fs -j /dev/sdf tune2fs 1.41.3 (12-Oct-2008) Creating journal inode: done This filesystem will be automatically checked every 21 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override.
Wed May 27 02:01:43 CEST 2009: New volume has been attached, resizing the ext3 filesystem.
Wed May 27 01:59:19 CEST 2009: Snapshot has been completed. Updates will follow shortly.
Wed May 27 01:40:10 CEST 2009: Currently at 94%. nginx is very nice to work with.
Wed May 27 01:27:29 CEST 2009: Currently at 90%. Just to clarify, no data will be at risk today.
Wed May 27 01:10:46 CEST 2009: Currently at 82%. Had bitbucket been hosting a centralized SCM, this would have been a nightmare.
Wed May 27 00:54:25 CEST 2009: Currently at 73^H5%. I just had an idea on how to make bitbucket work again, I could copy the data from the EBS volume to the temporary storage on the EC2 instance and do rsyncs to the permanent storage. In the event of a instance failure it would however kill all the data from the last rsync, and the chance of that is enought to make me discard the idea as unsafe. If data reaches the bucket, it should be safe.
Wed May 27 00:44:09 CEST 2009: Currently at 70%. Turned the status updates upside down. Updates will be put in at the top from now on.
Wed May 27 00:31:58 CEST 2009: Currently at 66%. I'm currently on irc at freenode in the #bitbucket channal, grab me if I can help you with something related to the downtime of bitbucket. No updates for 3 minutes while I make myself a coffee.
Wed May 27 00:28:52 CEST 2009: Currently at 63%.
ajaksu: madssj: you got a new paying user (hi) just from the status updates :)
Wed May 27 00:13:08 CEST 2009: Currently at 55^H7%. We need to use this snapshot when it's been created, I hope it isn't as slow as creating it.
Wed May 27 00:02:40 CEST 2009: Currently at 50%. Looks like this will take a hour. Shouldn't keep anyone from comitting locally or reading up on the lasest goddies or perhaps make a chair fight.
Tue May 26 23:45:32 CEST 2009: Creating a EBS snapshot to remount and scale the disk. Currently at 44%.
The linux kernel seem to be lying.
/dev/sdf 69G 53G 17G 76% /data/vol1
Mads - The bitbucket team