Discussion:
What uses these 50 GB?
Roland Olbricht
2014-08-17 17:28:33 UTC
Permalink
Hello everybody,

first of all thank you the development of Ext2/3/4. It works like a
charm and makes it possible to base applications on it.

However, now I have the first time where I need more information to
understand the behaviour of a ext4 installation on a 480 GB harddisk.
It holds a database with a size of 355 GB, as said by

"du -m":

...
355263 /opt/ssd

However, "df" says:

Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sdc 468346644 409888536 35015532 93% /opt/ssd

I do understand why there is a gap between "Used" plus "Available" and
"1K-blocks", but I don't understand why "Used" is so much bigger (54 GB
difference) than what "du -m" indicates.

I can rule out any issues with inodes; "df -i" indicates that less than
one percent is used.

I tried to understand more details by using "debugfs". I thought I get a
full list of used blocks with:

for i in *; do { echo "blocks $i" | sudo debugfs /dev/sdc | grep -vE
"^debugfs" | awk '{ for (i=1; i < NF; ++i) print $i; }'; }; done

which delivered 90938943 lines (containing block numbers allocated by
visible files). But

echo "testb 1 117330000" | sudo debugfs /dev/sdc | grep "marked in use"

has delivered 102595007 lines (containing block numbers marked as "used").

I tried to learn more about blocks marked as "marked in use" but not by
a known file, and

echo "icheck 98304 98305" | sudo debugfs /dev/sdc

returned

debugfs: icheck 98304 98305
Block Inode number
98304 <block not found>
98305 <block not found>

Could somebody explain to be what the purpose of these 11,656,064 blocks
is that don't belong to an inode but are still marked as used?

Do I have a chance other than reformatting the drive to get back this space?

Best regards,

Roland
Eric Sandeen
2014-08-17 18:00:47 UTC
Permalink
Post by Roland Olbricht
Hello everybody,
first of all thank you the development of Ext2/3/4. It works like a
charm and makes it possible to base applications on it.
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sdc 468346644 409888536 35015532 93% /opt/ssd
Could somebody explain to be what the purpose of these 11,656,064
blocks is that don't belong to an inode but are still marked as
used?
At least some is filesystem metadata. mkfs.ext4 preallocates inode tables,
bitmaps, etc, and that takes space from the filesystem. But that's
not supposed to be shown by default, with the "bsddf" behavior that's
hidden from the total available.
Post by Roland Olbricht
From above, 468346644*1024=479586963456; let's make a filesystem of
the same size:

# truncate --size=479586963456 fsfile
# mkfs.ext4 fsfile
# mount -o loop fsfile
# df mnt/
Filesystem 1K-blocks Used Available Use% Mounted on
/tmp/fsfile 460865340 71736 437359888 1% /tmp/mnt

Hm, that doesn't have the same "1k blocks" value as you saw.
What if we mount it with minixdf, which doesn't "hide" fs-internal
metadata from the totals?

# umount mnt/
# mount -o minixdf,loop fsfile mnt

# df mnt/
Filesystem 1K-blocks Used Available Use% Mounted on
/tmp/fsfile 468346644 7553040 437359888 2% /tmp/mnt

now the 1k-blocks total matches your output; you must be mounting with
minidxf for some reason.

468346644-460865340 is 7481304 1k blocks, or 7G, so that's some of it.
If you have one nonstandard mount option, perhaps there are other
tweaks you've made at mkfs time which could change the geometry from
what I see above. dumpe2fs -h output might show us that.

It could also be open but unlinked files, or unprocessed orphan inodes
after a crash. Have you run e2fsck?
Post by Roland Olbricht
debugfs: icheck 98304 98305
Block Inode number
98304 <block not found>
98305 <block not found>
icheck won't find an inode number for internal metadata; it's not associated
with any inode.
Post by Roland Olbricht
Do I have a chance other than reformatting the drive to get back this space?
It shouldn't be "lost" - repeating the same administration steps should
lead to the same space usage.

You could choose to rereate it with fewer inodes, to save some inode table
space...

-Eric
Roland Olbricht
2014-08-17 19:05:50 UTC
Permalink
Hello Eric,

thank you for the quick reply and the explanations.
Post by Eric Sandeen
dumpe2fs -h output might show us that.
Filesystem volume name: <none>
Last mounted on: /opt/ssd
Filesystem UUID: 75d6aae6-1746-4260-994b-148dfdb5af95
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index
filetype needs_recovery extent flex_bg sparse_super large_file huge_file
uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 29310976
Block count: 117212886
Reserved block count: 5860644
Free blocks: 115322570
Free inodes: 29310965
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 996
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Flex block group size: 16
Filesystem created: Wed Apr 16 06:31:53 2014
Last mount time: Wed Apr 16 06:48:26 2014
Last write time: Wed Apr 16 06:48:26 2014
Mount count: 1
Maximum mount count: -1
Last checked: Wed Apr 16 06:31:53 2014
Check interval: 0 (<none>)
Lifetime writes: 133 MB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
First orphan inode: 8388637
Default directory hash: half_md4
Directory Hash Seed: a094608e-671f-4a9c-81e8-81bb72eb4516
Journal backup: inode blocks
Journal features: journal_incompat_revoke
Journal size: 128M
Journal length: 32768
Journal sequence: 0x001d92ff
Journal start: 27755
Post by Eric Sandeen
It could also be open but unlinked files, or unprocessed orphan inodes
after a crash. Have you run e2fsck?
No, not yet. Shoud I do this routinely? Running

sudo e2fsck -fn /dev/sdc

gives as result:

Warning! /dev/sdc is mounted.
Warning: skipping journal recovery because doing a read-only filesystem
check.
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 8388637 has zero dtime. Fix? no

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: -(9255--9422) -(33793--33801) -(33808--33870)

... (really long list)

Fix? no

Free blocks count wrong (115322570, counted=14605339).
Fix? no

Inode bitmap differences: -8388637
Fix? no

Free inodes count wrong (29310965, counted=29310750).
Fix? no


/dev/sdc: ********** WARNING: Filesystem still has errors **********

/dev/sdc: 11/29310976 files (272.7% non-contiguous), 1890316/117212886
blocks

So I assume the best thing to do here would be to unmount the disk and
do a e2fsck?

Best regards,

Roland
Eric Sandeen
2014-08-17 19:36:23 UTC
Permalink
Post by Roland Olbricht
Hello Eric,
thank you for the quick reply and the explanations.
Post by Eric Sandeen
dumpe2fs -h output might show us that.
Filesystem volume name: <none>
...
Post by Roland Olbricht
Filesystem created: Wed Apr 16 06:31:53 2014
Last mount time: Wed Apr 16 06:48:26 2014
Last write time: Wed Apr 16 06:48:26 2014
Mount count: 1
Maximum mount count: -1
Last checked: Wed Apr 16 06:31:53 2014
Check interval: 0 (<none>)
Lifetime writes: 133 MB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
First orphan inode: 8388637
...

Ok, pretty standard, and it appears that you mkfs'd it in april,
mounted it, and have never unmounted it. (Nor has it ever crashed,
apparently).

Also, the last line above shows that there is at least one orphan
inode, meaning it's open but unlinked. Blocks aren't freed
until it's closed; that may be where all the space is.
Post by Roland Olbricht
Post by Eric Sandeen
It could also be open but unlinked files, or unprocessed orphan inodes
after a crash. Have you run e2fsck?
No, not yet. Shoud I do this routinely? Running
sudo e2fsck -fn /dev/sdc
Warning! /dev/sdc is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
and so the rest isn't useful. You'd have to unmount it to check it.

...
Post by Roland Olbricht
So I assume the best thing to do here would be to unmount the disk and do a e2fsck?
Probably so, or figure out who has that orphan inode open... if that process lets
go, the space may free up; if it doesn't let go, you won't be able to unmount...

-Eric
Roland Olbricht
2014-08-17 21:53:32 UTC
Permalink
Hello Eric,
Post by Eric Sandeen
Ok, pretty standard, and it appears that you mkfs'd it in april,
mounted it, and have never unmounted it. (Nor has it ever crashed,
apparently).
Also, the last line above shows that there is at least one orphan
inode, meaning it's open but unlinked. Blocks aren't freed
until it's closed; that may be where all the space is.
Thank you. That was indeed the cause of the problem. There was a single
file that has been deleted some month ago, but the process which kept it
open was still alive.

I'll add a check for open deleted files to my checklist.

Best regards,

Roland

Bodo Thiesen
2014-08-17 21:32:29 UTC
Permalink
* Roland Olbricht <***@gmx.de> hat geschrieben:

Hi Roland
Post by Roland Olbricht
Reserved block count: 5860644
Block size: 4096
Makes >20 GB. df doesn't show reserved clusters as free. That's why I
always mke2fs -m 0 [...].

Tip: It's a tunable parameter, but as for e2fsck, you *really* should
umount first.

Also, not mentioned yet: Files big enough, that not all cluster pointers /
extents fit in the inode itself, will take up additional meta data
clusters. However, for ext2 with extents feature (which you have), this
shouldn't really be much, I guess much much less than 1 GB.

Regards, Bodo
Loading...