ext3 / ext4 on USB flash drive?

[Mark Ballard]

I think this is really an attempt at user feedback, rather than user
discussion. But there's no such thing as a user-feedback mail list.

Nevertheless, others may find this pertinent: why doesn't mke2fs
handle USB's competently? And if it does, why doesn't it reassure me
so? And how can I handle a linux-formatted USB flash drive in the
absence of my system giving me any guidance?

I cannot accept the usual advice you see in the forums: simply format
it FAT16 or whatever. I need it to use a linux file system.

But linux may have problems, apparently, with the device:
http://lwn.net/Articles/428584/

Plenty of people say in the forums that ext3 and ext4 will wear your
USB out quickly, effectively thrashing it with their journaling. But
the above-linked article suggests it may depend on the device. Some
people say ext4 is simply okay, some that it is actually advisable to
use it. I'm none the wiser.

If it is true that ext4 will thrash your USB then it could important
if it is also true that, for example, the life-expectancy of a USB can
really as low as 10,000 writes:
http://linux.koolsolutions.com/2009/01/29/installing-linux-on-usb-part-3-which-linux-filesystem-for-usb-devices/.
I spoke to Integral Memory, the manufacturer of my USB drive. Their
technical support line said their device had a life expectancy of 1000
writes, and came with a two-year warranty. He didn't know what he was
talking about.

None of Integral's publically available spec-sheets tell you anything
useful. They give an approximate write-speed and that's about it.
Nothing about block size, whether optimized for linear write or not,
what method of wear distribution they use - all the things you are led
to believe may be important in deciding whether you should format your
USB ext4 or not.

Of course, I shouldn't have to know about any of these things as a
user anyway. But I'm left with no choice but to lower myself into this
arcane realm and get in a tangle.

One of the things I'm led to believe I must consider when formatting a
USB drive is its physical block size. This appears to be something of
a mystery, but someone reckons he has worked out a simple method of
discerning it: http://kim.oyhus.no/FlashBlockSize.html. Using that
method, and assuming Kim G. S. Øyhus is correct, my drive seems to
have 4k block size. Now perhaps if I format my drive with a filesystem
that uses a 4k block size I might stop it being mashed by ext4.
Perhaps.

It does seem strange to me though that mke2fs doesn't already do this
for me: discerns what's going on underneath the casing of my USB,
adjust its formatting parameters accordingly - and tell me what its
doing so I can be reassured. Or that the help pages at least tell me
definitively whether and in what circumstances it can be used with USB
so I don't have to waste so much time trying to unravel the mystery by
going like a pinball from one contradictory, incomplete or
well-meaning nincompoop forum message to another; and so I don't mash
my USB, and lose my data, because I should have listened to the
pessimists and not put an ext4 filesystem on my USB. I actually don't
want to believe them. But they shout loudest.

mb.

[Theodore Ts'o]

Post by Mark Ballard
I think this is really an attempt at user feedback, rather than user
discussion. But there's no such thing as a user-feedback mail list.
Nevertheless, others may find this pertinent: why doesn't mke2fs
handle USB's competently? And if it does, why doesn't it reassure me
so? And how can I handle a linux-formatted USB flash drive in the
absence of my system giving me any guidance?

The problem is that it really depends on the quality of the flash
drive. Two main things drive this:

1) Whether the flash used in the drive is SLC. MLC, or TLC. This
will tell you how many write cycles the flash can support. This is
where you will see numbers such as 10,000 write cycles, all the way
down to 1,000 write depending on the quality of the flash used.

2) The quality of the Flash Translation Layer (FTL). The important
thing to remember here is that not all parts of the storage device
will get used to the same amount. Some files will rarely change once
they are installed on the device; others will change a huge amount.
So a good Flash Translation Layer will spread out the wear so that if
you have a 120GB device, with flash cells that support 10,000 write
cycles, that ideally you can support up to 10,000 times 120GB == 120
TB worth of writes. Of course, that assumes 100% efficiency, which is
unrealistic. But some FTL's are incredibly bad, and in the worst
case, if they aren't spreading the wear around at all, if you write a
single block 1,000 times, and the flash only supports 1,000 write
cycles, then that block will go bad.

Unfortunately, the cheapest USB flash devices that you find in the
bargain bin at the checkout counter of the Micro Center tend to use
the crappiest flash media possible, and the crappist FTL's. Some even
will only have 512MB of flash even though it is advertised as having
8GB of flash, such that if you ever write more than 512MB worth of
files on the flash, you will start losing data. (Fortunately most
people don't actually use the full capacity of their USB sticks, which
is why the manufacturers can get away with this kind of fraud.)

So this is fundamentally a problem with the quality of the hardware,
and that's not something the file system can really compensate for.
And there's no way to tell whether a particular USB device has has a
high quality flash device, or is a craptastic flash device. It's not
like we can query the device for "I ripped off the purchaser" bit. :-)

Regards,

- Ted

[Eric Sandeen]

Post by Theodore Ts'o
So this is fundamentally a problem with the quality of the hardware,
and that's not something the file system can really compensate for.
And there's no way to tell whether a particular USB device has has a
high quality flash device, or is a craptastic flash device. It's not
like we can query the device for "I ripped off the purchaser" bit. :-)

Yes, this is the fundamental problem.

What's the best setting for the random blob of silicon on the end of
your usb plug? e2fsprogs can't call tech support, even to get bad
answers.

We'd like to help in software, but we can't; we have no reliable way
of knowing most of the necessary details of this class of hardware.
It's not exported in any way.

So unfortunately we are as in the dark as you are in this case.

-Eric

[Mark Ballard]

Post by Eric Sandeen
We'd like to help in software, but we can't; we have no reliable way
of knowing most of the necessary details of this class of hardware.
It's not exported in any way.
So unfortunately we are as in the dark as you are in this case.

This is incredible, Mr Sandeen. You mean USB flash manufacturers
(what's their body - the USB Implementer's Forum?) have simply not
provided a means for software to query the underlying hardware in a
USB flash? Have software producers asked them for this?

mb.

[Theodore Ts'o]

Post by Mark Ballard
This is incredible, Mr Sandeen. You mean USB flash manufacturers
(what's their body - the USB Implementer's Forum?) have simply not
provided a means for software to query the underlying hardware in a
USB flash? Have software producers asked them for this?

No, they haven't. And yes we have, since there are some things (such
as the erase block size) which would be useful for tuning file system
performance. And the technical people I've talked to at various Flash
manufacturers all agree it's pointless to hide this information, but
the product managers tend to be the roadblock. If you are buying
several million eMMC flash devices for a mobile handset (i.e., for an
Android device), you can find out this information, under NDA. But
otherwise, the only way you can find it out is by carrying out timing
attacks against the flash device.

Even if they did provide this sort of information, there are lots of
things which are not quantifiable. Cars don't have interface for
telling you whether they have the crash protection of a Ford Pinto, or
a Tesla S. Part of this is because product managers aren't going to
want to advertise that they have only provided 512MB worth of flash on
a device which is labelled as containing 8GB (and which will claim to
contain 8GB even if you query it via software; it's just that the
512MB + 4k write will cause some random block to disappear). But the
other part is that there isn't a good single metric for "crash safety"
(crash safety against head-on collisions? crash safety versus side
impact collisions? etc.) Similarly, even for a non-fradulent USB
stick or SD card, there is no single way to measure "FTL quality". An
FTL which is optimized for use in digital cameras (where writes are
usually sequential, and for large jpg images) may be horrible for use
a general purpose root file system.

There are sites that will do technical analysis and ratings for
various flash media (such as www.anandtech.com). For example a quick
google search turns up:

http://www.anandtech.com/show/4523/usb-30-flash-drive-roundup

These are going to be the pricier devices, though. They aren't going
to be the five dollar crap that you find at the checkout counter of a
computer store. But then again, you get what you pay for....

- Ted

[Mark Ballard]

From the little I have heard about control systems for cars, which was

some years ago, they were blockhead proprietary. The analogy would
only work if computing was customarily blackbox technology, which it
isn't. I'd be surprised if there were any branded flash drives that
contained less than their advertised amount of storage.

That leaves the question of what is going on under the hood in what is
probably the vast majority of devices where the flash isn't
fraudulent. And whether my system handles it correctly. My system
leaves me with no idea of either (though my hope holds out for some
tools I bookmarked recently). Reference to forums and specialist
websites gives genuine cause for doubt. Yet I thought it was usual for
system software to have a good angle on how its hardware was
constructed and what it was doing. I thought they worked in symbiosis,
and that this maintained by mutual necessity. I thought the symbiosis
was kept unassailably whole by a common purpose: the user.

What you say implies that this symbiosis has been broken by the
commercial greed of flash manufacturers. Or that it is by neglect on
their part, or lazyness, or some other cause of a fissure in industry
relations.

Whatever the reason, it raises another question, and that is what must
be done so that I can simply format my USB without a concern and get
back to my work.

even for a non-fradulent USB stick or SD card, there is no single way to measure "FTL quality". ... there are some things (such as the erase block size) which would be useful for tuning file system performance. And the technical people I've talked to at various Flash manufacturers all agree it's pointless to hide this information, but

the product managers tend to be the roadblock.
This is perhaps telling. One would imagine the USB Industry Forum
meeting the Association of (File) System Software Scribes or whatever
at routine collegiate meetings in Las Vegas hotels, and so on.

Which flash manufacturers have refused to collaborate? Why has the
fabled industry forum failed?

mb.

[Theodore Ts'o]

Post by Mark Ballard
isn't. I'd be surprised if there were any branded flash drives that
contained less than their advertised amount of storage.

The vast majority of flash sold, especially the cheap-grade flash
(i.e., SD Cards and USB sticks) sold through retail channels, is
probably unbranded. Because it's cheaper, and for most users, (a)
price is a feature, and (b) they are only using flash as a temporary
transport medium (e.g., here let me give you my slide presentation;
can I borrow a USB stick?), and (c) they are much more likely to lose
said flash device before it is likely to go bad, or even gets 100%
filled.

It's for the same reason that the quality of experience in airplanes
travel has degraded so badly. The market has spoken; and consumers
have said, at least by their actions, that price is important than
anything else.

Post by Mark Ballard
Whatever the reason, it raises another question, and that is what must
be done so that I can simply format my USB without a concern and get
back to my work.

Buy high quality flash which has been explicitly reviewed by a source
you trust. There isn't much else you can really do....

Post by Mark Ballard
This is perhaps telling. One would imagine the USB Industry Forum
meeting the Association of (File) System Software Scribes or whatever
at routine collegiate meetings in Las Vegas hotels, and so on.
Which flash manufacturers have refused to collaborate? Why has the
fabled industry forum failed?

They are collaborating --- with the mass buyers of their flash. If
you are a purchasing flash by the millions, then you can get all of
this information (under NDA), and you can dictate the quality of the
flash which is appropriate for your use case.

This even afflicted Microsoft's Windows Phone, where they had some
manufacturers provide an SD Card slot. This meant that end users
could replace their carefully tested-and-selected-for-performance SD
cards which was shipped with their phone with crap sold at the
checkout counter, and since the phone's root file system was stored on
the SD card, performance when into the crapper, and guess who the
customers blamed? Not the flash manufacturer, and not the handset
manufacturer for including a removable SD-card slot instead of using a
fixed eMMC flash device, but Microsoft.

As a result, many handset manufacutrers these days do *not* have an SD
card slot, and if they do, they don't allow the root file system to be
stored on the SD card, and the SD card can only be used for auxilary
or media storage (for which even really crappy flash is generally good
enough).

So the market is working; it's just working for the most common use
case, and the most common desire of the customers who are doing the
buying. And that means there will be high quality stuff that costs
$$$, and really cheap stuff where you get what you pay for, and
hardware manufacters who buy flash devices by the million unit order
will get better deals, and all of the low-level information under NDA.

All hail the free market.... as my libertarian friends would say,
"Huge success".

- Ted

[Mark Ballard]

Agh. This is great to have some of this clarified, Ted. It does
unfortunately reinorce my cynicism. But it also fills my heart. That
is, though it might be bad news to hear that I have most likely bought
a piece of crap, and that there's no way I can really tell what's
inside the cover unless either I do some kind of low-level alchemy of
a kind that mere users would normally be well advised to avoid, or the
manufacturer fesses up to its scam by publishing the specs; it is
nevertheless the case that this news has come from the filesystem
maintainer himself, and my faith in humanity is thus restored by this
simple proof of floss's egalitarian creed. Amen.

That said, the user forums are full of lost souls, despairing for want
of an answer to the simple question of whether they could well or
should not format their USB in ext3/4. Are you implying that it cannot
be known unless the manuf gives you the specs and if this is the case
and you've bought a bargain bucket flash drive (and assuming that
without a decent flash h/w controller it is true that ext3/4 will
thrash your drive) then you should stick to ext2? What then are the
features a USB flash spec you should look for as the minimum required
before installing ext4?

cheers

mb.

Post by Theodore Ts'o

Post by Mark Ballard
I think this is really an attempt at user feedback, rather than user
discussion. But there's no such thing as a user-feedback mail list.
Nevertheless, others may find this pertinent: why doesn't mke2fs
handle USB's competently? And if it does, why doesn't it reassure me
so? And how can I handle a linux-formatted USB flash drive in the
absence of my system giving me any guidance?

The problem is that it really depends on the quality of the flash
1) Whether the flash used in the drive is SLC. MLC, or TLC. This
will tell you how many write cycles the flash can support. This is
where you will see numbers such as 10,000 write cycles, all the way
down to 1,000 write depending on the quality of the flash used.
2) The quality of the Flash Translation Layer (FTL). The important
thing to remember here is that not all parts of the storage device
will get used to the same amount. Some files will rarely change once
they are installed on the device; others will change a huge amount.
So a good Flash Translation Layer will spread out the wear so that if
you have a 120GB device, with flash cells that support 10,000 write
cycles, that ideally you can support up to 10,000 times 120GB == 120
TB worth of writes. Of course, that assumes 100% efficiency, which is
unrealistic. But some FTL's are incredibly bad, and in the worst
case, if they aren't spreading the wear around at all, if you write a
single block 1,000 times, and the flash only supports 1,000 write
cycles, then that block will go bad.
Unfortunately, the cheapest USB flash devices that you find in the
bargain bin at the checkout counter of the Micro Center tend to use
the crappiest flash media possible, and the crappist FTL's. Some even
will only have 512MB of flash even though it is advertised as having
8GB of flash, such that if you ever write more than 512MB worth of
files on the flash, you will start losing data. (Fortunately most
people don't actually use the full capacity of their USB sticks, which
is why the manufacturers can get away with this kind of fraud.)
So this is fundamentally a problem with the quality of the hardware,
and that's not something the file system can really compensate for.
And there's no way to tell whether a particular USB device has has a
high quality flash device, or is a craptastic flash device. It's not
like we can query the device for "I ripped off the purchaser" bit. :-)
Regards,
- Ted

[Theodore Ts'o]

Post by Mark Ballard
That said, the user forums are full of lost souls, despairing for want
of an answer to the simple question of whether they could well or
should not format their USB in ext3/4. Are you implying that it cannot
be known unless the manuf gives you the specs and if this is the case
and you've bought a bargain bucket flash drive (and assuming that
without a decent flash h/w controller it is true that ext3/4 will
thrash your drive) then you should stick to ext2? What then are the
features a USB flash spec you should look for as the minimum required
before installing ext4?

The cheapest crap is going to be "unsafe at any speed". It might be
OK if you are using it for the occasional transfer of files from one
computer to another, but if you are going to be doing lots of updates,
your data is going to be at risk no matter what file system you use.

Some filesystems are more performant than others, so keep in mind that
a simple way to double the lifetime of some cheap crap is to slow down
the write speed by a factor of two. :-)

So I wouldn't so much focus on whether some storage device is "good
enough for ext3/4", but rather, "is it good enough given the value of
the data of the data I'm storing on it, and the read/write patterns
I'm going to be subjecting the drive to"?

Cheers,

- Ted