Register
It is currently Sat Oct 21, 2017 3:47 pm

Fragmented EXT3 filesystem. One so far...


All times are UTC


Post new topic Reply to topic  [ 6 posts ] 
Author Message
 PostPosted: Fri Oct 06, 2017 9:02 pm   
. . . .
User avatar

Joined: Mon Apr 21, 2003 3:56 pm
Posts: 8249
Below, my /dev/sdb1 is a Sandisk USB Flash drive that I VERY recently reformatted as EXT3. Lately, it had been having odd write issues like pauses for unusually long periods. I consider that to be a hint that something needs assistance, so I unmounted it, and ran fsck.

We have all been taught that EXT filesystems rarely need defragmenting due to the way the filesystem is handled under Linux.

Ok, I'm quite surprised from what I present here.
Code:
$ sudo fsck -fpv /dev/sdb1
fsck from util-linux 2.25.2

        3426 inodes used (0.06%, out of 6111232)
        2433 non-contiguous files (71.0%)
           3 non-contiguous directories (0.1%)
             # of inodes with ind/dind/tind blocks: 2525/1216/0
    16641004 blocks used (68.14%, out of 24421438)
           0 bad blocks
           1 large file

        3027 regular files
         390 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
        3417 files
[email protected]:~


What is the meaning of the word 'contiguous'?
Quote:
con·tig·u·ous
[kənˈtiɡyo͞oəs]
ADJECTIVE
sharing a common border; touching:
"the 48 contiguous states"
synonyms: adjacent · neighboring · adjoining · bordering · next-door · abutting · connecting · touching · in contact · proximate
next or together in sequence:
"five hundred contiguous dictionary entries"

'Non contiguous' is a fancy word for 'not connected together, not in a line'.
Also known as fragmented.

So the files on my EXT3 device at /dev/sdb1 are 71% fragmented.

I must be dreaming.
Is there any man page with context regarding defragmenting a filesystem?
Yup.
Code:
$ man -k defrag
e4defrag (8)         - online defragmenter for ext4 filesystem
[email protected]:~

Well now.
Code:
$ e4defrag --help
e4defrag: invalid option -- '-'
Usage   : e4defrag [-v] file...| directory...| device...
   : e4defrag  -c  file...| directory...| device...
[email protected]:~


Since EXT4 is regarded as a modification/update to EXT3, maybe I can get some ideas for my /dev/sdb1?
Code:
$ sudo e4defrag -cv /dev/sdb1
Filesystem is not ext4 filesystem
[email protected]:~

Well, no help for my EXT3 formatted /dev/sdb1, then.
Certainly, I will conclude that Linux now needs a defragmenter for EXT4 filesystems, and it looks like my EXT3 is in as bad a shape as any Windows fragmented disk drive I ever owned. 71% of my files are described as fragmented.

One item of note, I had just reformatted the device and simply copied the files onto the flash drive.

Or, am I missing something here?

_________________
eMachines T5246 AMD 64 X2 w/Antix 16.2 for AMD64
EeePC 900A w/ Antix 16.2 32 bit
Dell Inspiron 1545 w/ Fedora something recent for 32 bit


Top
 Profile  
 PostPosted: Mon Oct 09, 2017 3:04 pm   
Linux Guru
User avatar

Joined: Sat Apr 03, 2004 12:39 am
Posts: 12268
Location: Clinton Township, Michigan
@Mmmna, you're right; I would say that your filesystem is quite highly fragmented indeed.

This isn't always a problem on fast disks, whether moving head disks of the past or more recent solid state disk technology, but as the number of locations increases in order to present a full representation of a file, performance can, and will, suffer.

ext3 was introduced in 2001 as a replacement for ext2.

ext4 became stable in 2008; I'd say at this time that while ext3 may still be available, it's not even close to the best available technology (btrfs is now a common alternative even to ext4).

All of these filesystems do offer a journaling capability, which makes them reasonably reliable. Journaling by itself does not, however, resolve disk fragmentation.

One comment I found, comparing fragmentation of ext3 vs. ext4 said this:

"In ext3 file system, there is a block allocator for the disk for each block, and so, it is quite possible that fragmentation can occur.

However, in ext4 file system there is a multi-block allocator which can delay the writing of blocks to disk, so that it can property allocate several blocks at a time in a single chunk of disk to allow for a contiguous write - and so, it is less likely that fragmentation can occur (it is still possible, just less likely)".

So complete absence of disk fragmentation is not a GUARANTEE with either approach, though ext4 has a somewhat smarter algorithm for attempting to keep commonly accessed information located in a similar region of the disk "surface".

The use of tunefs, fsck, and sometimes additional utilities provided by particular implementations can be used to modify the blocks and contents of a file system. Generally speaking, these disks and their partitions need to be unmounted when the activity is performed, so these are either single user system mode operations or something you perform from another disk, another distribution, or system admin oriented distros.

I'd say that you have a good case for wanting to defragment these drives. I'd carefully read up on which specific utilities are available on the system(s) you're using and the best practices to use in order NOT to lose any valuable information. Check out reading on fsck for ext3, and also tunefs (and there MAY be a e3defrag tool, check to see what's available to you).

_________________
Brian Masinick
Distros: MX-16, antiX, Debian


Top
 Profile WWW YIM  
 PostPosted: Tue Oct 10, 2017 11:47 pm   
. . . .
User avatar

Joined: Mon Apr 21, 2003 3:56 pm
Posts: 8249
A few years back, I think this was about 2014-ish, for just a very short time (maybe a couple days), I chose to reformat a recently purchased 128 GB flash drive from FAT32 to EXT4. Easy enough to do.

Ultimately, that was not a great idea, back then. Maybe the details have since been ironed out, not sure at the moment.

The issue was that some normal function within EXT4 was causing activity on the flash drive, every 5 or so seconds, I think that went on for as long as the drive was inserted in the USB port. You shouldn't use flash devices for very frequent writes, because the memory locations only tolerate a finite amount of writes before the cell could fail. Sure, there is redundancy and all that, but because I chose to leave it in the USB port, persistent access seemed weak, especially since FAT32 doesn't do that, nor does EXT3. I don't need EXT4 so very badly, so back to FAT32 that device went.

I was glad I had chosen a flash drive which still used an activity LED (Sandisk: take a hint), because I might have never known that the drive was being accessed like that. I recall, fuzzily since the years passed, that the parameters could be adjusted, and I almost attempted some adjustments, but the device was managed by something like udev, and I was not familiar with dealing with that stuff. Couldn't find any warm and fuzzy responses to address my issue, back then, so I've ultimately dropped back to EXT3 on devices that will never see Windows, and FAT32 on devices that will POSSIBLY see Windows.

Still not comfortable with such device specific tuning and tweaking today, mostly because I can't seem to find any information explaining what functions might be acceptable for a user to adjust. Trying to adjust device specific parameters could be easy, if the magic behind device handling was predictable. I recently tried modifying /etc/fstab so that a device was mounted according to my wishes, and wouldn't you know, udev simply created a new fstab entry according to its OWN desires. That wasn't very helpful.

So, until I get a reasonably reliable, recently written 'how-to' (recent because system binaries can be totally revamped) , a how-to which is topically related to using Debian derived distros, I'll just continue to use the very trusted EXT3 on as many devices as I can. Not something where I can afford to mess around very much.

As for best practices, my important drives are periodically copied to an external hard disk drive. No, I'm not using backup software, just drag and drop, then rename the parent folder so I know when I made the backup.

I would read something about external device tuning, if I felt comfortable with the article.

_________________
eMachines T5246 AMD 64 X2 w/Antix 16.2 for AMD64
EeePC 900A w/ Antix 16.2 32 bit
Dell Inspiron 1545 w/ Fedora something recent for 32 bit


Last edited by mmmna on Thu Oct 12, 2017 10:52 pm, edited 1 time in total.

Top
 Profile  
 PostPosted: Wed Oct 11, 2017 1:19 pm   
Linux Guru
User avatar

Joined: Sat Apr 03, 2004 12:39 am
Posts: 12268
Location: Clinton Township, Michigan
Gotcha, that makes sense. If I happen to come across any really good articles that are detailed but also reasonable to read, understand, and digest, I'll share a pointer to them and let you decide whether they are worth reading or investigating for your purposes.

I have seen a few discussions over the years, but nothing that is super clear about why some file systems have frequent I/O operations and which of those operations are highly localized on the disk. From what I can gather about ext4, though it's a solid, well built, well maintained file system, I also believe that you are right - it was not created with serial USB jump drives, sticks, etc. in mind. It's possible that btrfs might be, but I'd have to do more research on that to know for certain.

What you have said about FAT32 is something I've heard or seen too. Apparently it does not have constant I/O, and that may be why FAT32 is a recommended filesystem for these types of drives. Again, if I find anything of interest, I'll share it and maybe it'll explain all of this clearly.

_________________
Brian Masinick
Distros: MX-16, antiX, Debian


Top
 Profile WWW YIM  
 PostPosted: Wed Oct 11, 2017 2:46 pm   
Linux Guru
User avatar

Joined: Sat Apr 03, 2004 12:39 am
Posts: 12268
Location: Clinton Township, Michigan
Here is some research I found this morning:

https://www.howtogeek.com/73178/what-fi ... usb-drive/ - recommends FAT32 primarily because it "works" everywhere.
https://www.quora.com/What-is-the-best- ... lash-drive - NTFS is mentioned for "security", but FAT32 still preferred for small flash drives.
https://en.wikipedia.org/wiki/Flash_file_system - definitions, origins and history of Flash file systems,
https://www.cs.helsinki.fi/group/nodes/ ... mories.pdf - Research on Flash formatted file systems.
http://shahidhussain.com/best-file-syst ... nal-drive/ - Yet another opinion piece on file systems.

The last article gives us a pretty solid reason to go with FAT or FAT32 if you are using an Android device. These are the two file formats that appear to work best with most Android systems.

If, however, you don't care about other types of systems and are mostly concerned with Linux storage only, then believe it or not, ext4 and btrfs are two of the better choices, even though you "saw the USB light frequently with ext4. The article claims that btrfs will have even better performance than ext4:
https://unix.stackexchange.com/question ... format-for

Summary: if you're conservative, want something that "works" nearly everywhere, FAT32 will do that, but it'll be slow if you're doing much in the way of interactive computing, and security on the filesystem itself is non-existent, so those are the trade-offs to consider before making a final decision.

_________________
Brian Masinick
Distros: MX-16, antiX, Debian


Top
 Profile WWW YIM  
 PostPosted: Thu Oct 12, 2017 10:53 pm   
. . . .
User avatar

Joined: Mon Apr 21, 2003 3:56 pm
Posts: 8249
Thanks, I'll check into these.

_________________
eMachines T5246 AMD 64 X2 w/Antix 16.2 for AMD64
EeePC 900A w/ Antix 16.2 32 bit
Dell Inspiron 1545 w/ Fedora something recent for 32 bit


Top
 Profile  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 6 posts ] 

All times are UTC


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron


Powered by phpBB © 2012 phpBB Group
© 2003 - 2012 USA LINUX USERS GROUP