- Site menu:
- Latest Discussions
- Latest Newswire
- Archives
- Submit Story
- Story HOWTO
- Syndicate
- E-mail Alerts
Test: Do Linux filesystems need defragmentation?
It was in a LXer discussion one of our readers suggested all those portage updates and temporary files in the compilation process lead to serious fragmentation of a Gentoo Linux system. That's true, when looking at the output of an 'emerge' process, there's a tremendous lot of files being created, copied or moved, and finally deleted. Not to speak about when the portage tree is updated. Probably, for normal usage, a Gentoo Linux system has one of the biggest numbers of this kind of file operations. Therefore, I figured, filesystems on a Gentoo Linux system were the ideal test to see if the Linux filesystems could live up to my expectations. However, since most people believe Linux filesystems don't need defragmentation, it was quite hard to find a tool which measures filesystem fragmentation. Nonetheless, I found one in the Gentoo forums. Here's the great part: You can use it yourself to measure filesystem fragmentation of your own filesystems!
I don't know under what license the content of Gentoo-forums is, so therefore I'll provide a short HOWTO. Credits go to Gentoo-user _droop_ for producing this perl script.
The argument should be the mount-point of one of your mounted filesystems if you want to know the fragmentation of a particular file system, like / or /usr or so. More interesting maybe is the ability to fill in any directory you like, for example I tested it on a directory where my P2P program occasionally puts some music; /home/kwint/music - only to find out my average mp3 in that dir consists of more than 50 fragments! It seems you must be root to do this scan, so I advise you to use sudo. A typical outcome would be: $ sudo ./frag.pl /root Now, let's look at some actual numbers, from my two Gentoo boxes. In both cases, the portage tree has been synchronized more than fifty times, and both boxes are in use for about one year and a half. The results are, less than 4% of the files on my /usr and /var partition are fragmented. On Gentoo, /var is where the actual compile takes place, once compiled Gentoo moves the files to /usr. The files over there have a maximum of consisting of an average of 1,11 fragments. I would consider that as a very neat score - especially since I use the -notail and -noatime options for ReiserFS. It seems those options make my ReiserFS more susceptible to file fragmentation. Back to my results, I'm sure the Windows defragmentation utility would say I don't need to defragment. But what if I _would_ like to defragment my filesystem, would that be possible? Well, though there's no official 'compiled' application for it as far as I could find, you can try Con Kolivas' script. For example, my /home partition contains 13G of files. Some of them are rather large like one-Gigabyte VMWare image files and some Linux iso's, all of it is on ReiserFS again by the way. Results were really bad: 6 fragments per file on average. This seemed like a good testing ground for Con's script, so I gave it a spin.
The results were quite stunning. I first tested it on my /home/kwint/music directory, where the average number of fragments per file were the 50 I mentioned above. After running Con's script this was reduced to 1.1, and the percentage of non-contiguous files dropped from 70% to 10%! This took only seven seconds to complete, and only 21 large files were touched. For your information, I'm running 2x5400rpm PATA disks in software RAID0 by means of EVMS here. Since this was a success, I decided I might as well 'defragment' my whole /home partition. This time, over 6000 files had to be defragmented, which of course took far more time, 22 minutes eventually. It was worth it, while the number of fragmented files only dropped from 7% to 8% - that must be because only some large files are defragmented, the average number of fragments per file dropped from 6 to 1,5. ConclusionThough I only tested two PC's, here's what I found: File fragmentation mainly happens in filesystems which contain some large files. You don't have to worry about your /usr or /var directory being fragmented, since they don't contain many large files. As the results show, it's worth the effort to try Con Kolivas defragmentation script.Please, if you try the scripts share your findings and react in the threads below! |
|
Comments (31)
I asked about this in 1995...
Wow, Bob, somebody gave you an utterly BS response.
Sorry to hear you took it to heart.
Maybe they were being serious, and doing the best they could in light of the limited options available under Linux.
In mainframe days, we paid a lot more attention to disk I/O, because nothing could help a business application more than well-tuned I/O. Mind you, that reflects typical mainframe workloads -- not to mention mainframe hardware.
In Unixland, cache and intelligent I/O ordering mitigate a lot of sins, but, even in Unixland, flying drive heads are a bad thing.
Sorry to hear you took it to heart.
Maybe they were being serious, and doing the best they could in light of the limited options available under Linux.
In mainframe days, we paid a lot more attention to disk I/O, because nothing could help a business application more than well-tuned I/O. Mind you, that reflects typical mainframe workloads -- not to mention mainframe hardware.
In Unixland, cache and intelligent I/O ordering mitigate a lot of sins, but, even in Unixland, flying drive heads are a bad thing.
Just for your info. Found it here:
http://www.reactos.org/forum/viewtopic.php?t=4759
---------------------------------------------------------------- Haos Lot of Linux file systems are highly fragmentation resistant.
So highly fragmentation resistant that one defrag program for linux does not have any direct interface with the file system. All it does is scans for a fragmented file from inode information(Linux internal block display so file system neutral) and tells the filesystem to move it and its magically defraged. No special operation required. General usage can trigger defrags.
Even just writing a extra block to a file can create a defrag event on some Linux filesystems.
Linux will defrag threw general use more than windows. So far that Vista has a background defrag program to keep its fragmentation under control.
Linux fragmention is a lot slower due to these minor changes.
Note Linux installed on FAT32(yes you can) Suxs on fragmentation about as much as Windows. There are little design problems in the way windows handles files. Nothing not fixable in Ros while keeping support for Windows.
It can take years at times to get Linux filesystem above 5 percent fragmented. Now you have seen the first reason. Fragmentation resistance in filesystems. Lot of people under linux reach the defrag point when the hardware needs replacing. So the idea of no need for defragmenting comes from the time to need to defrag comes around.
Windows does have a common cause of fragmentation called restore snapshots and altering swapfile sizes. Reactos should support swap partitions outbox. There is a important difference on fragmentation by using them. Spliting of tmp to its own partition can also have positive effects on fragmentation. Theses are little tricks Linux and Unix have been using for a long time due to great positive effects. MS did have support for swap partitions back in Windows 2000. But it default option still was swapfile.
Note HDD filled no room for play nothing can help you much then.
Please note it is MS fault on that run as Admin in the docs of the Creator of NT design Admin was never ment to be created alone.(Hello Windows NT 3.5 default user Admin level no requirement to create the extra user) User and Admin account should have always be created. Note MS decide that was too user unfriendly and removed it.
Now they have 1000s of programs that need it. Whos fault is it. Microsoft's. Buck stops with the person who decide to disobey the Creator advice.
Support for fragmentation resistant file systems have to be up there on the wish list.
---------------------------------------------------------------- Haos Lot of Linux file systems are highly fragmentation resistant.
So highly fragmentation resistant that one defrag program for linux does not have any direct interface with the file system. All it does is scans for a fragmented file from inode information(Linux internal block display so file system neutral) and tells the filesystem to move it and its magically defraged. No special operation required. General usage can trigger defrags.
Even just writing a extra block to a file can create a defrag event on some Linux filesystems.
Linux will defrag threw general use more than windows. So far that Vista has a background defrag program to keep its fragmentation under control.
Linux fragmention is a lot slower due to these minor changes.
Note Linux installed on FAT32(yes you can) Suxs on fragmentation about as much as Windows. There are little design problems in the way windows handles files. Nothing not fixable in Ros while keeping support for Windows.
It can take years at times to get Linux filesystem above 5 percent fragmented. Now you have seen the first reason. Fragmentation resistance in filesystems. Lot of people under linux reach the defrag point when the hardware needs replacing. So the idea of no need for defragmenting comes from the time to need to defrag comes around.
Windows does have a common cause of fragmentation called restore snapshots and altering swapfile sizes. Reactos should support swap partitions outbox. There is a important difference on fragmentation by using them. Spliting of tmp to its own partition can also have positive effects on fragmentation. Theses are little tricks Linux and Unix have been using for a long time due to great positive effects. MS did have support for swap partitions back in Windows 2000. But it default option still was swapfile.
Note HDD filled no room for play nothing can help you much then.
Please note it is MS fault on that run as Admin in the docs of the Creator of NT design Admin was never ment to be created alone.(Hello Windows NT 3.5 default user Admin level no requirement to create the extra user) User and Admin account should have always be created. Note MS decide that was too user unfriendly and removed it.
Now they have 1000s of programs that need it. Whos fault is it. Microsoft's. Buck stops with the person who decide to disobey the Creator advice.
Support for fragmentation resistant file systems have to be up there on the wish list.
Thanks Igor, that was interesting.
I also read ReiserFS is 'self defragmenting', but only when you don't use the -noatime and -notail options. Probably fragmentation is mainly a problem of FAT partitions, and I consider them antique. Nonetheless, they're still used on media cards, USB sticks etc. and until recently it was the best way to share HD-partitions between UNIX/Windows I think.
I also read ReiserFS is 'self defragmenting', but only when you don't use the -noatime and -notail options. Probably fragmentation is mainly a problem of FAT partitions, and I consider them antique. Nonetheless, they're still used on media cards, USB sticks etc. and until recently it was the best way to share HD-partitions between UNIX/Windows I think.
Quoting:they're still used on media cards, USB sticks etc.Where fragmentation presents near-zero performance impact. There is no head seek time. Only read-ahead optimizations result in a throughput hit, when non-contiguous reads can no longer be grouped together in a single request.
Quoting:they're still used on media cards, USB sticks etc.
You *want* fragmentation on those devices. As much as possible. In fact, the flash driver does just that underneath the filesystem driver. Fragment everything and scatter it all over the flash card. The reason is that Flash has only limited write capability. A sector wears out after 10.000-100.000 writes or so. So you need to spread it out and wear down the flash card evenly to prolong it's lifetime.
gus3: You're right. I should have said fragmentation used to be a problem on FAT filesystems on old HD's.
Quoting:Fragment everything and scatter it all over the flash card.Does this happen on the driver level, or does circuitry in the card take care of it? My (very shallow) understanding is the latter.
I'm not sure. The only thing I'm sure of is that it happens below the filesystem level.
Edit: Correction, it's both. It should be implemented at the filesystem level, but most removable flash devices (like USB sticks) have a controller chip on board that does wear-levelling so that a regular FAT32 filesystem can be used.
http://en.wikipedia.org/wiki/Flash_memory#Flash_file_systems
Edit: Correction, it's both. It should be implemented at the filesystem level, but most removable flash devices (like USB sticks) have a controller chip on board that does wear-levelling so that a regular FAT32 filesystem can be used.
http://en.wikipedia.org/wiki/Flash_memory#Flash_file_systems
> Wow, Bob, somebody gave you an utterly BS response.
Well, from the rest of things, it seems that the only bs was the "heads traveling over the full disk is a good thing".
I also ran a _lot_ of mainframe defrags, that and swapping tapes being the great labor on graveyard shift. Thus my concern from when I first put together a Linux system.
I'm sure the conversation is still in the Debian-User mailing list somewhere, I may look it up. My recollection may have slipped a bit over time.
{edit: not easily found through Google, bummer.}
All that said, I would say that the kit gloves with which defraging had to be handled in mainframe world (big sign: Never Kill A Defrag!) and the obvious benefits of a fragment-resistant file system, I don't think I'm going to be worrying about it any time soon on my machine.
I also ran a _lot_ of mainframe defrags, that and swapping tapes being the great labor on graveyard shift. Thus my concern from when I first put together a Linux system.
I'm sure the conversation is still in the Debian-User mailing list somewhere, I may look it up. My recollection may have slipped a bit over time.
{edit: not easily found through Google, bummer.}
All that said, I would say that the kit gloves with which defraging had to be handled in mainframe world (big sign: Never Kill A Defrag!) and the obvious benefits of a fragment-resistant file system, I don't think I'm going to be worrying about it any time soon on my machine.
>Well, from the rest of things, it seems that the only bs was the "heads traveling over the full disk is a good thing".
Sorry, that's what I was referring to.
And yes, the Linux filesystems are far more fragmentation-resistant that FAT32, although they can still fragment over time, and, especially, as disks approach capacity.
For example, my /usr directory structure is about 15% fragmented with 1.5 fragments per file. May not be too bad, depending on which files are fragmented how badly. Still, I hear the disk much more now than I did three years ago. Methinks some kind of action is in order.
Sorry, that's what I was referring to.
And yes, the Linux filesystems are far more fragmentation-resistant that FAT32, although they can still fragment over time, and, especially, as disks approach capacity.
For example, my /usr directory structure is about 15% fragmented with 1.5 fragments per file. May not be too bad, depending on which files are fragmented how badly. Still, I hear the disk much more now than I did three years ago. Methinks some kind of action is in order.
I ran the script too, 7.4% with 1.5 frags/file.
I am now wondering if ext3 journaling is going to cause problems with a defrag. It might need to be run in minimalist single-user mode, like fsck.
If you do, I'm interested in the tools/result. Please post.
I am now wondering if ext3 journaling is going to cause problems with a defrag. It might need to be run in minimalist single-user mode, like fsck.
If you do, I'm interested in the tools/result. Please post.
Darn, it says the same thing (mostly) that was said back in 95 and ext2.
http://en.wikipedia.org/wiki/Ext3#Defragmentation
I guess I should have noticed the article wasn't talking about ext3 and kept out of it... :^)
http://en.wikipedia.org/wiki/Ext3#Defragmentation
I guess I should have noticed the article wasn't talking about ext3 and kept out of it... :^)
It appears to me that "fragmentation" in the context of NTFS (and other extent-allocation filesystems, like ODS-11, and SGI's old EFS) has a different meaning than in block-allocation filesystems like BSD FFS, and modern derivatives, like EFS2, UFS, etc. A third meaning might come up in the context of log-structured filesytems (NetBSD's LFS).
NTFS keeps track of the actual disk blocks that belong to a file by "extents": a beginning block and a run length. The old VMS filesystem ODS-2 did the same, which isn't too surprising given that NTFS has a lot in common at the disk layout level with ODS-2. Wink wink, nudge nudge.
The BSD Fast File System (FFS) keeps track of individual block addresses that belong to a file. Small pieces of files could be kept in special "fragments", which were disk blocks that had pieces of up to 4 or 8 files in them.
SGI's EFS, which to me looked like FFS with extents instead of individual block numbers in it, had the same "fragmentation" as NTFS. Except SGI gave everyone a defragementer out of the box, and had a crontab entry to run it once a week or so.
So, a "fragmented" NTFS filesystem does not allow the allocation of large extents. A file header (inode) for a file will have a lot of 1 or 2 block "extents" associated with a file.
A "fragmented" BSD FFS filesystem might have a lot of small files kept in the specially-designated "fragment" disk blocks. Of course, it could also have the same kind of fragmentation that NTFS exhibits, where the disk blocks allocated to a file might not be in any kind of sequential order. But the on-disk layout of an FFS filesystem almost guaranteed this, with it's "cylinder groups" and multiple pre-allocated inodes.
There's huge semantic confusion around this issue: ensure that you speak of a common "fragmentation", and avoid BSing about non-equivalents.
NTFS keeps track of the actual disk blocks that belong to a file by "extents": a beginning block and a run length. The old VMS filesystem ODS-2 did the same, which isn't too surprising given that NTFS has a lot in common at the disk layout level with ODS-2. Wink wink, nudge nudge.
The BSD Fast File System (FFS) keeps track of individual block addresses that belong to a file. Small pieces of files could be kept in special "fragments", which were disk blocks that had pieces of up to 4 or 8 files in them.
SGI's EFS, which to me looked like FFS with extents instead of individual block numbers in it, had the same "fragmentation" as NTFS. Except SGI gave everyone a defragementer out of the box, and had a crontab entry to run it once a week or so.
So, a "fragmented" NTFS filesystem does not allow the allocation of large extents. A file header (inode) for a file will have a lot of 1 or 2 block "extents" associated with a file.
A "fragmented" BSD FFS filesystem might have a lot of small files kept in the specially-designated "fragment" disk blocks. Of course, it could also have the same kind of fragmentation that NTFS exhibits, where the disk blocks allocated to a file might not be in any kind of sequential order. But the on-disk layout of an FFS filesystem almost guaranteed this, with it's "cylinder groups" and multiple pre-allocated inodes.
There's huge semantic confusion around this issue: ensure that you speak of a common "fragmentation", and avoid BSing about non-equivalents.
A quick Google search reveals that there is a defrag utility for ext2 and some other file systems. The latest version listed appears to be defrag-0.73. I have no idea how well ti would work with ext3. It appears to be included in Debian.
One thing I would add is that while Linux may start fragmenting as the disk becomes full, it will still boot. I once filled up my hard drive and ran out of user disk space. The machine still booted and ran normally. If you get close to filling up a Windows system disk, you'll usually find that it will take 20 minutes or more for Windows to boot up, if it will at all.
One thing I would add is that while Linux may start fragmenting as the disk becomes full, it will still boot. I once filled up my hard drive and ran out of user disk space. The machine still booted and ran normally. If you get close to filling up a Windows system disk, you'll usually find that it will take 20 minutes or more for Windows to boot up, if it will at all.
> One thing I would add is that while Linux may start fragmenting as the disk becomes full, it will still boot.
Part of this is a standard 5%of the disk that is "reserved". So while it might reach the point of being full, it's never completely out of available space for the kernel and system processes.
A Very Good Idea(tm, reg us pat off).
Although, just a thought, with the bootable CD it becomes less of an issue. Rescue is possible through multiple vectors now. What's the best rescue disk for MS Windows? A Linux liveCD.
Part of this is a standard 5%of the disk that is "reserved". So while it might reach the point of being full, it's never completely out of available space for the kernel and system processes.
A Very Good Idea(tm, reg us pat off).
Although, just a thought, with the bootable CD it becomes less of an issue. Rescue is possible through multiple vectors now. What's the best rescue disk for MS Windows? A Linux liveCD.
Another data point:
I ran the "frag.pl" script on a ReiserFS (V3) that has my home directory on it. I formatted that partition when I moved from Slackware v9 to 10.1. I think I had 10.0 on my test machine at the time, so when 10.1 came out I moved to it quite quickly. This particular machine is on 24x7, in my basement. So, two years of home directory usage, about.
bash-3.1# ./frag.pl /home 6.80270065714389% non contiguous files, 1.61829190643809 average fragments.
it's unclear to me what this "frag.pl" script measures, and a ReiserFS filesystem may produce gibberish results.
I ran the "frag.pl" script on a ReiserFS (V3) that has my home directory on it. I formatted that partition when I moved from Slackware v9 to 10.1. I think I had 10.0 on my test machine at the time, so when 10.1 came out I moved to it quite quickly. This particular machine is on 24x7, in my basement. So, two years of home directory usage, about.
bash-3.1# ./frag.pl /home 6.80270065714389% non contiguous files, 1.61829190643809 average fragments.
it's unclear to me what this "frag.pl" script measures, and a ReiserFS filesystem may produce gibberish results.
> What's the best rescue disk for MS Windows?
UBCD4WIN. Available at http://www.ubcd4win.com/
You need a valid license for XP, and in theory you can't be running that license at the same time you use the disk.
UBCD4WIN. Available at http://www.ubcd4win.com/
You need a valid license for XP, and in theory you can't be running that license at the same time you use the disk.
Sorry, another thought occurs to me:
IDE disks now support "LBA", logical block addressing. Do you really know if you have "fragmented" or "non-contiguous" files on a disk with LBA turned on?
This same dilemma presented itself in the early 90s for BSD FFS systems like SunOS 4.x, when SCSI disks, which have always had something like LBA, became prevalent. The FFS "cylinder groups" where always a bit of a gamble, since SCSI disks could, and did, lie to you about heads/platters/sector size.
Note that NTFS, which keeps *extents* in its moral-equivalent-of-inodes ends up suffering when it can't find numerically-adjacent logically-addressed blocks. The number of extents per file header/inode is limited, and if a file occupies enough small extents, NTFS has to allocate a new header/inode, right? Lots of traffic through the $MFT equals a bottleneck for NTFS performance.
IDE disks now support "LBA", logical block addressing. Do you really know if you have "fragmented" or "non-contiguous" files on a disk with LBA turned on?
This same dilemma presented itself in the early 90s for BSD FFS systems like SunOS 4.x, when SCSI disks, which have always had something like LBA, became prevalent. The FFS "cylinder groups" where always a bit of a gamble, since SCSI disks could, and did, lie to you about heads/platters/sector size.
Note that NTFS, which keeps *extents* in its moral-equivalent-of-inodes ends up suffering when it can't find numerically-adjacent logically-addressed blocks. The number of extents per file header/inode is limited, and if a file occupies enough small extents, NTFS has to allocate a new header/inode, right? Lots of traffic through the $MFT equals a bottleneck for NTFS performance.
So what?
Thank you for not making any unfounded claims based on your findings. You could have said something like "my system is so much snappier now!"
That's great, you've found that some of your files are fragmented and then you found a utility to defragment them. You average number of fragments per file went down.
"As the results show, it's worth the effort to try Con Kolivas defragmentation script." Sure, that's true if you care about the average number of fragments per file. However, you did not explain why I should care about a metric like that.
That's great, you've found that some of your files are fragmented and then you found a utility to defragment them. You average number of fragments per file went down.
"As the results show, it's worth the effort to try Con Kolivas defragmentation script." Sure, that's true if you care about the average number of fragments per file. However, you did not explain why I should care about a metric like that.
Quoting:However, you did not explain why I should care about a metric like that.
It's almost impossible to benchmark in my opinion, though I _think_ I felt the difference back when I used WinXP. It's merely rest of mind you're buying; from the factors that could slow down your experience, at least file fragmentation isn't one of them anymore.
While your article is interesting in that it introduced me to the 'filefrag' util, (used
by the first script) and Colivas' script, I have to agree with chalex that you really
do not conclude with anything useful. Fragmentation isnt anything inheriently
bad, it just means that a file is not 100% contiguous on disk. Your own results
indicate that only few large files were mostly susceptible to fragmentation ,
which makes sense and shouldnt present much of a performance problem.
Fragmentation presented real problems on crufty old filesystems, like FAT based,
when after a while, it was hard to create a file that wasnt broken into chunks
all over the place. (fragmentation hurts you in two ways; it might waste your
read-ahead, and in seeking ) [It also can impact file recovery after a disaster,
but thats not performance]
Basicly, your conclusion is that 'it makes me feel better', which is fine, but dont pretend its anymore than that. And dont try to draw inferences from your old use of another operating system with an unspecified filesystem. Fragmentation was a problem that was solved long ago, and if you actually found that it was a signifigant problem with a filesystem, I would say that filesystem is probably not very good. At least you do provide the tools to allow someone to investigate further.
As far as benchmarking, a read through-put test wouldnt be too hard, just 'time find /foo -type f -exec cat {} ; > /dev/null' (find a way to invalidate the page cache-- excercise for the student) defrag and repeat the test.
Basicly, your conclusion is that 'it makes me feel better', which is fine, but dont pretend its anymore than that. And dont try to draw inferences from your old use of another operating system with an unspecified filesystem. Fragmentation was a problem that was solved long ago, and if you actually found that it was a signifigant problem with a filesystem, I would say that filesystem is probably not very good. At least you do provide the tools to allow someone to investigate further.
As far as benchmarking, a read through-put test wouldnt be too hard, just 'time find /foo -type f -exec cat {} ; > /dev/null' (find a way to invalidate the page cache-- excercise for the student) defrag and repeat the test.
Quoting:Basicly, your conclusion is that 'it makes me feel better', which is fine, but dont pretend its anymore than that.
That's true. The most important point for me was that the partitions that probably are important for system performance, /usr and /var, were not fragmented, like I assumed. I assumed so because a Slackware user pointed out Gentoo made his system slow 'because of filesystem fragmentation', and I guess that's no issue on my /usr and /var partitions. The rest of the article is 'but what if these results were worse, and I would like to defragment?" Maybe that wasn't clear enough. If files on my home partition - like my music files are a bit fragmented, that would probably not present a problem; let alone a Linux-iso which is uploaded through bittorrent.
It's a bid sad I can't un-defragment to do the benchmarking you suggest on the system I tested, but I might try it on my other PC. Probably I should test how much time it takes for VMWare to start WinXP from a fragmented and a defragmented image, but I'm afraid I can't give quantitative and representative results. Moreover, I'm no hero in the technical details as you might have understood by now. It's a bit the same like the RDSL scheduler maybe (also by Con Kolivas), where people _felt_ it made their system snappier, but Con was unable to present thorough benchmarks in fact showing / proving the improvements the people felt to Linus. That's the reason I didn't go into that and didn't make conclusions; it's really hard to do a proper benchmark and probably my feelings about the improvement in prefromance aren't a good measure either. Nonetheless, as indicated, I might try the read through-put test.
Quoting:a Slackware user pointed out Gentoo made his system slow 'because of filesystem fragmentation', and I guess that's no issue on my /usr and /var partitions.
More likely it was slow because there were lots and lots of small files scattered throughout the disk. That, I would expect, should be the result of multiple compile jobs (a "typical Gentoo workload" ;) ) - there are no (or not many) large files involved in that. It depends on the filesystem, but the typical block size (or "allocation unit" in Windows-speak) is 1kB or 4kB - you won't find many source files larger than that :) So file fragmentation should not be the issue.
Quoting:a Slackware user pointed out Gentoo made his system slow 'because of filesystem fragmentation'That was probably me. I did state that, or something closely resembling it, several months ago on these boards.
If I were to say the same right now, I would not have the courage of my conviction. I would hedge my bets by saying, "but I haven't moved Portage to its own filesystem yet, so take it with a grain of salt." However, I stand by the statement that my old 550 MHz P3 laptop has right now a snappier response running Slackware-current than my powerful 2.1GHz Athlon desktop had with the best-tuned Gentoo I could muster. And that's saying something.
Just a quick comment re: gus3. (as it touches on the lack of quantifiable measurements
and apples to oranges comparisions) Logicly, we would expect a 2.1g Athlon to perform
better than a 550m P3, but attempting to compare the two, running two completely
different distribution with possibly different kernels/filesystems/video cards/etc
doesnt really mean much. And when the metric is 'snappier', it gets even fuzzier
because even if you were comparing apples to apples (testing on the same hardware)
kernel configuration and version could make a tremendous difference.
So, I hope that 'And thats saying something.' means that further investigation is warrented, rather than case closed.
So, I hope that 'And thats saying something.' means that further investigation is warrented, rather than case closed.
@set:
Take it as you will, but when I switched from Gentoo to Slackware, I made a step backwards in software versions, so some "innovations" were gone, which should have resulted in slowdown, not speedup. Add to that the default i486 optimizations for Slackware (rather than the Athlon optimizations I gave to Gentoo), and it does make one wonder what external circumstance could cause Gentoo's slowdown.
Given the Portage system, my #1 suspect is filesystem fragmentation.
I use the same distro on all my systems (perhaps different versions, but all Slackware right now). Yes, they have different hardware, but I seriously doubt that it would affect a subjective "snappiness" judgment in the manner I observed.
Take it as you will, but when I switched from Gentoo to Slackware, I made a step backwards in software versions, so some "innovations" were gone, which should have resulted in slowdown, not speedup. Add to that the default i486 optimizations for Slackware (rather than the Athlon optimizations I gave to Gentoo), and it does make one wonder what external circumstance could cause Gentoo's slowdown.
Given the Portage system, my #1 suspect is filesystem fragmentation.
I use the same distro on all my systems (perhaps different versions, but all Slackware right now). Yes, they have different hardware, but I seriously doubt that it would affect a subjective "snappiness" judgment in the manner I observed.
gus3: You're right, it was you who started my '(de)frag journey', thanks for that BTW, it was interesting.
If you say you find Slackware snappier than Gentoo, I believe you. However, it seems file fragmentation wasn't a problem / issue on my system, and of course, as a 'real Gentoo user', I was glad to find out, we wouldn't like Gentoo being responsable for file fragmentation would we? ;)
I have tried to understand this whole (de)frag issue by reading a lot of discussions (I believe I spent several hours trying to catch up) - amongst other things on Gentoo forums, an old Linux-gazette benchmark, and 'phsolides' remarks in the other thread beneath this article, but the more I read, the more difficult and complex it all becomes (like phsolide suggests). It seems because of LBA, if two data blocks are logically next to each other, it doesn't have to mean they are physically next to each other. I also made a great blunder by forgetting I use EVMS (mainly a frontend to LVM2), which maps my 'binary data' over several virtual layers to the physical hard disk. I'm glad I only reported fragmentation numbers and no performance improvements or so.
Looking at what other users say, I became more confused, especially when it comes to ReiserFS. A lot of comments made on the issue were contradictive.
Anyway, the most important issues I 'learnt':
Both ReiserFS v3 and ext3 don't have defragmentation programs. ReiserFS4 will have its own defragmentation program. However, I understood ext3 isn't that susceptible to file fragmentaiton. On the other hand, a lot of people say 'ReiserFS v3 fragments like hell'. Most of the people agree putting /var, tmp and /usr/portage on different partitions help prevent fragmentation. The tails in ReiseFS partly prevent fragmentation. XFS doesn't pre-allocate blocks like most filesystems do these days, but it allocates them when the harddisk is flushed (synced I believe). Hans Reiser believes this is the way to go, since it prevents fragmentation, and this mechanism will be used in ReiserFS4. From Dino I understood file fragmentation really was an issue 'back in the old UNIX days'.
The connection of file fragmentation and performance, like chalex started this topic with, is also a difficult subject, and I'm afraid there's not much I can say about it, since I don't have the knowledge for it.
Maybe it would be a good idea to find the (kernel-module) maintainers of several of those filesystems and ask them for some comments on this whole issue.
If you say you find Slackware snappier than Gentoo, I believe you. However, it seems file fragmentation wasn't a problem / issue on my system, and of course, as a 'real Gentoo user', I was glad to find out, we wouldn't like Gentoo being responsable for file fragmentation would we? ;)
I have tried to understand this whole (de)frag issue by reading a lot of discussions (I believe I spent several hours trying to catch up) - amongst other things on Gentoo forums, an old Linux-gazette benchmark, and 'phsolides' remarks in the other thread beneath this article, but the more I read, the more difficult and complex it all becomes (like phsolide suggests). It seems because of LBA, if two data blocks are logically next to each other, it doesn't have to mean they are physically next to each other. I also made a great blunder by forgetting I use EVMS (mainly a frontend to LVM2), which maps my 'binary data' over several virtual layers to the physical hard disk. I'm glad I only reported fragmentation numbers and no performance improvements or so.
Looking at what other users say, I became more confused, especially when it comes to ReiserFS. A lot of comments made on the issue were contradictive.
Anyway, the most important issues I 'learnt':
Both ReiserFS v3 and ext3 don't have defragmentation programs. ReiserFS4 will have its own defragmentation program. However, I understood ext3 isn't that susceptible to file fragmentaiton. On the other hand, a lot of people say 'ReiserFS v3 fragments like hell'. Most of the people agree putting /var, tmp and /usr/portage on different partitions help prevent fragmentation. The tails in ReiseFS partly prevent fragmentation. XFS doesn't pre-allocate blocks like most filesystems do these days, but it allocates them when the harddisk is flushed (synced I believe). Hans Reiser believes this is the way to go, since it prevents fragmentation, and this mechanism will be used in ReiserFS4. From Dino I understood file fragmentation really was an issue 'back in the old UNIX days'.
The connection of file fragmentation and performance, like chalex started this topic with, is also a difficult subject, and I'm afraid there's not much I can say about it, since I don't have the knowledge for it.
Maybe it would be a good idea to find the (kernel-module) maintainers of several of those filesystems and ask them for some comments on this whole issue.
XFS Defrag
After reading some information about XFS a couple of years ago, I decided to use that filesystem. From what I understand, it creates eight virtual-partitions in each real partition it's installed on. The user never sees the eight partitions, but it's one of the things that really helps it. However, because the disk is split eight ways, it works best for larger disk drives - probably ones over 100GB being better than under 60. These days, it's pretty much every drive you can buy.
XFS allocates space on each of its partitions for new files and generally tries to keep them pretty well equally full (hence the preference for larger drives.) This tends to balance things out. However, unlike the other file systems, there IS a "defrag" utility for XFS. It's called "xfs_fsr", and can be run as you run other programs, although I'd probably stay away from complex downloads, 3-D image compilations, or other disk-intensive programs if possible - why work against yourself?
If you're using OpenSuSE, you'll want to install it as part of the "xfsdump" package. Once it's installed, you open a super user console window and type "xfs_fsr /dev/hda1" to start. In a few minutes, it's done and you've defrag'd the drive. No need to do a "init 3" or anything else.
XFS allocates space on each of its partitions for new files and generally tries to keep them pretty well equally full (hence the preference for larger drives.) This tends to balance things out. However, unlike the other file systems, there IS a "defrag" utility for XFS. It's called "xfs_fsr", and can be run as you run other programs, although I'd probably stay away from complex downloads, 3-D image compilations, or other disk-intensive programs if possible - why work against yourself?
If you're using OpenSuSE, you'll want to install it as part of the "xfsdump" package. Once it's installed, you open a super user console window and type "xfs_fsr /dev/hda1" to start. In a few minutes, it's done and you've defrag'd the drive. No need to do a "init 3" or anything else.
Yikes!... The comment I made within the previous post with angle brackets seems to have disappeared. So I should say "replace '/dev/hda1' with the drive or partition that you wish to defrag." Sorry it didn't show in the original post.
Danger, Will Robinson, danger!
"xfs_fsr" cost me MANY files! I lost close to 50 ripped CD tracks alone, plus digital camera pics, documents, GNOME settings...
Files got mixed together, so that The Bangles' "Walk Like an Egyptian" suddenly turned into Simon and Garfunkel's "Cecilia," in the middle of the song. Beethoven's 9th symphony was unplayable.
(Yes, yes, backups. I know.)
AFAICT, xfs_fsr hasn't been updated in a while, and it is not included as a standard XFS utility with most Linux distros.
"xfs_fsr" cost me MANY files! I lost close to 50 ripped CD tracks alone, plus digital camera pics, documents, GNOME settings...
Files got mixed together, so that The Bangles' "Walk Like an Egyptian" suddenly turned into Simon and Garfunkel's "Cecilia," in the middle of the song. Beethoven's 9th symphony was unplayable.
(Yes, yes, backups. I know.)
AFAICT, xfs_fsr hasn't been updated in a while, and it is not included as a standard XFS utility with most Linux distros.
Post a comment
A free account is needed to post. Write your comment here first: your text is kept in this browser while you sign up or log in, and is waiting for you when you come back.
Your message:Markdown: **bold**, *italic*, [text](https://link), - lists, > quotes, $x^2$ for math. Formatting help
Sign up and post (it is free) or Log in and post
Linux is a multi-user, multi-tasking OS. Unlike Windows, it does NOT try to use the first open space on a disk when you write a file, it looks for an open _block_ of space big enough for the file first. So each time a file is copied, if it was fragmented, it may not be after a while.
Second, with a multi-tasking OS, having the r/w heads sweeping across the disk is a _good_ thing.
Anyway, I haven't made any defragmentations in years of desktop use and continual updates/upgrades (Debian, don't you know) and have seen no degradation in service or disk speed or "excess disk access".