Reformatting to non-standard NTFS cluster sizes?


Y

Yousuf Khan

Has anyone experimented with non-standard cluster sizes on NTFS
partitions? Typically, the default behaviour is that when creating the
file system, the format utility chooses the ideal cluster size based on
the size of the partition. But typically the highest it ever chooses is
4KB.

http://technet.microsoft.com/en-us/library/cc767961.aspx

But there are options for choosing upto 64KB, which never get chosen by
default. I have a file system that I calculated has a simple average
file size of 115MB! I was thinking for files that large, I should choose
the highest possible cluster size available.

Yousuf Khan
 
Ad

Advertisements

J

Jason

On Sun, 15 Apr 2012 10:47:48 -0400 "Yousuf Khan" <bbbl67
@spammenot.yahoo.com> wrote in article said:
Has anyone experimented with non-standard cluster sizes on NTFS
partitions? Typically, the default behaviour is that when creating the
file system, the format utility chooses the ideal cluster size based on
the size of the partition. But typically the highest it ever chooses is
4KB.

http://technet.microsoft.com/en-us/library/cc767961.aspx

But there are options for choosing upto 64KB, which never get chosen by
default. I have a file system that I calculated has a simple average
file size of 115MB! I was thinking for files that large, I should choose
the highest possible cluster size available.

Yousuf Khan
I have partitions with large numbers of enormous .wav files I've
recorded. I have experimented with cluster sizes all the way to 64k but
see very little, if any, change in performance. That surprises me. I also
have several external HDD's (USB- and eSATA-attached) for backup and have
done the same tests with the same result.
 
W

Wolf K

On Sun, 15 Apr 2012 10:47:48 -0400 "Yousuf Khan"<bbbl67


I have partitions with large numbers of enormous .wav files I've
recorded. I have experimented with cluster sizes all the way to 64k but
see very little, if any, change in performance. That surprises me. I also
have several external HDD's (USB- and eSATA-attached) for backup and have
done the same tests with the same result.
NTFS was originally called HPFS, for High Performance File System. It
was designed to be fast and reliable with low cluster sizes, so as to
reduce blank space. NTFS inherited that ability, so you will see small
differences in performance when you sue larger clusters. Whether those
small diffs are worth it is your call. IMO, the advantage of larger
cluster sizes has been overtaken by improved hardware performance.

HPFS was devised was back the days when MS was creating OS/2 for IBM
(and itself). After the split, MS renamed its version NT. Up to NT 3.5,
both OS/2 and NT could read and write each other's file systems. Up to
Win 2000 there was still a directory labelled OS/2 buried in the
Windows/System tree.

HTH,
Wolf K.
 
K

Ken Springer

NTFS was originally called HPFS, for High Performance File System. It
was designed to be fast and reliable with low cluster sizes, so as to
reduce blank space. NTFS inherited that ability, so you will see small
differences in performance when you sue larger clusters. Whether those
small diffs are worth it is your call. IMO, the advantage of larger
cluster sizes has been overtaken by improved hardware performance.

HPFS was devised was back the days when MS was creating OS/2 for IBM
(and itself). After the split, MS renamed its version NT. Up to NT 3.5,
both OS/2 and NT could read and write each other's file systems. Up to
Win 2000 there was still a directory labelled OS/2 buried in the
Windows/System tree.
My question would be, does it affect the amount of data that can be stored?

Logically, the smaller the cluster size, it would require more entries
on the disk to keep track of which cluster belonged to which file, and
the order of the clusters. Larger sized clusters would require fewer
entries.

If your files are relatively small, I wouldn't think there'd be much
difference. But, what if you have huge amounts of large and larger
video, graphic, and audio files?


--
Ken

Mac OS X 10.6.8
Firefox 11.0
Thunderbird 11.0.1
LibreOffice 3.5.1.2
 
G

GMAN

On Sun, 15 Apr 2012 10:47:48 -0400 "Yousuf Khan" <bbbl67


I have partitions with large numbers of enormous .wav files I've
recorded. I have experimented with cluster sizes all the way to 64k but
see very little, if any, change in performance. That surprises me. I also
have several external HDD's (USB- and eSATA-attached) for backup and have
done the same tests with the same result.
It really has nothing to do with differences in performance, and more to do
with if you choose the wrong cluster size for the files you intend to store,
but the more wasted space per file.
Youd want a smaller clsuetr size if you were storing thousands of small MIDI
files, and larger sized clusters for large backup files or DV video captures.
 
R

Rod Speed

Ken Springer said:
My question would be, does it affect the amount of data that can be
stored?

Logically, the smaller the cluster size, it would require more entries on
the disk to keep track of which cluster belonged to which file, and the
order of the clusters. Larger sized clusters would require fewer entries.

If your files are relatively small, I wouldn't think there'd be much
difference. But, what if you have huge amounts of large and larger video,
graphic, and audio files?
But those drives by definition arent likely to be where you are obsessed
about space use.
 
Ad

Advertisements

Y

Yousuf Khan

I have partitions with large numbers of enormous .wav files I've
recorded. I have experimented with cluster sizes all the way to 64k but
see very little, if any, change in performance. That surprises me. I also
have several external HDD's (USB- and eSATA-attached) for backup and have
done the same tests with the same result.
Well, I have gone ahead done the reformat to one my partitions, I'm now
using 64K clusters on that. I'm in the middle of some rearrangement of
data on my hard drives. So what I've been going around doing is copying
and moving entire file systems worth of data from one drive to another.
When I did the transfer between a default 4K cluster drive and a 64K
cluster drive, the speed of transfer is not hugely different. However,
there was a huge difference in responsiveness of the drives. I was
watching the transfers (which lasted a couple of hours each) on the
Windows 7 Resource Monitor, and you could see that the 4K drives were
absolutely pegged at 100% disk activity, with disk queues that were
hovering >=1.0. But the drive that was using 64K clusters had disk
activity of 40-50%, and disk queues well <1.0. This worked in both
directions, whether I was copying off of the 64K drive, or copying to
the 64K drive, it was the same way.

Yousuf Khan
 
R

Rod Speed

GMAN said:
It really has nothing to do with differences in performance, and more to
do
with if you choose the wrong cluster size for the files you intend to
store,
but the more wasted space per file.
Youd want a smaller clsuetr size if you were storing thousands of small
MIDI
files, and larger sized clusters for large backup files or DV video
captures.
But again, with that last situation, you are likely to be choosing between
2TB and
3TB drives base on $/GB and the small saving in free space isnt likely to
matter.
All that really does is see you be able to squeeze one more file on the
drive at most.
 
W

Wolf K

My question would be, does [cluster size] affect the amount of data that can be stored?
Yes. The smaller the cluster size, the less wasted space.

Suppose a file is 87 K. With 4K clusters, that will take 22 clusters ==>
88K space for 87 K data. With 64K clusters, you'll need 2 ==> 128K
space for 87K data.

Now suppose your documents average about 87K in size....

HTH,
Wolf K.
 
Y

Yousuf Khan

My question would be, does it affect the amount of data that can be stored?

Logically, the smaller the cluster size, it would require more entries
on the disk to keep track of which cluster belonged to which file, and
the order of the clusters. Larger sized clusters would require fewer
entries.
You're thinking of the FAT system which has a global bitmap of clusters
assigned to files and folders. The bigger the drive and the smaller the
cluster size, the bigger the bitmap gets. In NTFS, it's more of an
extent-based system, which basically contains a starting and ending
cluster address of all extents belonging to a file. The metadata on the
disk only grows with the number of files, but not necessarily with the
size of the file system. So the size of the clusters won't make a
difference to the size of the metadata, just the number of files you put
into it.
If your files are relatively small, I wouldn't think there'd be much
difference. But, what if you have huge amounts of large and larger
video, graphic, and audio files?
However, you will get more slack with small files in a large-cluster
filesystem. Even the smallest files will take up a minimum of 64K with a
filesystem with 64K clusters.

However, I've found that the responsiveness of a large-cluster-size
filesystem, is much greater when you also have large files.

Yousuf Khan

Yousuf Khan
 
B

Brian Gregory [UK]

Yousuf Khan said:
Has anyone experimented with non-standard cluster sizes on NTFS
partitions? Typically, the default behaviour is that when creating the
file system, the format utility chooses the ideal cluster size based on
the size of the partition. But typically the highest it ever chooses is
4KB.

http://technet.microsoft.com/en-us/library/cc767961.aspx

But there are options for choosing upto 64KB, which never get chosen by
default. I have a file system that I calculated has a simple average file
size of 115MB! I was thinking for files that large, I should choose the
highest possible cluster size available.
lately I've always formatted my large external USB2 disks as NTFS with 32K
clusters.

It works fine, but I don't know if it really provides any benefit.
 
Ad

Advertisements

Y

Yousuf Khan

lately I've always formatted my large external USB2 disks as NTFS with 32K
clusters.

It works fine, but I don't know if it really provides any benefit.
But there must've been a reason you chose that size? Do you remember why?

Yousuf Khan
 
A

Arno

In comp.sys.ibm.pc.hardware.storage Yousuf Khan said:
Has anyone experimented with non-standard cluster sizes on NTFS
partitions? Typically, the default behaviour is that when creating the
file system, the format utility chooses the ideal cluster size based on
the size of the partition. But typically the highest it ever chooses is
4KB.

But there are options for choosing upto 64KB, which never get chosen by
default. I have a file system that I calculated has a simple average
file size of 115MB! I was thinking for files that large, I should choose
the highest possible cluster size available.
Yousuf Khan
First find out whether it makes a difference. I expect it does not,
as most of these clusters will be placed on disk consecutively
anyways. Usually, larger clusters are just to reduce administration
information (space gain) or to make large filesystems possible (such
as for FAT). My guess would be it causes more trouble than it is
worth.

Arno
 
Y

Yousuf Khan

First find out whether it makes a difference. I expect it does not,
as most of these clusters will be placed on disk consecutively
anyways. Usually, larger clusters are just to reduce administration
information (space gain) or to make large filesystems possible (such
as for FAT). My guess would be it causes more trouble than it is
worth.
So far, it's been working pretty well, it seems to reduce disk activity
immensely. I've got several disks that have nothing but huge files on
them, I'm thinking of converting them all over now, but that would
involve having other places to store the files that they got on them now
until the reformat can be done. I've converted about 3 file systems to
64K clusters already, but those were the smaller disks, which could be
temporarily accommodated on larger disks, but now the larger disks are
the ones that require it.

Yousuf Khan
 
C

Char Jackson

So far, it's been working pretty well, it seems to reduce disk activity
immensely. I've got several disks that have nothing but huge files on
them, I'm thinking of converting them all over now, but that would
involve having other places to store the files that they got on them now
until the reformat can be done. I've converted about 3 file systems to
64K clusters already, but those were the smaller disks, which could be
temporarily accommodated on larger disks, but now the larger disks are
the ones that require it.
Acronis Disk Director, and I presume other disk partitioning tools,
will allow you to change the cluster size nondestructively, so there
would be no need to move the data off of the drive. I'll leave the
risk/reward analysis up to the reader.
 
Y

Yousuf Khan

Acronis Disk Director, and I presume other disk partitioning tools,
will allow you to change the cluster size nondestructively, so there
would be no need to move the data off of the drive. I'll leave the
risk/reward analysis up to the reader.
Is it free?

Yousuf Khan
 
Ad

Advertisements

C

Char Jackson

Is it free?
Acronis isn't free, but since it can do this task it seems a good bet
that other similar programs can also do it, and some of them may be
free.
 
B

Brian Gregory [UK]

Yousuf Khan said:
But there must've been a reason you chose that size? Do you remember why?

Yousuf Khan
I think I just chose the same size as the allocation blocks were on some
FAT32 drives I had.
 
Ad

Advertisements

Y

Yousuf Khan

I think I just chose the same size as the allocation blocks were on some
FAT32 drives I had.
Oh-ok, it was just chosen based on precedent and tradition: "In my day,
we _always_ chose 32K clusters". :)

Yousuf Khan
 
Ad

Advertisements


Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top