Appendix: Sizing the Metadata Disk

 

Sun Support recommends that each metadata disk included in a file system hold no more than 750 Mbytes of metadata. If you are using one or more conventional disk devices as metadata disks, the disk itself will probably be much larger than that. In order to reduce the amount of metadata on each individual disk, metadata should be striped across all disks, using any stripe width equal to or greater than one. That way, metadata can be distributed across disk devices so no one disk exceeds 750 Mbytes in size. If the file system includes many files that are extremely active, the amount of metadata allocated to one disk may be reduced to as little as 350 Mbytes.

 

Metadata may also be round-robined over the metadata devices. Because the only significant file on the metadata disks is the .inodes file, round-robining data will effectively cause the metadata disks to behave as a concatenation. Reads and writes to inodes will always go to the device where the .inodes file is located. Consequently, that metadata device may end up being accessed constantly while other disks are barely used.  Sun Support therefore recommends round-robining metadata only on solid-state disks. These provide superior performance, although their prices are still sometimes too high for large file systems. Round-robining allows small solid-state disks of perhaps 500 Mbytes to essentially be filled before the next disk is used.

 

Because of the cost of solid-state disks, you may want to size such disks fairly closely to your anticipated need for metadata storage. 

 

The following must be considered when sizing a metadata disk: (from Daniel Synott, Craig Flaskerud and Nancee Melby)

 

1)     The .inodes file: each file and directory in the file system takes 512 bytes. If you are using segments, each segment also requires an inode. The .inodes file has an initial size of about 500 Kbytes which must also be added. Space from freed inodes may be reused but is not reclaimed.

2)     The bitmap (reservation map) takes the following amount of space:

      (((metadata+file data space)/DAU)/32k)

3)     According to documentation, each directory takes 16 Kbytes of space on the metadata device, but in my experience, the Dual Allocation Scheme implemented on the metadata disk devices reduces the space used by a directory to 4 Kbytes. Add to that any additional space required by directories containing very large numbers of files and 64 Kbytes for the root directory.

4)     Two copies of the superblock take 32 Kbytes.

5)     The first 16 DAUs of a file are pointed to by direct pointers in the inode. If a file is smaller than 16 DAUs, then all pointers are held in the inode, regardless of the size of the DAU. If the file exceeds 16 DAUs, then one block of 16k is allocated for indirect pointers. It holds 2048 direct pointers, each of which points to one full DAU. If all direct pointers are used, this block points to a file of at least 32 Mbytes for the smallest possible DAU of 16k (16k * 2056) and at most 132 Gbytes for the largest possible DAU of 64M (64M * 2056). If the size of the file exceeds the space pointed to by single indirect pointers, one block of 16k is allocated for double indirect pointers. This block contains 2048 pointers to 16k blocks containing single indirect pointers. Using one double indirect pointer allows you to point to between 65 Gbytes (32 Mbytes * 2049) and 270 Tbytes of file data. Triple indirect pointers may also be allocated as necessary, following the same logic. Setting the DAU too small can waste metadata disk space by forcing the allocation of double and triple indirect pointers that would not have been needed if the DAU had been set to a larger value.

6)     If ACLs are required, some space must also be allocated for them.

7)     Space must also be allocated for slop needed to make up for lost space and random management files. When a file is deleted, for example, the space used by its inode in the .inodes file is not reclaimed, although it may be reused.