Chapter 15. Concatenated Disk Device (CCD) configuration

Table of Contents

15.1. Install physical media
15.2. Configure Kernel Support
15.3. Disklabel each volume member of the CCD
15.4. Configure the CCD
15.5. Initialize the CCD device
15.6. Create a 4.2BSD/UFS filesystem on the new CCD device
15.7. Mount the filesystem

The CCD driver allows the user to concatenate several physical disks into one pseudo volume. While RAIDframe (see Chapter 16, NetBSD RAIDframe) also allows doing this to create RAID level 0 sets, it does not allow you to do striping across disks of different geometry, which is where CCD comes in handy. CCD also allows for an interleave to improve disk performance with a gained space loss. This example will not cover that feature.

The steps required to setup a CCD are as follows:

  1. Install physical media

  2. Configure kernel support

  3. Disklabel each volume member of the CCD

  4. Configure the CCD conf file

  5. Initialize the CCD device

  6. Create a filesystem on the new CCD device

  7. Mount the CCD filesystem

This example features a CCD setup on NetBSD/sparc 1.5. The CCD will reside on 4 SCSI disks in a generic external Sun disk pack chassis connected to the external 50 pin SCSI port.

15.1. Install physical media

This step is at your own discretion, depending on your platform and the hardware at your disposal.

From my DMESG:

Disk #1:
  probe(esp0:0:0): max sync rate 10.00MB/s
  sd0 at scsibus0 target 0 lun 0: <SEAGATE, ST32430N SUN2.1G, 0444> SCSI2 0/direct fixed
  sd0: 2049 MB, 3992 cyl, 9 head, 116 sec, 512 bytes/sect x 4197405 sectors

Disk #2
  probe(esp0:1:0): max sync rate 10.00MB/s
  sd1 at scsibus0 target 1 lun 0: <SEAGATE, ST32430N SUN2.1G, 0444> SCSI2 0/direct fixed
  sd1: 2049 MB, 3992 cyl, 9 head, 116 sec, 512 bytes/sect x 4197405 sectors

Disk #3
  probe(esp0:2:0): max sync rate 10.00MB/s
  sd2 at scsibus0 target 2 lun 0: <SEAGATE, ST11200N SUN1.05, 9500> SCSI2 0/direct fixed
  sd2: 1005 MB, 1872 cyl, 15 head, 73 sec, 512 bytes/sect x 2059140 sectors

Disk #4
  probe(esp0:3:0): max sync rate 10.00MB/s
  sd3 at scsibus0 target 3 lun 0: <SEAGATE, ST11200N SUN1.05, 8808 > SCSI2 0
  sd3: 1005 MB, 1872 cyl, 15 head, 73 sec, 512 bytes/sect x 2059140 sectors

15.2. Configure Kernel Support

The following kernel configuration directive is needed to provide CCD device support. It is enabled in the GENERIC kernel:

pseudo-device  ccd  4    # concatenated disk devices

In my kernel config, I also hard code SCSI ID associations to /dev device entries to prevent bad things from happening:

sd0     at scsibus0 target 0 lun ?
# SCSI disk drives
sd1     at scsibus0 target 1 lun ?
# SCSI disk drives
sd2     at scsibus0 target 2 lun ?
# SCSI disk drives
sd3     at scsibus0 target 3 lun ?
# SCSI disk drives
sd4     at scsibus0 target 4 lun ?
# SCSI disk drives
sd5     at scsibus0 target 5 lun ?
# SCSI disk drives
sd6     at scsibus0 target 6 lun ?
# SCSI disk drives

15.3. Disklabel each volume member of the CCD

Each member disk of the CCD will need a special file system established. In this example, I will need to disklabel:

/dev/rsd0c
/dev/rsd1c
/dev/rsd2c
/dev/rsd3c

Note

Always remember to disklabel the character device, not the block device, in /dev/r{s,w}d*

Note

On all platforms, the c slice is symbolic of the entire NetBSD partition and is reserved.

You will probably want to remove any pre-existing disklabels on the disks in the CCD. This can be accomplished in one of two ways with the dd(1) command:

# dd if=/dev/zero of=/dev/rsd0c bs=8k count=1
# dd if=/dev/zero of=/dev/rsd1c bs=8k count=1
# dd if=/dev/zero of=/dev/rsd2c bs=8k count=1
# dd if=/dev/zero of=/dev/rsd3c bs=8k count=1

If your port uses a MBR (Master Boot Record) to partition the disks so that the NetBSD partitions are only part of the overall disk, and other OSs like Windows or Linux use other parts, you can void the MBR and all partitions on disk by using the command:

# dd if=/dev/zero of=/dev/rsd0d bs=8k count=1
# dd if=/dev/zero of=/dev/rsd1d bs=8k count=1
# dd if=/dev/zero of=/dev/rsd2d bs=8k count=1
# dd if=/dev/zero of=/dev/rsd3d bs=8k count=1 

This will make all data on the entire disk inaccessible. Note that the entire disk is slice d on i386 (and some other ports), and c elsewhere (e.g. on sparc). See the kern.rawpartition sysctl - "3" means "d", "2" means "c".

The default disklabel for the disk will look similar to this:

# disklabel -r sd0
[...snip...]
bytes/sector: 512
sectors/track: 116
tracks/cylinder: 9
sectors/cylinder: 1044
cylinders: 3992
total sectors: 4197405
[..snip...]
3 partitions:
#        size   offset    fstype   [fsize bsize   cpg]
  c:  4197405       0     unused     1024  8192        # (Cyl.    0 - 4020*)

You will need to create one slice on the NetBSD partition of the disk that consumes the entire partition. The slice must begin at least one cylinder offset from the beginning of the disk/partition to provide space for the special CCD disklabel. The offset should be 1x sectors/cylinder (see following note). Therefore, the size value should be total sectors minus 1x sectors/cylinder. Edit your disklabel accordingly:

# disklabel -e sd0

Note

The offset of a slice of type ccd must be a multiple of the sectors/cylinder value.

Note

Be sure to export EDITOR=[path to your favorite editor] before editing the disklabels.

Note

The slice must be fstype ccd.

Because there will only be one slice on this partition, you can recycle the c slice (normally reserved for symbolic uses). Change your disklabel to the following:

3 partitions:
#        size   offset    fstype   [fsize bsize   cpg]
  c:  4196361     1044       ccd                       # (Cyl. 1 - 4020*)

Optionally you can setup a slice other than c to use, simply adjust accordingly below:

3 partitions:
#        size   offset    fstype   [fsize bsize   cpg]
  a:  4196361     1044       ccd                       # (Cyl. 1 - 4020*)
  c:  4197405       0     unused     1024  8192        # (Cyl. 0 - 4020*)

Be sure to write the label when you have completed. Disklabel will object to your disklabel and prompt you to re-edit if it does not pass its sanity checks.

15.4. Configure the CCD

Once all disks are properly labeled, you will need to generate a configuration file, /etc/ccd.conf. The file does not exist by default, and you will need to create a new one. The format is:

#ccd    ileave    flags   component    devices

Note

For the ileave, if a value of zero is used then the disks are concatenated, but if you use a value equal to the sectors/track number the disks are interleaved.

Example in this case:

# more /etc/ccd.conf
ccd0  0  none /dev/sd0c /dev/sd1c /dev/sd2c /dev/sd3c

Note

The CCD driver expects block device files as components. Be sure not to use character device files in the configuration.

15.5. Initialize the CCD device

Once you are confident that your CCD configuration is sane, you can initialize the device using the ccdconfig(8) command: Configure:

# ccdconfig -C -f /etc/ccd.conf

Unconfigure:

# ccdconfig -u -f /etc/ccd.conf

Initializing the CCD device will activate /dev entries: /dev/{,r}ccd#:

# ls -la  /dev/{,r}ccd0*
brw-r-----  1 root  operator   9, 0 Apr 28 21:35 /dev/ccd0a
brw-r-----  1 root  operator   9, 1 Apr 28 21:35 /dev/ccd0b
brw-r-----  1 root  operator   9, 2 May 12 00:10 /dev/ccd0c
brw-r-----  1 root  operator   9, 3 Apr 28 21:35 /dev/ccd0d
brw-r-----  1 root  operator   9, 4 Apr 28 21:35 /dev/ccd0e
brw-r-----  1 root  operator   9, 5 Apr 28 21:35 /dev/ccd0f
brw-r-----  1 root  operator   9, 6 Apr 28 21:35 /dev/ccd0g
brw-r-----  1 root  operator   9, 7 Apr 28 21:35 /dev/ccd0h
crw-r-----  1 root  operator  23, 0 Jun 12 20:40 /dev/rccd0a
crw-r-----  1 root  operator  23, 1 Apr 28 21:35 /dev/rccd0b
crw-r-----  1 root  operator  23, 2 Jun 12 20:58 /dev/rccd0c
crw-r-----  1 root  operator  23, 3 Apr 28 21:35 /dev/rccd0d
crw-r-----  1 root  operator  23, 4 Apr 28 21:35 /dev/rccd0e
crw-r-----  1 root  operator  23, 5 Apr 28 21:35 /dev/rccd0f
crw-r-----  1 root  operator  23, 6 Apr 28 21:35 /dev/rccd0g
crw-r-----  1 root  operator  23, 7 Apr 28 21:35 /dev/rccd0h

15.6. Create a 4.2BSD/UFS filesystem on the new CCD device

You may now disklabel the new virtual disk device associated with your CCD:

# disklabel -e ccd0

Once again, there will be only one slice, so you may either recycle the c slice or create a separate slice for use.

# disklabel -r ccd0
# /dev/rccd0c:
type: ccd
disk: ccd
label: default label
flags:
bytes/sector: 512
sectors/track: 2048
tracks/cylinder: 1
sectors/cylinder: 2048
cylinders: 6107
total sectors: 12508812
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0           # microseconds
track-to-track seek: 0  # microseconds
drivedata: 0
#        size   offset    fstype   [fsize bsize   cpg]
  c: 12508812        0    4.2BSD     1024  8192    16  # (Cyl. 0 - 6107*)

The filesystem will then need to be formatted:

# newfs /dev/rccd0c
Warning: 372 sector(s) in last cylinder unallocated
/dev/rccd0c:    12508812 sectors in 6108 cylinders of 1 tracks, 2048 sectors
        6107.8MB in 382 cyl groups (16 c/g, 16.00MB/g, 3968 i/g)

super-block backups (for fsck -b #) at:
[...]

15.7. Mount the filesystem

Once you have a created a file system on the CCD device, you can then mount the file system against a mount point on your system. Be sure to mount the slice labeled type ffs or 4.2BSD:

# mount /dev/ccd0c /mnt

Then:

# export BLOCKSIZE=1024; df
Filesystem  1K-blocks     Used   Avail Capacity  Mounted on
/dev/sd6a      376155   320290   37057    89%    /
/dev/ccd0c    6058800       1  5755859     0%    /mnt

Congratulations, you now have a working CCD. To configure the CCD device at boot time, set ccd=yes in /etc/rc.conf. You can adjust /etc/fstab to get the filesystem mounted at boot:

/dev/ccd0c  /home ffs     rw    1 2