Print this Page

Home Server Linux Installation

Index.

30-09-2011: Fixed internal link.
03-06-2011: Initial release.

Introduction.

This article shows a way to install Arch Linux on our server with software RAID (Redundant Array of Independent Disks). We will use RAID level 1, mirroring. RAID 1 is the only RAID level that can be used on the boot partition, because bootloaders (which read the boot partition) do not understand RAID, but a RAID 1 partition can be read like a normal partition. RAID 1 doesn’t protect your data, it will only protect you against a failing hard disk. It is assumed that you have some experience installing Arch Linux. It is also a good plan to connect the new server to the Internet which can even be done without any router or firewall because Arch doesn’t open any ports until you do that.

In this article the hard disks are 640 GB each, a normal size in the year that I created this server.  There will be four partitions, two 10 GB partitions for two operating system versions, one swap and one large partition for data storage. While Arch doesn’t need a second partition to try OS upgraded on, I choose to have a spare OS partition just in case. The partition layout will look like this:

  • /dev/sda1 + /dev/sdb1 = /dev/md0 (10 GB active OS root).
  • /dev/sda2 + /dev/sdb2 = /dev/md1 (10 GB spare OS root).
  • /dev/sda3 + /dev/sdb3 = 2 swap partitions, about 1 GB each.
  • /dev/sda4 + /dev/sdb4 = /dev/md2 (LVM space for data partitions).

The  steps to do the installation are:

  1. Boot the installer CD.
  2. Partition the hard disks.
  3. Create the RAID partitions.
  4. Install and configure Arch.
  5. Install Grub on both harddisks.
  6. Finish install and reboot.

Boot the Installer CD.

Arch Installer boot.

Boot the Arch Installer CD. At the syslinux prompt, hit enter. When the login prompt appears, login as root but don’t start the installer yet because we will partition the harddisk drives first and build the RAID arrays.

 

 

 

 

 

 

Partition the hard disks.

Use fdisk to partition the first hard disk.

[root@archiso ~]# fdisk /dev/sda
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklab
el
Building a new DOS disklabel with disk identifier 0xda23fd11.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): c
DOS compatibility flag is not set

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-83546, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-83546, default 83546): +1300

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (1302-83546, default 1302):
Using default value 1302
Last cylinder, +cylinders or +size{K,M,G} (1306-83546, default 83546): +1300
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (2603-83546, default 2603):
Last cylinder, +cylinders or _size{K,M,G} (2603-83546, default 83546): +245

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 4
First cylinder (2849-83546, default 2849):
Last cylinder, +cylinders or +size{K,M,G} (2849-83546, default 83546): 83540

Command (m for help): t
Partition (1-4): 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): t
Partition (1-4): 2
Hex code (type L to list codes): fd
Changed system type of partition 2 to fd (Linux raid autodetect)

Command (m for help): t
Partition (1-4): 3
Hex code (type L to list codes): 82
Changed system type of partition 3 to 82 (Linux swap / Solaris)

Command (m for help): t
Partition (1-4): 4
Hex code (type L to list codes): fd
Changed system type of partition 4 to fd (Linux raid autodetect)

Command (m for help): a
Partition (1-4): 1

Command (m for help): p

Disk /dev/sda: 687.2 GB, 687194767360 bytes
255 heads, 63 sectors/track, 83546 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xda23fd11

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        1301    10450251   fd  Linux raid autodetect
/dev/sda2            1302        2602    10450282+  fd  Linux raid autodetect
/dev/sda3            2603        2848     1975995   82  Linux swap / Solaris
/dev/sda4            2849       83540   648158490   fd  Linux raid autodetect

Command (m for help): w
Calling ioctl() to re-read partition table.
Syncing disks.
[root@archiso ~]#

Note that I didn’t use the disk up to the last cylinder. If you ever need a replacement it is possible that the new drive doesn’t have the exact same amount of cylinders. But, most likely you end up buying a larger replacement disk. We have now created three raid partitions and one swap partition and made the first partition active. Copy this partition table to the second disk:

[root@archiso ~]# sfdisk -d /dev/sda | sfdisk /dev/sdb

A whole lot of messages will follow, but in the end it should tell you that all went well. If not, start over again.

 

Create the RAID partitions.

First we need to load the RAID modules, then we can create the RAID arrays. We set the metadata for the array to version 0.90 while the default is 1.2. Version 1.2 is only supported by grub2 and Arch still uses the old stable grub.

[root@archiso ~]# modprobe raid1
[root@archiso ~]# mdadm -C /dev/md0 -l 1 --raid-devices 2 --metadata=0.90 \
                  /dev/sda1 /dev/sdb1
mdadm: array /dev/md0 started.
[root@archiso ~]# mdadm -C /dev/md1 -l 1 --raid-devices 2 --metadata=0.90 \
                  /dev/sda2 /dev/sdb2
mdadm: array /dev/md1 started.
[root@archiso ~]# mdadm -C /dev/md2 -l 1 --raid-devices 2 --metadata=0.90 \
                  /dev/sda4 /dev/sdb4
mdadm: array /dev/md2 started.
[root@archiso ~]#

The swap partitions are not in a RAID array because there is no point in putting swap memory on RAID. Instead we will use two swap partitions in our server so that the swap will be balanced over the two disks. Now start the arch installation:

[root@archiso ~]# /arch/setup

Start the installation by stepping thru the MAIN MENU

  1. Select source (ftp/http).
  2. Set clock
  3. Prepare Hard Drive(s).

Now we must be careful because we already created the partitions. In the “Prepare Hard Drive(s)” menu, choose menu option 3, “Manually Configure block devices, filesystems and mountpoints”. Next choose the /dev method. In the next screen you will see the raid devices /dev/md0, /dev/md1 and /dev/md2. You also see /dev/sda3 and /dev/sdb3, your swap devices. First create a ext3 filesystem on /dev/md0 as the root filesystem. Next configure /dev/sda3 and /dev/sdb3 as swap. We leave everything else empty for now. Select DONE from this menu and ignore the warning about the missing /boot partition. The root filesystem and swap partition will now be created, continue with the installation.

  1. Select Packages
  2. Install Packages
  3. Configure System

With “Select Packages”, just do the base install now, other packages will be installed when we need them. This will prevent that we end up with a bloated system with software which will only take up harddisk and backup space. When it comes to “Configure System” there are some important things to set or your RAID server won’t boot!

  • In /etc/rc.conf set USELVM=”yes”
  • In /etc/mkinitcpio.conf set MODULES=”raid1″
  • In /etc/mkinitcpio.conf set HOOKS=”base udev autodetect pata scsi sata mdadm filesystems”
  • Make any other changes you think you need such as the hostname etc.

Now before you are select Done, but after you changed these files, press Alt-F2 and login to the second terminal and give the following command:

[root@archiso ~]# mdadm --examine --scan >> /mnt/etc/mdadm.conf

That will add the RAID configuration to /etc/mdadm.conf on your new system. Go back to the first terminal with Alt-F1. Now select Done and mkinitcpio will run again with the new parameters and RAID configuration. Next, install the bootloader. Answer the two questions about RAID with yes (the default is No), to finish the installation. You may now reboot.

 

First boot and settings.

Now that the basic system is installed some changes need to be made to finish the RAID installation. Open /etc/mdadm.conf with your favorite editor and set the email address where to send RAID alarm messages to. You won’t get any mail yet because there is no mailer installed right now, that’s a different story. Then, add @mdadm to the end of the  DAEMONS=() line in /etc/rc.conf. Check it by starting by hand now “/etc/rc.d/mdadm start“.

If at this point you still see a lot of disk activity, don’t worry, most likely the RAID arrays are still being build. You can check that with:

[root@myhost etc]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda4[0] sdb4[1]
 648158400 blocks [2/2] [UU]
 [========>............]  resync = 44.2% (286779456/648158400) finish=521.2min \
          speed=11553K/sec

md1 : active raid1 sda2[0] sdb2[1]
 10450176 blocks [2/2] [UU]

md0 : active raid1 sda1[0] sdb1[1]
 10449152 blocks [2/2] [UU]

unused devices: <none>
[root@myhost etc]#

It is safe to continue the installation on the md2 array, it will get in sync sometime.

 

RAID and a failed disk.

Here is the status of the system when disk 1 is dead, in other words /dev/sda is missing. Do take note that /dev/sdb now shows as /dev/sda but the underscore on the blocks line should give you a hint that it’s the first drive that is gone.

[root@myhost ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda4[1]
 648158400 blocks [2/1] [_U]

md1 : active raid1 sda2[1]
 10450176 blocks [2/1] [_U]

md0 : active raid1 sda1[1]
 10449152 blocks [2/1] [_U]

unused devices: <none>
[root@myhost ~]#

Now, get a new drive and place it in your server. But first move the old second disk to the place of the first disk or we will have boot failures. If you put the new drive at the first position the systems BIOS will be confused because there is no MBR on the new disk. In our example, we couldn’t get a 640 GB drive, but we got a 900 GB drive instead. With the old surviving drive as first drive, the new one as second drive, turn the system on and it should boot without problems. The next thing we should do is rebuild the RAID arrays on the new drive. We start by copying the partition tables.

[root@myhost ~]# sfdisk -d /dev/sda | sfdisk /dev/sdb
Checking that no-one is using this disk right now ...
OK

Disk /dev/sdb: 117487 cylinders, 255 heads, 63 sectors/track

sfdisk: ERROR: sector 0 does not have an msdos signature
 /dev/sdb: unrecognized partition table type
Old situation:
No partitions found
New situation:
Units = sectors of 512 bytes, counting from 0

 Device Boot    Start       End   #sectors  Id  System
/dev/sdb1   *      2048  20900564   20898517  fd  Linux raid autodetect
/dev/sdb2      20900565  41801129   20900565  fd  Linux raid autodetect
/dev/sdb3      41801130  45753119    3951990  82  Linux swap / Solaris
/dev/sdb4      45753120 1342070099 1296316980  fd  Linux raid autodetect
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
[root@myhost ~]#

Now we can rebuild the array again starting with the unused partition (just to be safe).

[root@myhost ~]# mdadm /dev/md1 -a /dev/sdb2
mdadm: added /dev/sdb2
[root@myhost ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda4[1]
 648158400 blocks [2/1] [_U]

md1 : active raid1 sdb2[2] sda2[1]
 10450176 blocks [2/1] [_U]
 [>....................]  recovery =  1.1% (116160/10450176) finish=4.4min \
                          speed=38720K/sec

md0 : active raid1 sda1[1]
 10449152 blocks [2/1] [_U]

unused devices: <none>
[root@myhost ~]#

Now that this is working, do all other arrays.

[root@myhost ~]# mdadm /dev/md0 -a /dev/sdb1
mdadm: added /dev/sdb1
[root@myhost ~]# mdadm /dev/md2 -a /dev/sdb4
mdadm: added /dev/sdb4
[root@myhost ~]#

Bringing your new drive up to date will take hours to complete.

About swap, your system booted with a swap error because the new drive didn’t have partitions when powered on. You only have swap from the first (old) disk. Issue the following commands to reinstall your swap:

[root@myhost ~]# swapoff -a
[root@myhost ~]# mkswap /dev/sdb3
Setting up swapspace version 1, size = 1975988 KiB
no label, UUID=ecdb000d-45c8-4b82-8324-916109eae751
[root@myhost ~]# swapon -a
[root@myhost ~]#

Now wait until the disks are in sync again. In the meantime you should reinstall grub on both disks:

[root@myhost ~]# grub
Probing devices to guess BIOS drives. This may take a long time.

GNU GRUB  version 0.97  (640K lower / 3072K upper memory)

 [ Minimal BASH-like line editing is supported.  For the first word, TAB
 lists possible command completions.  Anywhere else TAB lists the possible
 completions of a device/filename. ]

grub> device (hd0) /dev/sda

grub> root (hd0,0)
 Filesystem type is ext2fs, partition type 0xfd

grub> setup (hd0)
 Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/e2fs_stage1_5" exists... yes
 Running "embed /boot/grub/e2fs_stage1_5 (hd0)"...  16 sectors are embedded.
succeeded
 Running "install /boot/grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/boot/grub/stage2 \
          /boot/grub/menu.lst"... succeeded
Done.

grub> device (hd0) /dev/sdb

grub> root (hd0,0)
 Filesystem type is ext2fs, partition type 0xfd

grub> setup (hd0)
 Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/e2fs_stage1_5" exists... yes
 Running "embed /boot/grub/e2fs_stage1_5 (hd0)"...  16 sectors are embedded.
succeeded
 Running "install /boot/grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/boot/grub/stage2 \
          /boot/grub/menu.lst"... succeeded
Done.

grub> quit
[root@myhost ~]#

 

Harddisk SMART monitoring.

Install the package smartmontools like this:

[root@myhost ~]# pacman -S smartmontools
resolving dependencies...
looking for inter-conflicts...

Targets (1): smartmontools-5.40-1

Total Download Size:    0.30 MB
Total Installed Size:   1.04 MB

Proceed with installation? [Y/n]
:: Retrieving packages from extra...
 smartmontools-5.40...   311.5K  656.9K/s 00:00:00 [######################] 100%
(1/1) checking package integrity                   [######################] 100%
(1/1) checking for file conflicts                  [######################] 100%
(1/1) installing smartmontools                     [######################] 100%
[root@myhost ~]#

Then edit /etc/smartd.conf and place a # character before the uncommented DEVICESCAN. Then add two lines like this (assuming you have SATA drives):

/dev/sda -a -d sat -I 194 -W 4,52,57 -R 5 -m admin@wpl.uk
/dev/sdb -a -d sat -I 194 -W 4,52,57 -R 5 -m admin@wpl.uk

Then add @smartd at he end of the DEVICES=() line in /etc/rc.conf and start smart by hand: “/etc/rc.d/smartd start“. That should start without problems. Again, you cannot receive any mail yet because there is no mailserver at this stage.

Permanent link to this article: http://www.mbse.eu/linux/homeserver/baseinstall/linuxinstall/