How to replace a failed disk of a RAID 5 array with mdadm on Linux

This is easy, once you know how it's done :-) These instructions were made on Ubuntu but they apply to many Linux distributions.

First of all, physically install your new disk and partition it so that it has the same (or a similar) structure as the old one you are replacing.

Then, install mdadm if you haven't already:

sudo apt-get install mdadm

Optional but might be a good idea if you can: reboot to make sure the whole md (multi-disk) software stack is loaded.

Let's check the status of our disks - mdadm should have discovered at least *something* on boot:

evy@evy-server:~$ cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]

md0 : inactive sdb2[1](S) sdc2[2](S)

3858200832 blocks

So this looks like it created one RAID device (/dev/md0) that consists of /dev/sdb2 and /dev/sdc2. We are need at least 3 disks for our RAID 5 so let's add another one:

evy@evy-server:~$ sudo mdadm --manage /dev/md0 --add /dev/sda3

mdadm: cannot get array info for /dev/md0

Hmm, this doesn't work because the RAID array in /dev/md0 is not active.
Let's activate it then:

evy@evy-server:~$ sudo mdadm --assemble /dev/md0

mdadm: no devices found for /dev/md0

Hmm, mdadm is not aware of the physical disks that /dev/md0 consists of. That's strange, because that information is clearly visible in /proc/mdstat, remember ?

A look at the man page reveals the --scan option, that tells mdadm to scan /proc/mdstat (and others) for RAID device information.

Let's try again:

evy@evy-server:~$ sudo mdadm --assemble /dev/md0 --scan

mdadm: /dev/md0 assembled from 2 drives - not enough to start the array while not clean - consider --force.

Ok, so we knew that - the RAID 5 is missing a disk.
But we need to activate it anyhow, to be able to add the missing disk !

So we force it:

evy@evy-server:~$ sudo mdadm --assemble /dev/md0 --scan --force

mdadm: SET_ARRAY_INFO failed for /dev/md0: Device or resource busy

Hmm, something's keeping our RAID array busy...
Let's just restart that RAID volume to fix that:

evy@evy-server:~$ sudo mdadm --stop /dev/md0

mdadm: stopped /dev/md0

And try again:

evy@evy-server:~$ sudo mdadm --assemble /dev/md0 --scan --force

mdadm: /dev/md0 has been started with 2 drives (out of 3).

Success !
Let's see what happened just now: 

evy@evy-server:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sdb2[1] sdc2[2]
      3858200832 blocks level 5, 64k chunk, algorithm 2 [3/2] [_UU]

Aha, there is movement !
The RAID 5 array is online, and the [_UU] flags show us that one disk is missing.

Here's some more info that demonstrates this nicely:

evy@evy-server:~$ sudo mdadm --detail  /dev/md0
/dev/md0:
        Version : 00.90
  Creation Time : Mon Feb  7 18:27:14 2011
     Raid Level : raid5
     Array Size : 3858200832 (3679.47 GiB 3950.80 GB)
  Used Dev Size : 1929100416 (1839.73 GiB 1975.40 GB)
   Raid Devices : 3
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Apr 25 13:04:09 2011
          State : clean, degraded
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 90b591c9:2fd9536f:86e10ba8:fe4e4a37
         Events : 0.172

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       18        1      active sync   /dev/sdb2
       2       8       34        2      active sync   /dev/sdc2

So let's try to add that missing disk, which had been our intent all along.

evy@evy-server:~$ sudo mdadm --manage /dev/md0 --add /dev/sda3
mdadm: added /dev/sda3

And this works.
Check it:

evy@evy-server:~$ sudo mdadm --detail /dev/md0
/dev/md0:
Version : 00.90
Creation Time : Mon Feb 7 18:27:14 2011
Raid Level : raid5
Array Size : 3858200832 (3679.47 GiB 3950.80 GB)
Used Dev Size : 1929100416 (1839.73 GiB 1975.40 GB)
Raid Devices : 3
Total Devices : 3
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Wed Jun 29 23:39:54 2011
State : clean, degraded, recovering
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 64K

Rebuild Status : 0% complete

UUID : 90b591c9:2fd9536f:86e10ba8:fe4e4a37
Events : 0.176

Number Major Minor RaidDevice State
3 8 3 0 spare rebuilding /dev/sda3
1 8 18 1 active sync /dev/sdb2
2 8 34 2 active sync /dev/sdc2

The RAID has already started rebuilding!

We can watch the progress like this:

evy@evy-server:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sda3[3] sdb2[1] sdc2[2]
3858200832 blocks level 5, 64k chunk, algorithm 2 [3/2] [_UU]
[>....................] recovery = 0.0% (673180/1929100416) finish=429.6min speed=74797K/sec

Aaaaaaah, there's no greater time to go to bed than when your RAID 5 is recovering :-)

Reverse engineered Logitech C110 color changing driver

I reverse engineered the windows driver of my Logitech C110 keyboard and remade it for Linux (with libusb) so that we can now change the color of the led lights in any shade between red and blue.

Attached is a proof of concept that I keep running all the time. It slowly, gradually makes the backlight color change from blue to red and back.

This open source Linux driver outperforms the proprietary windows driver because it enables continuously changing the backlight color in addition to setting just one, fixed color.

You can find it here: http://code.google.com/p/logitech-keyboard-change-color/downloads/list

The most interesting line of code (MILOC) is this one:

int written = usb_control_msg(handle, CONTROLFLAGS, 0x00000009, 0x00000307, 0x00000000, buffer, 0x00000005, 5000);

where:  

• handle is a pointer to the (properly initialized) USB device 046d:c22b
• #define CONTROLFLAGS 0x21 hold the RequestTypeReservedBits (as per the USB specs)
  
• unsigned char buffer [] = {0x07, 0x00, 0x00, 0x00, 0xFF}; is sent to the keyboard where:
  
• buffer[1] determines the led color (between 0x00 = blue and 0xFF = red)
• buffer[4] determines the led intensity  (between 0x00 = off and 0xFF = maximal) 

I hope you enjoy this keyboard as much as I do, and I hope Logitech starts shipping more of their products with Linux compatibility !

Or at least throwing some documentation online so we don't have to reverse engineer these things all the time :-)

Enjoy !