US8209587B1 - System and method for eliminating zeroing of disk drives in RAID arrays - Google Patents

show arrangements of data blocks on disks according to RAID-4 and RAID-5, respectively;

shows a network environment that includes a storage server according to an embodiment of the present invention;

is a diagram showing architecture of the storage server shown in

according to an embodiment of the present invention;

is a diagram showing a storage operating system of the storage server of

according to an embodiment of the present invention;

is a flow diagram of a method for using disk drives to perform a write operation according to a subtraction method without zeroing disk drives prior to using them according to one embodiment of the present invention;

is a flow diagram of a method for using disk drives to perform a write operation according to a recalculation method without zeroing disk drives prior to using them according to one embodiment of the present invention;

is a flow diagram of a method for optimizing write operations using inventive techniques of the present invention according to one embodiment of the present invention;

is a flow diagram of a method for optimizing write operations using inventive techniques of the present invention according to another embodiment of the present invention;

is a flow diagram of a method for optimizing write operations using inventive techniques of the present invention according to yet another embodiment of the present invention;

is a flow diagram of a method for optimizing write operations using inventive techniques of the present invention and a modified XPWrite command according to one embodiment of the present invention;

is a flow diagram of a method performed by a module executed on a drive that stores parity for a stripe to execute a modified XPWrite command according to an embodiment of the present invention; and

is a block diagram of the components of a disk configured to perform a write operation using a modified XPWrite command according to one embodiment of the present invention.

Referring now to

, they show arrangements of data blocks on storage devices according to RAID-4 and RAID-5 respectively. In

, data sent to a storage server from a client(s) for storage as part of a write operation may first be divided up into fixed-size, e.g., four Kilo Byte, blocks (e.g. D0, D1, etc.), which are then formed into groups that are stored as physical data blocks in a “stripe” (e.g. Stripe I, Stripe II, etc.) spread across multiple disks in an array. Row parity, e.g. an exclusive-OR (XOR) of the data in the stripe, is computed and may be stored in a parity protection block on disk D. The location of the row parity depends on the type of protection scheme or protocol implemented.

shows a RAID-4 scheme in which the row parity, e.g. P(0-2), P(3-5), P(6-8), and P(9-11) are stored in a dedicated disk (disk D).

shows a RAID-5 scheme in which the row parity is distributed across disks in the array.

As noted, a mechanism for eliminating a process of zeroing disk drives prior to adding them to a RAID system can be implemented in a storage server, such as the one shown in

.

shows a network environment that includes

210, a

200, and a collection of

250.

200 operates on behalf of

210 to store and manage shared files in a set of mass storage devices, such as magnetic or optical storage based disks or tapes. As previously described,

200 can also be a device that provides

210 with a block-level access to stored data, rather than a file-level access, or a device which provides clients with both file-level access and block-level access.

200 in

is coupled locally to a set of

210 over

230, such as a local area network (LAN), a wide area network (WAN), metropolitan are network (MAN) or the Internet. The

230 may be embodied as an Ethernet network or a Fibre Channel (FC) network. Each

210 may communicate with the

200 over the

230 by exchanging discrete frames or packets of data according to pre-defined protocols, such as TCP/IP.

Each of the

210 may be, for example, a conventional personal computer (PC), workstation, or the like. The

200 receives various requests from the

210 and responds to these requests. The

200 may have a distributed architecture; for example, it may include a separate N- (“network”) blade and D- (disk) blade (not shown). In such an embodiment, the N-blade is used to communicate with

210, while the D-blade includes the file system functionality and is used to communicate with the

250. The N-blade and D-blade communicate with each other using an internal protocol. Alternatively, the

200 may have an integrated architecture, where the network and data components are all contained in a single box and can be tightly integrated. The

200 further may be coupled through a switching fabric to other similar storage servers (not shown) which have their own local disk subsystems. In this way, all of the

250 can form a single storage pool, to which any client of any of the file servers has access.

Storage of information on the

200 is preferably implemented as one or more storage volumes that comprise physical

250 defining an overall logical arrangement of disk space. The disk drives within a volume are typically organized as one or more RAID groups (such as a

202 shown in

). The physical disks of each RAID group include those disks configured to store striped data (D) and those disks configured to store parity (P) for the data, in accordance with an illustrative RAID-4 and RAID-5 level configurations. However, other RAID level configurations (e.g., RAID-DP) are also contemplated.

A disk drive within a RAID group has a reserved area that stores configuration information for that particular drive and a copy of the configuration information for the entire RAID group (in

, the reserved

204 is designated on each disk to store configuration information). The configuration information may include a type of a RAID group, a number of disk drives within a RAID group, the disk drive's position within the RAID group, as well as other information describing the RAID group and the disk drive. The configuration information can also be persisted to the memory (shown in

) of the

200.

is a diagram showing the architecture of the

200 according to an embodiment of the present invention. The

200 includes one or more processor(s) 321 and

322 coupled to a

323. The

323 shown in

may represent any one or more separate physical buses and/or point-to-point connections, connected by appropriate bridges, adapters and/or controllers. The

323, therefore, may include, for example, a system bus, a Peripheral Component Interconnect (PCI) bus, a Hyper Transport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (sometimes referred to as “Firewire”).

The processor(s) 321 are the central processing units (CPUs) of the

200 and, thus, control the overall operation of the

200. In certain embodiments, the processor(s) 321 accomplish this by executing software, such as that described in more detail herein stored in

322. A

321 may include one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic devices (PLDs), or the like, or a combination of such devices.

322 comprises storage locations that are addressable by the

321 and adapters for storing software program code and data structures associated with the present invention. The

321 and adapters may, in turn, comprise processing elements and/or logic circuitry configured to execute the software code and manipulate various data structures.

322 can be a random access memory (RAM), a read-only memory (ROM), a flash memory, or the like, or a combination of such devices.

322 stores, among other components, the

324 of the

200. It will be apparent to those skilled in the art that other processing and memory means, including various computer readable media, may be used for storing and executing program instructions pertaining to the invention described herein.

322 also stores configuration information (also referred to herein as “labels”) of the disk drives.

In one implementation,

322 also stores a mask of each drive. In one embodiment, a mask is written to a checksum block associated with a data block that stores parity for a drive. This mask is referred to herein as a parity mask and is computed as a result of a logical operation on the masks of each disk drive storing data blocks included in the parity for a stripe where new data blocks are written. Thus, the mask that is written in the checksum block associated with a data block that stores parity for the stripe can be used to determine which data blocks are included in the parity for a particular stripe. A mask for each disk drive can be created by shifting a one bit by the position of the drive in the array. In one embodiment, a mask is a bitmask. For example, drive 0 would be assigned a binary mask of “0001b”, while drive 1 would be assigned a binary mask of “0010b” (a one bit shifted left by one bit), drive 2 would be assigned the mask of “0100b” (a one bit shifted left by two bits). A parity mask, such as for example, a mask of “0101b” would indicate that the parity block includes the data for

0 and 2 (“0101b” is the result of logically OR'ing the mask of

0, “0001b”, and the mask of drive 2, “0100b”). During a write operation, a parity mask is updated and written to a checksum block, which is associated with the parity block.

A portion of the

322 may be further organized as a “buffer cache” 350 for storing certain data structures as well as data blocks retrieved from

250 and to be written to the

250. A data block stored in

350 is appended with a checksum block, which could store up to 64 bytes of data. As described herein, the checksum block stores a unique disk drive ID, which is inserted prior to the data block being written to the disk, to indicate that the data block was written after the disk was added to the existing array. In addition, the checksum block may store a checksum for the associated data block, which can be computed by combining all the bytes for a block (e.g., 4 Kb) with a series of arithmetic or logical operations.

Also connected to the processor(s) 321 through the

323 are a

326 and a

327. The

326 cooperates with the

324 executing on the

200 to access data from

250. The

326 comprises a plurality of ports having input/output (I/O) interface circuitry that couples to the disks 250 (shown in

) over an I/O interconnect arrangement, such as a conventional high-performance, fibre channel (FC) link topology.

The

327 comprises a plurality of ports adapted to couple the

200 to one or more clients 210 (shown in

) over point-to-point links, wide area networks, virtual private networks implemented over a public network (Internet) or a shared local area network. The

327 thus may comprise the mechanical, electrical and signaling circuitry needed to connect the node to the network.

shows an example of the

324 of the

200. In the illustrative embodiment, the

324 can be the NetApp® Data ONTAP® storage operating system available from Network Appliance Inc., of Sunnyvale, Calif., that implements a Write Anywhere File Layout (WAFL®) file system. However, it is expressly contemplated that any appropriate storage operating system may be employed for use in accordance with the inventive principles described herein. As used herein, the term “storage operating system” generally refers to the computer-executable code operable on a computer to perform a storage function that manages data access. The

324 can also be implemented as a microkernel, an application program operating over a general-purpose operating system, such as UNIX® or Windows NT®, or as a general-purpose storage operating system with configurable functionality, which is configured for storage applications as described herein.

To facilitate access to the

250, the

324 implements a

430. The

430 logically organizes the information as a hierarchical structure of named directories and files on the disks. In one embodiment, the

430 is a write-anywhere file system that “virtualizes” the storage space provided by the

250. Each file may be implemented as a set of data blocks configured to store information, such as data, whereas a directory may be implemented as a specially formatted file in which names and links to other files and directories are stored.

324 further includes a

432, a

433, a

436, and a

435.

The

432 implements one or more of various high-level network protocols, such as Network File System (NFS), Common Internet File System (CIFS), Hypertext Transfer Protocol (HTTP) and/or Transmission Control Protocol/Internet Protocol (TCP/IP) to decode incoming client requests or encode outgoing responses to the client request in the appropriate protocol. The

433 includes one or more drivers, which implement one or more lower-level protocols to communicate over the network, such as Ethernet. The

432 and the associated

433 allow the

200 to communicate over the communication system 230 (e.g., with clients 210), as shown in

.

RAID controller module 436 (also referred to herein as a “storage module”) manages storage devices, such as disks, in a RAID system comprising one or more RAID groups.

436 manages data storage and retrieval in response to access requests from

210, which may include requests to write data and/or to read data. In one embodiment,

436 can be a software module implemented on the

200. In an alternative embodiment,

436 can be a separate enclosure implemented as hardware with its own controller or a processor, separate from, though in communication, with those that execute

430.

436 issues internal I/O commands to

250, in response to client requests. These commands can be a command to write data at a physical block number at

250 or a command to read data from a physical block number at

250.

According to an embodiment of the present invention, when

436 receives a request to perform a write operation, the

436 is configured to insert (in memory), into a checksum block of the data block indicated in the request, a unique ID of a disk drive where the data block will be written. Storage adapters, such as for example, storage adapter 326 (shown in

), transfer the data block(s) and the unique disk drive ID to the disk.

436 includes a

436 a.

436 a is configured to compare the disk drive ID stored in memory (which was stored in the configuration area) and a disk drive ID from the checksum block. If the two match, it indicates that data were written to the read data block after the disk was added to the array. If the two do not match, the data block was written prior to the disk being added to the array or the data block has become free and is removed from the parity block.

435 allows

200 to communicate with the

250. The

435 implements a lower-level storage device access protocol, such as Fibre Channel (FC) protocol, Small Computer Systems Interface (SCSI) protocol, Serial ATA (SATA), or Serial Attached SCSI (SAS).

Referring now to

, it illustrates a flow diagram of the steps to perform a write operation while eliminating the need to zero disk drives prior to using the disk drives. A person of ordinary skill in the art would understand that the steps described below do not need to be executed in the described order except where expressly required.

Initially, when a disk drive is added to a RAID group,

436 creates a unique disk drive ID for that disk drive. As described herein, in one implementation, a unique disk drive ID can be created using a unique RAID group ID and a count of the number of disks already added to the RAID group. A person of ordinary skill in the art would understand that other mechanisms could be used to create a unique disk drive ID. The unique disk drive ID is different for each disk drive.

436 then stores the unique disk drive ID in the configuration area on the disk.

200 receives a write request from

210 to modify a data block(s) in a RAID array.

200 passes the request to the

430.

430 identifies a logical block number(s) at which new data blocks will be written and sends a request to

436 to write the data blocks to the disk at the logical block number. The request includes a pointer to

350 that stores data to be written, a disk drive number, and a block number on the disk where data is to be written.

436 determines whether the parity mask for a stripe is stored in

322. If the parity mask is not stored in memory,

436 determines whether to use a subtraction method or a recalculation method to perform a write operation without using the parity mask of the stripe. The determination of which method to use is made by determining the number of read requests required for each method. If the recalculation method would require fewer reads, then it is used. Otherwise, the subtraction method is used.

If the subtraction method is used,

436 reads, into

322, data blocks to which data will be overwritten as well as a parity block for that stripe (at step 520). At

530,

436 determines whether the parity block is valid. The parity block is valid if it was written after the RAID group was created and the parity mask stored in the checksum block is not “0” and thus includes at least one data block for that stripe. In one implementation, to determine whether the parity block was written after the RAID group was created,

436 a compares the unique disk drive ID stored in the checksum block associated with the parity block with the unique disk drive ID kept in memory for the disk that stores parity. If the two match, it indicates that the parity block was written after the disk was added to the array, and thus stores valid data. Currently, the parity mask is equal to the mask read from the drive that stores parity for a stripe (step 540). This mask will be used to determine which data blocks in a stripe are included in the parity block for the stripe where new data will be written.

At

550, data blocks from the disk drives which were read and which are included in the parity mask are XOR'ed with the parity block. To determine which disk drives are included in the parity mask for a stripe, the following algorithm can be used.

436 performs:

If at

530 it is determined that the parity block for the stripe where data blocks will be modified is not valid (i.e., the unique drive ID stored in the configuration area for the drive that stores parity for a stripe does not match to the corresponding data stored in the checksum block associated with the parity block), it indicates that the parity block was written prior to the disk drive being added to the existing array and no data blocks in that stripe are included in the parity block. If no data blocks in the stripe are included in the parity block, these data blocks are invalid and can be ignored for purposes of parity computation. At

592,

436 assigns a zero value to the parity mask for a stripe.

At step 594,

436 computes new parity without using old data blocks that are being overwritten since these data blocks are invalid. In one implementation, the new parity is computed by XOR'ing new data to compute a new parity block. Thus, the present invention eliminates the need to use invalid data blocks to compute parity even without zeroing the invalid data blocks.

If the parity block is valid, a logical operation is performed on the new data blocks, old data blocks that are being read which are included in the parity, and old parity within the stripe (

550 and 560).

Once new parity has been computed for the stripe, at

570,

436 updates a parity mask for a stripe. The parity mask for a stripe is a result of a logical operation of masks of disk drives storing data blocks included in a parity block for that stripe. If none of the data blocks in a stripe to which new data are written were included in the parity block for that stripe, then parity mask was equal to zero before the new write operation. In one implementation, the parity mask is updated by performing a logical OR operation on the masks of disk drives where new data will be written. The mask of each data drive being written is logically OR'ed to the parity mask to produce a new parity mask. The new parity mask is inserted into a checksum block associated with the computed parity block.

According to an embodiment of the present invention, the updated parity mask may be cached in

322 in

350. On a subsequent write request, the cached parity mask can be used to determine which data blocks are valid without reading the data blocks or a parity block. This embodiment is described herein in reference to

.

At

580,

436 inserts, into a checksum block associated with a data block that will be written to a disk, a unique disk drive ID of that disk.

326 transfers the data block along with the unique disk drive ID from

350 to the

250.

436 a later compares the unique disk drive ID with the unique disk drive ID, which was stored in the configuration area on the disk and is kept in memory, to determine whether the data block is valid.

At

590,

436 inserts, into a checksum block associated with the computed parity block, a unique disk drive ID of the drive that stores parity for a stripe where the parity block will be written along with the new parity mask for a stripe.

326 transfers the parity block along with the unique disk drive ID and the parity mask from

350 onto the

250.

Assume that

200 receives a request to modify a number of data blocks in a RAID array. If the number of data blocks in a stripe being overwritten plus one is greater than the number of data blocks in the stripe not being overwritten,

436 uses a recalculation method to perform a write operation. Briefly, according to this method, data blocks that are not being overwritten are read into memory and XOR'ed with an XOR of new data blocks. The result of the XOR operation is written to a parity block for that stripe.

Referring now to

, it illustrates a flow diagram of a method for using disk drives to perform a write operation according to a recalculation method without zeroing disk drives prior to using them according to one embodiment of the present invention.

At

610,

436 reads into

322 data blocks to which data are not to be written. At

620,

436 determines whether any data blocks to which data are not to be written are valid. In one implementation, to determine whether a data block is valid,

436 reads a unique disk drive ID in the checksum block associated with the data block.

436 a compares the unique disk drive ID in the checksum block with the unique disk drive ID which was stored in the configuration area for that disk drive and is kept in memory. If the two match, it indicates that the data block is included in the parity block for that stripe and the data block was written after the disk drive was added to the disk array.

Optionally,

436 may check whether the parity block includes a data block(s) to which new data are not to be written. To this end,

436 may perform the following steps:

At step 640, parity is updated for the stripe where data blocks are modified. To this end,

436 performs a logical XOR operation on new data blocks and the data blocks read into memory that are valid and thus were included in parity. A person of ordinary skill in the art would understand that other techniques that produce a similar result can be used to update the parity.

If at

620, in the stripe where data blocks are modified, all data blocks are invalid, it indicates that these data blocks were written prior to creating a RAID group or prior to adding the disks to the RAID group or these data blocks have become free (unallocated) and were removed from the parity block in the stripe. Therefore, these data blocks can be ignored for purposes of parity computation. At

680,

436 assigns a zero value to the parity mask for the stripe.

436 computes new parity without using invalid data blocks that will be modified (step 690). In one implementation,

436 computes parity by XOR'ing new data.

At

650,

436 updates the current mask of the parity mask. The current mask is a result of the logical operation performed on the mask of the disk drives that store data blocks included into the parity. If none of the data blocks are included in the parity block for the stripe where new data are written, the parity mask is assigned a zero value. In one implementation, to update the current mask, the current mask of the parity mask for a stripe is logically OR'ed with the masks of disk drives to which new data will be written. The result may be first written onto a checksum block associated with the parity block in

350.

At

660,

436 inserts, into a checksum block associated with a data block that will be written to a disk, a unique disk drive ID for that disk.

326 transfers the data block along with the unique disk drive ID from

350 to the

250. The unique disk drive ID will be later compared by

436 a with the unique disk drive ID stored in the configuration area for that disk and kept in memory

At

670,

436 inserts, a unique disk drive ID of the drive that stores parity for a stripe, into a checksum block associated with the computed parity block that will be written to the drive that stores parity for a stripe.

326 transfers the parity block along with the unique disk drive ID and the parity mask from

350 to the drive that stores

250.

Referring now to

, it illustrates a flow diagram of a method for optimizing write operations using inventive techniques of the present invention according to one embodiment of the present invention.

Initially, at

710, a determination is made whether the parity mask for the stripe is cached in

322. If the mask is not cached in

322, a determination is made whether the number of read operations that need to be performed using a subtraction method exceeds the number of read operations that need to be performed using a recalculation method (step 715). If so, the write operation is performed using a recalculation method as described in

. Alternatively, if the number of read operations needed to be performed using a recalculation method is equal to or exceeds the number of read operations needed to be performed using the subtraction method, the write operation is performed using a subtraction method, as described in

.

If the parity mask is cached in

322, the parity mask can be used to determine which data blocks are valid, e.g., included into a parity block in a stripe where new data will be written. This information will be used to eliminate the need to read invalid data blocks to perform a write operation.

At

720, a determination is made whether the number of read operations that would have to be performed using a recalculation method is less than the number of read operations that would have to be performed using a subtraction method. To this end, the cached parity mask for the stripe is used to determine data blocks from which drives are included in the parity for the stripe. In one implementation, the following steps can be performed by RAID controller module 436:

436 uses this information to determine a number of read operations that need to be performed by a subtraction method or a recalculation method to perform a write operation. It at

720, the number of read operations for a recalculation method is greater than or equal to the number of read operations for a subtraction method, then the write operation is performed using a subtraction method (as described in reference to

). Alternatively, the write operation is performed using a recalculation method.

To this end, at

730, data blocks that meet the following two requirements are read into memory 322: (1) data blocks that are included in a parity block for the stripe where new data are written; and (2) data blocks to which new data are not to be written. Data blocks that are not included in the parity block are not read since they are invalid and can be ignored for parity computation.

At

740, data blocks that were read are logically XOR'ed. At

750, new data blocks are logically XOR'ed in memory. To compute new parity, the result of the XOR operation is XOR'ed with the result of the logical operation at

740.

At

760, the parity mask for the stripe is updated. In one implementation, the mask is updated by being OR'ed with the masks of the drives where new data are written. The mask is written into a checksum block associated with the computed parity block.

At

770,

436 inserts, into a checksum block associated with the data block stored in

350 that will be written to a disk, a unique disk drive ID for that disk.

326 transfers the data block along with the unique disk drive ID from

350 onto the

250. The unique disk drive ID will be later compared by

436 a with the unique disk drive ID kept in memory that is stored in the configuration area on the disk.

At

780,

436 inserts, into a checksum block associated with the computed parity block a unique disk drive ID for the drive that stores parity for the stripe where the parity block will be written.

326 transfers the parity block along with the unique disk drive ID and the parity mask from

350 onto the

250.

Referring now to

, it illustrates a flow diagram of a method for optimizing write operations using inventive techniques of the present invention according to another embodiment of the present invention. According to this embodiment, the write operation is performed using a subtraction method since the number of read operations to be performed using the recalculation method is greater than the number of read operations to be performed using the subtraction method (step 782).

As described herein, when a parity mask for a stripe is cached in

322, it can be used to determine which data blocks are valid in a stripe where new data are written.

At

784,

436 reads into

322 data blocks that meet the following two criteria: (1) data blocks that are being overwritten; and (2) included in a parity block in the stripe where new data are written. Data blocks that are not included in the parity block do not need to be read and can be ignored for parity computation.

At

786, a parity block for the stripe where new data will be written is read into memory. Then data blocks that are read into the memory are logically XOR'ed, at

788. At

790 new data blocks are XOR'ed in memory. To compute new parity, the result of the XOR operation in

790 is XOR'ed with the result of the operation in

788.

At

792, the parity mask for the stripe is updated. In one implementation, the parity mask is OR'ed with the masks of the drives where new data will be written in the stripe. The mask is written into a checksum block associated with the computed parity block.

At

794,

436 inserts, into a checksum block associated with the data block that will be written to a disk drive, a unique disk drive ID for that disk.

326 transfers the data block along with the unique disk drive ID from

350 to the

250. The unique disk drive ID will be later compared by

436 a with the unique disk drive ID kept in memory that is stored in the configuration area on the disk.

At

796,

436 inserts, into a checksum block associated with the computed parity block a unique disk drive ID for the drive that stores parity for a stripe where the parity block will be written.

326 transfers the parity block along with the unique disk drive ID and the parity mask for the stripe from

350 onto the

250.

According to yet another optimization technique, a write operation can be performed by eliminating read operations if none of the data blocks being overwritten are valid (i.e., their unique disk drive ID does not compare to the unique disk drive ID which was stored in the configuration information for the respective disk drives) but some other data blocks in the stripe not being overwritten are valid. To implement this embodiment,

436 issues a command to a disk drive that stores parity. In one embodiment, the command is a modified XPWrite command. XPWrite command is provided by SCSI standard, described in SCSI Block Commands, second generation, published draft of Sep. 22, 2006 from the T10 Technical Committee (see www.t10.org). XPWrite command includes a logical block address (LBA) of the block where to write data as well as a number of blocks for writing the data. As will be described below, XPWrite command is modified so that data stored in the checksum block associated with the new data block is not XOR'ed with the parity block. Rather, it is written directly to the parity block. Although the present invention is described in the context of the modified XPWrite command, a person of ordinary skill in the art would understand that any command that produces a same result can be used.

Referring now to

, it illustrates a flow diagram of a method for optimizing write operations using a modified XPWrite command according to one embodiment of the present invention.

Initially, at

910, it was determined that the parity mask for the stripe is cached in

322. The parity mask is used to determine which data blocks are valid (e.g., included in the parity block for the stripe where new data are to be written). At

920, a determination is made whether data blocks that will be modified are not valid and some other data blocks in a stripe are valid. In one embodiment, the determination is made by performing a logical AND operation on the mask of each data drive and the parity mask for the stripe. If a result of the logical operation does not contain a logical “1”, it indicates that a data block is not included in the parity, and thus does not need to be read for purposes of parity computation. If any of the data blocks that are being overwritten in the stripe are valid, or all of the data blocks in the stripe not being overwritten are invalid, the write operation is performed according to the steps recited in

.

At

930, new data blocks are XOR'ed in

322.

436 inserts a unique disk drive ID of the drive that stores parity for the stripe.

436 inserts the parity mask for the stripe into the checksum block associated with the parity block. The parity mask is updated to include the masks of drives being written.

436 then issues a modified XPWrite command to the disk that stores parity (step 940).

436 writes, to the data drives, the new data blocks along with the checksum block, which includes the unique drive ID.

Referring now to

, it illustrates components residing on each

250 in a RAID group, such as

202 shown in

, to execute a modified XPWrite command issued by

436. Each

250 in a

202 includes a

1100, which implements a

1130. In response to receiving a modified XPWrite command,

1130 is configured to read data from the data block indicated in the command and to request new data from

200.

1130 also receives the XOR of new data blocks and computes new parity by XOR'ing the new data blocks with the data read in memory. Steps performed by

1130 will be described in more detail in reference to

. In a preferred embodiment,

1130 is implemented as firmware embedded into a hardware device, such as

1100. In another embodiment,

1130 can be implemented as software stored in the

1120. In another embodiment,

1130 may reside in a controller between

200 and the disk drive. The

1130 does not need to reside in the disk drive.

Referring to

, it illustrates the steps performed by

1130 at the parity disk to execute a modified XPWrite command issued by

436 at the

200.

At

1010,

1130 receives the command issued by

436. The command includes a block number at which data will be written as well as the number of blocks that the data occupies. In addition, the command includes an Operation Code (opcode) indicating the type of the command.

Command executing

1130 reads (step 1020), into

1120, a data block(s) indicated in the command. Command executing

1130 requests data from storage driver module 435 (step 1030). In one implementation,

435 sends, to command executing

1130, XOR of data blocks that will be written to disk drives and the checksum block to be written to disk.

Command executing

1130 receives the result of the XOR operation (step 1040) and the checksum block. Then, command executing

1130 computes new parity by XOR'ing the result of the XOR operation of the previous step with the data read into the disk memory 1120 (step 1050). At

1060,

1130 writes the computed parity to the parity disk. At

1070,

1130 writes the checksum block to the parity disk. According to an embodiment of the present invention, the checksum block is written directly to the drive that stores parity for a stripe without being XOR'ed with the parity.

According to this embodiment, as a result of the execution of XPWrite command by the parity disk, the necessity for reading the old parity block by

436 and XOR'ing the parity block with new data is eliminated. Instead,

1130 at the

250 reads the old parity block, XOR's the old parity block with the XOR of new data, and writes the result of the operation to the drive that stores parity for a stripe.

According to another embodiment of the present invention, write operations can be optimized if unallocated (or free data blocks) are removed from a parity block for that stripe. According to an embodiment of the present invention,

430 includes a write allocation module 442 (shown in

) that performs write allocation of blocks in a volume in response to an event in the file system (e.g., dirtying of the blocks in a file). To allocate data blocks, the

442 uses the block allocation data structures (one

440 is shown in

) to select free blocks (i.e., data blocks that are not allocated) to which to write the dirty blocks (i.e., data blocks that store data). In one embodiment, the block

440 can be implemented as a bitmap, in which a logical “1” indicates that a data block is allocated and logical “0” indicates that a data block is not allocated or vice versa.

When performing write allocation,

430 traverses a small portion of each disk (corresponding to a few blocks in depth within each disk) to “lay down” a plurality of stripes per RAID group. When performing write allocation within the volume, the

442 typically works down a RAID group, allocating all free blocks within the stripes it passes over. This is efficient from a RAID system point of view in that more blocks are written per stripe.

As previously described, zeroing data blocks can be a time-consuming process. According to an embodiment of the present invention, since data from unallocated data blocks is removed from parity in a corresponding stripe, new parity can be computed without using the free data blocks even without zeroing those data blocks.

illustrates the steps to implement this embodiment of the present invention.

As described herein,

430 maintains block

440 in which the

430 keeps track of allocated and unallocated data blocks.

436 periodically receives an indication from

430 which data blocks have become unallocated or free (step 810).

At

820, to remove unallocated data blocks from a parity block,

436 XOR's unallocated data blocks with the parity block for that stripe. In one embodiment, the process of removing unallocated data blocks from a parity block is performed during idle time. As used herein, “idle time” refers to a period of time during which requests from

210 are expected to be low (such as, for example, during weekends, holidays, and off-peak hours).

In another embodiment, the

430 may remove unallocated data blocks from a parity block at the same time that data are written to other data blocks in the same stripe.

436 then updates the parity block by XOR'ing all data blocks in the stripe except for the unallocated data blocks.

At

830,

436 clears the unique disk ID in a data block (s) that is being removed from parity. At

840, the parity mask for the stripe is updated. In one implementation, to this end, a logical AND operation is performed on the current parity mask and a complement of the mask of the drive that stores a free data block. A complement is a logical function that converts each logical “1” bit to a logical “0” bit and each logical “0” bit to a logical “1” bit. For example, if the mask stored in the checksum block on the drive that stores parity for a stripe were “0111b” (indicating that the data blocks for

0, drive 1, and drive 2 were contained in the parity), and the data for drive 1 were removed from parity, then the mask “0111b” would be ANDed with the complement of the mask for drive 1. The mask for drive 1 is 0010b.” The complement of the mask for drive 1 is “1101b”. Thus, the result of the logical AND operation on “0111b” and “1101b” is “0101b.” This removes the mask of the drive that stores free data blocks from the parity mask.

A media error on a disk occurs when data cannot be read from a particular block or a number of blocks from that disk. Referring now to Table 1, it illustrates an array of disks storing data blocks. In this example, one stripe (Stripe 0) is shown for purposes of explanation only. A person of ordinary skill in the art would understand that the disk array shown in Table 1 may include more than one stripe. Disk 1 through

4 store data.

0 stores row parity. Thus, the exemplary disk array is implemented using RAID-4 parity scheme.

Assume that Disk 1 failed (i.e., the disk experienced an operation problem that renders the disk unable to respond to read-write operations). Disk 2 stores a data block that has a media error. In general, to recover data from a data block on Disk 2 that has a media error,

436 would have to perform the following steps:

Read data stored on Disk 1, Disk 3, and

4 for

0;

Read parity from

0 for

0;

XOR data stored on Disk 1, Disk 3, and

4 for

0;

XOR the result of the previous step and parity for

0.

The present invention provides the same benefits of not reading data blocks from the failed disk drive, but, unlike the referenced patent application, it does not require the data block on Disk 1 for that stripe to be zeroed. This is accomplished by determining whether a data block on a failed drive in a stripe where a media error was detected is valid. In one implementation,

436 performs a logical AND operation on the parity mask for the stripe and the mask of the failed drive. If a result of the logical operation does not contain a logical bit “1”, it indicates that data from the failed drive are not included in the parity block for that stripe and was written prior to adding the drive to the RAID group or before the RAID group was formed, or was removed from parity after the data block became free. As such, the data from the failed drive can be ignored and need not be read to recover data from another data block in the stripe. This allows

200 to recover data from the data block that has a media error in an array that has a failed drive without zeroing the failed drive in RAID-4 or RAID-5. Furthermore, this embodiment allows

200 to recover data in a dual parity RAID array with two failed drives and a third drive failure or a media error on a third disk.

According to an embodiment of the present invention, instead of reading data blocks from all disks in a stripe,

436 determines whether a data block in a stripe is included in the parity block for that stripe. To this end, in one implementation,

436 performs a logical AND operation on the parity mask for the stripe and the mask of the disk drive that needs to be read for purposes of reconstructing data on the failed drive. If a result of the logical operation does not contain a logical bit “1”, it indicates that data from that disk drive are not included in the parity block for that stripe. Therefore, the data were written prior to adding the disk to the RAID array or prior to forming the array, or was removed from the parity for the stripe when the data block became free. As such, this data block does not need to be read and can be ignored for purposes of reconstructing data on the failed drive. Additionally, if the data block to be reconstructed is invalid, then it does not need to be reconstructed and no reads or writes are required for that stripe.

According to an embodiment of the present invention, the number of read and write operations is reduced by determining which data blocks were written prior to creating a RAID group or being added to the RAID group, or have become free and were removed from the parity for the stripe and thus do not need to be copied. According to the present invention, data blocks that are not included in the parity block for the same stripe were written prior to adding their respective drives to the RAID array or these data blocks have become free. To determine whether a data block is included in the parity, in one implementation,

436 performs a logical AND operation on the parity mask for a stripe and the mask of the disk drive that includes a data block that is about to be copied onto another disk. If a result of a logical operation does not contain a logical bit “1”, it indicates that the data block is not included in the parity block for that stripe. These data blocks are not being copied. Hence, this reduces the number of I/O operations that would otherwise have to be performed to copy data blocks from a sick disk. This also reduces the reliance on the sick disk and allows the valid data to be copied onto a spare disk faster.

1. A method performed by a computer connected to a storage array having a plurality of storage devices, comprising:

determining by a processor of the computer whether a block in a stripe is to be included in a parity block for the stripe;

in response to determining that the block in the stripe is to be included in the parity block, determining whether the block is valid;

in response to determining that the block is invalid, ignoring the block to update the parity block for the stripe; and

in response to determining that the block is valid, updating the parity block using the block in the stripe.

2. The method of

, further comprising assigning a first mask to each storage device of the plurality of storage devices in the storage array.

3. The method of

, further comprising updating a second mask of a particular storage device to which new data are written that stores parity for the stripe in the array.

4. The method of

, wherein each storage device of the plurality of storage devices comprises a disk drive.

5. The method of

, further comprising:

inserting, into a checksum block of a new data block of the stripe, a unique storage device identification (ID) of a storage device of the plurality of storage devices where the new data block will be written; and

writing the new data block and an associated checksum block to the storage device identified by the unique storage device ID.

6. The method of

, further comprising:

inserting, into a checksum block associated with the parity block, a unique storage device identification (ID) of a storage device of the plurality of storage devices where the parity block will be written; and

writing the parity block and the checksum block to the storage device where the parity block will be written.

7. The method of

wherein determining whether the block is valid comprises:

storing a unique storage device identification (ID) on a storage device of the plurality of storage devices in the storage array storing the block;

responsive to receiving an access request that comprises a write operation and associated data, reading a data block in the stripe that is not to be modified and an associated checksum block; and

comparing the unique storage device ID to the data stored in the checksum block.

8. A computer configured to perform input/output (I/O) operations for a storage array having a plurality of storage devices, comprising:

a processor configured to execute one or more processes, the one or more processes, when executed, configured to:

determine whether a block in a stripe is to be included in a parity block for the stripe;

determine, in response to determining that the block in the stripe is to be included in the parity block, whether the block is valid;

ignore, in response to determining that the block is invalid, the block to update the parity block for the stripe; and

update, in response to determining that the block is valid, the parity block using the block in the stripe.

9. The method of

, wherein the unique storage device ID is read from a memory of the computer.

10. The computer of

, further comprising a storage module coupled to the processor, the storage module configured to insert, in a checksum block associated with a data block that stores data in an access request, a unique identification (ID) of a storage device of the plurality of storage devices in the storage array where the data in the access request will be written.

11. The computer of

, further comprising a storage module coupled to the processor, the storage module configured to insert, in a checksum block associated with the parity block, a unique identification (ID) of a storage device of the plurality of storage devices where the parity block will be written.

12. The computer of

, further comprising a memory coupled to the processor for storing a mask of each storage device of the plurality of storage devices in the storage array.

13. The computer of

, wherein each storage device of the plurality of storage devices comprises a disk drive.

14. The computer of

further comprising:

a storage module coupled to the processor and configured to store a unique storage device identification (ID) on a storage device of the plurality of storage devices in the storage array; and

responsive to receiving an access request that comprises a write operation and associated data, the storage module further configured to read a data block in the stripe that is not to be modified and an associated checksum block, and further configured to compare the unique storage device ID to the data stored in the checksum block.

15. A computer-readable storage medium stored with program instructions for execution by a processor, the computer-readable storage medium comprising:

program instructions that determine whether a block in a stripe is to be included in a parity block for the stripe;

program instructions that determine, in response to determining that the block in the stripe is to be included in the parity block, whether the block is valid;

program instructions that ignore, in response to determining that the block is invalid, the block to update the parity block for the stripe; and

program instructions that update, in response to determining that the block is valid, the parity block using the block.