Scroll to Top

When implementing an LSI Nytro WarpDrive (NWD) or Nytro MegaRAID (NMR) PCIe flash card in a Linux server, you need to modify quite a few variables to get the best performance out of these cards.

In the Linux server, device assignments sometimes change after reboots. Sometimes, the PCIe flash card can be assigned /dev/sda. Other times, it can be assigned /dev/sdd, or any device name. This variability can wreak havoc when modifying the Linux environment variables. To get around this issue, assignments by the SCSI address should be used so all of the Linux performance variables will persist properly across reboots. If using a filesystem, use the device UUID address in the mount statement in /etc/fstab to persist the mount command across reboots.

Cut and paste the script
The first step to solving is to cut and paste the following script, except the SCSI address (highlighted in yellow), into /etc/rc.local. You’ll need to enter the SCSI address of the PCIe card before executing the script.

nwd_getdevice.sh
ls -al /dev/disk/by-id |grep 'scsi-3600508e07e726177965e06849461a804 ' |grep /sd > nwddevice.txt
awk '{split($11,arr,"/"); print arr[3]}' nwddevice.txt > nwd1device.txt
variable1=$(cat nwd1device.txt)
echo "4096" > /sys/block/$variable1/queue/nr_requests
echo "512" > /sys/block/$variable1/device/queue_depth
echo "deadline" > /sys/block/$variable1/queue/scheduler
echo "2" > /sys/block/$variable1/queue/rq_affinity
echo 0 > /sys/block/$variable1/queue/rotational
echo 0 > /sys/block/$variable1/queue/add_random
echo 1024 > /sys/block/$variable1/queue/max_sectors_kb
echo 0 > /sys/block/$variable1/queue/nomerges
blockdev --setra 0 /dev/$variable1

The highlighted SCSI address above needs to be modified with the SCSI address of the PCIe flash card. To get the address, issue this command:

ls –al /dev/disk/by-id

When you install the Nytro PCIe flash card, Linux will assign a name to the device. For example, the device name can be listed as /dev/sdX, and X can be any letter. The output from the ls command above will show the SCSI address for this PCIe device. Don’t use the address containing “-partX” in it. Be sure to note this SCSI address since you will need it to create the script below. Include a single space between the SCSI address and the closing single quote in the script.

Create nwd_getdevice.sh file
Next, copy the code and create a file called “nwd_getdevice.sh” with the modified SCSI address.

After saving this file, change file permissions to “execute” and then place this command in the /etc/rc.local file:

/path/nwd_getdevice.sh

Test the script
To test this script, execute it on the command line exactly how you stated it in the rc.local file. The next time the system reboots, the settings will be set to the appropriate device.

Multiple PCIe flash cards
If you plan to deploy multiple LSI PCIe flash cards in the server, the easiest way is to duplicate all of commands in the nwd_getdevice.sh script and paste them at the end. Then change the SCSI address of the next card and overlay the SCSI address in the newly pasted area. You can follow this procedure for as many LSI PCIe flash cards as are installed in the server. For example:

nwd_getdevice.sh
ls -al /dev/disk/by-id |grep 'scsi-1stscsiaddr83333365e06849461a804 ' |grep /sd > nwddevice.txt
awk '{split($11,arr,"/"); print arr[3]}' nwddevice.txt > nwd1device.txt
variable1=$(cat nwd1device.txt)
echo "4096" > /sys/block/$variable1/queue/nr_requests
echo "512" > /sys/block/$variable1/device/queue_depth
echo "deadline" > /sys/block/$variable1/queue/scheduler
echo "2" > /sys/block/$variable1/queue/rq_affinity
echo 0 > /sys/block/$variable1/queue/rotational
echo 0 > /sys/block/$variable1/queue/add_random
echo 1024 > /sys/block/$variable1/queue/max_sectors_kb
echo 0 > /sys/block/$variable1/queue/nomerges
blockdev --setra 0 /dev/$variable1
ls -al /dev/disk/by-id |grep 'scsi-2ndscsiaddr1234566666654444444444 ' |grep /sd > nwddevice.txt
awk '{split($11,arr,"/"); print arr[3]}' nwddevice.txt > nwd1device.txt
variable1=$(cat nwd1device.txt)
echo "4096" > /sys/block/$variable1/queue/nr_requests
echo "512" > /sys/block/$variable1/device/queue_depth
echo "deadline" > /sys/block/$variable1/queue/scheduler
echo "2" > /sys/block/$variable1/queue/rq_affinity
echo 0 > /sys/block/$variable1/queue/rotational
echo 0 > /sys/block/$variable1/queue/add_random
echo 1024 > /sys/block/$variable1/queue/max_sectors_kb
echo 0 > /sys/block/$variable1/queue/nomerges
blockdev --setra 0 /dev/$variable1

Final thoughts
The most important step in implementing Nytro PCIe flash cards under Linux is aligning the card on a boundary, which I cover in Part 1 of this series.  This step alone can deliver a 3x performance gain or more based on our in-house tests as well as testing from some of our customers. The rest of this series walks you through the process of setting up these aligned flash cards using a file system, ASM or RAW device and, finally, persisting all the Linux performance variables to the card so these settings are persisted across reboots.

Links to the other posts in this series:

How to maximize PCIe flash performance under Linux

Part 1: Aligning PCIe flash devices
Part 2: Creating the RAW device or filesystem
Part 3: Oracle ASM

Rick Stehno has 36 years of IT industry experience and for the past 24 years has worked with Oracle relational databases. He has been at LSI for six years, benchmarking Oracle databases for the company’s SAN storage and flash technology divisions.... Read more

Tags: , , , , , , , , , , ,
Views: (1979)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>