Setup NCM for Linux

alawrence112 · ‎07-06-2016

Step 1: Installing Linux NCM

Note: Must be on one of the following: RHEL 6.5, RHEL 6.7, RHEL 7.0, and RHEL 7.1

(I’ve tested on 6.7 and 7.1)

Get NCM from InfoSight: https://infosight.nimblestorage.com/InfoSight/#software/Integration+Kits/Connection+Manager+%28NCM%29+for+Linux

Install Dependencies:

yum install device-mapper-multipath sg3_utils iscsi-initiator-utils

Install mcm as root (or with sudo):


chmod 755 [path-to-ncm]

./[path-to-ncm]

==============================================================================================================

Post Installation Steps for User

==============================================================================================================


Please add following LVM filters into "devices" section of /etc/lvm/lvm.conf


preferred_names = ["^/dev/nimblestorage/", "^/dev/mpath/", "^/dev/mapper/mpath", "^/dev/[hs]d"]

filter = [ "r|^/dev/nimblestorage/lower_tier_devices/.*|" ]

global_filter = [ "r|^/dev/nimblestorage/lower_tier_devices/.*|" ]

NOTE: For SAN Boot host, please re-generate initramfs using below command after adding LVM filters.

dracut --add "multipath ncm" --force /boot/initramfs-`uname -r`-ncm.img

where `uname -r` yeilds current running kernel version.

Step 2: Configure iSCSI Network Adapters

Setup Storage NICs on the linux guest (NICs that are on the same subnet(s) as your storage).

Add these to your guest if you are virtual via VMWare/HyperV/Xen/KVM/etc... then configure them.


Example:

vim /etc/sysconfig/network-scripts/ifcfg-eno33559296


TYPE=Ethernet

BOOTPROTO=none

IPV4_FAILURE_FATAL=no

IPADDR="10.1.0.133"

NETMASK="255.255.255.0"

MTU="9000"

IPV6INIT=no

NAME=iscsi-2

UUID=787d979e-a2ff-48eb-96ca-86e368eb254c

DEVICE=eno50338560

ONBOOT=yes

You might need to bring the adapter up if you just added it.

ifup eth0

ifup eno33559296

Step 3: Create an Initiator Group

Get the IQN from your Linux guest:

cat /etc/iscsi/initiatorname.iscsi

Create a iSCSI initiator group on the nimble with the IQN

Screen Shot 2016-07-06 at 9.34.15 PM.png

Step 4: Create an Volume on Nimble

Create a volume on the Nimble for your needs (perf policy, size, etc…) using the newly made iSCSI initiator group for access.

Screen Shot 2016-07-06 at 9.36.20 PM.png

Step 5: Connect to the Volume in Linux

ncmadm --rescan [iSCSI Discovery IP]

(Example output)

[root@ajl-rhel6 ~]# ncmadm --rescan 10.1.0.17

Discovery IP :

10.1.0.17

List of discovered devices:

target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.10:3260,2460

target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.19:3260,2460

target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.11:3260,2460

target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.20:3260,2460

target:iqn.2007-11.com.nimblestorage:ajl-linux6-ncm-das-v2d35dd4571510ccf.0000032c.5c64d18a portal:10.1.0.17:3260,2460

Login to the new devices:

target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.10:3260,2460

target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.19:3260,2460

target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.20:3260,2460

target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.11:3260,2460

target:iqn.2007-11.com.nimblestorage:ajl-linux6-ncm-das-v2d35dd4571510ccf.0000032c.5c64d18a portal:10.1.0.17:3260,2460

Rescan initiated. Use ncmadm -l to list Nimble device

Showing the list command


[root@ajl-rhel6 ~]# ncmadm -l

size=150G uuid=2f7773dfa9b4d0ad36c9ce9008ad1645c mount-device: /dev/nimblestorage/AJL-Linux6-NCM-DAS-2f7773dfa9b4d0ad36c9ce9008ad1645c

target: iqn.2007-11.com.nimblestorage:ajl-linux6-ncm-das-v2d35dd4571510ccf.0000032c.5c64d18a connection_mode=automatic

 ` + array_id=1 device=/dev/dm-3

   |-    7:0:0:0   sdb   8:16  session=5   sess_st=LOGGED_IN  dev_st=running    iface=default    tgtportal=10.1.0.17        data_ip=10.1.0.11       

   |-    8:0:0:0   sdc   8:32  session=6   sess_st=LOGGED_IN  dev_st=running    iface=default    tgtportal=10.1.0.17        data_ip=10.1.0.10

Step 6: Format a Multipath Disk

Find it in /dev/mapper or /dev/nimblestorage (mpatha in this case). If it is a different location then /dev/mapper you'll need to modify the below commands to your location.

[root@ajl-rhel6 ~]# ls /dev/mapper/

2f7773dfa9b4d0ad36c9ce9008ad1645c control mpatha vg_ajlrhel6-lv_home vg_ajlrhel6-lv_root vg_ajlrhel6-lv_swap

Create the filesystem

fdisk /dev/mapper/[mpathXX]

Create it how you want. My example is a primary partition, the first one, and set type to be LVM as I prefer LVMs. You might choose a different type or to use parted and GPT vs fdisk and MBR. I will be covering fdisk. If you do not want LVM, don't change the partition type, leave it as 83.

Example:

[root@ajl-rhel6 ~]# fdisk /dev/mapper/mpatha

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel with disk identifier 0xb4e794c7.

Changes will remain in memory only, until you decide to write them.

After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to

switch off the mode (command 'c') and change display units to

sectors (command 'u').

Command (m for help): m

Command action

a toggle a bootable flag

b edit bsd disklabel

c toggle the dos compatibility flag

d delete a partition

l list known partition types

m print this menu

n add a new partition

o create a new empty DOS partition table

p print the partition table

q quit without saving changes

s create a new empty Sun disklabel

t change a partition's system id

u change display/entry units

v verify the partition table

w write table to disk and exit

x extra functionality (experts only)

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-19581, default 1):

Using default value 1

Last cylinder, +cylinders or +size{K,M,G} (1-19581, default 19581):

Using default value 19581

Command (m for help): t

Selected partition 1

Hex code (type L to list codes): l

0 Empty 24 NEC DOS 81 Minix / old Lin bf Solaris

1 FAT12 39 Plan 9 82 Linux swap / So c1 DRDOS/sec (FAT-

2 XENIX root 3c PartitionMagic 83 Linux c4 DRDOS/sec (FAT-

3 XENIX usr 40 Venix 80286 84 OS/2 hidden C: c6 DRDOS/sec (FAT-

4 FAT16 <32M 41 PPC PReP Boot 85 Linux extended c7 Syrinx

5 Extended 42 SFS 86 NTFS volume set da Non-FS data

6 FAT16 4d QNX4.x 87 NTFS volume set db CP/M / CTOS / .

7 HPFS/NTFS 4e QNX4.x 2nd part 88 Linux plaintext de Dell Utility

8 AIX 4f QNX4.x 3rd part 8e Linux LVM df BootIt

9 AIX bootable 50 OnTrack DM 93 Amoeba e1 DOS access

a OS/2 Boot Manag 51 OnTrack DM6 Aux 94 Amoeba BBT e3 DOS R/O

b W95 FAT32 52 CP/M 9f BSD/OS e4 SpeedStor

c W95 FAT32 (LBA) 53 OnTrack DM6 Aux a0 IBM Thinkpad hi eb BeOS fs

e W95 FAT16 (LBA) 54 OnTrackDM6 a5 FreeBSD ee GPT

f W95 Ext'd (LBA) 55 EZ-Drive a6 OpenBSD ef EFI (FAT-12/16/

10 OPUS 56 Golden Bow a7 NeXTSTEP f0 Linux/PA-RISC b

11 Hidden FAT12 5c Priam Edisk a8 Darwin UFS f1 SpeedStor

12 Compaq diagnost 61 SpeedStor a9 NetBSD f4 SpeedStor

14 Hidden FAT16 <3 63 GNU HURD or Sys ab Darwin boot f2 DOS secondary

16 Hidden FAT16 64 Novell Netware af HFS / HFS+ fb VMware VMFS

17 Hidden HPFS/NTF 65 Novell Netware b7 BSDI fs fc VMware VMKCORE

18 AST SmartSleep 70 DiskSecure Mult b8 BSDI swap fd Linux raid auto

1b Hidden W95 FAT3 75 PC/IX bb Boot Wizard hid fe LANstep

1c Hidden W95 FAT3 80 Old Minix be Solaris boot ff BBT

1e Hidden W95 FAT1

Hex code (type L to list codes): 8e

Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.

The kernel still uses the old table. The new table will be used at

the next reboot or after you run partprobe(8) or kpartx(8)

Syncing disks.

At this point you have two options.

Option 1: Reboot to clear up partition table.

Option 2: Run

kpartx -a /dev/mapper/[mpath]

  Example:  kpartx -a /dev/mapper/mpathb

After reboot/kpartx you will now see a /dev/mapper/mapthap1

Step 6: Create an LVM

(if you do not want an LVM you can skip to step 7 and see the file system format command)

I'm a fan of LVM's. As such, that is what I did next. See the example for pvcreate/vgcreate/lvcreate using all the space. If you want to do differently check out redhat's guide.

[root@ajl-rhel6 ~]# pvcreate /dev/mapper/mpathap1

WARNING: Ignoring duplicate config value: preferred_names

Physical volume "/dev/mapper/mpathap1" successfully created

[root@ajl-rhel6 ~]# vgcreate IOtesting /dev/mapper/mpathap1

WARNING: Ignoring duplicate config value: preferred_names

Volume group "IOtesting" successfully created

[root@ajl-rhel6 ~]# lvcreate -n IOTesting -l 100%FREE IOtesting

WARNING: Ignoring duplicate config value: preferred_names

Logical volume "IOTesting"created.

Bring your LVM's online

vgchange -a y [volume group name]

  Example: vgchange -a y IOtesting

or all at once

Example: vgchange -a y

Step 7: Format the Filesystem

mkfs.ext4 -L [somelabel] /dev/[vg]/[lv]

I highly encourage the use of labels when formatting the filesystem as you can mount by them. VERY useful when things jump around. If you are not dong an LVM this is a must as your iscsi disks names can move around at boot time (eg /dev/sdc -> /dev/sde). Labels will make your life better.

[root@ajl-rhel6 ~]# mkfs.ext4 -L IOTST /dev/IOtesting/IOTesting

mke2fs 1.41.12 (17-May-2010)

Discarding device blocks: done

Filesystem label=

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

Stride=0 blocks, Stripe width=0 blocks

9830400 inodes, 39320576 blocks

1966028 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=4294967296

1200 block groups

32768 blocks per group, 32768 fragments per group

8192 inodes per group

Superblock backups stored on blocks:

32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,

4096000, 7962624, 11239424, 20480000, 23887872

Writing inode tables: done

Creating journal (32768 blocks): done

Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 36 mounts or

180 days, whichever comes first. Use tune2fs -c or -i to override.

Creating auto mount (fstab)

Step 8: Create an fstab Entry

Create a location to mount the new filesystem

mkdir /[some location]

  Example: mkdir /IOTesting

Edit fstab

_netdev is critical. If you do defaults and not _netdev your LVM will not go active at boot time and you will have no filesystem to write to!

vi /etc/fstab

Mount by LVM point:

/dev/IOtesting/IOTesting        /IOTesting      ext4    _netdev        0 0

Mount by filesystem label:

LABEL=TST2      /IOtst2 ext4    _netdev        0 0

Verify and mount the entries created in fstab

mount -a

Step 9: View the Fruit of Your Labor

[root@ajl-rhel6 ~]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/vg_ajlrhel6-lv_root

50G 2.3G 45G 5% /

tmpfs 1.9G 0 1.9G 0% /dev/shm

/dev/sda1 477M 41M 411M 10% /boot

/dev/mapper/vg_ajlrhel6-lv_home

45G 52M 43G 1% /home

/dev/mapper/IOtesting-IOTesting

148G 60M 140G 1% /IOTesting

/dev/mapper/IOtesting2-IOTesting2

148G 60M 140G 1% /IOtst2

Setup NCM for Linux

Setup NCM for Linux

alawrence112

Hewlett Packard Enterprise International