CentOS7下编译安装SPDK iSCSI Target

痛定思痛。 2022-09-29 06:33 279阅读 0赞

# 操作环境 #

Centos 7

# 操作步骤 #

1.下载SPDK

#git clone https://github.com/spdk/spdk.git

2.安装依赖包

# dnf install -y gcc gcc-c++ CUnit-devel libaio-devel openssl-devel

3.下载安装DPDK

#cd spdk
    #wget http://fast.dpdk.org/rel/dpdk-17.02.tar.xz
    #tar xf dpdk-17.02.tar.xz
    #cd dpdk-17.02 && make install T=x86_64-native-linuxapp-gcc DESTDIR=.
    #make DPDK_DIR=./dpdk-17.02/x86_64-native-linuxapp-gcc
    #sudo scripts/setup.sh

4.在安装完成后，我们可以在spdk/app/iscsi\_tgt/目录下面找到iscsi\_tgt可执行文件

![Center][]

5. 编辑iscsi tgt配置文件，可以根据spdk/etc/spdk/iscsi.conf.in文件，进行修改

# iSCSI target configuration file
    #
    # Please write all parameters using ASCII.
    # The parameter must be quoted if it includes whitespace.
    #
    # Configuration syntax:
    # Leading whitespace is ignored.
    # Lines starting with '#' are comments.
    # Lines ending with '\' are concatenated with the next line.
    # Bracketed ([]) names define sections
    
    [Global]
      # Shared Memory Group ID. SPDK applications with the same ID will share memory.
      # Default: <the process PID>
      #SharedMemoryID 0
    
      # Users can restrict work items to only run on certain cores by
      #  specifying a ReactorMask.  Default is to allow work items to run
      #  on core 0.
      #ReactorMask 0xFFFF
    
      # Tracepoint group mask for spdk trace buffers
      # Default: 0x0 (all tracepoint groups disabled)
      # Set to 0xFFFFFFFFFFFFFFFF to enable all tracepoint groups.
      #TpointGroupMask 0x0
    
      # syslog facility
      LogFacility "local7"
    
    [iSCSI]
      # node name (not include optional part)
      # Users can optionally change this to fit their environment.
      NodeBase "iqn.2016-06.io.spdk"
    
    # AuthFile /usr/local/etc/spdk/auth.conf //注释掉
    
      MinConnectionsPerCore 4
      # Power saving related variable, this parameter defines how long an iSCSI
      # connection must be idle before moving it to a state where it will consume
      # less power. This variable is defined in terms of microseconds. We set default
      # value as 5ms.
      MinConnectionIdleInterval 5000
    
      # Socket I/O timeout sec. (0 is infinite)
      Timeout 30
    
      # authentication information for discovery session
      DiscoveryAuthMethod Auto
    
      #MaxSessions 128
      #MaxConnectionsPerSession 2
    
      # iSCSI initial parameters negotiate with initiators
      # NOTE: incorrect values might crash
      DefaultTime2Wait 2
      DefaultTime2Retain 60
    
      ImmediateData Yes
      ErrorRecoveryLevel 0
    
    [Rpc]
      # Defines whether to enable configuration via RPC.
      # Default is disabled.  Note that the RPC interface is not
      # authenticated, so users should be careful about enabling
      # RPC in non-trusted environments.
      Enable No
      # Listen address for the RPC service.
      # May be an IP address or an absolute path to a Unix socket.
      Listen 127.0.0.1
    
    # Users must change the PortalGroup section(s) to match the IP addresses
    #  for their environment.
    # PortalGroup sections define which TCP ports the iSCSI server will use
    #  to listen for incoming connections.  These are also used to determine
    #  which targets are accessible over each portal group.
    [PortalGroup1]
     Portal DA1 10.10.200.61:3260 //设置为iscsi target server ip
    
    # Users must change the InitiatorGroup section(s) to match the IP
    #  addresses and initiator configuration in their environment.
    # Netmask can be used to specify a single IP address or a range of IP addresses
    #  Netmask 192.168.1.20   <== single IP address
    #  Netmask 192.168.1.0/24 <== IP range 192.168.1.*
    [InitiatorGroup1]
      InitiatorName ALL
     Netmask 10.10.200.0/24 //设置为iscsi initiator 的ip范围
    
    # NVMe configuration options
    #[Nvme]
    # NVMe Device Whitelist
    # Users may specify which NVMe devices to claim by their PCI
    # domain, bus, device, and function. The format is dddd:bb:dd.f, which is
    # the same format displayed by lspci or in /sys/bus/pci/devices. The second
    # argument is a "name" for the device that can be anything. The name
    # is referenced later in the Subsystem section.
    #
    # Alternatively, the user can specify ClaimAllDevices. All
    # NVMe devices will be claimed and named Nvme0, Nvme1, etc.
    #  BDF 0000:00:00.0 Nvme0
    #  BDF 0000:01:00.0 Nvme1
    
      # The number of attempts per I/O when an I/O fails. Do not include
      # this key to get the default behavior.
    #  NvmeRetryCount 4
      # The maximum number of NVMe controllers to claim. Do not include this key to
      # claim all of them.
    #  NumControllers 2
      # Registers the application to receive timeout callback and to reset the controller.
      ResetControllerOnTimeout Yes
      # Timeout value.
    #  NvmeTimeoutValue 30
      # Set how often the admin queue is polled for asynchronous events.
      # Units in microseconds.
    #  AdminPollRate 100000
    
    # Users may change this section to create a different number or size of
    #  malloc LUNs.
    # If the system has hardware DMA engine, it will use an IOAT
    # (i.e. Crystal Beach DMA) channel to do the copy instead of memcpy.
    # Of course, users can disable offload even it is available.
    #[Malloc]
      # Number of Malloc targets
    #  NumberOfLuns 3
      # Malloc targets are 128M
    #  LunSizeInMB 128
      # Block size. Default is 512 bytes.
    #  BlockSize 4096
    
    # Users may not want to use offload even it is available.
    # Users may use the whitelist to initialize specified devices, IDS
    #  uses BUS:DEVICE.FUNCTION to identify each Ioat channel.
    [Ioat]
      Disable Yes
      Whitelist 00:04.0
      Whitelist 00:04.1
    
    # Users must change this section to match the /dev/sdX devices to be
    #  exported as iSCSI LUNs. The devices are accessed using Linux AIO.
    [AIO] AIO /dev/sdb //设置硬盘盘符
    #  AIO /dev/sdc
    
    # The Split virtual block device slices block devices into multiple smaller bdevs.
    #[Split]
      # Syntax:
      #   Split <bdev> <count> [<size_in_megabytes>]
    
      # Split Malloc1 into two equally-sized portions, Malloc1p0 and Malloc1p1
    #  Split Malloc1 2
    
      # Split Malloc2 into eight 1-megabyte portions, Malloc2p0 ... Malloc2p7,
      # leaving the rest of the device inaccessible
    #  Split Malloc2 8 1
    
    # Users should change the TargetNode section(s) below to match the
    #  desired iSCSI target node configuration.
    # TargetName, Mapping, LUN0 are minimum required
    [TargetNode1]
      TargetName disk1
      TargetAlias "Data Disk1"
      Mapping PortalGroup1 InitiatorGroup1
    # AuthMethod Auto # AuthGroup AuthGroup1 //注释掉
      # Enable header and data digest
      # UseDigest Header Data
      UseDigest Auto
      # Use the first malloc target
    #  LUN0 Malloc0
      # Using the first AIO target
     LUN0 AIO0 //设置LUN映射
      # Using the second storage target
    #  LUN2 AIO1
      # Using the third storage target
    #  LUN3 AIO2
      QueueDepth 128
    
    #[TargetNode2]
    #  TargetName disk2
    #  TargetAlias "Data Disk2"
    #  Mapping PortalGroup1 InitiatorGroup1
    #  AuthMethod Auto
    #  AuthGroup AuthGroup1
    #  UseDigest Auto
    #  LUN0 Nvme0
    #  QueueDepth 32

6.启动iscsi tgt

[root@localhost iscsi_tgt]# ./iscsi_tgt 
    Starting Intel(R) DPDK initialization ... 
    [ DPDK EAL parameters: iscsi -c 1 --file-prefix=spdk2697 --base-virtaddr=0x1000000000 --proc-type=auto ]
    EAL: Detected 8 lcore(s)
    EAL: Auto-detected process type: PRIMARY
    EAL: No free hugepages reported in hugepages-1048576kB
    EAL: Probing VFIO support...
    Occupied cpu core mask is 0x1
    Occupied cpu socket mask is 0x1
    device 0 HDD UNIT
    device 0: LUN0 AIO0
    Total cores available: 1
    Using net framework default
    Reactor started on core 0 on socket 0

7.从iscsi initiator端连接tgt

[root@lnmp-190 nodes]# iscsiadm -m discovery -t sendtargets -p 10.10.200.61
    10.10.200.61:3260,1 iqn.2016-06.io.spdk:disk1
    [root@lnmp-190 nodes]# iscsiadm -m node -T iqn.2016-06.io.spdk:disk1 -l
    Logging in to [iface: default, target: iqn.2016-06.io.spdk:disk1, portal: 10.10.200.61,3261] (multiple)
    Login to [iface: default, target: iqn.2016-06.io.spdk:disk1, portal: 10.10.200.61,3261] successful.
    [root@lnmp-190 nodes]# fdisk -l
    
    Disk /dev/sda: 53.7 GB, 53687091200 bytes
    64 heads, 32 sectors/track, 51200 cylinders
    Units = cylinders of 2048 * 512 = 1048576 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x000d3782
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sda1   *           2         501      512000   83  Linux
    Partition 1 does not end on cylinder boundary.
    /dev/sda2             502       51200    51915776   8e  Linux LVM
    Partition 2 does not end on cylinder boundary.
    
    Disk /dev/mapper/vg_lnmp190-lv_root: 48.9 GB, 48930750464 bytes
    255 heads, 63 sectors/track, 5948 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x00000000
    
    
    Disk /dev/mapper/vg_lnmp190-lv_swap: 4227 MB, 4227858432 bytes
    255 heads, 63 sectors/track, 514 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x00000000
    
    
    Disk /dev/sdd: 4783.5 GB, 4783519825920 bytes
    255 heads, 63 sectors/track, 581563 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 4096 bytes / 1048576 bytes
    Disk identifier: 0x00000000

查看iscsi 磁盘，以上信息中的sdd为iscsi硬盘

[root@lnmp-190 nodes]# iscsiadm -m session -P 3 | grep "Attached scsi disk" | awk '{print $4}'
    sdd

[Center]: /images/20220708/1eba9422349745df9614fb36f864db79.png