ubuntu14.04 infiniband set problems

July 8, 2014, 5:25 am

≫ Next: Re: 56 GbE over a ConnectX-3 VPI adapter

≪ Previous: Re: NFS over RDMA on CentOS 6.3 Problem

$

0

0

1 first install the driver

MLNX_OFED_LINUX-2.2-1.0.1-ubuntu14.04-x86_64.iso

2  ibhosts

can see

Ca	: 0x002590ffff17b868 ports 1 "node03 HCA-1"
Ca	: 0x002590ffff17bd40 ports 1 "node02 HCA-1"
Ca	: 0x002590ffff17b998 ports 1 "node05 HCA-1"

3  ibping one of them

have problems:

ibping -G 0x002590ffff90579c

ibwarn: [4132] sa_rpc_call: only lid routes are supported

ibwarn: [4132] ib_path_query_via: sa call path_query failed

ibping: iberror: failed: can't resolve destination port 0x002590ffff90579c

4 how to test this nodes can communicate with each other

  set each with /etc/network/interfaces like

auto ib0

iface ib0 inet static

address 192.168.2.10-80 #from node01 to node08

netmask 255.255.255.0

post-up echo connected > /sys/class/net/ib0/mode

        post-up /sbin/ifconfig $IFACE mtu 65520

gateway 192.168.2.215

5 i don't know how to do .look up for help sincerely . Thank you

---

Re: 56 GbE over a ConnectX-3 VPI adapter

July 8, 2014, 6:26 am

≫ Next: Am i able to use SFP+ 10Gbase-SR with MCX312A-XCBT adapter?

≪ Previous: ubuntu14.04 infiniband set problems

$

0

0

for back-2-back 56GbE connections between ConnectX-3 and ConnectX-3 Pro, the solution provided in this post is still relevent

Am i able to use SFP+ 10Gbase-SR with MCX312A-XCBT adapter?

July 9, 2014, 6:08 am

≫ Next: Re: Mixing OFED 1.5.3 and 2.2 in the same network?

≪ Previous: Re: 56 GbE over a ConnectX-3 VPI adapter

$

0

0

Greetings.

I recently bought the MCX312A-XCBT adapter not knowing which optical module i was going to use. It turned out that the sfp+ 10gbase-SR is actually needed. The question is does this adapter support this module. Documentation http://www.mellanox.com/related-docs/user_manuals/ConnectX-3_Ethernet_Single_and_Dual_SFP+_Port_Adapter_Card_User_Manual… states that 10GBASE-CX4, 10GBASE-R, and 1000BASE-R is supported.  But there is actually no 10GBASE-R standart it doesn't exist or i don't know about it. Maybe 10GBASE-R means typo 10gbase-SR or LR ER or KR?

Re: Mixing OFED 1.5.3 and 2.2 in the same network?

July 9, 2014, 6:53 am

≫ Next: Is there a way to show what SFP adapters are installed without pulling them from the card?

≪ Previous: Am i able to use SFP+ 10Gbase-SR with MCX312A-XCBT adapter?

$

0

0

So I have been delving into this and I've found the following:

 

I have managed to compile ofed 1.5.3 in CentOS 6.5 after following this post instructions (el5.10 ofed build problem.)

 

My infiniband nodes have these cards:

• computing nodes:
  • Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]
  • CA type: MT26428
  • Firmware version: 2.9.1100
• GPFS servers:
  • Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]
  • CA type: MT26428
  • Firmware version: 2.9.1000
• Other GPFS clients:
  • Mellanox Technologies MT27500 Family [ConnectX-3]
  • CA type: MT4099
  • Firmware version: 2.10.300 (upgradable up to 2.11.500)

 

And, according to the release notes:

2.2

MLNX_OFED Rev 2.2-1.0.1 supports the following Mellanox network adapter cards:

• Connect-IB® (Rev 10.10.3000 and above)

• ConnectX®-3 Pro (Rev 2.31.5050 and above)

• ConnectX®-3 (Rev 2.31.5050 and above)

• ConnectX®-2 (Rev 2.9.1200 and above)

ConnectX®-2 does not support all the new functionality of MLNX_OFED 2.0.3-XXX. For the complete list of the supported features per HCA, please refer to the MLNX_OFED User Manual.

 

1.5.3

Mellanox supports the following adapters with MLNX_OFED_LINUX 1.5.3:

  - Mellanox Technologies HCAs (SDR and DDR Modes are Supported):

  - ConnectX-2 / ConnectX-2 EN (fw-ConnectX2 Rev 2.9.1000)

  - ConnectX-3 VPI / ConnectX-3 EN (fw-4099 Rev 2.11.0500)

 

 

So, it seems that none of my current cards are compatible with ofed 2.2, and only some of my servers are compatible with the latest 1.5.3.

 

What should I do? Should I keep everything at 1.5.3? Should I update everything to 2.2? Would I win anything for moving to 2.2?

 

And when buying new nodes (with new cards)? Should we install them with 2.2? Should we downgrade them to 1.5.3? Can both 1.5.3 and 2.2 coexist in the same network? Will 1.5.3 last? Is it already "dead"?

 

Thanks in advance,

 

Txema

Is there a way to show what SFP adapters are installed without pulling them from the card?

July 9, 2014, 7:54 am

≫ Next: Re: Re: Mixing OFED 1.5.3 and 2.2 in the same network?

≪ Previous: Re: Mixing OFED 1.5.3 and 2.2 in the same network?

$

0

0

I currently have ConnectX-2 cards installed on my servers with SFP adapters installed. I would like to query the SFPs and find out what they are without pulling them from the machines.

 

The servers are running Ubuntu and ethtool does not report any information about the SFPs. Is there a tool or command that I can use to obtain this information?

Re: Re: Mixing OFED 1.5.3 and 2.2 in the same network?

July 10, 2014, 1:24 am

≫ Next: Re: Am i able to use SFP+ 10Gbase-SR with MCX312A-XCBT adapter?

≪ Previous: Is there a way to show what SFP adapters are installed without pulling them from the card?

$

0

0

MLNX OFED 2.2 is preferred and it has significant improvements in both stability and performance of IPoIB. However if you have older Linux versions that are not supported by 2.2 (like RHEL 5.x kernels) then you have to use 1.5.3 on those systems. Both versions can coexist, just be aware that they have different default settings for IPoIB modes – it is CM for 1.5.3 and UD for 2.2. Links between 1.5.3 and 2.2 systems (with default settings) will negotiate to UD so performance will be limited by performance of UD on 1.5.3 which is lower than CM on 1.5.3. Setting 2.2 systems to CM will give you better performance on links with 1.5.3 systems. If both ends are 2.2 preferred mode is UD (default): although both UD and CM have good performance on 2.2, UD has a bit higher performance and better scales with a big number of connections. On firmware: MLNX OFED installation by default will do firmware upgrade on your Mellanox branded cards (OEM cards by HP or others will not be upgraded). You can check card PSID with ibv_devinfo (see http://www.mellanox.com/page/firmware_HCA_FW_identification) – for Mellanox cards PSID always starts with “MT”, for example: MT_1100110019. With actual PSID you can always find proper fw at http://www.mellanox.com/page/firmware_download but perhaps easiest is to allow MLNX OFED installation script to do the upgrades to versions required. Once f/w is upgraded – your ConnecX2 and ConnectX3 cards will work just fine.

Re: Am i able to use SFP+ 10Gbase-SR with MCX312A-XCBT adapter?

July 10, 2014, 2:13 am

≫ Next: Re: Am i able to use SFP+ 10Gbase-SR with MCX312A-XCBT adapter?

≪ Previous: Re: Re: Mixing OFED 1.5.3 and 2.2 in the same network?

$

0

0

MCX312A-XCBT can be used with both SR and LR modules - for example with these Mellanox modules (also qualified with similar modules from other major vendors):

MFM1T02A-LR  Mellanox optical module, ETH 10GbE, 10Gb/s, SFP+, LC-LC, 1310nm, LR up to 10km

MFM1T02A-SR  Mellanox optical module, ETH 10GbE, 10Gb/s, SFP+, LC-LC, 850nm, SR up to 300m

BTW, they also can be used with 1GE modules:

MC3208011-SX  Mellanox® Optical module, ETH 1GbE, 1Gb/s, SFP, LC-LC, SX 850nm, up to 500m

MC3208411-T     Mellanox® module, ETH 1GbE, 1Gb/s, SFP, Base-T, up to 100m

Re: Am i able to use SFP+ 10Gbase-SR with MCX312A-XCBT adapter?

July 10, 2014, 3:37 am

≫ Next: Re: Am i able to use SFP+ 10Gbase-SR with MCX312A-XCBT adapter?

≪ Previous: Re: Am i able to use SFP+ 10Gbase-SR with MCX312A-XCBT adapter?

$

0

0

Thanks a lot!

---

Re: Am i able to use SFP+ 10Gbase-SR with MCX312A-XCBT adapter?

July 10, 2014, 4:26 am

≫ Next: Re: Mixing OFED 1.5.3 and 2.2 in the same network?

≪ Previous: Re: Am i able to use SFP+ 10Gbase-SR with MCX312A-XCBT adapter?

$

0

0

I'm not able to buy exactly MFM1T02A-SR because they are out of stock everywhere in Moscow. Sales manager just told me that this module is Finistar FTLX8571D3BCL-ME branded Mellanox. FTLX8571D3BCL (whithout -ME at the end) is in stock and i am looking farward to buying it. Do you happen to know will it actually fit?

Re: Mixing OFED 1.5.3 and 2.2 in the same network?

July 10, 2014, 6:13 am

≫ Next: Re: Am i able to use SFP+ 10Gbase-SR with MCX312A-XCBT adapter?

≪ Previous: Re: Am i able to use SFP+ 10Gbase-SR with MCX312A-XCBT adapter?

$

0

0

Thanks Andre,

 

My kernels are all pretty new and should all accept ofed 2.2 (CentOS 6.3 and 6.5, and also CentOS 6.2 that will be soon updated to 6.5). All supported according 2.2-1.0.1 release notes.

 

My problem comes with my cards. They are all 2~3 years old and there seems to be no more firmware updates for them. The mellanox ofed installer says they are up to date or cannot be updated with newer firmwares:

 

• Compute:

Image type:      ConnectX

FW Version:      2.9.1100

Device ID:       26428

PSID:            IBM08B0130009

 

Mellanox equivalent model: No match

max firm connectX2 = 2.9.1000

http://old.mellanox.com/content/pages.php?pg=firmware_table_IBM

Latest release: 08 aug 11

 

• Dell GPFS servers:

Image type:      ConnectX

FW Version:      2.9.1000

Device ID:       26428

PSID:            MT_0D90110009

 

Mellanox equivalent model:

Single 4X IB QDR Port, PCIe Gen2 x8, Tall Bracket, RoHS-R6 HCA Card, QSFP Connector

Max firmware available: 2.9.1000

Latest release:09 June 11

 

 

• IBM spare nodes

Image type:      FS2

FW Version:      2.11.500

Device ID:       4099

PSID:            IBM1020110028

 

Mellanox equivalent model:

MT_1020110028

ConnectX-3 VPI adapter card; dual-port QSFP; FDR10 IB (40Gb/s) and 10GigE; PCIe3.0x8 8GT/s; RoHS R6 PCI DevID: 4099

Max firmware available: 2.31.5050

http://old.mellanox.com/content/pages.php?pg=firmware_table_IBM

Max firmware available for the IBM OEM version: 2.11.0500

Latest release:15 Nov 12

 

 

While 2.2 release notes require:

- ConnectX-3: 2.31.5050 or above

- ConnectX-2: 2.9.1200 or above

 

But 1.5.3 requirements are fine:

- ConnectX-2 / ConnectX-2 EN (fw-ConnectX2 Rev 2.9.1000)

- ConnectX-3 VPI / ConnectX-3 EN (fw-4099 Rev 2.11.0500)

 

 

So, I suppose I should keep everything with 1.5.3, and install 2.2 in new nodes but setting them to CM while old nodes exist.

 

Thanks,

 

Txema

Re: Am i able to use SFP+ 10Gbase-SR with MCX312A-XCBT adapter?

July 11, 2014, 12:58 am

≫ Next: SX6036: Subnet Manager

≪ Previous: Re: Mixing OFED 1.5.3 and 2.2 in the same network?

$

0

0

I don't have details on FTLX8571D3BCL vs. FTLX8571D3BCL-ME testing, but I suspect that "-ME" stands for Mellanox. I don't think you will have problems with Finisar in general. BTW, firmware release notes is a good source for supported/tested cables and modules, see for example:

http://www.mellanox.com/pdf/firmware/ConnectX3Pro-FW-2_31_5050-release_notes.pdf

Thanks

Andre

SX6036: Subnet Manager

July 12, 2014, 3:49 pm

≫ Next: Re: Is there a way to show what SFP adapters are installed without pulling them from the card?

≪ Previous: Re: Am i able to use SFP+ 10Gbase-SR with MCX312A-XCBT adapter?

$

0

0

The SX6036 comes with an onboard subnet manager (capable of managing 648 nodes).

 

Am I correct in assuming that one of the nodes could be a SX6025 externally managed switch?

Re: Is there a way to show what SFP adapters are installed without pulling them from the card?

July 13, 2014, 4:42 am

≫ Next: Re: ubuntu14.04 infiniband set problems

≪ Previous: SX6036: Subnet Manager

$

0

0

Hello,

 

There isn't a tool for that as far as I know if the NIC runs on ethernet. if it is running on IB you can use the latest ibdiagnet tool that comes from ibutils2 and use the flag --get_cable_info

Re: ubuntu14.04 infiniband set problems

July 13, 2014, 4:49 am

≫ Next: Re: Big problem under CentOS 6.5 and the HP NC542m Network cards! I need help! I

≪ Previous: Re: Is there a way to show what SFP adapters are installed without pulling them from the card?

$

0

0

Hello,

 

Please try the following :

 

1.take node03 HCA-1 and run "ibstat" in order to get the Port GUID

2.run "ibping -S" to enter Server mode

3.From any other node in the fabric, use "ibping -G" and use node03 Port GUID as the argument.

 

Additionally, please make sure your perftest versions from both servers are identical

Re: Big problem under CentOS 6.5 and the HP NC542m Network cards! I need help! I

July 13, 2014, 5:30 am

≫ Next: Re: Big problem under CentOS 6.5 and the HP NC542m Network cards! I need help! I

≪ Previous: Re: ubuntu14.04 infiniband set problems

$

0

0

Mario,

this thread is very long with lots of details. i just stepped in and in am trying to understand the asset of things:

is you problem has to do with port mapping? means: are you having issues with ethX not consistently connecting to the same switch blade?

 

if this is the case, you may way to harden the association of ethX interface with udev device (with is eventually your physical MEZ card) - I've seen this happening on various operating systems before. Linux mostly.

 

i will follow up with you later on with specific guidelines on how to set the udev rules. you can also google this around as it is an operating system issue mostly.

 

cheers,

---

Re: Big problem under CentOS 6.5 and the HP NC542m Network cards! I need help! I

July 13, 2014, 5:55 am

≫ Next: Re: SX6036: Subnet Manager

≪ Previous: Re: Big problem under CentOS 6.5 and the HP NC542m Network cards! I need help! I

$

0

0

As stated above you can force the mapping of any nic to a specified ethX interface. once you have made sure you have the appropriate ifcfg-ethX script (and a set MAC) you can use udev to force the OS to map the mlx device and the ethX HCA :

 

/etc/udev/rules.d/70-persistent-net.rules:

 

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:02:c9:12:86:90", ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"

Reboot.

Re: SX6036: Subnet Manager

July 13, 2014, 9:22 am

≫ Next: Re: ubuntu14.04 infiniband set problems

≪ Previous: Re: Big problem under CentOS 6.5 and the HP NC542m Network cards! I need help! I

$

0

0

Thank you

Re: ubuntu14.04 infiniband set problems

July 13, 2014, 6:09 pm

≫ Next: Re: ubuntu14.04 infiniband set problems

≪ Previous: Re: SX6036: Subnet Manager

$

0

0

i run "ibping -S＂

ｉｔ　ｉｎｄｅｅｄ　ｃａｎ　ｐｉｎｇ　ｆｏｒｍ　ｏｔｈｅｒ　ｎｏｄｅｓ　ｔｏ　ｔｈｅ　ｓｅｔｕｐ　ｎｏｄｅ

ｂｕｔ　ｃａｎ＇ｔ　ｆｒｏｍ　ｔｈｅ　ｏｐｏｓｉｔｅ　ｓｉｄｅ

Re: ubuntu14.04 infiniband set problems

July 13, 2014, 11:50 pm

≫ Next: Re: ubuntu14.04 infiniband set problems

≪ Previous: Re: ubuntu14.04 infiniband set problems

$

0

0

ibping is a utility that uses a server and client type of of testing. if you want to ping the opposite side you will simply have to reverse the roles of the hosts.

Re: ubuntu14.04 infiniband set problems

July 14, 2014, 12:57 am

≫ Next: Remove infiniband related drivers?

≪ Previous: Re: ubuntu14.04 infiniband set problems

$

0

0