现在服务器使用infiniband的越来越多,因为infiniband本身能提供比传统以太网更高的带宽， infiniband的开销比以太网要小，对于节点间通信大量数据传输比以太网效率要更高。这几天搞了下~遇到Mellanox和思科交换机连接的一些小问题.查看文档简单知道了下infiniband一些基本信息。
infiniband卡(HCA)和交换机互联有那些线:
- QSFP 铜缆线
- QSFP 光缆线
- QSFP 光转发器

这里看下思路连接线分类
Product Type Connector Type Cisco QSFP-40G-SR-BD 40GBASE-SR-BiDi, duplex MMF LC Cisco QSFP-40G-SR4 40GBASE-SR4, 4lanes, 850 nm MMF MPO-12 Cisco FET-40G Fabric Extender, 4lanes, 850 nm MMF MPO-12 Cisco QSFP-40G-CSR4 40GBASE-CSR4, 4 lanes, 850 nm MMF MPO-12 Cisco WSP-Q40GLR4L 40GBASE-LR4-Lite, 1310 nm, SMF LC Cisco QSFP-40GE-LR4 40GBASE-LR4, 1310 nm, SMF LC Cisco QSFP-40G-LR4 40GBASE-LR4, 1310 nm, SMF with OTU3 data-rate support LC Cisco QSFP-4SFP10G-CU (1M, 3M, 5M) QSFP to 4 SFP+ copper break-out cables - Cisco QSFP-4X10G-AC (7M, 10M) Cisco QSFP-H40G-CU (1M, 3M, 5M) QSFP to QSFP copper direct-attach cables Cisco QSFP-H10G-ACU (7M, 10M) QSFP-4X10G-AOC (1M, 2M, 3M, 5M, 7M, 10M) QSFP to four SFP+ active optical breakout cables QSFP-H40G-AOC (1M, 2M, 3M, 5M, 7M, 10M, 15M) QSFP to QSFP active optical cables |
从上面的可以看出
QSFP-QSFP的铜缆线最长支持3M
QSFP-QSFP的光缆线最长支持10M
如果距离大于10M,这里就需要借助QSFP 光转发器+MPO/MTO线进行配合
所以大家在采购Infiniband相关配件的时候,最好做好规划。以免后期安装麻烦。
客户采购的是Mellanox的卡和线交换机是思科的,这里就遇到问题,思科的交换机默认之人自己的模块。相应的信息如下
xxxx(config)# show inter e2/1
Ethernet2/1 is down (Transceiver validation failed)
admin state is up, Dedicated Interface
Hardware: 10000/40000 Ethernet, address: 88f0.31bd.4602 (bia 88f0.31bd.4602)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is access
auto-duplex, 40 Gb/s, media type is 40G
Beacon is turned off
Auto-Negotiation is turned off
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped never
Last clearing of "show interface" counters never
0 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
RX
0 unicast packets 0 multicast packets 0 broadcast packets
0 input packets 0 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 0 multicast packets 0 broadcast packets
0 output packets 0 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
xxxx(config)#show inter trans
Ethernet2/1
transceiver is present
type is QSFP-40G-CR4
name is Mellanox -- mekkanox的线
part number is 00KF018
revision is A1
serial number is 2F0184B405W
nominal bitrate is 14100 MBit/sec per channel
Link length supported for copper is 3 m
cisco id is --
cisco extended id number is 0
Ethernet2/7
transceiver is present
type is QSFP-40G-SR4
name is CISCO --思科的光转发器
part number is FTL410QE2C-C1
revision is B
serial number is FIS183101DM
nominal bitrate is 10300 MBit/sec per channel
Link length supported for 50/125um OM2 fiber is 30 m
Link length supported for 50/125um OM3 fiber is 100 m
cisco id is --
cisco extended id number is 16 |
