- 论坛徽章:
- 0
|
Applies To Operating Systems/Solaris/Solaris 2.x
Attachments (none)
Document Content INFODOC ID: 15572
Question:
Can I configure two ethernet interfaces on the same subnet?
Answer:
The answer is Yes and No! Here are the problems you will encounter and some of the possible solutions. First clarify the
configuration. This document refers to a configuration similar to the one below where two interfaces are connected to the same physical subnet with
different host addresses. It does NOT refer to two interfaces with different network numbers (i.e., acting as a router) which is the more usual
configuration and does not have any issues.
+------------+ +-----------------------+
| | | |
| | | host |
| host | | |
| p4u-14a | | |
| | | p4u-3000a |
| | | |
| | | |
+------------+ +-----------------------+
| le0 | hme0 | hme1
| (192.156.183.20) |(129.156.183.17) | (129.156.183.20
| | |
| | |
| | |
|===================================================================|
This is subnet 129.156.183.0 with a default netmask of 255.255.255.0.
Why would you want to do this?
There are two reasons:
1. For redundancy - If hme0 on the server breaks, you can still access it via hme1.
2. To provide extra bandwidth to the server - The above example is drawn as shared ethernet (i.e., 10base5 coax).
If the network were an ethernet switch, you could, in theory, have a 10Mbit link to each interface.
What are the problems?
1. Current ethernet cards from Sun do not have their own MAC or Ethernet address. Each interface inherits a MAC
address derived from the HOSTID of the system. This means that each interface has the SAME MAC address.
This IS a perfectly legal thing to do. It's called Host Addressing or something similar.
This is fine when the interfaces are attached to different subnets when the Sun is configured as a router. However, when
the interfaces are connected to the same subnet we have a problem because the MAC address is used at the Datalink
Layer to send packets to the correct destination. If you have two destinations with the same address, it won't work!
The usual symptoms are very poor performance (if it works at all).
If you look at the ARP table of a client (type arp -a), you will see something like this:
Device IP Address Mask Flags Phys Addr
------ -------------------- --------------- ----- ---------------
be0 podsix 255.255.255.255 08:00:20:19:69:ca
be0 clem 255.255.255.255 08:00:20:1c:b6:b4
be0 whippet 255.255.255.255 08:00:20:11:4a:fd
be0 pingpong 255.255.255.255 08:00:20:19:0c:d3
be0 peezero 255.255.255.255 08:00:20:1c:b5:9b
be0 skid 255.255.255.255 08:00:20:71:de:c6
.
.
etc
If the Phys Addr (MAC) appears against more than one hostname, or a host does not have a MAC address in the table
this might be the problem.
If you type ifconfig -a as root on the server, you will see the MAC address allocated to each interface:
lo0: flags=849 mtu 8232
inet 127.0.0.1 netmask ff000000
hme0: flags=863 mtu 1500
inet 129.156.183.17 netmask ffffff00 broadcast 129.156.183.255
ether 8:0:1:1:1:6c
hme1: flags=863 mtu 1500
inet 129.156.183.208 netmask ffffff00 broadcast 129.156.183.255
ether 8:0:0:0:0:6c
In the above example, the MAC addresses are different. They would be the same by default. You can change them
with the ifconfig command:
ifconfig hme1 ether 8:0:0:0:0:6c
You have to choose an address which does not exist on this subnet (i.e., make one up !). Do not use an address that
starts with 01:00:5e: as this is the multicast address.
To make the changes permanent add something like the following to /etc/rcS.d/S30rootusr.sh:
/sbin/ifconfig lo0 127.0.0.1 up 2>;&1 >;/dev/null
/sbin/ifconfig hme0 plumb
/sbin/ifconfig hme0 ether 8:0:1:1:1:6c
/sbin/ifconfig hme1 plumb
/sbin/ifconfig hme1 ether 8:0:0:0:0:6c
2. Now that you have different MAC addresses, your client can connect to either of the IP addresses shown in the drawing and
get a connection. However, look closely at what's happening (with snoop -v). You will see that the packets
going to 129.156.183.17 have a destination MAC address of 8:0:1:1:1:6c and packets going to 129.156.183.208 have a
MAC address of 8:0:0:0:0:6c, but ALL the packets comming FROM the server have a source MAC address of 8:0:1:1:1:6c.
This means that the ingoing packets are going to both hme0 and hme1 but all the packets leaving the server are going
via hme0 only.
This defeats the objectives of configuring two interfaces on the same subnet! There is a filter for snoop which shows
this quite clearly (See mac_snoop_filter). Why does this happen? If you take a look at the routing table on the server,
you will see:
# netstat -rn
Routing Table:
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ------ ---------
127.0.0.1 127.0.0.1 UH 0 0 lo0
129.156.183.0 129.156.183.17 U 3 12 hme0
129.156.183.0 129.156.183.208 U 2 0 hme1
224.0.0.0 129.156.183.17 U 3 0 hme0
default 129.156.183.2 UG 0 27
Notice that there are two routes to the local network 129.156.183.0. These are added when the interface is plumbed in.
Because hme0 was plumbed first, it is first in the routing table and used before the other interface. If the hme0 interface
had failed, you could remove the route to this interface manually or by configuring the interface down, and then the
other route would be used.
To overcome this problem you can do the following, but it's not recommended. Remove the routes to the local
net:
route delete 129.156.183.0 129.156.183.17 0
route delete 129.156.183.0 129.156.183.208 0
Now add routes to each host on the local subnet via specific interfaces;
route add 129.156.183.1 129.156.183.17 0
route add 129.156.183.2 129.156.183.17 0
route add 129.156.183.3 129.156.183.17 0
route add 129.156.183.4 129.156.183.17 0
route add 129.156.183.5 129.156.183.17 0
.
etc
route add 129.156.183.20 129.156.183.208 0
route add 129.156.183.21 129.156.183.208 0
route add 129.156.183.22 129.156.183.208 0
route add 129.156.183.23 129.156.183.208 0
route add 129.156.183.24 129.156.183.208 0
route add 129.156.183.25 129.156.183.208 0
etc
This will achieve a sort of load balancing by routing packets from some hosts to one interface and the packets to
another host via another.
This won't achieve any interface redundancy because the broadcasts can only go out of one interface and that might
be the one that is broken (so ARP might not work). Also, the machine might rely on another system for NIS+/NFS,
which can't be contacted because the interface that this machine is reached from is not there.
There is no automatic switchover!
Rob Cotter and Tom Bauer have written further information on Network Load-Sharing with Multihomed Servers on Ethernet
Switches.
What are interface groups?
This is new in Solaris[TM] 2.6 and overcomes the limitations of the above. When a connection is made to the server, an IP cache
entry is made which contains the interface that the client connected to. All packets going to that client will leave the server
via that interface. The cache entry is valid for the length of the ARP entry for that host.
If the same client now connects to the other interface of the server, the return packets will leave via the interface in the cache,
not via the interface they arrived on.
Another client which connects to the server's second interface will have its return packets leave via the second interface.
This will give good load balancing over a number of interfaces.
If one interface fails, then any client server sessions that go through that interface will hang. If the client has
a session through the interface that is still working, then the interface will not hang unless the server is dependent on a service
that is using the failed interface. Once the cache entry expires, a new connection from the client to the server through the
remaining interface will create a new cache entry for this host.
If a session was hung because the packets back to the client were leaving via the failed interface but were arriving via the
working interface, then the new cache entry will make the packets leave via the working interface instead.
Of course, any sessions where the packets arrived on the failed interface will still be hung as the client can't contact the
interface. However, it could start a new connection via the working interface.
This is illustrated by the following snoops:
Client "p4u-14a" connects to "p4u-3000a" interface hme0 - return packets from same interface
(29)MAC: p4u-14a -->; BROADCAST ARP:ETHER:
(30)MAC: p4u-3000a-hme0 -->; p4u-14a ARP:ETHER:
(31)MAC: p4u-14a -->; p4u-3000a-hme0 IP: 129.156.183.19, -->; 129.156.183.17, TCP:TELNET:ETHER:
(32)MAC: p4u-3000a-hme0 -->; p4u-14a IP: 129.156.183.17, -->; 129.156.183.19, TCP:TELNET:ETHER:
(33)MAC: p4u-14a -->; p4u-3000a-hme0 IP: 129.156.183.19, -->; 129.156.183.17, TCP:TELNET:ETHER:
Client "p4u-14a" connects to "p4u-3000a" interface hme1 - return packets via hme0
(230)MAC: p4u-14a -->; BROADCAST ARP:ETHER:
(231)MAC: p4u-3000a-hme1 -->; p4u-14a ARP:ETHER:
(232)MAC: p4u-14a -->; p4u-3000a-hme1 IP: 129.156.183.19, -->; 129.156.183.208, TCP:TELNET:ETHER:
(233)MAC: p4u-3000a-hme0 -->; p4u-14a IP: 129.156.183.208, -->; 129.156.183.19, TCP:TELNET:ETHER:
(234)MAC: p4u-14a -->; p4u-3000a-hme1 IP: 129.156.183.19, -->; 129.156.183.208, TCP:TELNET:ETHER:
Client "p4u-12b" connects to "p4u-3000a" interface hme1 - return packets from same interface
(302)MAC: p4u-12b -->; BROADCAST ARP:ETHER:
(303)MAC: p4u-3000a-hme1 -->; p4u-12b ARP:ETHER:
(304)MAC: p4u-12b -->; p4u-3000a-hme1 IP: 129.156.183.92, -->; 129.156.183.208, TCP:TELNET:ETHER:
(305)MAC: p4u-3000a-hme1 -->; p4u-12b IP: 129.156.183.208, -->; 129.156.183.92, TCP:TELNET:ETHER:
(306)MAC: p4u-12b -->; p4u-3000a-hme1 IP: 129.156.183.92, -->; 129.156.183.208, TCP:TELNET:ETHER:
(307)MAC: p4u-12b -->; p4u-3000a-hme1 IP: 129.156.183.92, -->; 129.156.183.208, TCP:TELNET:ETHER:
Client "p4u-12b" connects to "p4u-3000a" interface hme0 - return packets hme1
(345)MAC: p4u-12b -->; BROADCAST ARP:ETHER:
(346)MAC: p4u-3000a-hme0 -->; p4u-12b ARP:ETHER:
(347)MAC: p4u-12b -->; p4u-3000a-hme0 IP: 129.156.183.92, -->; 129.156.183.17, TCP:TELNET:ETHER:
(34MAC: p4u-3000a-hme1 -->; p4u-12b IP: 129.156.183.17, -->; 129.156.183.92, TCP:TELNET:ETHER:
(349)MAC: p4u-12b -->; p4u-3000a-hme0 IP: 129.156.183.92, -->; 129.156.183.17, TCP:TELNET:ETHER:
(350)MAC: p4u-12b -->; p4u-3000a-hme0 IP: 129.156.183.92, -->; 129.156.183.17, TCP:TELNET:ETHER:
(351)MAC: p4u-3000a-hme1 -->; p4u-12b IP: 129.156.183.17, -->; 129.156.183.92, TCP:TELNET:ETHER:
Interface hme0 is unplugged! Client p4u-14a is hung !
(377)MAC: p4u-14a -->; p4u-3000a-hme1 IP: 129.156.183.19, -->; 129.156.183.208, TCP:TELNET:ETHER:
(37MAC: p4u-14a -->; p4u-3000a-hme0 IP: 129.156.183.19, -->; 129.156.183.17, TCP:TELNET:ETHER:
(379)MAC: p4u-14a -->; p4u-3000a-hme0 IP: 129.156.183.19, -->; 129.156.183.17, TCP:TELNET:ETHER:
(380)MAC: p4u-14a -->; BROADCAST ARP:ETHER:
(381)MAC: p4u-14a -->; p4u-3000a-hme1 IP: 129.156.183.19, -->; 129.156.183.208, TCP:TELNET:ETHER:
(382)MAC: p4u-3000a-hme1 -->; p4u-14a ARP:ETHER:
(383)MAC: p4u-14a -->; p4u-3000a-hme0 IP: 129.156.183.19, -->; 129.156.183.17, TCP:TELNET:ETHER:
(384)MAC: p4u-14a -->; p4u-3000a-hme0 IP: 129.156.183.19, -->; 129.156.183.17, TCP:TELNET:ETHER:
(385)MAC: p4u-14a -->; p4u-3000a-hme0 IP: 129.156.183.19, -->; 129.156.183.17, TCP:TELNET:ETHER:
(386)MAC: p4u-14a -->; p4u-3000a-hme0 IP: 129.156.183.19, -->; 129.156.183.17, TCP:TELNET:ETHER:
After timeout and a new session forces a new cache entry:
(393)MAC: p4u-14a -->; p4u-3000a-hme1 IP: 129.156.183.19, -->; 129.156.183.208, TCP:TELNET:ETHER:
(394)MAC: p4u-3000a-hme1 -->; p4u-14a IP: 129.156.183.208, -->; 129.156.183.19, TCP:TELNET:ETHER:
(395)MAC: p4u-14a -->; p4u-3000a-hme1 IP: 129.156.183.19, -->; 129.156.183.208, TCP:TELNET:ETHER:
Note: Experimentation has shown the cache entry recovers more quickly if you try and
connect to the working interface from the "hung" client
This feature is turned on by default in 2.6. To turn off the feature:
ndd -set /dev/ip ip_enable_groups_ifs 0
|
|
免费注册
发表于 2003-01-11 15:54
发表于 2003-01-14 16:45
