公司11gRac环境出现问题，数据库不能mount,连接上去看了下
[oracle@bcrac12 ~]$ sqlplus / AS sysdba SQL*Plus: Release 11.2.0.3.0 Production ON Thu Sep 4 11:30:42 2014 Copyright (c) 1982, 2011, Oracle. ALL rights reserved. Connected TO: Oracle DATABASE 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production WITH the Partitioning, REAL Application Clusters, OLAP, DATA Mining AND REAL Application Testing options SQL> SELECT open_mode FROM v$database; SELECT open_mode FROM v$database * ERROR at line 1: ORA-01507: DATABASE NOT mounted SQL> ALTER databasem mount; ALTER databasem mount * ERROR at line 1: ORA-00940: invalid ALTER command SQL> ALTER DATABASE mount; ALTER DATABASE mount * ERROR at line 1: ORA-00600: internal error code, arguments: [kccsbck_first], [1], [853858596], [], [], [], [], [], [], [], [], [] |
以为两个节点参数不一致导致
连接到节点2上查看，参数都一样，忽然发现通过两个ip地址连接上来的计算机名一样
[oracle@bcrac12 ~]$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.85 bcrac11
192.168.0.87 bcrac12
10.0.0.86 bcrac11-priv
10.0.0.88 bcrac12-priv
192.168.0.86 bcrac11-vip
192.168.0.88 bcrac12-vip
192.168.0.82 racscan
[oracle@bcrac12 ~]$ ping bcrac12
PING bcrac12 (192.168.0.87) 56(84) bytes of data.
64 bytes from bcrac12 (192.168.0.87): icmp_seq=1 ttl=64 time=0.016 ms
64 bytes from bcrac12 (192.168.0.87): icmp_seq=2 ttl=64 time=0.022 ms
[oracle@bcrac12 ~]$ ifconfig
eth1 Link encap:Ethernet HWaddr 52:54:00:41:B4:83
inet addr:192.168.0.87 Bcast:192.168.255.255 Mask:255.255.0.0
inet6 addr: fe80::5054:ff:fe41:b483/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2309967 errors:0 dropped:352826 overruns:0 frame:0
TX packets:291719 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:302406495 (288.3 MiB) TX bytes:43811251 (41.7 MiB)
eth1:1 Link encap:Ethernet HWaddr 52:54:00:41:B4:83
inet addr:192.168.0.82 Bcast:192.168.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth1:2 Link encap:Ethernet HWaddr 52:54:00:41:B4:83
inet addr:192.168.0.85 Bcast:192.168.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth2 Link encap:Ethernet HWaddr 52:54:00:0E:2E:48
inet addr:10.0.0.88 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe0e:2e48/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:18170303 errors:0 dropped:352826 overruns:0 frame:0
TX packets:16977728 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:11969135166 (11.1 GiB) TX bytes:19066312409 (17.7 GiB)
eth2:1 Link encap:Ethernet HWaddr 52:54:00:0E:2E:48
inet addr:169.254.74.94 Bcast:169.254.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:3710613 errors:0 dropped:0 overruns:0 frame:0
TX packets:3710613 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2348775015 (2.1 GiB) TX bytes:2348775015 (2.1 GiB) |
发现两个public ip都在一个节点上。手工停止到节点2能正常连接到节点1上，顺手把节点2的给重启了，我擦public又漂移到节点1了
[oracle@bcrac11 ~]$ ifconfig
eth1 Link encap:Ethernet HWaddr 52:54:00:6D:90:0B
inet addr:192.168.0.85 Bcast:192.168.255.255 Mask:255.255.0.0
inet6 addr: fe80::5054:ff:fe6d:900b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2599998 errors:0 dropped:353225 overruns:0 frame:0
TX packets:515571 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:647650276 (617.6 MiB) TX bytes:95190214 (90.7 MiB)
eth1:1 Link encap:Ethernet HWaddr 52:54:00:6D:90:0B
inet addr:192.168.0.82 Bcast:192.168.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth1:2 Link encap:Ethernet HWaddr 52:54:00:6D:90:0B
inet addr:192.168.0.87 Bcast:192.168.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth2 Link encap:Ethernet HWaddr 52:54:00:24:26:01
inet addr:10.0.0.86 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe24:2601/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:19014819 errors:0 dropped:353238 overruns:0 frame:0
TX packets:16168605 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:19330567969 (18.0 GiB) TX bytes:11735179732 (10.9 GiB)
eth2:1 Link encap:Ethernet HWaddr 52:54:00:24:26:01
inet addr:169.254.98.184 Bcast:169.254.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:5225101 errors:0 dropped:0 overruns:0 frame:0
TX packets:5225101 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4822155302 (4.4 GiB) TX bytes:4822155302 (4.4 GiB)
[oracle@bcrac11 ~]$ exit |
节点1服务正常
[grid@bcrac11 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE bcrac11
ora.LISTENER.lsnr
ONLINE ONLINE bcrac11
ora.OCR.dg
ONLINE ONLINE bcrac11
ora.asm
ONLINE ONLINE bcrac11 Started
ora.gsd
OFFLINE OFFLINE bcrac11
ora.net1.network
ONLINE ONLINE bcrac11
ora.ons
ONLINE ONLINE bcrac11
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE bcrac11
ora.bcrac11.vip
1 ONLINE ONLINE bcrac11
ora.bcrac12.vip
1 ONLINE INTERMEDIATE bcrac11 FAILED OVER
ora.cvu
1 ONLINE ONLINE bcrac11
ora.oc4j
1 ONLINE ONLINE bcrac11
ora.racdb.db
1 ONLINE ONLINE bcrac11 Open
2 ONLINE OFFLINE
ora.scan1.vip
1 ONLINE ONLINE bcrac11 |
通过ifconfig down掉子网ip
[root@bcrac11 ~]# ifdown eth1:2
usage: ifdown <device name>
[root@bcrac11 ~]# ifconfig eth1:2
eth1:2 Link encap:Ethernet HWaddr 52:54:00:6D:90:0B
inet addr:192.168.0.87 Bcast:192.168.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
[root@bcrac11 ~]# ifconfig eth1:2 down
[root@bcrac11 ~]# ifconfig
eth1 Link encap:Ethernet HWaddr 52:54:00:6D:90:0B
inet addr:192.168.0.85 Bcast:192.168.255.255 Mask:255.255.0.0
inet6 addr: fe80::5054:ff:fe6d:900b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2601182 errors:0 dropped:353308 overruns:0 frame:0
TX packets:515759 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:647761299 (617.7 MiB) TX bytes:95223061 (90.8 MiB)
eth1:1 Link encap:Ethernet HWaddr 52:54:00:6D:90:0B
inet addr:192.168.0.82 Bcast:192.168.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth2 Link encap:Ethernet HWaddr 52:54:00:24:26:01
inet addr:10.0.0.86 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe24:2601/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:19018024 errors:0 dropped:353321 overruns:0 frame:0
TX packets:16170847 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:19333333709 (18.0 GiB) TX bytes:11736318494 (10.9 GiB)
eth2:1 Link encap:Ethernet HWaddr 52:54:00:24:26:01
inet addr:169.254.98.184 Bcast:169.254.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:5225734 errors:0 dropped:0 overruns:0 frame:0
TX packets:5225734 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4822748419 (4.4 GiB) TX bytes:4822748419 (4.4 GiB) |
通过priv网段登录到节点2 重启网卡服务后集群正常
[root@bcrac12 ~]# /u01/11.2.0/grid/bin/crsctl status res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE bcrac11
ONLINE ONLINE bcrac12
ora.LISTENER.lsnr
ONLINE ONLINE bcrac11
ONLINE ONLINE bcrac12
ora.OCR.dg
ONLINE ONLINE bcrac11
ONLINE ONLINE bcrac12
ora.asm
ONLINE ONLINE bcrac11 Started
ONLINE ONLINE bcrac12 Started
ora.gsd
OFFLINE OFFLINE bcrac11
OFFLINE OFFLINE bcrac12
ora.net1.network
ONLINE ONLINE bcrac11
ONLINE ONLINE bcrac12
ora.ons
ONLINE ONLINE bcrac11
ONLINE ONLINE bcrac12
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE bcrac11
ora.bcrac11.vip
1 ONLINE ONLINE bcrac11
ora.bcrac12.vip
1 ONLINE ONLINE bcrac12
ora.cvu
1 ONLINE ONLINE bcrac11
ora.oc4j
1 ONLINE ONLINE bcrac11
ora.racdb.db
1 ONLINE ONLINE bcrac11 Open
2 ONLINE ONLINE bcrac12 Open
ora.scan1.vip
1 ONLINE ONLINE bcrac11 |
