联系：手机/微信(+86 17813235971) QQ(107644445)
标题：硬件故障数据库异常恢复
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
硬件故障数据库crash
有客户由于硬件故障导致数据库异常ORA-00345 ORA-00312 ORA-27070 OSD-04016
Tue Feb 05 16:58:26 2019 Thread 1 advanced to log sequence 17139 (LGWR switch) Current log# 12 seq# 17139 mem# 0: S:\ORADATA\ORCL\REDO12A.LOG Current log# 12 seq# 17139 mem# 1: S:\ORADATA\ORCL\REDO12B.LOG Tue Feb 05 19:47:24 2019 Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_lgwr_2420.trc: ORA-00345: redo log write error block 152097 count 8 ORA-00312: online log 12 thread 1: 'S:\ORADATA\ORCL\REDO12A.LOG' ORA-27070: async read/write failed OSD-04016: 异步 I/O 请求排队时出错。 O/S-Error: (OS 1) 函数不正确。 ORA-00345: redo log write error block 152097 count 8 ORA-00312: online log 12 thread 1: 'S:\ORADATA\ORCL\REDO12B.LOG' ORA-27070: async read/write failed OSD-04016: 异步 I/O 请求排队时出错。 O/S-Error: (OS 1) 函数不正确。 ORA-00345: redo log write error block 152105 count 1 ORA-00312: online log 12 thread 1: 'S:\ORADATA\ORCL\REDO12A.LOG' ORA-27070: async read/write failed OSD-04016: 异步 I/O 请求排队时出错。 O/S-Error: (OS 1) 函数不正确。
直接启动数据库报错
修复好硬件之后,直接启动数据库报ORA-00600 kcratr_scan_lastbwr错误
Fri Feb 08 20:58:15 2019 alter database mount exclusive Successful mount of redo thread 1, with mount id 1527506791 Database mounted in Exclusive Mode Lost write protection disabled Completed: alter database mount exclusive alter database open Beginning crash recovery of 1 threads Started redo scan Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_ora_3672.trc (incident=41353): ORA-00600: ??????, ??: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], [] Incident details in: c:\oracle\diag\rdbms\orcl\orcl\incident\incdir_41353\orcl_ora_3672_i41353.trc Aborting crash recovery due to error 600 Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_ora_3672.trc: ORA-00600: ??????, ??: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], [] Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_ora_3672.trc: ORA-00600: ??????, ??: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], [] ORA-600 signalled during: alter database open... Fri Feb 08 20:58:24 2019 Trace dumping is performing id=[cdmp_20190208205824] Fri Feb 08 20:59:04 2019 alter database open Beginning crash recovery of 1 threads Started redo scan Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_ora_1696.trc (incident=41354): ORA-00600: 内部错误代码, 参数: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], [] Incident details in: c:\oracle\diag\rdbms\orcl\orcl\incident\incdir_41354\orcl_ora_1696_i41354.trc Aborting crash recovery due to error 600 Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_ora_1696.trc: ORA-00600: 内部错误代码, 参数: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], [] Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_ora_1696.trc: ORA-00600: 内部错误代码, 参数: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], [] ORA-600 signalled during: alter database open ...
recover database报错
执行recover database报错ORA-00600 6101,ORA-00600 kdourp_inorder2,ORA-00600 ktbsdp1,ORA-00600 3020
Fri Feb 08 21:09:20 2019 ALTER DATABASE RECOVER database Media Recovery Start started logmerger process Parallel Media Recovery started with 4 slaves Fri Feb 08 21:09:21 2019 Recovery of Online Redo Log: Thread 1 Group 12 Seq 17139 Reading mem 0 Mem# 0: S:\ORADATA\ORCL\REDO12A.LOG Mem# 1: S:\ORADATA\ORCL\REDO12B.LOG Fri Feb 08 21:09:21 2019 Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr02_3780.trc (incident=49379): ORA-00600: internal error code, arguments: [6101], [17], [21], [0], [], [], [], [], [], [], [], [] Incident details in: c:\oracle\diag\rdbms\orcl\orcl\incident\incdir_49379\orcl_pr02_3780_i49379.trc Fri Feb 08 21:09:21 2019 Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr01_2040.trc (incident=49371): ORA-00600: internal error code, arguments: [kdourp_inorder2], [34], [0], [0], [44], [], [], [], [], [], [], [] Incident details in: c:\oracle\diag\rdbms\orcl\orcl\incident\incdir_49371\orcl_pr01_2040_i49371.trc Fri Feb 08 21:09:21 2019 Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr03_1068.trc (incident=49387): ORA-00600: internal error code, arguments: [ktbsdp1], [], [], [], [], [], [], [], [], [], [], [] Incident details in: c:\oracle\diag\rdbms\orcl\orcl\incident\incdir_49387\orcl_pr03_1068_i49387.trc Fri Feb 08 21:09:24 2019 Trace dumping is performing id=[cdmp_20190208210924] Slave exiting with ORA-10562 exception Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr03_1068.trc: ORA-10562: Error occurred while applying redo to data block (file# 4, block# 1716972) ORA-10564: tablespace USERS ORA-01110: data file 4: 'S:\ORADATA\ORCL\USERS01.DBF' ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 204127 ORA-00600: internal error code, arguments: [ktbsdp1], [], [], [], [], [], [], [], [], [], [], [] Slave exiting with ORA-10562 exception Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr02_3780.trc: ORA-10562: Error occurred while applying redo to data block (file# 4, block# 1738552) ORA-10564: tablespace USERS ORA-01110: data file 4: 'S:\ORADATA\ORCL\USERS01.DBF' ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 211606 ORA-00600: internal error code, arguments: [6101], [17], [21], [0], [], [], [], [], [], [], [], [] Slave exiting with ORA-10562 exception Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr01_2040.trc: ORA-10562: Error occurred while applying redo to data block (file# 4, block# 1725898) ORA-10564: tablespace USERS ORA-01110: data file 4: 'S:\ORADATA\ORCL\USERS01.DBF' ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 73907 ORA-00600: internal error code, arguments: [kdourp_inorder2], [34], [0], [0], [44], [], [], [], [], [], [], [] Recovery Slave PR03 previously exited with exception 10562 Fri Feb 08 21:09:28 2019 Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr04_2608.trc (incident=49395): ORA-00600: internal error code, arguments: [3020], [4], [1739291], [18516507], [], [], [], [], [], [], [], [] ORA-10567: Redo is inconsistent with data block (file# 4, block# 1739291, file offset is 1363369984 bytes) ORA-10564: tablespace USERS ORA-01110: data file 4: 'S:\ORADATA\ORCL\USERS01.DBF' ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 211552 Incident details in: c:\oracle\diag\rdbms\orcl\orcl\incident\incdir_49395\orcl_pr04_2608_i49395.trc Slave exiting with ORA-600 exception Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr04_2608.trc: ORA-00600: internal error code, arguments: [3020], [4], [1739291], [18516507], [], [], [], [], [], [], [], [] ORA-10567: Redo is inconsistent with data block (file# 4, block# 1739291, file offset is 1363369984 bytes) ORA-10564: tablespace USERS ORA-01110: data file 4: 'S:\ORADATA\ORCL\USERS01.DBF' ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 211552 Media Recovery failed with error 448 Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr00_1548.trc: ORA-00283: recovery session canceled due to errors ORA-00448: normal completion of background process Slave exiting with ORA-283 exception Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr00_1548.trc: ORA-00283: recovery session canceled due to errors ORA-00448: normal completion of background process ORA-10562 signalled during: ALTER DATABASE RECOVER database ...
出现上述问题主要是由于硬件突然故障,数据写丢失导致相关问题.
处理思路
RMAN> recover datafile 1; 启动 recover 于 09-2月 -19 使用通道 ORA_DISK_1 正在开始介质的恢复 介质恢复完成, 用时: 00:00:01 完成 recover 于 09-2月 -19 RMAN> recover datafile 2; 启动 recover 于 09-2月 -19 使用通道 ORA_DISK_1 正在开始介质的恢复 介质恢复完成, 用时: 00:00:01 完成 recover 于 09-2月 -19 RMAN> recover datafile 3; 启动 recover 于 09-2月 -19 使用通道 ORA_DISK_1 正在开始介质的恢复 介质恢复完成, 用时: 00:00:02 完成 recover 于 09-2月 -19 RMAN> recover datafile 4; 启动 recover 于 09-2月 -19 使用通道 ORA_DISK_1 正在开始介质的恢复 无法恢复介质 RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: recover 命令 (在 02/09/2019 21:48:19 上) 失败 ORA-00283: recovery session canceled due to errors RMAN-11003: 在分析/执行 SQL 语句期间失败: alter database recover if needed datafile 4 ORA-00283: 恢复会话因错误而取消 ORA-10562: Error occurred while applying redo to data block (file# 4, block# 172 5913) ORA-10564: tablespace USERS ORA-01110: 数据文件 4: 'S:\ORADATA\ORCL\USERS01.DBF' ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 73907 ORA-00600: 内部错误代码, 参数: [kdourp_inorder2], [34], [43], [44], [44], [], [] , [], [], [], [], [] SQL> recover datafile 4; ORA-00283: 恢复会话因错误而取消 ORA-10562: Error occurred while applying redo to data block (file# 4, block# 1725913) ORA-10564: tablespace USERS ORA-01110: 数据文件 4: 'S:\ORADATA\ORCL\USERS01.DBF' ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 73907 ORA-00600: 内部错误代码, 参数: [kdourp_inorder2], [34], [43], [44], [44], [], [], [], [], [], [], [] --通过bbed修改异常文件,屏蔽文件恢复,直接open库 SQL> alter database open; 数据库已更改。
数据库open之后,逻辑方式导出数据,重建新库,导入数据.
