RMAN备份+Tivoli软件+ L25带库备份时出现了故障，请专家过来看一下

sucrolee

这个问题让我头痛好多天了
具体情况是这样的
本来这里备份跑得很好。后来由我接手，我刚接触这个，麻烦来了。
有几天在删除大量的数据，所以数据库很慢，RMAN跑全备和归档备份，但跑全备是每天都跑，以前还行，但那几天删除大量数据的时候，一天全备都跑不完，归档备份也没有跑完（当然也不排除这个时候问题就已经出现了），后来我看了一下有8个RMAN进程。
于是我从ORACLE中删除了所有的RMAN的SESSION，然后又在web界面上把这几个备份的session也cancel了。
后来继续备份，但可怜的是备份还是不行。 （只是做归档备份）
后来重启了tivoli. 但备份还是做不了。
现在备份还跑不了，归档日志已经很多，相信过不了几天，硬盘就会撑爆了。希望有兄弟能给我指点一下迷津。
RMAN的日志
input archive log thread=1 sequence=15548 recid=15543 stamp=626311567
channel sbt1: starting piece 1 at 01-JUL-07
released channel: sbt1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on sbt1 channel at 07/01/2007 12:00:44
ORA-19502: write error on file "arch_7714_1", blockno 198657 (blocksize=512)
ORA-27030: skgfwrt: sbtwrite2 returned error
ORA-19511: Error received from media manager layer, error text:
   ANS0266I (RC2302) The dsmEndTxn vote is ABORT, so check the reason field.
Recovery Manager complete.

tivoli的相关日志
[email=root@PDBKSERV]root@PDBKSERV[/email] # /etc/init.d/tivoli  start
[email=root@PDBKSERV]root@PDBKSERV[/email] # ANR0900I Processing options file dsmserv.opt.
Tivoli Storage Manager for Sun Solaris 8/9
Version 5, Release 2, Level 3.0
Licensed Materials - Property of IBM
(C) Copyright IBM Corporation 1990, 2003. All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.
ANR7800I DSMSERV generated at 09:51:57 on Jun 22 2004.
ANR7801I Subsystem process ID is 27510.
ANR0990I Server restart-recovery in progress.
[email=root@PDBKSERV]root@PDBKSERV[/email] # ANR0200I Recovery log assigned capacity is 112 megabytes.
ANR0201I Database assigned capacity is 212 megabytes.
ANR0306I Recovery log volume mount in progress.
[email=root@PDBKSERV]root@PDBKSERV[/email] # ANR0353I Recovery log analysis pass in progress.
ANR0354I Recovery log redo pass in progress.
ANR0355I Recovery log undo pass in progress.
ANR0352I Transaction recovery complete.
ANR1635I The server machine GUID, 36.86.ff.70.a8.26.11.da.ad.11.0a.19.04.15.aa.77, has initialized.
ANR2100I Activity log process has started.
ANR4726I The NAS-NDMP support module has been loaded.
ANR4726I The Centera support module has been loaded.
ANR8300E I/O error on library LIBL25 (OP=40146C01, CC=203, KEY=02, ASC=80, ASCQ=01,
SENSE=70.00.02.00.00.00.00.0E.00.00.00.00.80.0
01.00.00.00.00.00.00.,
Description=Manual intervention required).  Refer to Appendix D in the 'Messages' manual for recommended ac
ction.
ANR1412W Volume KWD902L2 access mode is "unavailable".
ANR1412W Volume KWD903L2 access mode is "unavailable".
ANR1412W Volume KWD904L2 access mode is "unavailable".
ANR1414W Volume KWD907L2 access mode is "read-only" due to previous write error.
ANR8440E Initialization failed for SCSI library LIBL25; will retry in 2 minute(s).
ANR1412W Volume KWD915L2 access mode is "unavailable".
ANR1305I Disk volume /opt/tivoli/tsm/server/bin/sarch00.dsm varied online.
ANR1305I Disk volume /opt/tivoli/tsm/server/bin/sbkup00.dsm varied online.
ANR2803I License manager started.
ANR8200I TCP/IP driver ready for connection with clients on port 1500.
ANR8285I Shared Memory driver ready for connection with clients on port 1510
ANR8190I HTTP driver ready for connection with clients on port 1580.
ANR2560I Schedule manager started.
ANR0993I Server initialization complete.
ANR0916I TIVOLI STORAGE MANAGER distributed by Tivoli is now ready for use.
ANR2827I Server is licensed to support Tivoli Space Manager for a quantity of 20.
ANR2860I Server is licensed to support Tivoli Disaster Recovery Manager.
ANR2827I Server is licensed to support Tivoli Data Protection for Oracle for a quantity of 20.
ANR2827I Server is licensed to support Tivoli Data Protection for MS SQL Server for a quantity of 20.
ANR2827I Server is licensed to support Tivoli Data Protection for MS Exchange for a quantity of 20.
ANR2827I Server is licensed to support Tivoli Data Protection for Lotus Notes for a quantity of 20.
ANR2827I Server is licensed to support Tivoli Data Protection for Lotus Domino for a quantity of 20.
ANR2827I Server is licensed to support Tivoli Data Protection for Informix for a quantity of 20.
ANR2827I Server is licensed to support Tivoli Data Protection for R/3 for a quantity of 20.
ANR2827I Server is licensed to support Tivoli Data Protection for IBM ESS for a quantity of 20.
ANR2827I Server is licensed to support Tivoli Data Protection for IBM ESS R/3 for a quantity of 20.
ANR2827I Server is licensed to support Tivoli Data Protection for EMC Symmetrix for a quantity of 20.
ANR2827I Server is licensed to support Tivoli Data Protection for EMC Symmetrix R/3 for a quantity of 20.
ANR2828I Server is licensed to support Tape Library Sharing.
ANR2827I Server is licensed to support Managed System for LAN for a quantity of 20.
ANR2827I Server is licensed to support Managed System for SAN for a quantity of 20.
ANR2827I Server is licensed to support Managed Library for a quantity of 20.
ANR2828I Server is licensed to support Tivoli Data Protection for NDMP.
ANR2827I Server is licensed to support Tivoli Data Protection for WAS for a quantity of 20.
ANR2828I Server is licensed to support Tivoli Storage Manager for Data Retention.
TSM:SERVER1>
[email=root@PDBKSERV]root@PDBKSERV[/email] #
[email=root@PDBKSERV]root@PDBKSERV[/email] #
[email=root@PDBKSERV]root@PDBKSERV[/email] # ANR0407I Session 1 started for administrator ADMIN (WebBrowser) (HTTP 10.25.4.50(350).
ANR0405I Session 1 ended for administrator ADMIN (WebBrowser).
ANR2017I Administrator ADMIN issued command: QUERY PROCESS
ANR0406I Session 28 started for node PD_CCB_ORA (TDP Oracle SUN) (Tcp/Ip 192.1.1.5(35559)).
ANR0403I Session 28 ended for node PD_CCB_ORA (TDP Oracle SUN).
ANR2017I Administrator ADMIN issued command: QUERY SESSION
ANR0406I Session 40 started for node PD_CCB_ORA (TDP Oracle SUN) (Tcp/Ip 192.1.1.5(35581)).
ANR0403I Session 40 ended for node PD_CCB_ORA (TDP Oracle SUN).
ANR2017I Administrator ADMIN issued command: QUERY SESSION
ANR2750I Starting scheduled command BKUPDB ( backup db type=full devclass=LTO2DEV volumenames=KWD900L2,KWD901L2 scratch=no ).
ANR2017I Administrator ADMIN issued command: BACKUP DB type=full devclass=LTO2DEV volumenames=KWD900L2,KWD901L2 scratch=no
ANR0984I Process 1 for DATABASE BACKUP started in the BACKGROUND at 09:30:26.
ANR2753I (BKUPDB):ANR2280I Full database backup started as process 1.
ANR2756I Scheduled command BKUPDB started successfully.
ANR1409W Volume KWD900L2 already in use - skipped.
ANR1409W Volume KWD901L2 already in use - skipped.
ANR4583E Database backup terminated - output media full and scratch media could not be mounted.
ANR0985I Process 1 for DATABASE BACKUP running in the BACKGROUND completed with completion state FAILURE at 09:30:28.
ANR8300E I/O error on library LIBL25 (OP=40146C01, CC=203, KEY=02, ASC=80, ASCQ=01,
SENSE=70.00.02.00.00.00.00.0E.00.00.00.00.80.0
01.00.00.00.00.00.00.,
Description=Manual intervention required).  Refer to Appendix D in the 'Messages' manual for recommended ac
ction.
ANR8441E Initialization failed for SCSI library LIBL25.
ANR0407I Session 51 started for administrator ADMIN (WebBrowser) (HTTP 10.25.4.50(3555)).
ANR0405I Session 51 ended for administrator ADMIN (WebBrowser).
ANR2017I Administrator ADMIN issued command: QUERY PROCESS
ANR0406I Session 82 started for node PD_CCB_ORA (TDP Oracle SUN) (Tcp/Ip 192.1.1.5(35712)).
ANR8300E I/O error on library LIBL25 (OP=40146C01, CC=203, KEY=02, ASC=80, ASCQ=01,
SENSE=70.00.02.00.00.00.00.0E.00.00.00.00.80.0
01.00.00.00.00.00.00.,
Description=Manual intervention required).  Refer to Appendix D in the 'Messages' manual for recommended ac
ction.
ANR8441E Initialization failed for SCSI library LIBL25.
ANR1401W Mount request denied for volume KWD906L2 - mount failed.
ANR8300E I/O error on library LIBL25 (OP=40146C01, CC=203, KEY=02, ASC=80, ASCQ=01,
SENSE=70.00.02.00.00.00.00.0E.00.00.00.00.80.0
01.00.00.00.00.00.00.,
Description=Manual intervention required).  Refer to Appendix D in the 'Messages' manual for recommended ac
ction.
ANR8441E Initialization failed for SCSI library LIBL25.
ANR1401W Mount request denied for volume KWD906L2 - mount failed.
ANR2017I Administrator ADMIN issued command: QUERY SESSION
ANR8300E I/O error on library LIBL25 (OP=40146C01, CC=203, KEY=02, ASC=80, ASCQ=01,
SENSE=70.00.02.00.00.00.00.0E.00.00.00.00.80.0
01.00.00.00.00.00.00.,
Description=Manual intervention required).  Refer to Appendix D in the 'Messages' manual for recommended ac
ction.
ANR8441E Initialization failed for SCSI library LIBL25.
ANR1401W Mount request denied for volume KWD906L2 - mount failed.
ANR0525W Transaction failed for session 82 for node PD_CCB_ORA (TDP Oracle SUN) - storage media inaccessible.
ANR0403I Session 82 ended for node PD_CCB_ORA (TDP Oracle SUN).
ANR0406I Session 87 started for node PD_CCB_ORA (TDP Oracle SUN) (Tcp/Ip 192.1.1.5(35730)).
ANR0403I Session 87 ended for node PD_CCB_ORA (TDP Oracle SUN).
ANR0406I Session 88 started for node PD_CCB_ORA (TDP Oracle SUN) (Tcp/Ip 192.1.1.5(35851)).
ANR8300E I/O error on library LIBL25 (OP=40146C01, CC=203, KEY=02, ASC=80, ASCQ=01,
SENSE=70.00.02.00.00.00.00.0E.00.00.00.00.80.0
01.00.00.00.00.00.00.,
Description=Manual intervention required).  Refer to Appendix D in the 'Messages' manual for recommended ac
ction.
ANR8441E Initialization failed for SCSI library LIBL25.
ANR1401W Mount request denied for volume KWD906L2 - mount failed.
ANR8300E I/O error on library LIBL25 (OP=40146C01, CC=203, KEY=02, ASC=80, ASCQ=01,
SENSE=70.00.02.00.00.00.00.0E.00.00.00.00.80.0
01.00.00.00.00.00.00.,
Description=Manual intervention required).  Refer to Appendix D in the 'Messages' manual for recommended ac
ction.
ANR8441E Initialization failed for SCSI library LIBL25.
ANR1401W Mount request denied for volume KWD906L2 - mount failed.
ANR8300E I/O error on library LIBL25 (OP=40146C01, CC=203, KEY=02, ASC=80, ASCQ=01,
SENSE=70.00.02.00.00.00.00.0E.00.00.00.00.80.0
01.00.00.00.00.00.00.,
Description=Manual intervention required).  Refer to Appendix D in the 'Messages' manual for recommended ac
ction.
ANR8441E Initialization failed for SCSI library LIBL25.
ANR1401W Mount request denied for volume KWD906L2 - mount failed.
ANR0525W Transaction failed for session 88 for node PD_CCB_ORA (TDP Oracle SUN) - storage media inaccessible.
ANR0403I Session 88 ended for node PD_CCB_ORA (TDP Oracle SUN).
ANR0406I Session 89 started for node PD_CCB_ORA (TDP Oracle SUN) (Tcp/Ip 192.1.1.5(35865)).
ANR0403I Session 89 ended for node PD_CCB_ORA (TDP Oracle SUN).
ANR0407I Session 90 started for administrator ADMIN (WebBrowser) (HTTP 10.25.4.50(3590)).
ANR0405I Session 90 ended for administrator ADMIN (WebBrowser).
ANR2017I Administrator ADMIN issued command: QUERY PROCESS
ANR2017I Administrator ADMIN issued command: QUERY SESSION

阿拉丁神灯

硬件设备有问题~
解决办法：
1，检查带库和主机的物理连接，重起带库，重起Tivoli软件~
2，如果1不能解决问题，打电话给硬件厂商获得支持~

lotusin

原帖由 sucrolee 于 2007-7-8 03:49 发表
这个问题让我头痛好多天了
具体情况是这样的
本来这里备份跑得很好。后来由我接手，我刚接触这个，麻烦来了。
有几天在删除大量的数据，所以数据库很慢，RMAN跑全备和归档备份，但跑全备是每天都跑，以前还行 ...

arch_7714_1是oracle的归档日志名，还是tovil里的名字？如果是归档日志名，可能归档日志有问题。

herowangzj

物理设备连接问题,检查libL25这个设备的状态先.

科学球

带机坏了不能读写磁带，导致大量磁带状态变为不可用，最终变成没有磁带可用。所以不能备份。

ASC=80, ASCQ=01为带库错误代码，查Sun L25维护手册，可以确认带机具体故障。

ANR1412W Volume KWD902L2 access mode is "unavailable".
ANR1412W Volume KWD903L2 access mode is "unavailable".
ANR1412W Volume KWD904L2 access mode is "unavailable".
ANR1414W Volume KWD907L2 access mode is "read-only" due to previous write error

这些磁带可能都需要TSM重新处理，我不太懂TSM，帮不了你这个。

sucrolee

小弟对这些真的是门都没有入，请各位回贴的高人不吝指教
1。如何检查带库是否在正常工作
2。带库中有十几盘磁带，UNAVALIABLE的好像只有几盘，这是为什么
3。如何重启带库
这个问题让我头痛好久了，但我又是刚接手，公司又没有人能够帮我，再不解决的话，数据库的机器归档日志一满，我将会很惨。生产系统都会停下来的。
我们跑的是电信系统，一旦系统停下来，公司将会面临巨额的赔偿。

leirenyuan

从带库上找问题吧。
也许磁头脏了，机械臂卡住了，卡带等等。
人工干预以下或许就好了。