如何用awk把日志内容分段提取

niezhisheng

想用awk把下面日志分段提取出来,每段开头都有java.sql.SQL,但不能确保未来所有日志都会以java.sql.SQL开头,可以确定的是每段开头包含字符"Exception",日志样本如下:



java.sql.SQLException: ORA-06502: PL/SQL: 数字或值错误
ORA-06512: 在"CIQUSR.TRT_MIGOODS", line 162
ORA-06512: 在"CIQUSR.TRT_MIDECL_WEB", line 603
ORA-06512: 在line 1

        at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134)
        at oracle.jdbc.ttc7.TTIoer.processError(TTIoer.java:289)
        at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:573)
        at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1891)
        at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:1093)
        at oracle.jdbc.driver.OracleStatement.executeNonQuery(OracleStatement.java:2047)
        at oracle.jdbc.driver.OracleStatement.doExecuteOther(OracleStatement.java:1940)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2709)
        at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:589)
        at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:656)
        at com.itown.dataexchange.SHTransDataWorker.executePrMid(SHTransDataWorker.java:108
        at com.itown.dataexchange.SHTransDataWorker.SendDeclData(SHTransDataWorker.java:406)
        at com.itown.dataexchange.SHTransDataWorker.doWork(SHTransDataWorker.java:129)
        at com.itown.dataexchange.SHTransDataWorker.run(SHTransDataWorker.java:66)
        at java.lang.Thread.run(Unknown Source)
java.sql.BatchUpdateException: 批处理中出现错误: ORA-01438: 值大于此列指定的允许精确度

        at oracle.jdbc.dbaccess.DBError.throwBatchUpdateException(DBError.java:571)
        at oracle.jdbc.dbaccess.DBError.throwBatchUpdateException(DBError.java:626)
        at oracle.jdbc.driver.OracleStatement.executeBatch(OracleStatement.java:6841)
        at com.itownet.shanghai.sql.PooledStatement.executeBatch(PooledStatement.java:50)
        at com.itownet.shanghai.dbaccess.DataHandle.handleTranscation(DataHandle.java:156)
        at com.itown.dataexchange.SHTransDataWorker.SendDeclData(SHTransDataWorker.java:400)
        at com.itown.dataexchange.SHTransDataWorker.doWork(SHTransDataWorker.java:129)
        at com.itown.dataexchange.SHTransDataWorker.run(SHTransDataWorker.java:66)
        at java.lang.Thread.run(Unknown Source)
java.sql.SQLException: ORA-06502: PL/SQL: 数字或值错误
ORA-06512: 在"CIQUSR.TRT_MIGOODS", line 162
ORA-06512: 在"CIQUSR.TRT_MIDECL_WEB", line 603
ORA-06512: 在line 1

        at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134)
        at oracle.jdbc.ttc7.TTIoer.processError(TTIoer.java:289)
        at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:573)
        at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1891)
        at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:1093)
        at oracle.jdbc.driver.OracleStatement.executeNonQuery(OracleStatement.java:2047)
        at oracle.jdbc.driver.OracleStatement.doExecuteOther(OracleStatement.java:1940)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2709)
        at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:589)
        at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:656)
        at com.itown.dataexchange.SHTransDataWorker.executePrMid(SHTransDataWorker.java:108
        at com.itown.dataexchange.SHTransDataWorker.SendDeclData(SHTransDataWorker.java:406)
        at com.itown.dataexchange.SHTransDataWorker.doWork(SHTransDataWorker.java:129)
        at com.itown.dataexchange.SHTransDataWorker.run(SHTransDataWorker.java:66)
        at java.lang.Thread.run(Unknown Source)

[ 本帖最后由 niezhisheng 于 2006-2-21 13:29 编辑 ]

CUDev

想用awk把下面日志分段提取出来

不知道要达到什么样的分段效果？

niezhisheng

达到的效果是在每段前加上<value>,每段结尾加上</value>,如下
<value>
java.sql.SQLException: ORA-06502: PL/SQL: 数字或值错误
ORA-06512: 在"CIQUSR.TRT_MIGOODS", line 162
ORA-06512: 在"CIQUSR.TRT_MIDECL_WEB", line 603
ORA-06512: 在line 1

        at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134)
        at oracle.jdbc.ttc7.TTIoer.processError(TTIoer.java:289)
        at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:573)
        at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1891)
        at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:1093)
        at oracle.jdbc.driver.OracleStatement.executeNonQuery(OracleStatement.java:2047)
        at oracle.jdbc.driver.OracleStatement.doExecuteOther(OracleStatement.java:1940)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2709)
        at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:589)
        at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:656)
        at com.itown.dataexchange.SHTransDataWorker.executePrMid(SHTransDataWorker.java:108
        at com.itown.dataexchange.SHTransDataWorker.SendDeclData(SHTransDataWorker.java:406)
        at com.itown.dataexchange.SHTransDataWorker.doWork(SHTransDataWorker.java:129)
        at com.itown.dataexchange.SHTransDataWorker.run(SHTransDataWorker.java:66)
        at java.lang.Thread.run(Unknown Source)
</value>

[ 本帖最后由 niezhisheng 于 2006-2-21 13:28 编辑 ]

johnny_jiang

原帖由 niezhisheng 于 2006-2-21 11:15 发表
达到的效果是在每段前加上<value>,每段结尾加上</value>,如下
<value>
java.sql.SQLException: ORA-06502: PL/SQL: 数字或值错误
ORA-06512: 在"CIQUSR.TRT_MIGOODS", line 162
...

1. 
   
2. 
   
3. # awk
   
4. # pickup.awk
   
5. 
   
6. BEGIN { count=1 }
   
7. 
   
8. $1~/\w*Exception:/ { if (count != 1) { print "</value>" } print "<value>";count+=1 }
   
9. { print }
   
10. 
    
11. END { print "</value>" }
    
12. 
    

复制代码

awk -f pickup.awk XXX

试试看，行不行,

unixkiller

很简单的C代码就能实现了

shitou9000

原帖由 johnny_jiang 于 2006-2-21 22:53 发表



[code]

# awk
# pickup.awk

BEGIN { count=1 }

$1~/\w*Exception:/ { if (count != 1) { print "</value>" } print "<value>";count+=1 }
{ print }

END {  ...

可以用 有点看不懂。可以解释一下吗？

李某人

楼主解释解释
看不太明白

寂寞烈火

try:

1. 
   
2. cat filename|sed '/java.sql.*Exception/s/.*/<\/value>\n<value>\n&/' |sed -e '1d' -e '$s/.*/&\n<\/value>/'

复制代码

johnny_jiang

原帖由 shitou9000 于 2006-2-22 10:48 发表

可以用 有点看不懂。可以解释一下吗？

BEGIN { count=1 }        # 设置变量count


$1~/\w*Exception:/ { if (count != 1) { print "</value>" } print "<value>";count+=1 }

# 如果$1符合正则,判断count是否为1,如果为1说明是第一段记录,那么不用添加</value>结束标志.
# 输出<value>,记数器+1

{ print } # 输出当前行

END { print "</value>" } # 输出结束标志

niezhisheng

谢谢 johnny_jiang  兄我用你的方法已经把问题解决.
其实我完整的目的是要把日志生成一个xml格式的文件,但如果一来就说得太多想帮忙的人反而更加糊涂,下面是我完整的源代码
echo '<?xml version="1.0" encoding="UTF-8"?>'
echo '<DataPage>'
echo '<cols>'
echo '<col isnumber="0" alarmtype="">'主机名'</col>'
echo '<col isnumber="0" alarmtype="">'日志名'</col>'
echo '<col isnumber="0" alarmtype="enum">'日志内容'</col>'
echo '<col isnumber="0" alarmtype="">'获取时间'</col>'
echo '</cols>'
echo '<units>'

echo '<unit></unit>'
echo '<unit></unit>'
echo '<unit></unit>'
echo '<unit></unit>'
echo '</units>'
DATE=`date +"%Y-%m-%d %T"`
HOSTN=`hostname`
cd ~/monitor/shells/
echo '<data>'
awk 'BEGIN {count=1}
$1~/\w*Exception:/{if(count !=1){print "</value><value>'"$DATE"'</value></row>"}  print "<row><value>'"$HOSTN"'</value><value>exception.log</value><value>";count+=1}
{print}
END{print "</value><value>'"$DATE"'</value></row>"}' exception.log
echo '</data>'
echo '</DataPage>'

[ 本帖最后由 niezhisheng 于 2006-2-22 16:39 编辑 ]