xargs awk使用问题

ghostgorst

有xml文件格式如下:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?> <!DOCTYPE bill_run SYSTEM "DTD/telfort_invoice2_4_4.dtd" [ <!ENTITY DL1 ""> <!ENTITY DL2 ""> <!ENTITY D1150L2 "IDD"> <!ENTITY RMKT5B11L2 "sss"> <!ENTITY RMKT5N12L2 "sdfs"> <!ENTITY RMKT5I12L2 "sdf"> <!ENTITY RMKT5B12L2 "dfg"> ]> <!-- xmllib generated on Aug  3 2009 --> <bill_run bill_run_id="3283" date="20090731" > &nbsp;&nbsp;<O2id> &nbsp;&nbsp;&nbsp;&nbsp;<release>1.0</release>    &nbsp;&nbsp;</O2id> &nbsp;&nbsp;<O2info b_type="MM" lang="NL">    &nbsp;&nbsp;</O2info> &nbsp;&nbsp;<bill b_ref_no="" cid="100228020"> &nbsp;&nbsp;&nbsp;&nbsp;<rend pap="1" pdf="3"/> &nbsp;&nbsp;&nbsp;&nbsp;<prop> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<ppos></ppos>       &nbsp;&nbsp;&nbsp;&nbsp;</prop> &nbsp;&nbsp;&nbsp;&nbsp;<cmf>       &nbsp;&nbsp;&nbsp;&nbsp;</cmf> &nbsp;&nbsp;&nbsp;&nbsp;<msg type="MKT"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<ln>       &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</ln>       &nbsp;&nbsp;&nbsp;&nbsp;</msg> &nbsp;&nbsp;&nbsp;&nbsp;<list type="srvnotov">      &nbsp;&nbsp;&nbsp;&nbsp;</list> &nbsp;&nbsp;&nbsp;&nbsp;<front>       &nbsp;&nbsp;&nbsp;&nbsp;</front> &nbsp;&nbsp;&nbsp;&nbsp;<xmlsum/> &nbsp;&nbsp;&nbsp;&nbsp;<xmldtl> &nbsp;&nbsp;&nbsp;&nbsp;</xmldtl> &nbsp;&nbsp;</bill> </bill_run>

ly5066113

如果 ${BILLING_HOME}/log/.file_path.log 中的所有文件是一次性给 awk 处理的，那么这个脚本是没有问题的，但显然是分批处理的，所以结果就不对了。

ghostgorst

用下面的shell脚本,把多个xml文件合并成一个,结果有问题,<bill_run>节点出现了多次.

cat ${BILLING_HOME}/log/.file_path.log|xargs awk  '/^<?xml/,/^]>/{ &nbsp;&nbsp;&nbsp;&nbsp;if(!a[$0]++) &nbsp;&nbsp;&nbsp;&nbsp;if(NR==FNR) h=h"n"$0 &nbsp;&nbsp;&nbsp;&nbsp;else sub(/n]>/,"n"$0"n]>",h)    &nbsp;&nbsp;&nbsp;&nbsp;next &nbsp;&nbsp;&nbsp;} &nbsp;&nbsp;/<bill /,/</bill>/{print;next} &nbsp;&nbsp;/^</bill_run>/{next} &nbsp;&nbsp;NR==FNR&&!a[$0]++ &nbsp;&nbsp;END{ &nbsp;&nbsp;&nbsp;&nbsp;print "</bill_run>" &nbsp;&nbsp;&nbsp;&nbsp;print h > t".xml" &nbsp;&nbsp;}' t=$$ > $$

cat $$.xml $$ > merge.xml
${BILLING_HOME}/log/.file_path.log中存放的是xml文件的绝对路径:总共有21个,这里没全列出来,太多了
telfort1:/kfbuild/x144378/billing/log>cat ${BILLING_HOME}/log/.file_path.log
/kfbuild/x144378/billing/BILL/REAL_BILL/XML/00/20090508/TBM/20090508-3101000802010.xml
/kfbuild/x144378/billing/BILL/REAL_BILL/XML/00/20090508/TBM/20090508-3101000908137.xml
/kfbuild/x144378/billing/BILL/REAL_BILL/XML/01/20090508/TBM/20090508-3101000966135.xml
/kfbuild/x144378/billing/BILL/REAL_BILL/XML/02/20090508/TBM/20090508-3101000804442.xml


但是相同的shell,只是文件名不一样合并结果却是正确的

cat merge.log|xargs awk  '/^<?xml/,/^]>/{ &nbsp;&nbsp;if(!a[$0]++) &nbsp;&nbsp;&nbsp;&nbsp;if(NR==FNR) h=h "n" $0 &nbsp;&nbsp;&nbsp;&nbsp;else sub(/n]>/,"n"$0"n]>",h)    &nbsp;&nbsp;&nbsp;&nbsp;next &nbsp;&nbsp;} &nbsp;&nbsp;/<bill /,/</bill>/{print;next} &nbsp;&nbsp;/^</bill_run>/{next} &nbsp;&nbsp;NR==FNR&&!a[$0]++ &nbsp;&nbsp;END{ &nbsp;&nbsp;&nbsp;&nbsp;print "</bill_run>" &nbsp;&nbsp;&nbsp;&nbsp;print h > t".xml" &nbsp;&nbsp;&nbsp;&nbsp;}' t=$$  > $$

cat $$.xml $$ > merge.xml

文件merge.log中存放了59个xml文件,
telfort1:/kfbuild/x144378/billing/BILL/REAL_BILL/XML/00/20090508/MM>cat merge.log
20090508-3101000798010.xml
20090508-3101000804020.xml
20090508-3101000804030.xml
20090508-3101000804040.xml
20090508-3101000804050.xml
20090508-3101000804060.xml
20090508-3101000804070.xml
20090508-3101000804080.xml



请大家点拨下这是什么原因啊?第一个脚本不对是因为文件包括了路径,超出了awk能接受的参数长度了么?

[ 本帖最后由 ghostgorst 于 2009-8-28 11:16 编辑 ]

ghostgorst

cat ${BILLING_HOME}/log/.file_path.log|xargs awk  
你意思是这里.file_path.log里文件被自动分批给awk处理了?
但是为什么会被自动分批呢?有什么办法可以制止么?merge.log文件比.file_path.log多也没被分批啊.

[ 本帖最后由 ghostgorst 于 2009-8-28 11:29 编辑 ]

blackold

xargs的作用之一就重组命令行，防止命令参数过长。

ghostgorst

因为这个原因,cat ${BILLING_HOME}/log/.file_path.log|xargs awk  .file_path.log里文件被分批处理了么?

NR==FNR&&!a[$0]++
但是我加上了NR==FNR这个条件了啊,难道分批处理之后FNR要重新计数么?不知道我自己定义个变量,作用类似FNR,能解决这个问题么?

blackold

是否分批处理取决于参数个数与环境限制。

nhw_cs

原帖由 ghostgorst 于 2009-8-28 12:08 发表
因为这个原因,cat ${BILLING_HOME}/log/.file_path.log|xargs awk  .file_path.log里文件被分批处理了么?

NR==FNR&&!a[$0]++
但是我加上了NR==FNR这个条件了啊,难道分批处理之后FNR要重新计数么?不知道我自 ...

你的XML文件总数大约多少个? 比如不超过1000个, 给xargs 加上-n参数试验一下看看结果有无问题:


cat ${BILLING_HOME}/log/.file_path.log|xargs -n 1000 awk  .....

ghostgorst

总共大概是26万个,每次合并1000个左右,有可能2000的

ghostgorst

试了下!不行哎!
cat ${BILLING_HOME}/log/.file_path.log|xargs -n 300 awk  '/^<?xml/,/^]>/{
    if(!a[$0]++)
    if(NR==FNR) h=h"n"$0
    else sub(/n]>/,"n"$0"n]>",h)   
    next
   }
  /<bill /,/</bill>/{print;next}
  /^</bill_run>/{next}
  NR==FNR&&!a[$0]++
  END{
    print "</bill_run>"
    print h > t".xml"
  }' t=$$ > $$