合并文本中相同的字段

mcshell

   这是shell版块中的以为朋友提的，文本中相同字段合并的问题，也见了一些，可是这种合并，感觉没有思路。求解思路。。。
etc/xml/validation/dataFileFormatDefs.xml:
  destinations:
    dtsystem@app8002:/mnt/configs/production02/default/itm
    dtsystem@app8003:/mnt/configs/production03/default/itm
    dtsystem@app8004:/mnt/configs/production04/default/itm
    dtsystem@app8005:/mnt/configs/production05/default/itm
etc/xml/validation/dataFileFormatDefs.xml:
  destinations:
    dtsystem@app8002:/opt/demandtec/production/etc/xml/validation
    dtsystem@app8003:/opt/demandtec/production/etc/xml/validation
    dtsystem@app8004:/opt/demandtec/production/etc/xml/validation
    dtsystem@app8005:/opt/demandtec/production/etc/xml/validation
    dtsystem@pns8001:/opt/demandtec/production/etc/xml/validation
buildNumber.txt:
  destinations:
    dtsystem@app8002:/opt/demandtec/production/lib
    dtsystem@app8003:/opt/demandtec/production/lib
    dtsystem@app8004:/opt/demandtec/production/lib

上面的文本，我要的结果是：

etc/xml/validation/dataFileFormatDefs.xml:
  destinations:
    dtsystem@app8002:/mnt/configs/production02/default/itm
    dtsystem@app8003:/mnt/configs/production03/default/itm
    dtsystem@app8004:/mnt/configs/production04/default/itm
    dtsystem@app8005:/mnt/configs/production05/default/itm
    dtsystem@app8002:/opt/demandtec/production/etc/xml/validation
    dtsystem@app8003:/opt/demandtec/production/etc/xml/validation
    dtsystem@app8004:/opt/demandtec/production/etc/xml/validation
    dtsystem@app8005:/opt/demandtec/production/etc/xml/validation
    dtsystem@pns8001:/opt/demandtec/production/etc/xml/validation
buildNumber.txt:
  destinations:
    dtsystem@app8002:/opt/demandtec/production/lib
    dtsystem@app8003:/opt/demandtec/production/lib
    dtsystem@app8004:/opt/demandtec/production/lib

这只是文件的一个片段，要求如果出现相同的连续两行（类似下面2行），保留第一次出现，其余出现都删除。需要查找的连续的两行中，第一行需要以小写字母开头，冒号结尾；第二行就是原封不动的“  destinations:”
etc/xml/validation/dataFileFormatDefs.xml:
  destinations:

sjdy521

怎么看起来格式像YAML，可以考虑按照YAML做分析，有现成的模块

mcshell

回复 2# sjdy521


    因为文本是一行一行读取的，我用hash去重后，，顺序就乱了。。。

kk861123

mcshell 发表于 2012-09-20 10:31
回复 2# sjdy521

哪里重复了？
try~

1. #!/usr/bin/perl
   
2. use strict;
   
3. use warnings;
   
4. use Data::Dumper;
   
5. 
   
6. my $str = do{local $/; <DATA>};
   
7. my @chunks = split /^(?=\S)/m,$str;
   
8. my %xml;
   
9. while (@chunks) {
   
10.     my $text = shift @chunks;
    
11.     open my $fh, "<", \$text or die($!);
    
12.     my $title  = <$fh>;     # etc/....
    
13.        $title .= <$fh>;     #   dest...
    
14.     push @{ $xml{$title} },<$fh>;
    
15.     close($fh);
    
16. }
    
17. 
    
18. while (my($k,$v) = each %xml) {
    
19.     print $k, @{$v};
    
20. }
    
21. 
    
22. __DATA__
    
23. etc/xml/validation/dataFileFormatDefs.xml:
    
24.   destinations:
    
25.     dtsystem@app8002:/mnt/configs/production02/default/itm
    
26.     dtsystem@app8003:/mnt/configs/production03/default/itm
    
27.     dtsystem@app8004:/mnt/configs/production04/default/itm
    
28.     dtsystem@app8005:/mnt/configs/production05/default/itm
    
29. etc/xml/validation/dataFileFormatDefs.xml:
    
30.   destinations:
    
31.     dtsystem@app8002:/opt/demandtec/production/etc/xml/validation
    
32.     dtsystem@app8003:/opt/demandtec/production/etc/xml/validation
    
33.     dtsystem@app8004:/opt/demandtec/production/etc/xml/validation
    
34.     dtsystem@app8005:/opt/demandtec/production/etc/xml/validation
    
35.     dtsystem@pns8001:/opt/demandtec/production/etc/xml/validation
    
36. buildNumber.txt:
    
37.   destinations:
    
38.     dtsystem@app8002:/opt/demandtec/production/lib
    
39.     dtsystem@app8003:/opt/demandtec/production/lib
    
40.     dtsystem@app8004:/opt/demandtec/production/lib
    

复制代码

mcshell

谢谢了。。~可以的。。我研究下你的脚本回复 4# kk861123


   

rubyish

这可以吗？

1. use 5.016;
   
2. my @a = split /\n(?=^[a-z])/m, do { local $/; <DATA> };
   
3. my %h;
   
4. for (@a) {
   
5.     my ( $a, $b, $c ) = split /\n/, $_, 3;
   
6.     say for $h{$a}++ ? () : ( $a, $b ), $c;
   
7. }

复制代码

mcshell

回复 6# rubyish


嗯，，可以的。。感觉code很简洁

cdtits

没见重复行啊

moperyblue

1. sed '/^[a-z].*:/{N;s/\n/,/}' file|awk '!a[$0]++'|tr , \\n

复制代码

sunzhiguolu

1. #!/usr/bin/perl
   
2. use strict;
   
3. use warnings;
   
4. 
   
5. my ($pos, $cnt, $flag, @aData) = (0) x 3;
   
6. while (<DATA>){
   
7.     if (/\A(etc|build)/){
   
8.         $pos = $1 eq 'etc' ? 0 : 1;
   
9.         $flag = defined ($aData[$pos]) ? 1 : 0;
   
10.         $cnt = 0 if ($flag);
    
11.     }
    
12.     next if ($flag and $cnt++ < 2);
    
13.     $aData[$pos] .= $_;
    
14. }
    
15. 
    
16. print @aData;
    
17. 
    
18. __DATA__
    
19. etc/xml/validation/dataFileFormatDefs.xml:
    
20.   destinations:
    
21.     dtsystem@app8002:/mnt/configs/production02/default/itm
    
22.     dtsystem@app8003:/mnt/configs/production03/default/itm
    
23.     dtsystem@app8004:/mnt/configs/production04/default/itm
    
24.     dtsystem@app8005:/mnt/configs/production05/default/itm
    
25. etc/xml/validation/dataFileFormatDefs.xml:
    
26.   destinations:
    
27.     dtsystem@app8002:/opt/demandtec/production/etc/xml/validation
    
28.     dtsystem@app8003:/opt/demandtec/production/etc/xml/validation
    
29.     dtsystem@app8004:/opt/demandtec/production/etc/xml/validation
    
30.     dtsystem@app8005:/opt/demandtec/production/etc/xml/validation
    
31.     dtsystem@pns8001:/opt/demandtec/production/etc/xml/validation
    
32. buildNumber.txt:
    
33.   destinations:
    
34.     dtsystem@app8002:/opt/demandtec/production/lib
    
35.     dtsystem@app8003:/opt/demandtec/production/lib
    
36.     dtsystem@app8004:/opt/demandtec/production/lib

复制代码