[正则表达]如何找这样的重复行？

peterdocter

<h2>1080</h2>:1080
<h2>1080</h2>:1080s
<h2>12–step</h2>:12-step
<h2><a href="entry://18–wheel.er">18–wheel.er</a></h2>:18-wheeler
<h2><a href="entry://18–wheel.er">18–wheel.er</a></h2>:18-wheelers
<h2><a href="entry://twenty–twenty">twen.ty–twen.ty</a></h2>:20/20
<h2>2,4,5–T</h2>:2,4,5-T
<h2>24–7</h2>:24-7
<h2>24–7</h2>:24/7
<h2>2,4–D</h2>:2,4-D
<h2>2–D</h2>:2-D
<h2>2–D</h2>:2-Ds
<h2>2WD</h2>:2WD
<h2>3–D</h2>:3-D
<h2>3–D</h2>:3-Ds
<h2>4–H</h2>:4-H
<h2>4–H</h2>:4-H'er
<h2>4–H</h2>:4-Her
<h2>4–H</h2>:4-H'ers
如：
重复内容：
<h2>1080</h2>:1080
<h2>1080</h2>:1080s
就是看<h2>1080</h2>，是重复了。
<h2>12–step</h2>:12-step这个没有重复了！
这三行都是重复的：
<h2>4–H</h2>:4-H
<h2>4–H</h2>:4-H'er
<h2>4–H</h2>:4-Her
<h2>4–H</h2>:4-H'ers
是否sed无法实现？awk又不熟悉...

Shell_HAT

1. awk -F: '!a[$1]++' urfile

复制代码

peterdocter

回复 2# Shell_HAT
多谢，不过不重复的内容出现了。只要想重复内容，也就是过滤不重复！

   

cao627

1. awk -F: '{if(++a[$1]==1)b[$1]=$0;if(a[$1]==2){print b[$1];f[$1]=1}}f[$1]' file
   

复制代码

peterdocter

回复 4# cao627
比上面好一点点，但是数据是这样时就有问题
<h2><a href="entry://ability">abil.i.ty</a></h2>:ability
<h2><a href="entry://ab incunabulis">ab in.cu.na.bu.lis</a></h2>:ab incunabulis
<h2><a href="entry://Abington">Ab.ing.ton</a></h2>:Abington
<h2><a href="entry://ab initio">ab in.i.tio</a></h2>:ab initio
<h2><a href="entry://abiogenesis">abio.gen.e.sis</a></h2>:abiogeneses
<h2><a href="entry://abiogenesis">abio.gen.e.sis</a></h2>:abiogenesis
<h2><a href="entry://abiogenesis">abio.gen.e.sis</a></h2>:abiogenesises
<h2><a href="entry://abiogenic">abio.gen.ic</a></h2>:abiogenic
<h2><a href="entry://abiogenic">abio.gen.ic</a></h2>:abiogenically

1-4非重复内容还是出现！
   

关阴月飞

回复 5# peterdocter

1. awk -F'</h2>:' 'FNR==NR{a[$1]++;next}a[$1]>1' file file

复制代码

cao627

@peterdocter
楼上正解；关键是你没说楚：“判断重不重复是取一行上<h2>和</h2>之间的内容 ”

peterdocter

回复 6# 关阴月飞

多谢！已经实测过，可行！