awk去除部分内容相同的重复行

sellie

知道怎么用awk去除文本文件的重复行. 如果去除部分内容重复的行呢? 行中有分隔符, 比如

111,aaa
222,bbb
333,ccc
444,aaa
555,bbb
666,ccc

现想把4-6行去除,因为他们的第二个字段内容和1-3行重复了.

谢谢.

Herowinter

回复 1# sellie

1. awk -F',' '!a[$2]++' i
   
2. 111,aaa
   
3. 222,bbb
   
4. 333,ccc

复制代码

sellie

再问一下,这个是bash的写法,请问csh下怎么写?

Herowinter 发表于 2014-06-26 15:42
回复 1# sellie

Herowinter

回复 3# sellie
你有awk吗？gawk or nawk?awk版本？

   

sellie

awk --version
GNU Awk 3.1.5
Copyright (C) 1989, 1991-2005 Free Software Foundation.

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.


回复 4# Herowinter


   

Herowinter

回复 3# sellie
csh改成这样可以的。

1. awk -F',' '!($2 in a){a[$2];print}' i

复制代码

你也可以用5楼的sort方法，如果你不是要严格保持源文件的纪录
出现顺序的话。

sellie

谢谢. 完美解决.
BTW, sort的方案也可以去重,但我不想改变文本顺序.
回复 7# Herowinter