如何删除文件中重复的字符

learnllehs

大家好，刚接触shell不就，在这里请教大家如何能够删除文件中的重复内容。
待处理文件如下：
we avoid  avoid bois
ccccc  ddddd 1123 1123
fdsaflsj  we

希望去重后的结果如下:
we avoid  bois
ccccc  ddddd 1123
fdsaflsj

在这先谢谢给位前辈的帮助啦~

重楼君

1. with open('file.txt', 'r') as f:
   
2.     for line in f:
   
3.         i = line.split()
   
4.         j = []
   
5.         for x in i:
   
6.             if not x in j:
   
7.                 j.append(x)
   
8.         print ' '.join(j)
   
9. 
   

复制代码

回复 1# learnllehs


   

sunzhiguolu

再来一个,

1. perl -007 -pe 'while(s/(?=(\w+))(.+?)\b\1\b/$2/gs){}' f

复制代码

we avoid   bois
ccccc  ddddd 1123
fdsaflsj

sunzhiguolu

1. perl -ane 'print"$_ " for grep {!$h{$_}++} @F;print"\n"' f

复制代码

we avoid bois
ccccc ddddd 1123
fdsaflsj

sunzhiguolu

1. perl -0777 -pe 'while(s/(\w+)(.*?)\b\1\b/$1$2/s){}' f

复制代码

we avoid   bois
ccccc  ddddd 1123
fdsaflsj

moperyblue

貌似楼主的需求不是单行字符串去重

1. sed -r ':a;s/(\b\S+\b) +(.*) *\1/\1 \2/;ta;N;ba'

复制代码

sunzhiguolu

1. perl -ane 'print"$_ " for grep {!$h{$_}++} @F;print"\n";%h=()' f

复制代码

we avoid bois
ccccc ddddd 1123
fdsaflsj we

sunzhiguolu

1. perl -pe 's/(\w+)\h+\1/$1/' f

复制代码

we avoid bois
ccccc  ddddd 1123
fdsaflsj  we

moperyblue

1. awk '{for(i=1;i<=NF;i++)if(!a[$i]++)printf (i==1?"":FS) $i;print""}'

复制代码

seanking1987

回复 8# Herowinter

thanks,其实更准确的说我是想问为什么要 print xxoo,后来查了一下awk  print和printf的区别，现在知道原因了。

The performance difference is probably not measurable.
The print function outputs a newline at the end; printf does not unless requested.
The print code converts the arguments to strings and then sends them to the output separated by the OFS (output field separator).
The printf code might need to convert the string to a double before formatting it using a double format (%16.8g or something), and similar operations.
The system call used is going to write(2) or something similar for both, but there'll be code (probably <stdio.h>) layered above that.