如何删除文件中重复的字符

learnllehs

大家好，刚接触shell不就，在这里请教大家如何能够删除文件中的重复内容。
待处理文件如下：
we avoid  avoid bois
ccccc  ddddd 1123 1123
fdsaflsj  we

希望去重后的结果如下:
we avoid  bois
ccccc  ddddd 1123
fdsaflsj

在这先谢谢给位前辈的帮助啦~

关阴月飞

1. awk '{for(i=0;i++<NF;)if(!a[$i]++)printf $i FS;print xxoo}' urfile

复制代码

learnllehs

可以的，非常感谢您的帮助！！！ 谢谢  回复 2# 关阴月飞


   

haooooaaa

1. sed -r 's#(\b\w+\b)\s+([^\b\1\b]\s+){0,}\b\1\b#\1#g'

复制代码

1. awk '{for(i=1;i<=NF;i++)if(split($0,t,"\\<"$i"\\>")>2)sub($i,"");$1=$1;print}'

复制代码

seanking1987

回复 2# 关阴月飞


你好，能解释下print xxoo啥意思吗，百度都没搜到

WilliBhamlll

1. sed -r ':1;s/(\S+) (.*)\1/\1\2/;t1' file

复制代码

Herowinter

1. awk '{for(i=1;i<=NF;i++)if(!a[$i]++)printf i==1?$i:FS$i;print ""}' i
   
2. we avoid bois
   
3. ccccc ddddd 1123
   
4. fdsaflsj

复制代码

Herowinter

回复 5# seanking1987

别太认真，随便一个字符串而已，
只是论坛很多大神偏好于使用"xxoo".
   

seanking1987

回复 8# Herowinter

thanks,其实更准确的说我是想问为什么要 print xxoo,后来查了一下awk  print和printf的区别，现在知道原因了。

The performance difference is probably not measurable.
The print function outputs a newline at the end; printf does not unless requested.
The print code converts the arguments to strings and then sends them to the output separated by the OFS (output field separator).
The printf code might need to convert the string to a double before formatting it using a double format (%16.8g or something), and similar operations.
The system call used is going to write(2) or something similar for both, but there'll be code (probably <stdio.h>) layered above that.


   

moperyblue

1. awk '{for(i=1;i<=NF;i++)if(!a[$i]++)printf (i==1?"":FS) $i;print""}'

复制代码