shell 能否区分 单字节双字节

chichushanren

试图找出文本中的中文英文字符，有无好办法？
或者通过区分单双字节来判断？shell如何实现？
请大侠给点思路。谢谢！

Shell_HAT

1. sed "s/[\x21-\x7E][\x80-\xFF]\|[\x80-\xFF][\x21-\x7E]//g" a.txt >b.txt

复制代码

chichushanren

回复 2# Shell_HAT

vi test.txt

asdfksadfasdf王

sed "s/[\x21-\x7E][\x80-\xFF]\|[\x80-\xFF][\x21-\x7E]//g" test.txt > 1.txt
sed: -e expression #1, char 51: Invalid collation character


报错啊！！！！
   

chichushanren

继续求教！1！！

jason680

回复 3# chichushanren

# cat test.txt
asdfksadfasdf王小明abc

# perl -F'/\b/' -lane '{print join ",",@F}' test.txt
asdfksadfasdf,王小明,abc

# perl -F'/\b/' -lane '{print join "\n",@F}' test.txt
asdfksadfasdf
王小明
abc

   

jason680

回复 3# chichushanren

# cat test.txt
asdf王小明abc

# awk -vFS="" '{N=split($0,a,//);for(n=1;n<=N;n++)print length(a[n]),a[n]}' test.txt
1 a
1 s
1 d
1 f
3 王
3 小
3 明
1 a
1 b
1 c

Note: length = 3 for unicode chinese word

chichushanren

回复 6# jason680

不行啊！！！！
more test1.txt
asdf王小明abc

awk -vFS="" '{N=split($0,a,//);for(n=1;n<=N;n++)print length(a[n]),a[n]}' test1.txt
1 a
1 s
1 d
1 f
1 王
1 小
1 明
1 a
1 b
1 c

rdcwayx

jason680 发表于 2012-09-26 13:46
回复 3# chichushanren

# cat test.txt

$ awk -V |head -1
GNU Awk 4.0.1

$ awk -vFS="" '{N=split($0,a,//);for(n=1;n<=N;n++)print length(a[n]),a[n]}' test.txt
1 a
1 s
1 d
1 f
1 王
1 小
1 明
1 a
1 b
1 c

jason680

@chichushanren
@rdcwayx

awk version issue

man gawk
       As of version 3.1.5, gawk is multibyte aware.  This means that index(),
       length(),  substr()  and  match()  all work in terms of characters, not
       bytes.

awk use other way to identify, or use perl command easily

$ gawk  '{N=split($0,a,"");for(n=1;n<=N;n++)print length(a[n]),a[n]}' test.txt
1 a
1 s
1 d
1 f
1 王
1 小
1 明
1 a
1 b
1 c

$ gawk  '{N=split($0,a,"");for(n=1;n<=N;n++)if(a[n]~/[a-z]/)print "w="a[n];else print "CN="a[n]}' test.txt
w=a
w=s
w=d
w=f
CN=王
CN=小
CN=明
w=a
w=b
w=c