两个文件中不区分大小写和重音的单词匹配

Question

我得到了两个未引用的单列TSV文件(从数据库导出(，其中有几千个人的名字，我需要找到两个文件中出现的名字。这两个文件都以UTF-8和CRLF结尾，并以BOM表0xEF 0xBB 0xBF开头。

一个简单的join或comm命令就可以完成任务，但名称上有一些不同：

# cat file1.tsv
A.  Einstein
Louis Pasteur 
Diego Armando Maradona
Isaac Newton
Frava D’onä
D Rüge
Françoise Barré-Sinoussi

# cat file2.tsv
Diego Maradona
Albert Einstein
Francoise, BARRE  SINOUSSI
Louis Pasteur
frava d'ona
Marie-Louise Von FRANZ
Dimitri Rüge

file2.tsv中的预期匹配为：

Diego Maradona
Albert Einstein
Francoise, BARRE  SINOUSSI
Louis Pasteur
frava d'ona
Dimitri Rüge

---

我写了这个bashsedawkgrep脚本，它动态生成用于匹配姓氏的正则表达式：

#!/bin/bash
# U+0300 = 0xCC80 = 52352
# U+033F = 0xCCBF = 52415
# U+0340 = 0xCD80 = 52608
# U+036E = 0xCDAE = 52654
_COMBINING_CHARS_=()
for i in {52352..52415} {52608..52654}
do
hex=$(printf %04X "$i")
_COMBINING_CHARS_+=( "$(printf 'x'"${hex:0:2}"'x'"${hex:2:2}")" )
done
_COMBINING_CHARS_ERE_=$(IFS='|'; printf %s "${_COMBINING_CHARS_[*]}")

# Function that removes the BOM, CRLF, and COMBINING characters:
sanitize() {
LANG=C sed -E 
-e $'1s/^xEFxBBxBF//' 
-e $'s/r$//' 
-e "s/$_COMBINING_CHARS_ERE_//g" 
-- "$@"
}

# Function that generates a regex for the _lastname_:
toERE() {
awk '
{
if ( $0 ~ /,/) {
n = split($0, a, ",");
$0 = a[n];
} else {
$0 = $NF
}
sub("^[[:space]]+","");
sub("[[:space]]+$","");
gsub("[[:space:]-]+"," ");
}
{
ere = ""
sep = "";
for ( nf = 1; nf <= NF; nf++ ) {
n = split($nf, c, "");
for ( i = 1; i <= n; i++ ) {
ere = ere "[[=" c[i] "=]]"
}
ere = sep ere 
sep = "[[:space:]-]+"
}
print ere "[[:space:]]*$"
}
' < <(sanitize "$@")
}

grep -E -f <(toERE "$1") <(sanitize "$2")

---

不幸的是，给定输入的结果是：

grep: illegal byte sequence

UTF-8多字节字符似乎是个问题，但我想不出用awk处理它的方法

Answer 1

agrep怎么样？man agrep:agrep-在文件中搜索字符串或正则表达式，具有近似匹配功能。它并不像我们将看到的那样完美：

$ while IFS= read -r line
do 
echo -n "$line: "
agrep -B -y  "$line" file1
done < file2

输出：

Diego A. Maradona: agrep: 1 word matches within 6 errors
Maradona, Diego Armando
Albert Einstein: agrep: 1 word matches within 5 errors
A. Einstein
Louis Pasteur: Louis Pasteur
frava dona: agrep: 2 words match within 4 errors
Maradona, Diego Armando
Fräva Dona

很好的例子，因为我们已经在最后三行看到了一个问题。

Answer 2

建议以下技巧：

cat file1.csv file1.csv | sort | uniq -d

解释

cat file1.csv file1.csv一个接一个地组合机器人程序文件

sort将相似的线路放在一起

uniq -d仅打印具有重复的行

Answer 3

^{最后我完成了ruby的工作，但我发布了awk解决方案}

有两个问题：

• POSIX[= =]不适用于变音符号

• awk不知道多字节字符

通过将输入文件转换为ASCII可以解决此问题。iconv可以在一定程度上准确地处理罗马尼亚字符，这正是我所需要的：

#!/bin/bash
to_ascii() {
case $(uname) in
Darwin)
iconv -f UTF-8 -t UTF-8-MAC "$@" |
iconv -f UTF-8 -t ASCII//TRANSLIT//IGNORE
;;
Linux)
iconv -f UTF-8 -t ASCII//TRANSLIT//IGNORE "$@"
;;
esac |
sed $'s/r$//'
}

现在我们只需要做一点规范化，并在家族名称中找到完美的匹配：

awk '
{
gsub("-+","-");
gsub("27+","27");
gsub("[.[:space:]]+"," ");
sub("^[[:space:]]+","");
sub("[[:space:]]+$","");
}
{
if ($0 ~ /,/) {
n = split($0,a,"[[:space:]]*,[[:space:]]*");
lastname = a[n];
} else {
lastname = $NF;
}
gsub("[-[:space:]]+"," ",lastname);
lastname = tolower(lastname);
}
FNR == NR {
keys[lastname] = $0;
next;
}
{
count = 0;
for (n in keys) {
if (n == lastname) {
matches[++count] = keys[n]
}
}
if (count > 0) {
print $0
for (i = 1; i <= count; i++) {
print "t" matches[i]
}
}
}
' <(to_ascii "$2") <(to_ascii "$1")

输出

A Einstein
Albert Einstein
Louis Pasteur
Louis Pasteur
Diego Armando Maradona
Diego Maradona
Frava D'ona
frava d'ona
D Ruge
Dimitri Ruge
Francoise Barre-Sinoussi
Francoise, BARRE SINOUSSI