从列表中删除标点符号

Question

我正在抽取《独立宣言》的样本，并计算其中单词长度的频率。

文件中的示例文本：

"When in the Course of human events it becomes necessary for one people to dissolve the political bands which have connected them with another and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires 
that they should declare the causes which impel them to the separation."

注意：单词长度不能包含任何标点符号，例如 string.punctuation 中的任何内容。

预期成果(样本(：

Length Count
1 16
2 267
3 267
4 169
5 140
6 112
7 99
8 68
9 61
10 56
11 35
12 13
13 9
14 7
15 2

我目前坚持从已转换为列表的文件中删除标点符号。

这是我到目前为止尝试过的：

import sys
import string
def format_text(fname):
        punc = set(string.punctuation)
        words = fname.read().split()
        return ''.join(word for word in words if word not in punc)
try:
    with open(sys.argv[1], 'r') as file_arg:
        file_arg.read()
except IndexError:
    print('You need to provide a filename as an arguement.')
    sys.exit()
fname = open(sys.argv[1], 'r')
formatted_text = format_text(fname)
print(formatted_text)

Answer 1

您可以从单词中删除标点符号，也可以避免将所有文件读入内存：

punc = string.punctuation
return ' '.join(word.strip(punc) for line in fname for word in line.split())

如果要从Nature's中删除'，则需要翻译：

from string import punctuation
# use ord of characters you want to replace as keys and what you want to replace them with as values
tbl = {ord(k):"" for k in punctuation}
return ' '.join(line.translate(tbl) for line in fname)

要获取频率，请使用计数器字典：

from collections import Counter
freq = Counter(len(word.translate(tbl)) for line in fname for word in line.split())

或者根据您的方法：

freq = Counter(len(word.strip(punc)) for line in fname for word in line.split())

以上面问题中的行为例：

lines =""""When in the Course of human events it becomes necessary for one people to dissolve the political bands which have connected them with another and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires
that they should declare the causes which impel them to the separation."""
from collections import Counter
freq = Counter(len(word.strip(punctuation)) for line in lines.splitlines() for word in line.split())
print(freq.most_common()) 

输出键/值对的元组，从看到最多的字长度一直到最小，键是长度，第二个元素是频率：

[(3, 15), (2, 12), (4, 9), (5, 9), (6, 9), (7, 7), (8, 5), (9, 3), (1, 1), (10, 1)]

如果要输出从 1 个字母单词开始的频率，而不按顺序排序：

mx = max(freq.values())
for i in range(1, mx+1):
    v = freq[i]
    if v:
        print("length {} words appeared {} time/s.".format(i, v) )

输出：

length 1 words appeared 1 time/s.
length 2 words appeared 12 time/s.
length 3 words appeared 15 time/s.
length 4 words appeared 9 time/s.
length 5 words appeared 9 time/s.
length 6 words appeared 9 time/s.
length 7 words appeared 7 time/s.
length 8 words appeared 5 time/s.
length 9 words appeared 3 time/s.
length 10 words appeared 1 time/s.

对于缺少的键，与普通字典不同的计数器字典不会返回 keyError if v而是返回 0 的值，因此对于文件中出现的字长，该值仅为 True。

如果要打印清理后的数据，将所有逻辑放入函数中：

def clean_text(fname):
    punc = string.punctuation
    return [word.strip(punc) for line in fname for word in line.split()]

def get_freq(cleaned):
    return Counter(len(word) for word in cleaned)

def freq_output(d):
    mx = max(d.values())
    for i in range(1, mx + 1):
        v = d[i]
        if v:
            print("length {} words appeared {} time/s.".format(i, v))
try:
    with open(sys.argv[1], 'r') as file_arg:
        file_arg.read()
except IndexError:
    print('You need to provide a filename as an arguement.')
    sys.exit()
fname = open(sys.argv[1], 'r')
formatted_text = clean_text(fname)
print(" ".join(formatted_text))
print()
freq = get_freq(formatted_text)
freq_output(freq) 

哪个在问题代码段输出上运行：

~$ python test.py test.txt
When in the Course of human events it becomes necessary for one people  
to dissolve the political bands which have connected them with another
and to assume among the powers of the earth the separate and equal station 
 to which the Laws of Nature and of Nature's God entitle them a decent 
respect to the opinions of mankind requires that they should declare 
the causes which impel them to the separation
length 1 words appeared 1 time/s.
length 2 words appeared 12 time/s.
length 3 words appeared 15 time/s.
length 4 words appeared 9 time/s.
length 5 words appeared 9 time/s.
length 6 words appeared 9 time/s.
length 7 words appeared 7 time/s.
length 8 words appeared 5 time/s.
length 9 words appeared 3 time/s.
length 10 words appeared 1 time/s.

如果您只关心频率输出，则一次性完成所有操作：

import sys
import string

def freq_output(fname):
    from string import punctuation
    tbl = {ord(k): "" for k in punctuation}
    d = Counter(len(word.strip(punctuation)) for line in fname for word in line.split())
    d = Counter(len(word.translate(tbl)) for line in fname for word in line.split())
    mx = max(d.values())
    for i in range(1, mx + 1):
        v = d[i]
        if v:
            print("length {} words appeared {} time/s.".format(i, v))

try:
    with open(sys.argv[1], 'r') as file_arg:
        file_arg.read()
except IndexError:
    print('You need to provide a filename as an arguement.')
    sys.exit()
fname = open(sys.argv[1], 'r')
freq_output(fname)

使用任何正确的方法d .

Answer 2

您可以使用翻译去除标点符号：

import string
words = fname.read().translate(None, string.punctuation).split()

在 Python 中去除字符串标点符号的最佳方法

PY2.7：

import string
from collections import defaultdict
from collections import Counter
def s1():
    with open("myfile.txt", "r") as f:
        counts = defaultdict(int)
        for line in f:
            words = line.translate(None, string.punctuation).split()
            for length in map(len, words):
                counts[length] += 1
    return counts
def s2():
    with open("myfile.txt", "r") as f:
        counts = Counter(length for line in f for length in map(len, line.translate(None, string.punctuation).split()))
    return counts

---

print s1()
defaultdict(<type 'int'>, {1: 111, 2: 1169, 3: 1100, 4: 1470, 5: 1425, 6: 1318, 7: 1107, 8: 875, 9: 938, 10: 108, 11: 233, 12: 146})
print s2()
Counter({4: 1470, 5: 1425, 6: 1318, 2: 1169, 7: 1107, 3: 1100, 9: 938, 8: 875, 11: 233, 12: 146, 1: 111, 10: 108})

在python 2.7中，使用计数器比手动构建字典慢，因为计数器更新的实现方式。

%timeit s1()
100 loops, best of 3: 4.42 ms per loop
%timeit s2()
100 loops, best of 3: 9.27 ms per loop

PY3：

我认为在python 3.2中，计数器已更新，并且变得与手动构建计数器字典相同或更快。

此外，python3的翻译也变得不那么冗长：

import string
from collections import defaultdict
from collections import Counter
strip_punct = str.maketrans('','',string.punctuation)
def s1():
    with open("myfile.txt", "r") as f:
        counts = defaultdict(int)
        for line in f:
            words = line.translate(strip_punct).split()
            for length in map(len, words):
                counts[length] += 1
    return counts
def s2():
    with open("myfile.txt", "r") as f:
        counts = Counter(length for line in f for length in map(len, line.translate(strip_punct).split()))
    return counts

---

print(s1())
defaultdict(<class 'int'>, {1: 111, 2: 1169, 3: 1100, 4: 1470, 5: 1425, 6: 1318, 7: 1107, 8: 875, 9: 938, 10: 108, 11: 233, 12: 146})
print(s2())
Counter({4: 1470, 5: 1425, 6: 1318, 2: 1169, 7: 1107, 3: 1100, 9: 938, 8: 875, 11: 233, 12: 146, 1: 111, 10: 108})

---

%timeit s1()
100 loops, best of 3: 11.4 ms per loop
%timeit s2()
100 loops, best of 3: 11.2 ms per loop

Answer 3

您可以使用正则表达式：

import re
def format_text(fname, pattern):
    words = fname.read()
    return re.sub(p, '', words)
p = re.compile(r'[!&:;",.]')
fh = open('C:/Projects/ExplorePy/test.txt')
text = format_text(fh, p)

根据需要应用 split((，并且可以优化模式。