Hadoop Streaming破坏了python生成的Avro

Question

我有一个相当简单的脚本，它以JSON形式获取Twitter数据并将其转换为Avro文件。

from avro import schema, datafile, io
import json, sys
from types import *
def main():
    if len(sys.argv) < 2:
        print "Usage: cat input.json | python2.7 JSONtoAvro.py output"
        return
    s = schema.parse(open("tweet.avsc").read())
    f = open(sys.argv[1], 'wb')
    writer = datafile.DataFileWriter(f, io.DatumWriter(), s, codec = 'deflate')
    failed = 0
    for line in sys.stdin:
        line = line.strip()
    try:
        data = json.loads(line)
    except ValueError as detail:
        continue
    try:
        writer.append(data)
    except io.AvroTypeException as detail:
        print line
        failed += 1
writer.close()
print str(failed) + " failed in schema"
if __name__ == '__main__':
    main()

从那里，我用它来提供一个基本的Hadoop流脚本(也是在Python中)，它只是提取推文的某些元素。但是，当我这样做时，脚本的输入似乎是扭曲的JSON。通常JSON会失败，因为在tweet正文中间或用户定义的描述中有一些错误的u。

下面是我用来调用Hadoop Streaming的bash脚本:

jars=/usr/lib/hadoop/lib/avro-1.7.1.cloudera.2.jar,/usr/lib/hive/lib/avro-mapred-1.7.1.cloudera.2.jar
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar 
    -files $jars,$HOME/sandbox/hadoop/streaming/map/tweetMapper.py,$HOME/sandbox/hadoop/streaming/data/keywords.txt,$HOME/sandbox/hadoop/streaming/data/follow-r3.txt 
     -libjars $jars 
     -input /user/ahanna/avrotest/avrotest.json.avro 
     -output output 
     -mapper "tweetMapper.py -a" 
     -reducer org.apache.hadoop.mapred.lib.IdentityReducer 
     -inputformat org.apache.avro.mapred.AvroAsTextInputFormat 
     -numReduceTasks 1

我在扯头发试图解决这个问题。如有任何建议，不胜感激。

Answer 1

不确定你是否还在寻找答案。u似乎是unicode字符。试试

   resp = json.dumps(line) 
   data = json.loads(resp)

如果是unicode表示导致错误，dump将解决它。