小贝子编程

星火正在检索超过20条记录



我有一个名为students的文件。Json，看起来像这样:

{"uid":1,"name":"Michael","sid":1}
{"uid":2,"name":"Andy", "age":30,"sid":1}
{"uid":3,"name":"Jsaxsustin", "age":19,"sid":1}
{"uid":4,"name":"Andasxsay", "age":30,"sid":2}
{"uid":5,"name":"Jusewedtin", "age":19,"sid":1}
{"uid":6,"name":"Anwereddy", "age":30,"sid":3}
{"uid":7,"name":"Justdwedwein", "age":19,"sid":2}
{"uid":8,"name":"Andwedewy", "age":30,"sid":1}
{"uid":9,"name":"Justidedwn", "age":19,"sid":1}
{"uid":10,"name":"Anddwdey", "age":30,"sid":3}
{"uid":11,"name":"Michael","sid":1}
{"uid":12,"name":"Andy", "age":30,"sid":1}
{"uid":13,"name":"Jsaxsustin", "age":19,"sid":1}
{"uid":14,"name":"Andasxsay", "age":30,"sid":2}
{"uid":15,"name":"Jusewedtin", "age":19,"sid":1}
{"uid":16,"name":"Anwereddy", "age":30,"sid":3}
{"uid":17,"name":"Justdwedwein", "age":19,"sid":2}
{"uid":18,"name":"Andwe2fr3fdewy", "age":30,"sid":1}
{"uid":19,"name":"Justide4y45y54dwn", "age":19,"sid":1}
{"uid":20,"name":"Anddwd45y45yey", "age":30,"sid":3}
{"uid":21,"name":"Justdw45y45yedin", "age":19,"sid":1}
{"uid":22,"name":"An45y45ydy", "age":30,"sid":1}
{"uid":23,"name":"Jsaxsus4y54ytin", "age":19,"sid":1}
{"uid":24,"name":"Andas45y4y5xsay", "age":30,"sid":2}
{"uid":25,"name":"Jusewe4y5dtin", "age":19,"sid":1}
{"uid":26,"name":"Anwere45y45yddy", "age":30,"sid":3}
{"uid":27,"name":"Justdwe4y4y5dwein", "age":19,"sid":2}
{"uid":28,"name":"Andwede45ywy", "age":30,"sid":1}
{"uid":29,"name":"Justided45y45wn", "age":19,"sid":1}
{"uid":30,"name":"Anddwde4t4y", "age":30,"sid":3}
{"uid":31,"name":"Mich4y554ael","sid":1}
{"uid":32,"name":"An45ydy", "age":30,"sid":1}
{"uid":33,"name":"Jsaxsudfsstin", "age":19,"sid":1}
{"uid":34,"name":"Andasxssdfdsay", "age":30,"sid":2}
{"uid":35,"name":"Jusewedtsdfdsin", "age":19,"sid":1}
{"uid":36,"name":"Anweredsfdsdy", "age":30,"sid":3}
{"uid":37,"name":"Justdwedfsdwein", "age":19,"sid":2}
{"uid":38,"name":"Andwedewy", "age":30,"sid":1}
{"uid":39,"name":"Jdsfdsfustidedwn", "age":19,"sid":1}
{"uid":40,"name":"Ansdfdsey", "age":30,"sid":3}
{"uid":41,"name":"Jussdsdtdwedin", "age":19,"sid":1}

然后我使用火花导入数据，并使用sqlContext对其进行sql查询。为了检索信息，我使用以下scala代码(几乎取自scala上的spark示例):

val sqlContext = new org.apache.spark.sql.SQLContext(sc);
import sqlContext.implicits._
val students = sqlContext.read.json("../myTests/students.json");
students.registerTempTable("students");
sqlContext.sql("SELECT * FROM students").show;

这不是返回列表的所有项，它只返回20条记录(前20条记录)…如果在最后一行，而不是

sqlContext.sql("SELECT * FROM students").show;

我把

sqlContext.sql("SELECT * FROM students where uid>10").show;

它给我从10到30的20条记录。简而言之，我想看20多张唱片，我怎么能做到呢?这可能吗?我已经看了文档，并没有读到任何关于这个....我知道这可能有点傻，但在我进入更严重的事情之前，我正在做一些测试……我下载了spark，并将其作为独立运行，只是为了查看scala…

我的spark初始化显示这个(只有错误，我改变了配置，所以它不会显示我的信息)

15/07/17 11:37:55 ERROR Shell: Failed to locate the winutils binary in the hadoo
p binary path
java.io.IOException: Could not locate executable nullbinwinutils.exe in the Ha
doop binaries.
        at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
        at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:363)
        at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)
        at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:104)
        at org.apache.hadoop.security.Groups.<init>(Groups.java:86)
        at org.apache.hadoop.security.Groups.<init>(Groups.java:66)
        at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Group
s.java:280)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupI
nformation.java:271)
        at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(Use
rGroupInformation.java:248)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(
UserGroupInformation.java:763)
        at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGrou
pInformation.java:748)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGr
oupInformation.java:621)
        at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils
.scala:2162)
        at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils
.scala:2162)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2162)
        at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:212)
        at org.apache.spark.repl.SparkIMain.<init>(SparkIMain.scala:118)
        at org.apache.spark.repl.SparkILoop$SparkILoopInterpreter.<init>(SparkIL
oop.scala:187)
        at org.apache.spark.repl.SparkILoop.createInterpreter(SparkILoop.scala:2
17)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$Spark
ILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:949)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$Spark
ILoop$$process$1.apply(SparkILoop.scala:945)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$Spark
ILoop$$process$1.apply(SparkILoop.scala:945)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClass
Loader.scala:135)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$pr
ocess(SparkILoop.scala:945)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSub
mit$$runMain(SparkSubmit.scala:664)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:16
9)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _ / _ / _ `/ __/  '_/
   /___/ .__/_,_/_/ /_/_   version 1.4.0
      /_/
Using Scala version 2.10.4 (Java HotSpot(TM) Client VM, Java 1.7.0_21)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc.
SQL context available as sqlContext.
scala> studentsstudentsstudentsstudents

谢谢

假设你想看到100行，你也可以使用

df.show(100)
sqlContext.sql("SELECT * FROM students where uid>10").show(100)

方法的链接

sqlContext。sql("SELECT * FROM students")返回一个DataFrame实例，参考DataFrame API。要检索不同的行数，可以调用"。take(n)":

sqlContext.sql("SELECT * FROM students where uid>10").take(30).foreach(println)

相关内容

  • 没有找到相关文章

最新更新


热门标签：

javascript python java c# php android html jquery c++ css ios sql mysql arrays asp.net json python-3.x ruby-on-rails .net sql-server django objective-c excel regex ruby linux ajax iphone xml vba spring asp.net-mvc database wordpress string postgresql wpf windows xcode bash git oracle list vb.net multithreading eclipse algorithm macos powershell visual-studio image forms numpy scala function api selenium