Sqoop connect with SQL Anywhere 17

Question

我正在尝试让sqoop与sqlanywhere17的sajdbc4.jar一起工作。我不断收到库错误，因为命令无法找到 jar 所需的.so相关文件。

现在，我创建了一个符号链接，用于sajdbc4.jar -> /opt/sqlanywhere17/java/sajdbc4.jar/usr/hdp/current/sqoop-client/lib/。

ln -s  /opt/sqlanywhere17/java/sajdbc4.jar /usr/hdp/current/sqoop-client/lib/sajdbc4.jar 

我收到此错误：

线程 "main" java.lang.UnsatisfiedLinkError： no dbjdbc17 in java.library.path

遵循本指南和本文档后：

cat /opt/sqlanywhere17/bin64/sa_config.sh >> ~/.bashrc
cat /opt/sqlanywhere17/bin64/sa_config.sh >> ~/.bash_profile

并检查echo $LD_LIBRARY_PATH：

/opt/sqlanywhere17/lib64:/opt/sqlanywhere17/bin64/jre180/lib/amd64/client:/opt/sqlanywhere17/bin64/jre180/lib/amd64/server:/opt/sqlanywhere17/bin64/jre180/lib/amd64:/opt/sqlanywhere17/bin64/jre180/lib/amd64/native_threads:

并检查java.library.path的：/opt/sqlanywhere17/lib64

命令行：

java -XshowSettings:properties

<小时 />

Property settings:
awt.toolkit = sun.awt.X11.XToolkit
com.sap.vm.codeline = 81_REL
com.sap.vm.compressedoops = true
com.sap.vm.profilingserver = true
com.sap.vm.type = opt
com.sap.vm.version = 10
file.encoding = UTF-8
file.encoding.pkg = sun.io
file.separator = /
java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
java.awt.printerjob = sun.print.PSPrinterJob
java.class.path = .
java.class.version = 52.0
java.endorsed.dirs = /opt/sqlanywhere17/bin64/jre180/lib/endorsed
java.ext.dirs = /opt/sqlanywhere17/bin64/jre180/lib/ext-sap
/opt/sqlanywhere17/bin64/jre180/lib/ext
/usr/java/packages/lib/ext
java.home = /opt/sqlanywhere17/bin64/jre180
java.io.tmpdir = /tmp
java.library.path = /opt/sqlanywhere17/lib64
/opt/sqlanywhere17/bin64/jre180/lib/amd64/client
/opt/sqlanywhere17/bin64/jre180/lib/amd64/server
/opt/sqlanywhere17/bin64/jre180/lib/amd64
/opt/sqlanywhere17/bin64/jre180/lib/amd64/native_threads

/usr/java/packages/lib/amd64
/usr/lib64
/lib64
/lib
/usr/lib

<小时 />

完全错误：

19/10/22 08:30:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.5.3.0-37
19/10/22 08:30:07 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
19/10/22 08:30:07 INFO manager.SqlManager: Using default fetchSize of 1000
Exception in thread "main" java.lang.UnsatisfiedLinkError: no dbjdbc17 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at sap.jdbc4.sqlanywhere.IDriver.try_load(IDriver.java:451)
at sap.jdbc4.sqlanywhere.IDriver.<clinit>(IDriver.java:413)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:873)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.tool.EvalSqlTool.run(EvalSqlTool.java:64)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.main(Sqoop.java:243)
[root@server1 dag_scripts]# echo $LD_LIBRARY_PATH
/opt/sqlanywhere17/lib64:/opt/sqlanywhere17/bin64/jre180/lib/amd64/client:/opt/sqlanywhere17/bin64/jre180/lib/amd64/server:/opt/sqlanywhere17/bin64/jre180/lib/amd64:/opt/sqlanywhere17/bin64/jre180/lib/amd64/native_threads:

<小时 /><小时 />

Answer 1

此解决方案目前似乎可以解决问题：

• ln -s /opt/sqlanywhere17/java/sajdbc4.jar /usr/hdp/current/sqoop-client/lib/sajdbc4.jar

• 根据 32 位/64 位要求选择/opt/sqlanywhere17/lib32或/opt/sqlanywhere17/lib64/

• 创建指向当前 JDK 路径和 JVM 路径的符号链接

• 打开访问/opt/sqlanywhere17/java/的权限(如果您以 root 用户身份安装)

• 确保用户是"hadoop"组的一部分 - 用于 sqoop 文件夹权限 (.env)

• 将默认导出添加到运行 sqoop 命令的任何用户

  ln -s /opt/sqlanywhere17/lib64/* /usr/jdk64/jdk1.8.0_112/jre/lib/amd64/
  ln -s /opt/sqlanywhere17/lib64/* /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64/jre/lib/amd64/
  chmod 755 -R /opt/sqlanywhere17/*
  usermod -a -G hadoop user
  #For user executing sqoop:
  cat /opt/sqlanywhere17/bin64/sa_config.sh >> ~/.bashrc
  

---

Sqoop 用于测试的命令：

对于 sajdbc4.jar：

sqoop eval --connect 'jdbc:sqlanywhere:Host=192.168.1.11:1234;ServerName=DemoServer;DatabaseName=demo' --driver sap.jdbc4.sqlanywhere.IDriver --username testuser --P --query "SELECT * from database.table"

对于 jconn4：

你也可以把 jconn4.jar 放到 sqoop 的 lib 目录中。

/usr/hdp/current/sqoop-client/lib/sajdbc4.jar

https://wiki.scn.sap.com/wiki/display/SYBCON/Software+Developer+Kit+for+ASE+driver+versions+for+Windows

sqoop eval --connect 'jdbc:sybase:Tds:192.168.1.11:1234' --driver com.sybase.jdbc4.jdbc.SybDriver --username testuser --P --query "SELECT * from database.table"

---

对于在 DBCPConnectionPool 中运行 SqlAnywhere17 的 NiFi：

删除所有符号链接和.bashrc配置文件设置，而是将 sqlanywhere (/opt/sqlanywhere17/bin64/sa_config.sh)的配置附加到"用于 nifi-env.sh 的 NiFi 节点高级配置片段(安全阀)"中。我能够让库正确加载。连接现在在 NiFi 中工作！

唯一有趣的错误是，如果我使用驱动程序创建 2 个 DBCPConnectionPool 的池，其中一个查询将失败并说：failed to process session due to Native Library /opt/sqlanywhere17/lib64/libdbjdbc17.so.1 already loaded in another classloader; Processor Administratively Yielded for 1 sec: java.lang.UnsatisfiedLinkError: Native Library /opt/sqlanywhere17/lib64/libdbjdbc17.so.1 already loaded in another classloader

所以目前我们使用 sajdbc4 的 1 个连接.jar我们的解决方法是使用 jconn4.jar 用于另一个。