i用 openssl
加密文件,然后将其放在HDFS上,我使用了AES/ECB,128位和盐选项,并且在某些研究中,我发现openSSL使用PKCS5 PADDING作为默认值默认为隐形丝载体类。这是我的加密过程:
# echo -n "password" > .pw
# openssl enc -aes-128-ecb -salt -in .pw -out .pw.enc
# hdfs dfs -put .pw.enc /user/user1/
sqoop版本是1.4.6
命令:
sqoop import
-Dorg.apache.sqoop.credentials.loader.class=org.apache.sqoop.util.password.CryptoFileLoader
-Dorg.apache.sqoop.credentials.loader.crypto.passphrase=sqoop
--connect jdbc:oracle:thin:@host/database
--username user1
--password-file /user/user1/.pw.enc
--table db.table1
--hive-import
--hive-overwrite
--hive-table hivedb.table1
--hive-drop-import-delims
给出:
17/03/08 15:10:37 WARN tool.BaseSqoopTool: Failed to load password file
java.io.IOException: Can't decrypt the password
at org.apache.sqoop.util.password.CryptoFileLoader.loadPassword(CryptoFileLoader.java:151)
at org.apache.sqoop.util.CredentialsUtil.fetchPasswordFromLoader(CredentialsUtil.java:81)
at org.apache.sqoop.util.CredentialsUtil.fetchPassword(CredentialsUtil.java:66)
at org.apache.sqoop.tool.BaseSqoopTool.applyCredentialsOptions(BaseSqoopTool.java:1042)
at org.apache.sqoop.tool.BaseSqoopTool.applyCommonOptions(BaseSqoopTool.java:997)
at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:875)
at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435)
at org.apache.sqoop.Sqoop.run(Sqoop.java:131)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:966)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:824)
at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:436)
at javax.crypto.Cipher.doFinal(Cipher.java:2165)
at org.apache.sqoop.util.password.CryptoFileLoader.loadPassword(CryptoFileLoader.java:149)
... 12 more
Error while loading password file: Can't decrypt the password
我也尝试手动给出其他隐形磁中加载参数,并将本地文件传递给--password-file
。
我可以用openssl
成功解密文件。我不能解密Java程序(?)
我看到填充有一个问题,但我不知道它是什么,以及如何使用某种填充方法或其他要做的任何事情加密文件,我没有经验丰富。
类中也有org.apache.sqoop.credentials.loader.crypto.iterations
参数,指示PBKDF2迭代的数量,但我不知道它是否更改任何内容。
感谢您的任何帮助。
我不是SQoop和Hadoop的专家,但从您的例外开始
CryptoFileLoader.loadPassword(CryptoFileLoader.java:151)
我看了一个cryptofofileloader.java
的源代码在我看来,情况与您的工作有些不同:使用 pbkdf2 算法将密码存储在加密文件中,该算法不等于应用AES-128-ECB。来自Wikipedia:
pbkdf2应用伪随机函数,例如基于哈希的消息身份验证代码(HMAC),以及输入密码或密码以及盐值以及盐值,并多次重复该过程以产生派生的键,然后可以用作其使用,然后将其用作派生的键后续操作中的加密密钥。附加的计算工作使密码的破解更加困难,被称为关键拉伸。
无法从OpenSSL命令行进行PBKDF2。我使用Java进行了一个小测试,这可能是替代性
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.security.NoSuchAlgorithmException;
import javax.crypto.Cipher;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.SecretKeySpec;
public class Test {
/*Default is AES in electronic code book with padding.*/
private static String DEFAULT_ALG = "AES/ECB/PKCS5Padding";
/*Default salt is not much secure, use your own!*/
private static String DEFAULT_SALT = "SALT";
/*Iterate 10000 times by default.*/
private static int DEFAULT_ITERATIONS = 10000;
/*One of valid key sizes for default algorithm (AES).*/
private static int DEFAULT_KEY_LEN = 128;
public static void main(String[] args) throws IOException {
String inputFileName = "C:\temp\in.txt"; /*Enter your input (plain) file path */
String outputFileName = "C:\temp\out.bin"; /*Enter your output (encrypted) file path */
String passPhrase = "mypassphrase"; /*Enter your passphrase */
String salt = DEFAULT_SALT;
String alg = DEFAULT_ALG;
int iterations = DEFAULT_ITERATIONS;
int keyLen = DEFAULT_KEY_LEN;
SecretKeyFactory factory = null;
try {
factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
} catch (NoSuchAlgorithmException e) {
throw new IOException("Can't load SecretKeyFactory", e);
}
SecretKeySpec key = null;
try {
String algOnly = alg.split("/")[0];
key = new SecretKeySpec(
factory.generateSecret(
new PBEKeySpec(passPhrase.toCharArray(), salt.getBytes(), iterations, keyLen)).getEncoded(),
algOnly);
} catch (Exception e) {
throw new IOException("Can't generate secret key", e);
}
Cipher crypto = null;
try {
crypto = Cipher.getInstance(alg);
} catch (Exception e) {
throw new IOException("Can't initialize the decryptor", e);
}
Path inputFileLocation = Paths.get(inputFileName);
byte[] decrypted = Files.readAllBytes(inputFileLocation);
byte[] encrypted;
try {
crypto.init(Cipher.ENCRYPT_MODE, key);
encrypted = crypto.doFinal(decrypted);
} catch (Exception e) {
throw new IOException("Can't decrypt the password", e);
}
Path outputFileLocation = Paths.get(outputFileName);
Files.write(outputFileLocation, encrypted);
}
}
如西蒙妮(Simone)的答案中 - openssl和java实现之间的加密算法存在差异。这就是为什么您可以毫无问题地使用openssl解密(因为它再次调用(不同的)算法)。
经过大量的挖掘,我从(Dave Thompson)找到了这个答案:
简短答案:openssl enc(无-k for raw)的使用不是pbkdf2;它几乎是PBKDF1,带有迭代计数1。
似乎有两种方式解决这个问题:
a)在Java中找到可以解密OpenSSL正在做的事情的东西 - 此答案中的帖子中引用了一个Java库" Bouncycastle"(如果您很高兴使用它而不是标准的隐性firtofile)OpenSSL使用的算法。
或
b)找到实现PBKDF2的其他命令行实用程序,而不是OpenSSL。Nabble.com发布中也提到了不同语言的许多实现。
。(值得为戴夫(Dave)归功于所引用的关键观察)