如何在Java中获得UTF8字符集的完整列表

Question

我想添加一个测试套件，将运行在整个Unicode字符集。是否有一种方法来获得Unicode字符的完整列表?大多数在线资源讨论如何编码和解码，但没有找到讨论如何获得完整列表的有用材料。

Answer 1

TL;DR:您可能需要跳到"可见代码点";

---

所有代码点

每个Unicode字符(码点)都可以用UTF-8编码。正如维基百科所说:

UTF-8能够使用1到4个1字节(8位)编码单元对Unicode中所有1,112,064个有效字符编码点进行编码。

Unicode由1,114,112个码位组成，范围从0_hex到10FFFF_hex。

那么，要获取所有UTF-8字符:

// Build string with every Unicode character
int[] codePoints = new int[0x110000]; // 0 - 0x10FFFF
for (int i = 0; i < codePoints.length; i++)
codePoints[i] = i;
String allChars = new String(codePoints, 0, codePoints.length);
// Convert to UTF-8
byte[] allUtf8Sequences = allChars.getBytes(StandardCharsets.UTF_8);
// Print statistics
System.out.printf("Code points: %d = 0x%1$x%n", codePoints.length);
System.out.printf("Java chars : %d = 0x%1$x%n", allChars.length());
System.out.printf("  Surrogate pairs: %d = 0x%1$x%n", allChars.length() - codePoints.length);
System.out.printf("UTF-8 bytes: %d = 0x%1$x%n", allUtf8Sequences.length);
System.out.printf("  Average bytes per code point: %.2f%n", (double) allUtf8Sequences.length / codePoints.length);

输出

Code points: 1114112 = 0x110000
Java chars : 2162688 = 0x210000
Surrogate pairs: 1048576 = 0x100000
UTF-8 bytes: 4384642 = 0x42e782
Average bytes per code point: 3.94

---

可见代码点

请注意，并非所有代码点当前都是由Unicode定义的。如果您希望限制为已定义的字符，请使用Character.isDefined(codePoint)。

您可能也不想跳过控制字符和空白字符。要跳过所有这些，只检查可见字符，我们可以使用Character.getType(codePoint):

检查字符类型。

// Build string with visible Unicode characters
int[] codePoints = new int[Character.MAX_CODE_POINT + 1];
int count = 0;
for (int codePoint = 0; codePoint < codePoints.length; codePoint++) {
switch (Character.getType(codePoint)) {
case Character.UNASSIGNED:
case Character.CONTROL:     // Cc
case Character.FORMAT:      // Cf
case Character.PRIVATE_USE: // Co
case Character.SURROGATE:   // Cs
case Character.SPACE_SEPARATOR:     // Zs
case Character.LINE_SEPARATOR:      // Zl
case Character.PARAGRAPH_SEPARATOR: // Zp
break; // Skip
default:
codePoints[count++] = codePoint;
}
}
String chars = new String(codePoints, 0, count);
// Convert to UTF-8
byte[] utf8bytes = chars.getBytes(StandardCharsets.UTF_8);
// Print statistics
System.out.printf("Code points: %d = 0x%1$x%n", count);
System.out.printf("Java chars : %d = 0x%1$x%n", chars.length());
System.out.printf("  Surrogate pairs: %d = 0x%1$x%n", chars.length() - count);
System.out.printf("UTF-8 bytes: %d = 0x%1$x%n", utf8bytes.length);
System.out.printf("  Average bytes per code point: %.2f%n", (double) utf8bytes.length / count);

输出

Code points: 143679 = 0x2313f
Java chars : 231980 = 0x38a2c
Surrogate pairs: 88301 = 0x158ed
UTF-8 bytes: 517331 = 0x7e4d3
Average bytes per code point: 3.60