关于Xml大文件的解析小结

柔光的暖阳◎ 2023-07-08 09:02 19阅读 0赞

1. 对于大文件，很多时候我们是不能直接通过普通的读文件解析的。这篇文章主要是总结下解析xml大文件的思路和代码。

2. **主要思路**，其实就是通过封装一个切割文件的工具类。如：每次读取部分文件内容，比如10M。Xml标签定位、标签匹配。

1) 假设有如下的test.xml (1.42kb)

<?xml version="1.0" encoding="UTF-8"?>
    <Data>
      <Header>
        <ContentDate>2020-02-03T01:00:00Z</ContentDate>
        <FileContent>BAS</FileContent>
        <DeltaStart>2020-02-02T17:00:00Z</DeltaStart>
      </Header>
      <Records>
    <Record>
    	   <BAS>123</BAS>
    	   <Entity>
    		  <LegalName>1 AVSUPER FUND</LegalName>
    		  <OtherEntityNames>
    			 <OtherEntityName xml:lang="1 en" type="TRADING_OR_OPERATING_NAME">1 AvSuper Pty Ltd as trustee for AvSuper Fund</OtherEntityName>
    		  </OtherEntityNames>
    	   </Entity>	   
    	  </Record>
    	  <Record>
    	   <BAS>456</BAS>
    	   <Entity>
    		  <LegalName>2 AVSUPER FUND</LegalName>
    		  <OtherEntityNames>
    			 <OtherEntityName xml:lang="2 en" type="TRADING_OR_OPERATING_NAME">2 AvSuper Pty Ltd as trustee for AvSuper Fund</OtherEntityName>
    		  </OtherEntityNames>
    	   </Entity>
    	  </Record>
    	  <Record>
    	   <BAS>789</BAS>
    	   <Entity>
    		  <LegalName>3 AVSUPER FUND</LegalName>
    		  <OtherEntityNames>
    			 <OtherEntityName xml:lang="3 en" type="TRADING_OR_OPERATING_NAME">3 AvSuper Pty Ltd as trustee for AvSuper Fund</OtherEntityName>
    		  </OtherEntityNames>
    	   </Entity>	   
    	  </Record>
    	  <Record>
    	   <BAS>1022</BAS>
    	   <Entity>
    		  <LegalName>4 AVSUPER FUND</LegalName>
    		  <OtherEntityNames>
    			 <OtherEntityName xml:lang="4 en" type="TRADING_OR_OPERATING_NAME">4 AvSuper Pty Ltd as trustee for AvSuper Fund</OtherEntityName>
    		  </OtherEntityNames>
    	   </Entity>	   
    	  </Record>
      </Records>
    </Data>

2) **读取大文件**。

public MappedBiggerFileReader(String fileName, int arraySize) throws IOException {
    	this.fileIn = new FileInputStream(fileName);
    	FileChannel fileChannel = fileIn.getChannel();
    	this.fileLength = fileChannel.size();
    	this.number = (int) Math.ceil((double) fileLength / (double) Integer.MAX_VALUE);
    	this.mappedBufArray = new MappedByteBuffer[number];// memory File Mapping Array
    	long preLength = 0;
    	long regionSize = (long) Integer.MAX_VALUE;// size of mapping region
    	for (int i = 0; i < number; i++) {
    		// mapping contiguous areas of files to memory file mapping arrays
    		if (fileLength - preLength < (long) Integer.MAX_VALUE) {
    			regionSize = fileLength - preLength;// the size of the last area
    		}
    		mappedBufArray[i] = fileChannel.map(FileChannel.MapMode.READ_ONLY, preLength, regionSize);
    		preLength += regionSize;// the beginning of the next area
    	}
    	this.arraySize = arraySize;
    }

测试：

public static void main(String[] args) throws IOException {
    	MappedBiggerFileReader reader = new MappedBiggerFileReader(
    			"D:\\test\\test.xml", 1 * 1024); // 1Kb
    	while (reader.read() != -1) {
    		System.out.println("===========================");
    		System.out.println(new String(reader.getArray()));
    	}
    	reader.close();
    }

注：等号上面读取了1kb的内容。

![20200228145558892.png][]

3) **Xml标签定位**。

从test.xml可以看出，我们要取的是<Record>的内容，所以我们现在需要截取上一步读取的1kb里面的<Record>和</Record>之间的所有内容了。

public static Range getRangeForTags(StringBuffer sfb, String tag) {
    	Range range = new Range();
    	range.setFrom(sfb.indexOf("<" + tag));
    	range.setTo(sfb.lastIndexOf("</" + tag + ">"));
    	return range;
    }

测试：

public static void main(String[] args) throws IOException {
    	MappedBiggerFileReader reader = new MappedBiggerFileReader(
    			"D:\\test\\test.xml", 1 * 1024); // 1Kb
    	StringBuffer cache = new StringBuffer();
    	while (reader.read() != -1) {
    		cache.append(new String(reader.getArray()));
    		Range range = Strkit.getRangeForTags(cache, "Record");
    		System.out.println(range);
    	}
    	reader.close();
    }

![20200228170737687.png][]

4) **Xml标签匹配查询**。

上一步取得了我们想要的标签的index，那么现在就是匹配里面的<Record></Record>。

public static List<String> getSubUtil(String soap,String rgex){
    	List<String> list = new ArrayList<String>();
    	Pattern pattern = Pattern.compile(rgex);
    	Matcher m = pattern.matcher(soap);
    	while (m.find()) {
    		int i = 1;
    		list.add(m.group(i));
    	}
    	return list;
    }

测试：

public static void main(String[] args) throws IOException {
    	StringBuffer cache = new StringBuffer();
    	cache.append("<Records>"
    			+ "<Record>\r\n" + 
    			"	   <BAS>123</BAS>\r\n" + 
    			"	   <Entity>\r\n" + 
    			"		  <LegalName>1 AVSUPER FUND</LegalName>\r\n" + 
    			"		  <OtherEntityNames>\r\n" + 
    			"			 <OtherEntityName xml:lang=\"1 en\" type=\"TRADING_OR_OPERATING_NAME\">1 AvSuper Pty Ltd as trustee for AvSuper Fund</OtherEntityName>\r\n" + 
    			"		  </OtherEntityNames>\r\n" + 
    			"	   </Entity>	   \r\n" + 
    			"	  </Record><Record>\r\n" + 
    			"	   <BAS>456</BAS>\r\n" + 
    			"	   <Entity>\r\n" + 
    			"		  <LegalName>2 AVSUPER FUND</LegalName>\r\n" + 
    			"		  <OtherEntityNames>\r\n" + 
    			"			 <OtherEntityName xml:lang=\"2 en\" type=\"TRADING_OR_OPERATING_NAME\">2 AvSuper Pty Ltd as trustee for AvSuper Fund</OtherEntityName>\r\n" + 
    			"		  </OtherEntityNames>\r\n" + 
    			"	   </Entity>\r\n" + 
    			"	  </Record>"
    			+ "</Records>");
    
    	StringBuffer texts = new StringBuffer();
    	String node = "Record";
    	String rgex = "<" + node + "" + "([\\s\\S]*?)</" + node + ">";
    	List<String> elemTexts = StrSplitUtil.getSubUtil(cache.toString(), rgex);
    	elemTexts.forEach(str -> {
    		str = str.replace("s><" + node,"");
    		String t = "<" + node + str + "</" + node + ">";
    		texts.append(t);
    	});
    	System.out.println(texts);
    }

注：去掉了首尾的<Records>。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBU3RyaXZlcg_size_16_color_FFFFFF_t_70][]

5) **集成的解析工具类**。

结合第3步获取到的index来删除cache里面的内容，实现Xml切割的核心思路。

public void parse() {
    	MappedBiggerFileReader reader = null;
    	try {
    		reader = new MappedBiggerFileReader(filePath, size * 1024);
    		StringBuffer cache = new StringBuffer();
    		while (reader.read() != -1) {
    			cache.append(new String(reader.getArray()));
    			Range range = Strkit.getRangeForTags(cache, node);
    
    			if (range.getFrom() >= 0) {
    				StringBuffer texts = new StringBuffer();
    
    				String rgex = "<" + node + "" + "([\\s\\S]*?)</" + node + ">";
    				List<String> elemTexts = StrSplitUtil.getSubUtil(cache.toString(), rgex);
    				elemTexts.forEach(str -> {
    					str = str.replace("s>\n<" + node + "", "");
    					String t = "<" + node + str + "</" + node + ">";
    					texts.append(t);
    				});
    				texts.insert(0, "<Root>");
    				texts.append("</Root>");
    
    				System.out.println("===============");
    				System.out.println(texts);
    				cache = cache.delete(0, range.getTo());
    			}
    		}
    	} catch (Exception e) {
    		e.printStackTrace();
    	}
    }

测试：

public static void main(String[] args) {
    	new XmlParseUtil("D:\\test\\test.xml", "Record", 1).parse();
    }

注：切割后的每一个Record前后都加上了<Root></Root>，方便之后再使用[XstreamUtil进行Xml解析][XstreamUtil_Xml]。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBU3RyaXZlcg_size_16_color_FFFFFF_t_70 1][]

3. 附上[源码下载][Link 1]，有什么不懂或者觉得有问题的欢迎留言讨论。

注：

1. 如果遇到：Xstream NumberFormatException: Zero length string...

可能是因为你测试的xml文件中，有部分数据节点存在attribute但是这个attribute的值是空的。

[20200228145558892.png]: https://img-blog.csdnimg.cn/20200228145558892.png
[20200228170737687.png]: https://img-blog.csdnimg.cn/20200228170737687.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBU3RyaXZlcg_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200228163041193.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBU3RyaXZlcg==,size_16,color_FFFFFF,t_70
[XstreamUtil_Xml]: https://blog.csdn.net/BAStriver/article/details/104063006
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBU3RyaXZlcg_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200228171221996.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBU3RyaXZlcg==,size_16,color_FFFFFF,t_70
[Link 1]: https://download.csdn.net/download/BAStriver/12198584