13、Solr4.8.0源码分析(13)之LuceneCore的索引修复
Solr4.8.0源码分析(13)之LuceneCore的索引修复
题记：今天在公司研究elasticsearch,突然看到一篇博客说elasticsearch具有索引修复功能，顿感好奇，于是点进去看了下，发现原来是Lucene Core自带的功能。说实话之前学习Lucene文件格式的时候就想做一个索引文件解析和检测的工具，也动手写了一部分，最后没想到发现了一个已有的工具，正好对照着学习下。
索引的修复主要是用到CheckIndex.java这个类，可以直接查看类的Main函数来了解下。
1、 CheckIndex的使用；
首先使用以下命令来查看lucenecore.jar怎么使用：
1 192:lib rcf$ java -cp lucene-core-4.8-SNAPSHOT.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex
2
3 ERROR: index path not specified
4
5 Usage: java org.apache.lucene.index.CheckIndex pathToIndex [-fix] [-crossCheckTermVectors] [-segment X] [-segment Y] [-dir-impl X]
6
7 -fix: actually write a new segments_N file, removing any problematic segments
8 -crossCheckTermVectors: verifies that term vectors match postings; THIS IS VERY SLOW!
9 -codec X: when fixing, codec to write the new segments_N file with
10 -verbose: print additional details
11 -segment X: only check the specified segments. This can be specified multiple
12 times, to check more than one segment, eg '-segment _2 -segment _a'.
13 You can't use this with the -fix option
14 -dir-impl X: use a specific FSDirectory implementation. If no package is specified the org.apache.lucene.store package will be used.
15
16 **WARNING**: -fix should only be used on an emergency basis as it will cause
17 documents (perhaps many) to be permanently removed from the index. Always make
18 a backup copy of your index before running this! Do not run this tool on an index
19 that is actively being written to. You have been warned!
20
21 Run without -fix, this tool will open the index, report version information
22 and report any exceptions it hits and what action it would take if -fix were
23 specified. With -fix, this tool will remove any segments that have issues and
24 write a new segments_N file. This means all documents contained in the affected
25 segments will be removed.
26
27 This tool exits with exit code 1 if the index cannot be opened or has any
28 corruption, else 0.
当敲java -cp lucene-core-4.8-SNAPSHOT.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex 这个就能看到相当于help的信息啦，但是为什么这里用这么一串奇怪的命令呢？通过java -help来查看-cp 以及 -ea就可以发现，-cp其实等同于-classpath 提供类和jar的搜索路径，-ea等同于-enableassertions提供是否启动断言设置。所以上述的命令其实可以简化为java -cp lucene-core-4.8-SNAPSHOT.jar org.apache.lucene.index.CheckIndex 。
首先来检查下索引的情况：可以看出信息蛮清楚明了的。
1 userdeMacBook-Pro:lib rcf$ java -cp lucene-core-4.8-SNAPSHOT.jar -ea:org.apache.lucene.index... org.apache.lucene.index.CheckIndex ../../../../../../solr/Solr/test/data/index
2
3 Opening index @ ../../../../../../solr/Solr/test/data/index
4
5 Segments file=segments_r numSegments=7 version=4.8 format= userData={commitTimeMSec=1411221019854}
6 1 of 7: name=_k docCount=18001
7 codec=Lucene46
8 compound=false
9 numFiles=10
10 size (MB)=0.493
11 diagnostics = {timestamp=1411221019346, os=Mac OS X, os.version=10.9.4, mergeFactor=10, source=merge, lucene.version=4.8-SNAPSHOT Unversioned directory - rcf - 2014-09-20 21:11:36, os.arch=x86_64, mergeMaxNumSegments=-1, java.version=1.7.0_60, java.vendor=Oracle Corporation}
12 no deletions
13 test: open reader.........OK
14 test: check integrity.....OK
15 test: check live docs.....OK
16 test: fields..............OK [3 fields]
17 test: field norms.........OK [1 fields]
18 test: terms, freq, prox...OK [36091 terms; 54003 terms/docs pairs; 18001 tokens]
19 test: stored fields.......OK [54003 total field count; avg 3 fields per doc]
20 test: term vectors........OK [0 total vector count; avg 0 term/freq vector fields per doc]
21 test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 SORTED; 0 SORTED_SET]
22
23 2 of 7: name=_l docCount=1000
24 codec=Lucene46
25 compound=false
26 numFiles=10
27 size (MB)=0.028
28 diagnostics = {timestamp=1411221019406, os=Mac OS X, os.version=10.9.4, source=flush, lucene.version=4.8-SNAPSHOT Unversioned directory - rcf - 2014-09-20 21:11:36, os.arch=x86_64, java.version=1.7.0_60, java.vendor=Oracle Corporation}
29 no deletions
30 test: open reader.........OK
31 test: check integrity.....OK
32 test: check live docs.....OK
33 test: fields..............OK [3 fields]
34 test: field norms.........OK [1 fields]
35 test: terms, freq, prox...OK [2002 terms; 3000 terms/docs pairs; 1000 tokens]
36 test: stored fields.......OK [3000 total field count; avg 3 fields per doc]
37 test: term vectors........OK [0 total vector count; avg 0 term/freq vector fields per doc]
38 test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 SORTED; 0 SORTED_SET]
39
40 3 of 7: name=_m docCount=1000
41 codec=Lucene46
42 compound=false
43 numFiles=10
44 size (MB)=0.028
45 diagnostics = {timestamp=1411221019478, os=Mac OS X, os.version=10.9.4, source=flush, lucene.version=4.8-SNAPSHOT Unversioned directory - rcf - 2014-09-20 21:11:36, os.arch=x86_64, java.version=1.7.0_60, java.vendor=Oracle Corporation}
46 no deletions
47 test: open reader.........OK
48 test: check integrity.....OK
49 test: check live docs.....OK
50 test: fields..............OK [3 fields]
51 test: field norms.........OK [1 fields]
52 test: terms, freq, prox...OK [2002 terms; 3000 terms/docs pairs; 1000 tokens]
53 test: stored fields.......OK [3000 total field count; avg 3 fields per doc]
54 test: term vectors........OK [0 total vector count; avg 0 term/freq vector fields per doc]
55 test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 SORTED; 0 SORTED_SET]
56
57 4 of 7: name=_n docCount=1000
58 codec=Lucene46
59 compound=false
60 numFiles=10
61 size (MB)=0.028
62 diagnostics = {timestamp=1411221019552, os=Mac OS X, os.version=10.9.4, source=flush, lucene.version=4.8-SNAPSHOT Unversioned directory - rcf - 2014-09-20 21:11:36, os.arch=x86_64, java.version=1.7.0_60, java.vendor=Oracle Corporation}
63 no deletions
64 test: open reader.........OK
65 test: check integrity.....OK
66 test: check live docs.....OK
67 test: fields..............OK [3 fields]
68 test: field norms.........OK [1 fields]
69 test: terms, freq, prox...OK [2002 terms; 3000 terms/docs pairs; 1000 tokens]
70 test: stored fields.......OK [3000 total field count; avg 3 fields per doc]
71 test: term vectors........OK [0 total vector count; avg 0 term/freq vector fields per doc]
72 test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 SORTED; 0 SORTED_SET]
73
74 5 of 7: name=_o docCount=1000
75 codec=Lucene46
76 compound=false
77 numFiles=10
78 size (MB)=0.028
79 diagnostics = {timestamp=1411221019629, os=Mac OS X, os.version=10.9.4, source=flush, lucene.version=4.8-SNAPSHOT Unversioned directory - rcf - 2014-09-20 21:11:36, os.arch=x86_64, java.version=1.7.0_60, java.vendor=Oracle Corporation}
80 no deletions
81 test: open reader.........OK
82 test: check integrity.....OK
83 test: check live docs.....OK
84 test: fields..............OK [3 fields]
85 test: field norms.........OK [1 fields]
86 test: terms, freq, prox...OK [2002 terms; 3000 terms/docs pairs; 1000 tokens]
87 test: stored fields.......OK [3000 total field count; avg 3 fields per doc]
88 test: term vectors........OK [0 total vector count; avg 0 term/freq vector fields per doc]
89 test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 SORTED; 0 SORTED_SET]
90
91 6 of 7: name=_p docCount=1000
92 codec=Lucene46
93 compound=false
94 numFiles=10
95 size (MB)=0.028
96 diagnostics = {timestamp=1411221019739, os=Mac OS X, os.version=10.9.4, source=flush, lucene.version=4.8-SNAPSHOT Unversioned directory - rcf - 2014-09-20 21:11:36, os.arch=x86_64, java.version=1.7.0_60, java.vendor=Oracle Corporation}
97 no deletions
98 test: open reader.........OK
99 test: check integrity.....OK
100 test: check live docs.....OK
101 test: fields..............OK [3 fields]
102 test: field norms.........OK [1 fields]
103 test: terms, freq, prox...OK [2002 terms; 3000 terms/docs pairs; 1000 tokens]
104 test: stored fields.......OK [3000 total field count; avg 3 fields per doc]
105 test: term vectors........OK [0 total vector count; avg 0 term/freq vector fields per doc]
106 test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 SORTED; 0 SORTED_SET]
107
108 7 of 7: name=_q docCount=1000
109 codec=Lucene46
110 compound=false
111 numFiles=10
112 size (MB)=0.027
113 diagnostics = {timestamp=1411221019863, os=Mac OS X, os.version=10.9.4, source=flush, lucene.version=4.8-SNAPSHOT Unversioned directory - rcf - 2014-09-20 21:11:36, os.arch=x86_64, java.version=1.7.0_60, java.vendor=Oracle Corporation}
114 no deletions
115 test: open reader.........OK
116 test: check integrity.....OK
117 test: check live docs.....OK
118 test: fields..............OK [3 fields]
119 test: field norms.........OK [1 fields]
120 test: terms, freq, prox...OK [2001 terms; 3000 terms/docs pairs; 1000 tokens]
121 test: stored fields.......OK [3000 total field count; avg 3 fields per doc]
122 test: term vectors........OK [0 total vector count; avg 0 term/freq vector fields per doc]
123 test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 SORTED; 0 SORTED_SET]
124
125 No problems were detected with this index.
由于我的索引文件是正常的，那么通过网上的例子来查看下错误的情况下是什么样子的，并且-fix是怎么样子的效果：
来自网友：http://blog.csdn.net/laigood/article/details/8296678
1 Segments file=segments_2cg numSegments=26 version=3.6.1 format=FORMAT_3_1 [Lucene 3.1+] userData={translog_id=1347536741715}
2 1 of 26: name=_59ct docCount=4711242
3 compound=false
4 hasProx=true
5 numFiles=9
6 size (MB)=6,233.694
7 diagnostics = {mergeFactor=13, os.version=2.6.32-71.el6.x86_64, os=Linux, lucene.version=3.6.1 1362471 - thetaphi - 2012-07-17 12:40:12, source=merge, os.arch=amd64, mergeMaxNumSegments=-1, java.version=1.6.0_24, java.vendor=Sun Microsystems Inc.}
8 has deletions [delFileName=_59ct_1b.del]
9 test: open reader.........OK [3107 deleted docs]
10 test: fields..............OK [25 fields]
11 test: field norms.........OK [10 fields]
12 test: terms, freq, prox...OK [36504908 terms; 617641081 terms/docs pairs; 742052507 tokens]
13 test: stored fields.......ERROR [read past EOF: MMapIndexInput(path="/usr/local/sas/escluster/data/cluster/nodes/0/indices/index/5/index/_59ct.fdt")]
14 java.io.EOFException: read past EOF: MMapIndexInput(path="/usr/local/sas/escluster/data/cluster/nodes/0/indices/index/5/index/_59ct.fdt")
15 at org.apache.lucene.store.MMapDirectory$MMapIndexInput.readBytes(MMapDirectory.java:307)
16 at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:400)
17 at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:253)
18 at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:492)
19 at org.apache.lucene.index.IndexReader.document(IndexReader.java:1138)
20 at org.apache.lucene.index.CheckIndex.testStoredFields(CheckIndex.java:852)
21 at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:581)
22 at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064)
23 test: term vectors........OK [0 total vector count; avg 0 term/freq vector fields per doc]
24 FAILED
25 WARNING: fixIndex() would remove reference to this segment; full exception:
26 java.lang.RuntimeException: Stored Field test failed
27 at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:593)
28 at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064)
29
30
31 WARNING: 1 broken segments (containing 4708135 documents) detected
32 WARNING: 4708135 documents will be lost
在检查结果中可以看到，分片5的_59ct.fdt索引文件损坏，.fdt文件主要存储lucene索引中存储的fields，所以在检查test: stored fields时出错。
下面的警告是说有一个损坏了的segment，里面有4708135个文档。
在原来的命令基础上加上-fix参数可以进行修复索引操作（ps：在进行修改前最好对要修复的索引进行备份，不要在正在执行写操作的索引上执行修复。）
1 java -cp lucene-core-3.6.1.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /usr/local/sas/escluster/data/cluster/nodes/0/indices/index/5/index/ -fix
2 NOTE: will write new segments file in 5 seconds; this will remove 4708135 docs from the index. THIS IS YOUR LAST CHANCE TO CTRL+C!
3 5...
4 4...
5 3...
6 2...
7 1...
8 Writing...
9 OK
10 Wrote new segments file "segments_2ch"
还可以通过检查某一个segment：
1 userdeMacBook-Pro:lib rcf$ java -cp lucene-core-4.8-SNAPSHOT.jar -ea:org.apache.lucene.index... org.apache.lucene.index.CheckIndex ../../../../../../solr/Solr/test/data/index -segment _9
2
3 Opening index @ ../../../../../../solr/Solr/test/data/index
4
5 Segments file=segments_r numSegments=7 version=4.8 format= userData={commitTimeMSec=1411221019854}
6
7 Checking only these segments: _9:
8 No problems were detected with this index.
还可以通过-verbose查看更多详细信息，这里就不在详述。
2、 CheckIndex的源码；
接着我们再来学习下CheckIndex的源码是怎么来实现以上功能的，检查索引的功能主要集中在checkindex()函数上，
1 public Status checkIndex(List<String> onlySegments) throws IOException {
2 ...
3 final int numSegments = sis.size(); //获取segment个数
4 final String segmentsFileName = sis.getSegmentsFileName(); //获取segment_N名字
5 // note: we only read the format byte (required preamble) here!
6 IndexInput input = null;
7 try {
8 input = dir.openInput(segmentsFileName, IOContext.READONCE);//读取segment_N文件
9 } catch (Throwable t) {
10 msg(infoStream, "ERROR: could not open segments file in directory");
11 if (infoStream != null)
12 t.printStackTrace(infoStream);
13 result.cantOpenSegments = true;
14 return result;
15 }
16 int format = 0;
17 try {
18 format = input.readInt(); //读取segment_N version
19 } catch (Throwable t) {
20 msg(infoStream, "ERROR: could not read segment file version in directory");
21 if (infoStream != null)
22 t.printStackTrace(infoStream);
23 result.missingSegmentVersion = true;
24 return result;
25 } finally {
26 if (input != null)
27 input.close();
28 }
29
30 String sFormat = "";
31 boolean skip = false;
32
33 result.segmentsFileName = segmentsFileName;//segment_N名字
34 result.numSegments = numSegments; //segment个数
35 result.userData = sis.getUserData(); //获取user信息，如userData={commitTimeMSec=1411221019854}
36 String userDataString;
37 if (sis.getUserData().size() > 0) {
38 userDataString = " userData=" + sis.getUserData();
39 } else {
40 userDataString = "";
41 }
42 //获取版本信息，如version=4.8
43 String versionString = null;
44 if (oldSegs != null) {
45 if (foundNonNullVersion) {
46 versionString = "versions=[" + oldSegs + " .. " + newest + "]";
47 } else {
48 versionString = "version=" + oldSegs;
49 }
50 } else {
51 versionString = oldest.equals(newest) ? ( "version=" + oldest ) : ("versions=[" + oldest + " .. " + newest + "]");
52 }
53
54 msg(infoStream, "Segments file=" + segmentsFileName + " numSegments=" + numSegments
55 + " " + versionString + " format=" + sFormat + userDataString);
56
57 if (onlySegments != null) {
58 result.partial = true;
59 if (infoStream != null) {
60 infoStream.print("\nChecking only these segments:");
61 for (String s : onlySegments) {
62 infoStream.print(" " + s);
63 }
64 }
65 result.segmentsChecked.addAll(onlySegments);
66 msg(infoStream, ":");
67 }
68
69 if (skip) {
70 msg(infoStream, "\nERROR: this index appears to be created by a newer version of Lucene than this tool was compiled on; please re-compile this tool on the matching version of Lucene; exiting");
71 result.toolOutOfDate = true;
72 return result;
73 }
74
75
76 result.newSegments = sis.clone();
77 result.newSegments.clear();
78 result.maxSegmentName = -1;
79 //开始遍历segment，检查segment
80 for(int i=0;i<numSegments;i++) {
81 final SegmentCommitInfo info = sis.info(i); //获取segment信息
82 int segmentName = Integer.parseInt(info.info.name.substring(1), Character.MAX_RADIX);
83 if (segmentName > result.maxSegmentName) {
84 result.maxSegmentName = segmentName;
85 }
86 if (onlySegments != null && !onlySegments.contains(info.info.name)) {
87 continue;
88 }
89 Status.SegmentInfoStatus segInfoStat = new Status.SegmentInfoStatus();
90 result.segmentInfos.add(segInfoStat);
91 //获取segments编号，名字，document个数，如下信息 1 of 7: name=_k docCount=18001
92 msg(infoStream, " " + (1+i) + " of " + numSegments + ": name=" + info.info.name + " docCount=" + info.info.getDocCount());
93 segInfoStat.name = info.info.name;
94 segInfoStat.docCount = info.info.getDocCount();
95
96 final String version = info.info.getVersion();
97 if (info.info.getDocCount() <= 0 && version != null && versionComparator.compare(version, "4.5") >= 0) {
98 throw new RuntimeException("illegal number of documents: maxDoc=" + info.info.getDocCount());
99 }
100
101 int toLoseDocCount = info.info.getDocCount();
102
103 AtomicReader reader = null;
104
105 try {
106 final Codec codec = info.info.getCodec(); //获取codec信息，如codec=Lucene46
107 msg(infoStream, " codec=" + codec);
108 segInfoStat.codec = codec;
109 msg(infoStream, " compound=" + info.info.getUseCompoundFile());//获取复合文档格式标志位：compound=false
110 segInfoStat.compound = info.info.getUseCompoundFile();
111 msg(infoStream, " numFiles=" + info.files().size());
112 segInfoStat.numFiles = info.files().size(); //获取段内文件个数numFiles=10
113 segInfoStat.sizeMB = info.sizeInBytes()/(1024.*1024.);//获取segment大小如size (MB)=0.493
114 if (info.info.getAttribute(Lucene3xSegmentInfoFormat.DS_OFFSET_KEY) == null) {
115 // don't print size in bytes if its a 3.0 segment with shared docstores
116 msg(infoStream, " size (MB)=" + nf.format(segInfoStat.sizeMB));
117 }
118 //获取诊断信息，diagnostics = {timestamp=1411221019346, os=Mac OS X, os.version=10.9.4, mergeFactor=10,
119 //source=merge, lucene.version=4.8-SNAPSHOT Unversioned directory - rcf - 2014-09-20 21:11:36,
120 //os.arch=x86_64, mergeMaxNumSegments=-1, java.version=1.7.0_60, java.vendor=Oracle Corporation}
121 Map<String,String> diagnostics = info.info.getDiagnostics();
122 segInfoStat.diagnostics = diagnostics;
123 if (diagnostics.size() > 0) {
124 msg(infoStream, " diagnostics = " + diagnostics);
125 }
126 //判断是否有document删除，如输出no deletions 或者 has deletions [delFileName=_59ct_1b.del]
127 if (!info.hasDeletions()) {
128 msg(infoStream, " no deletions");
129 segInfoStat.hasDeletions = false;
130 }
131 else{
132 msg(infoStream, " has deletions [delGen=" + info.getDelGen() + "]");
133 segInfoStat.hasDeletions = true;
134 segInfoStat.deletionsGen = info.getDelGen();
135 }
136
137 //通过新建SegmentReader对象来检查是否能获取索引reader，如果扔出错误说明不能打开。例如输出test: open reader.........OK，
138 if (infoStream != null)
139 infoStream.print(" test: open reader.........");
140 reader = new SegmentReader(info, DirectoryReader.DEFAULT_TERMS_INDEX_DIVISOR, IOContext.DEFAULT);
141 msg(infoStream, "OK");
142
143 segInfoStat.openReaderPassed = true;
144 //通过checkIntegrity来检查文件的完整性，例如输出：test: check integrity.....OK
145 //checkIntegrity函数是通过CodecUtil.checksumEntireFile()实现来检查文件的完整。
146 if (infoStream != null)
147 infoStream.print(" test: check integrity.....");
148 reader.checkIntegrity();
149 msg(infoStream, "OK");
150
151 //检查document数量是否正确，如果有删除的document时候，通过reader.numDocs()
152 //== info.info.getDocCount() - info.getDelCount()以及统计以及检测livedocs的个数是否于reader.numDocs() 一致。
153 //注意：没有删除的document时候，livedocs为null；
154 //当没有删除的document时候，reader.maxDoc() == info.info.getDocCount()来检查是否一致。
155 //输出结果比如：test: check live docs.....OK
156 //solr的管理界面可以显示： 现有document，删除的document以及所有的document数量，获取方法就是如下。
157 if (infoStream != null)
158 infoStream.print(" test: check live docs.....");
159 final int numDocs = reader.numDocs();
160 toLoseDocCount = numDocs;
161 if (reader.hasDeletions()) {
162 if (reader.numDocs() != info.info.getDocCount() - info.getDelCount()) {
163 throw new RuntimeException("delete count mismatch: info=" + (info.info.getDocCount() - info.getDelCount()) + " vs reader=" + reader.numDocs());
164 }
165 if ((info.info.getDocCount()-reader.numDocs()) > reader.maxDoc()) {
166 throw new RuntimeException("too many deleted docs: maxDoc()=" + reader.maxDoc() + " vs del count=" + (info.info.getDocCount()-reader.numDocs()));
167 }
168 if (info.info.getDocCount() - numDocs != info.getDelCount()) {
169 throw new RuntimeException("delete count mismatch: info=" + info.getDelCount() + " vs reader=" + (info.info.getDocCount() - numDocs));
170 }
171 Bits liveDocs = reader.getLiveDocs();
172 if (liveDocs == null) {
173 throw new RuntimeException("segment should have deletions, but liveDocs is null");
174 } else {
175 int numLive = 0;
176 for (int j = 0; j < liveDocs.length(); j++) {
177 if (liveDocs.get(j)) {
178 numLive++;
179 }
180 }
181 if (numLive != numDocs) {
182 throw new RuntimeException("liveDocs count mismatch: info=" + numDocs + ", vs bits=" + numLive);
183 }
184 }
185
186 segInfoStat.numDeleted = info.info.getDocCount() - numDocs;
187 msg(infoStream, "OK [" + (segInfoStat.numDeleted) + " deleted docs]");
188 } else {
189 if (info.getDelCount() != 0) {
190 throw new RuntimeException("delete count mismatch: info=" + info.getDelCount() + " vs reader=" + (info.info.getDocCount() - numDocs));
191 }
192 Bits liveDocs = reader.getLiveDocs();
193 if (liveDocs != null) {
194 // its ok for it to be non-null here, as long as none are set right?
195 // 这里好像有点问题，当delete document不存在时候，liveDocs应该为null。
196 for (int j = 0; j < liveDocs.length(); j++) {
197 if (!liveDocs.get(j)) {
198 throw new RuntimeException("liveDocs mismatch: info says no deletions but doc " + j + " is deleted.");
199 }
200 }
201 }
202 msg(infoStream, "OK");
203 }
204 if (reader.maxDoc() != info.info.getDocCount()) {
205 throw new RuntimeException("SegmentReader.maxDoc() " + reader.maxDoc() + " != SegmentInfos.docCount " + info.info.getDocCount());
206 }
207
208 // Test getFieldInfos()
209 // 获取域状态以及数量 如：test: fields..............OK [3 fields]
210 if (infoStream != null) {
211 infoStream.print(" test: fields..............");
212 }
213 FieldInfos fieldInfos = reader.getFieldInfos();
214 msg(infoStream, "OK [" + fieldInfos.size() + " fields]");
215 segInfoStat.numFields = fieldInfos.size();
216
217 // Test Field Norms
218 // 获取Field Norms的状态以及数量 test: field norms.........OK [1 fields]
219 segInfoStat.fieldNormStatus = testFieldNorms(reader, infoStream);
220
221 // Test the Term Index
222 // 获取Field Index的状态以及数量 test: terms, freq, prox...OK [36091 terms; 54003 terms/docs pairs; 18001 tokens]
223 segInfoStat.termIndexStatus = testPostings(reader, infoStream, verbose);
224
225 // Test Stored Fields
226 // 获取Stored Field 的状态 test: stored fields.......OK [54003 total field count; avg 3 fields per doc]
227 segInfoStat.storedFieldStatus = testStoredFields(reader, infoStream);
228
229 // Test Term Vectors
230 // 获取Term Field 的状态 test: stored fields.......OK [54003 total field count; avg 3 fields per doc]
231 segInfoStat.termVectorStatus = testTermVectors(reader, infoStream, verbose, crossCheckTermVectors);
232
233 // 获取Doc Value 的状态 test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 SORTED; 0 SORTED_SET]
234 segInfoStat.docValuesStatus = testDocValues(reader, infoStream);
235
236 // Rethrow the first exception we encountered
237 // This will cause stats for failed segments to be incremented properly
238 if (segInfoStat.fieldNormStatus.error != null) {
239 throw new RuntimeException("Field Norm test failed");
240 } else if (segInfoStat.termIndexStatus.error != null) {
241 throw new RuntimeException("Term Index test failed");
242 } else if (segInfoStat.storedFieldStatus.error != null) {
243 throw new RuntimeException("Stored Field test failed");
244 } else if (segInfoStat.termVectorStatus.error != null) {
245 throw new RuntimeException("Term Vector test failed");
246 } else if (segInfoStat.docValuesStatus.error != null) {
247 throw new RuntimeException("DocValues test failed");
248 }
249
250 msg(infoStream, "");
251
252 } catch (Throwable t) {
253 msg(infoStream, "FAILED");
254 String comment;
255 comment = "fixIndex() would remove reference to this segment";
256 msg(infoStream, " WARNING: " + comment + "; full exception:");
257 if (infoStream != null)
258 t.printStackTrace(infoStream);
259 msg(infoStream, "");
260 result.totLoseDocCount += toLoseDocCount;
261 result.numBadSegments++;
262 continue;
263 } finally {
264 if (reader != null)
265 reader.close();
266 }
267
268 // Keeper
269 result.newSegments.add(info.clone());
270 }
271
272 if (0 == result.numBadSegments) {
273 result.clean = true;
274 } else
275 msg(infoStream, "WARNING: " + result.numBadSegments + " broken segments (containing " + result.totLoseDocCount + " documents) detected");
276
277 if ( ! (result.validCounter = (result.maxSegmentName < sis.counter))) {
278 result.clean = false;
279 result.newSegments.counter = result.maxSegmentName + 1;
280 msg(infoStream, "ERROR: Next segment name counter " + sis.counter + " is not greater than max segment name " + result.maxSegmentName);
281 }
282
283 if (result.clean) {
284 msg(infoStream, "No problems were detected with this index.\n");
285 }
286
287 return result;
288 }
其中关于testFieldNorms这几个的源码明天继续学习