Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ReadV2 in TsFile #144

Merged
merged 34 commits into from
Mar 7, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
44d26bb
TMP save 01-10
Jan 10, 2018
2fee2f0
TMP save 01-16
Jan 16, 2018
9a2b0b0
add getSize() to TimeValuePair
Jan 17, 2018
e1dc833
modify interface of SeriesChunk
Jan 18, 2018
ef5b406
fix a bug in last commit
Jan 18, 2018
a9d19e3
add close() method to TimeValuePair interface
Jan 19, 2018
a66a86f
add single series reader for one TsFile
Jan 19, 2018
fd3fa8e
add constructor for TsFileIOWriter
liukun4515 Jan 20, 2018
2026d94
add Serializable to TimeValuePair class
Jan 20, 2018
c9e7bde
Merge branch 'f_zjr_read_rebuild_pr' of github.com:thulab/tsfile into…
Jan 20, 2018
2be7808
add startRowGroup and endRowGroup function for tsfile
liukun4515 Jan 21, 2018
720513c
add datatype to TsPrimitiveType
Jan 21, 2018
efd1fcf
Merge branch 'f_zjr_read_rebuild_pr' of github.com:thulab/tsfile into…
Jan 21, 2018
87ee4e7
remove unused lines
Jan 24, 2018
05393db
fix a bug in SeriesReaderFromSingleFile where the fileStream is not c…
Jan 24, 2018
59e453d
add getByType for TsPrimitiveType
Jan 25, 2018
999b00b
change log level from info to debug
Jan 25, 2018
8784e02
cancel fill tsfile
liukun4515 Jan 26, 2018
f2e7723
remove cache and query in seieswriteimpl
liukun4515 Jan 29, 2018
d0c1eb1
add clear() function to Cache
Jan 29, 2018
5f9f0bb
remove logback.xml
Jan 30, 2018
8a20462
add error log in flush timeseries data
liukun4515 Feb 1, 2018
bd77c43
add error in flush timeseries data
liukun4515 Feb 1, 2018
aff995d
remove device id from FileSchema
kr11 Feb 1, 2018
3b4399a
Remove unused import
kr11 Feb 1, 2018
ffe2bbd
remove the checkIsDeltaExist function, add checkIsTimeSeriesExist fun…
liukun4515 Feb 1, 2018
d39d885
remove the get memory function
liukun4515 Feb 1, 2018
2c4e109
add more constructor to SeriesReader
Feb 2, 2018
a079888
Merge branch 'f_zjr_read_rebuild_pr' of github.com:thulab/tsfile into…
Feb 2, 2018
dca0ac6
repair some codes according to comments from Pull Request
Feb 5, 2018
6a65f5c
modify the code according to the comments in pr
liukun4515 Feb 6, 2018
bdf10fe
modify PageReader
Feb 6, 2018
2b9e7cc
fix some codes according to comments
Feb 27, 2018
3e5b35e
Merge branch 'f_zjr_read_rebuild_pr' of github.com:thulab/tsfile into…
Feb 27, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add error log in flush timeseries data
  • Loading branch information
liukun4515 committed Feb 1, 2018
commit 8a20462665597bb92b3b40b0a9047db42a635420
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
package cn.edu.tsinghua.tsfile.common.utils;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.util.ArrayList;
Expand Down Expand Up @@ -86,17 +85,4 @@ public void reset() {
list.clear();
totalSize = 0;
}

/**
* transform all {@link PublicBAOS} to list of {@link ByteArrayInputStream}
*
* @return a list of {@link ByteArrayInputStream} contain all page data
*/
public List<ByteArrayInputStream> transform() {
List<ByteArrayInputStream> ret = new ArrayList<>();
for (PublicBAOS baos : list) {
ret.add(baos.transformToInputStream());
}
return ret;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
package cn.edu.tsinghua.tsfile.timeseries.readV2.datatype;

public class Test {

public static void main(String[] args) {
// TODO Auto-generated method stub
new TsPrimitiveType.TsBoolean(false);
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,6 @@ public interface IPageWriter {
void writePage(ListByteArrayOutputStream listByteArray, int valueCount, Statistics<?> statistics,
long maxTimestamp, long minTimestamp) throws PageException;

/**
* query all pages which have been packaged
*
* @return left is all pages data, right is the name of compression
*/
Pair<List<ByteArrayInputStream>, CompressionTypeName> query();

/**
* write the page to specified IOWriter
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,9 @@ public void writePage(ListByteArrayOutputStream listByteArray, int valueCount, S
// compress the input data
if (this.minTimestamp == -1)
this.minTimestamp = minTimestamp;
if(this.minTimestamp==-1){
LOG.error("Write page error, {}, minTime:{}, maxTime:{}",desc,minTimestamp,maxTimestamp);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里如果出错了,还继续执行下去?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

需要,这个是当时测试出来的错误,如果出问题想用error日志记录下来。

}
this.maxTimestamp = maxTimestamp;
int uncompressedSize = listByteArray.size();
ListByteArrayOutputStream compressedBytes = compressor.compress(listByteArray);
Expand Down Expand Up @@ -76,22 +79,16 @@ public void writePage(ListByteArrayOutputStream listByteArray, int valueCount, S
estimateMaxPageMemSize());
}

public Pair<List<ByteArrayInputStream>, CompressionTypeName> query() {


List<ByteArrayInputStream> backupPageList = buf.transform();
Pair<List<ByteArrayInputStream>, CompressionTypeName> ret = new Pair<List<ByteArrayInputStream>, CompressionTypeName>(
backupPageList, compressor.getCodecName());
return ret;
}

private void resetTimeStamp() {
if (totalValueCount == 0)
minTimestamp = -1;
}

@Override
public void writeToFileWriter(TsFileIOWriter writer, Statistics<?> statistics) throws IOException {
if(minTimestamp==-1){
LOG.error("Write page error, {}, minTime:{}, maxTime:{}",desc,minTimestamp,maxTimestamp);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里如果出错了,还继续执行下去?

}
writer.startSeries(desc, compressor.getCodecName(), desc.getType(), statistics, maxTimestamp, minTimestamp);
long totalByteSize = writer.getPos();
writer.writeBytesToStream(buf);
Expand Down