PARQUET-1292 Adding constructors to ProtoParquetWriter with writeSpecsCompliant flag#473
PARQUET-1292 Adding constructors to ProtoParquetWriter with writeSpecsCompliant flag#473chawlakunal wants to merge 2 commits intoapache:masterfrom
Conversation
…ag so that a specs compliant writer can be instantiated
| } | ||
|
|
||
| /** | ||
| * Create a new {@link ProtoParquetWriter}. The default block size is 50 MB.The default |
There was a problem hiding this comment.
I believe the default block size is 128MB by default (see https://github.com/apache/parquet-mr/blob/0d55abd05b0e5027c18e60d1ac3b22998dd00951/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetWriter.java#L41
|
Thanks @chawlakunal for the commit, it looks good to me. I believe there is a slight mistake in a comment (see my review above). |
|
@BenoitHanotte Happy to contribute!!1 I have created a JIRA for this but I am unable to assign it to myself. |
|
@chawlakunal, I added you the list of contributors so that you can assign JIRA-s to yourself. I assigned this one to you to make sure it works. |
|
@BenoitHanotte Can this be approved and merged? |
|
Hi, |
|
Hi @lukasnalezenec |
|
+1 on finding a way to pass the Right now we have to do something like: Configuration conf = new Configuration();
ProtoWriteSupport.setWriteSpecsCompliant(conf, true);
ParquetWriter<MyMessage> writer
= new ParquetWriter<>(file, new ProtoWriteSupport<MyMessage>(cls), CompressionCodecName.GZIP, 256 * 1024 * 1024, 1 * 1024 * 1024, 1 * 1024 * 1024, true, false, ParquetWriter.DEFAULT_WRITER_VERSION, conf); |
|
I am still waiting for comments from @lukasnalezenec |
|
I think that problem is that there was no ProtoParquetWriter constructor with conf parameter. I would take existing constructors and add those two new constructors: We might also extend some other existing constructors with conf parameter. (I am not sure if it is necessary) After this change we might turn on the parameter like this: Configuration conf = new Configuration();
ProtoWriteSupport.setWriteSpecsCompliant(conf, true);
ParquetWriter<MyMessage> writer = new ProtoParquetWriter<>(file, cls, conf);`If you disagree with me, I'm open to another solution. |
|
@lukasnalezenec From my perspective, it looks great! |
|
Hello @costimuraru @chawlakunal |
|
@BenoitHanotte I'll work on that sometime this week. |
|
@chawlakunal any luck? |
|
@chawlakunal Any news on this? |
This PR adds constructors to ProtoParquetWriter with flags for specs compliant writer (based on #411). Now a specs compliant ProtoParquetWriter can be created as