File tree Expand file tree Collapse file tree 1 file changed +4
-3
lines changed
Expand file tree Collapse file tree 1 file changed +4
-3
lines changed Original file line number Diff line number Diff line change @@ -449,6 +449,10 @@ Here are the details of all the sources in Spark.
449449 <td><b>File source</b></td>
450450 <td>
451451 <code>path</code>: path to the input directory, and common to all file formats.
452+ <br/>
453+ <code>maxFilesPerTrigger</code>: maximum number of new files to be considered in every trigger (default: no max)
454+ <br/>
455+ <code>latestFirst</code>: whether to processs the latest new files first, useful when there is a large backlog of files(default: false)
452456 <br/><br/>
453457 For file-format-specific options, see the related methods in <code>DataStreamReader</code>
454458 (<a href="api/scala/index.html#org.apache.spark.sql.streaming.DataStreamReader">Scala</a>/<a href="api/java/org/apache/spark/sql/streaming/DataStreamReader.html">Java</a>/<a href="api/python/pyspark.sql.html#pyspark.sql.streaming.DataStreamReader">Python</a>).
@@ -1076,9 +1080,6 @@ Here are the details of all the sinks in Spark.
10761080 <td>Append</td>
10771081 <td>
10781082 <code>path</code>: path to the output directory, must be specified.
1079- <code>maxFilesPerTrigger</code>: maximum number of new files to be considered in every trigger (default: no max)
1080- <br/>
1081- <code>latestFirst</code>: whether to processs the latest new files first, useful when there is a large backlog of files(default: false)
10821083 <br/><br/>
10831084 For file-format-specific options, see the related methods in DataFrameWriter
10841085 (<a href="api/scala/index.html#org.apache.spark.sql.DataFrameWriter">Scala</a>/<a href="api/java/org/apache/spark/sql/DataFrameWriter.html">Java</a>/<a href="api/python/pyspark.sql.html#pyspark.sql.DataFrameWriter">Python</a>).
You can’t perform that action at this time.
0 commit comments