Skip to content

Commit 595ffd7

Browse files
author
assafmendelson
committed
File source options for spark 2.1 appeared under File sink
1 parent 0ebb3b8 commit 595ffd7

File tree

1 file changed

+4
-3
lines changed

1 file changed

+4
-3
lines changed

docs/structured-streaming-programming-guide.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -449,6 +449,10 @@ Here are the details of all the sources in Spark.
449449
<td><b>File source</b></td>
450450
<td>
451451
<code>path</code>: path to the input directory, and common to all file formats.
452+
<br/>
453+
<code>maxFilesPerTrigger</code>: maximum number of new files to be considered in every trigger (default: no max)
454+
<br/>
455+
<code>latestFirst</code>: whether to processs the latest new files first, useful when there is a large backlog of files(default: false)
452456
<br/><br/>
453457
For file-format-specific options, see the related methods in <code>DataStreamReader</code>
454458
(<a href="api/scala/index.html#org.apache.spark.sql.streaming.DataStreamReader">Scala</a>/<a href="api/java/org/apache/spark/sql/streaming/DataStreamReader.html">Java</a>/<a href="api/python/pyspark.sql.html#pyspark.sql.streaming.DataStreamReader">Python</a>).
@@ -1076,9 +1080,6 @@ Here are the details of all the sinks in Spark.
10761080
<td>Append</td>
10771081
<td>
10781082
<code>path</code>: path to the output directory, must be specified.
1079-
<code>maxFilesPerTrigger</code>: maximum number of new files to be considered in every trigger (default: no max)
1080-
<br/>
1081-
<code>latestFirst</code>: whether to processs the latest new files first, useful when there is a large backlog of files(default: false)
10821083
<br/><br/>
10831084
For file-format-specific options, see the related methods in DataFrameWriter
10841085
(<a href="api/scala/index.html#org.apache.spark.sql.DataFrameWriter">Scala</a>/<a href="api/java/org/apache/spark/sql/DataFrameWriter.html">Java</a>/<a href="api/python/pyspark.sql.html#pyspark.sql.DataFrameWriter">Python</a>).

0 commit comments

Comments
 (0)