Skip to content

Commit 9cd7ced

Browse files
EarthsonEarthson
committed
merge upstream
2 parents 97cbe0c + b9dfdcc commit 9cd7ced

File tree

711 files changed

+23171
-9459
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

711 files changed

+23171
-9459
lines changed

‎.rat-excludes‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,3 +86,5 @@ org.apache.spark.scheduler.SparkHistoryListenerFactory
8686
.*parquet
8787
LZ4BlockInputStream.java
8888
spark-deps-.*
89+
.*csv
90+
.*tsv

‎LICENSE‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -264,7 +264,7 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
264264
(New BSD license) Protocol Buffer Java API (org.spark-project.protobuf:protobuf-java:2.4.1-shaded - http://code.google.com/p/protobuf)
265265
(The BSD License) Fortran to Java ARPACK (net.sourceforge.f2j:arpack_combined_all:0.1 - http://f2j.sourceforge.net)
266266
(The BSD License) xmlenc Library (xmlenc:xmlenc:0.52 - http://xmlenc.sourceforge.net)
267-
(The New BSD License) Py4J (net.sf.py4j:py4j:0.9 - http://py4j.sourceforge.net/)
267+
(The New BSD License) Py4J (net.sf.py4j:py4j:0.9.1 - http://py4j.sourceforge.net/)
268268
(Two-clause BSD-style license) JUnit-Interface (com.novocode:junit-interface:0.10 - http://github.com/szeiger/junit-interface/)
269269
(BSD licence) sbt and sbt-launch-lib.bash
270270
(BSD 3 Clause) d3.min.js (https://github.com/mbostock/d3/blob/master/LICENSE)

‎NOTICE‎

Lines changed: 53 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -610,7 +610,59 @@ Vis.js uses and redistributes the following third-party libraries:
610610

611611
===============================================================================
612612

613-
The CSS style for the navigation sidebar of the documentation was originally
613+
The CSS style for the navigation sidebar of the documentation was originally
614614
submitted by Óscar Nájera for the scikit-learn project. The scikit-learn project
615615
is distributed under the 3-Clause BSD license.
616616
===============================================================================
617+
618+
For CSV functionality:
619+
620+
/*
621+
* Copyright 2014 Databricks
622+
*
623+
* Licensed under the Apache License, Version 2.0 (the "License");
624+
* you may not use this file except in compliance with the License.
625+
* You may obtain a copy of the License at
626+
*
627+
* http://www.apache.org/licenses/LICENSE-2.0
628+
*
629+
* Unless required by applicable law or agreed to in writing, software
630+
* distributed under the License is distributed on an "AS IS" BASIS,
631+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
632+
* See the License for the specific language governing permissions and
633+
* limitations under the License.
634+
*/
635+
636+
/*
637+
* Copyright 2015 Ayasdi Inc
638+
*
639+
* Licensed under the Apache License, Version 2.0 (the "License");
640+
* you may not use this file except in compliance with the License.
641+
* You may obtain a copy of the License at
642+
*
643+
* http://www.apache.org/licenses/LICENSE-2.0
644+
*
645+
* Unless required by applicable law or agreed to in writing, software
646+
* distributed under the License is distributed on an "AS IS" BASIS,
647+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
648+
* See the License for the specific language governing permissions and
649+
* limitations under the License.
650+
*/
651+
652+
653+
===============================================================================
654+
For dev/sparktestsupport/toposort.py:
655+
656+
Copyright 2014 True Blade Systems, Inc.
657+
658+
Licensed under the Apache License, Version 2.0 (the "License");
659+
you may not use this file except in compliance with the License.
660+
You may obtain a copy of the License at
661+
662+
http://www.apache.org/licenses/LICENSE-2.0
663+
664+
Unless required by applicable law or agreed to in writing, software
665+
distributed under the License is distributed on an "AS IS" BASIS,
666+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
667+
See the License for the specific language governing permissions and
668+
limitations under the License.

‎R/README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,16 @@
11
# R on Spark
22

33
SparkR is an R package that provides a light-weight frontend to use Spark from R.
4+
### Installing sparkR
5+
6+
Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This can be done by running the script `$SPARK_HOME/R/install-dev.sh`.
7+
By default the above script uses the system wide installation of R. However, this can be changed to any user installed location of R by setting the environment variable `R_HOME` the full path of the base directory where R is installed, before running install-dev.sh script.
8+
Example:
9+
```
10+
# where /home/username/R is where R is installed and /home/username/R/bin contains the files R and RScript
11+
export R_HOME=/home/username/R
12+
./install-dev.sh
13+
```
414

515
### SparkR development
616

‎R/install-dev.sh‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,12 +35,19 @@ LIB_DIR="$FWDIR/lib"
3535
mkdir -p $LIB_DIR
3636

3737
pushd $FWDIR > /dev/null
38+
if [ ! -z "$R_HOME" ]
39+
then
40+
R_SCRIPT_PATH="$R_HOME/bin"
41+
else
42+
R_SCRIPT_PATH="$(dirname $(which R))"
43+
fi
44+
echo "USING R_HOME = $R_HOME"
3845

3946
# Generate Rd files if devtools is installed
40-
Rscript -e ' if("devtools" %in% rownames(installed.packages())) { library(devtools); devtools::document(pkg="./pkg", roclets=c("rd")) }'
47+
"$R_SCRIPT_PATH/"Rscript -e ' if("devtools" %in% rownames(installed.packages())) { library(devtools); devtools::document(pkg="./pkg", roclets=c("rd")) }'
4148

4249
# Install SparkR to $LIB_DIR
43-
R CMD INSTALL --library=$LIB_DIR $FWDIR/pkg/
50+
"$R_SCRIPT_PATH/"R CMD INSTALL --library=$LIB_DIR $FWDIR/pkg/
4451

4552
# Zip the SparkR package so that it can be distributed to worker nodes on YARN
4653
cd $LIB_DIR

‎R/pkg/NAMESPACE‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,10 +35,14 @@ exportMethods("arrange",
3535
"count",
3636
"cov",
3737
"corr",
38+
"covar_samp",
39+
"covar_pop",
3840
"crosstab",
3941
"describe",
4042
"dim",
4143
"distinct",
44+
"drop",
45+
"dropDuplicates",
4246
"dropna",
4347
"dtypes",
4448
"except",
@@ -271,14 +275,15 @@ export("as.DataFrame",
271275
"createExternalTable",
272276
"dropTempTable",
273277
"jsonFile",
274-
"read.json",
275278
"loadDF",
276279
"parquetFile",
277280
"read.df",
281+
"read.json",
278282
"read.parquet",
279283
"read.text",
280284
"sql",
281-
"table",
285+
"str",
286+
"tableToDF",
282287
"tableNames",
283288
"tables",
284289
"uncacheTable")

0 commit comments

Comments
 (0)