Repository navigation
[Proposal]Create a repository for SparkLoad‘s dependencies #4101
Description
Motivation
Recently, we have introduced the Spark Load, which currently needs to upload many jar packages to the Yarn cluster before load. These jar packages include $DORIS_HOME/lib/palo-fe.jar(the Dpp runtime dependency) and all jars in the $SPARK_HOME/jars folder(the Spark dependencies), which usually takes 2~3 minutes to be uploaded.
Currently, these jars are uploaded to the temporary directories in HDFS. The palo-fe.jar is uploaded to {working_dir}/jobs/DB_ID/LABEL/JOB_ID/configs. Other jars are packaged as zip file and uploaded to {stage_dir}/APPLICATION_ID/__spark_lib__.zip.
In most cases, the jar packages uploaded by two different load are completely same, which means we don't have to upload these jar packages every time. Secondly, the jar packages should be stored in one directory so that we can manage them easily. Moreover, we can put all jars in a zip file in the compilation phase.
Therefore, as a proposal, I suggest to create a repository for all dependencies of Spark Load in HDFS.
The repository structure
Repository/
|-lib_{version}.zip
| {All spark dependencies}
| |-roaringbitmap.jar
| |-activation-1.1.1.jar
| |-aircompressor-0.10.jar
| |-...
| {All dpp dependencies}
| |-spark-dpp.jar
|-lib_{version}.zip
|-lib_{version}.zip
|-...
The Repository/ directory is the parent dir of all zip files. When a spark load is submitted, fe will first compare the version between remote zip files in the repository and local zip file, and only upload when fe can not find the right version.
We must set the spark config spark.yarn.archive to the right zip file which fe found in the above step so that the yarn cluster will know where to find these dependencies.
Note that, the spark-dpp.jar is built by spark-dpp sub-modules. The difference between palo-fe.jar and spark-dpp.jar is that spark-dpp.jar contain other third-party libraries that palo-fe.jar depends on. You can see the details about spark-dpp sub-modules of fe in this issue #4098 .
Meanwhile, it is neccessary to set AppResourceHdfsPath argument of spark-submit to lib.jar file.
Works to do
[] Build lib.zip in compliation phase
[] Add a repository property for spark load
[] Add a prepare phase for spark load to check and upload dependencies
Activity
[-][Proposal]Create a jar package's repository for Spark Load[/-][+][Proposal]Create a repository for SparkLoad‘s dependencies[/+]wangbo commentedon Jul 17, 2020
Some Questions:
1 Do we need to delete useless version Jar?
2 In Multiple Doris Clusters case,how dose every cluster knows its version, whether they share the same directory?
3 How to deal the case when FE Gray release, every FE has different code?
xy720 commentedon Jul 21, 2020
1、I am not going to remove the old version yet，because it's not a big problem for now.
2、At present, I plan to save a DppVersion in FeConstant, fe will try to find or create a subfolders in repository named as DppVersion before load. Different Doris clusters will have its own repository directory.
3、Fe will soon have a submodule of spark load，for every cluster we maintain a DppVersion in FeConstant, represents a version of spark load submodule(Just like the meta version). Before Fe starts to submit spark load, it will first look for the subfolders which named as DppVersion under its repository, and find each library under this folder. Then it will compare each library with MD5, and determine whether it needs to upload again
Please see the comment in the following:
xy720 commentedon Jul 21, 2020
The repository structure will be like this:
1、The archive represents a remote directory contains the libraries which are in the same dppVersion.
2、The library(lib) represents a remote library for Spark-2x/Spark-dpp.
Every time when fe upload its zip, it creates a subfolder like "__archive_1_2_0" in repository, and put the zip file into it.
The zip file is named like "__lib_md5sum_spark-dpp.zip", generated by the jar packages. For example, spark-dpp.zip is generated by spark-dpp.jar, and spark-2x.zip is generated by jars under
$SPARK_HOME/jars.If upload is needed, by default fe will find the local zip file with the following config:
spark_dpp_resource_local_path = $DORIS_HOME + "/lib/spark-dpp.zip"spark_resource_local_path = "{user_setting}"if the config
spark_resource_local_pathis empty, fe will try find zip file at$SPARK_HOME/jars/spark-2x.zip[feature](cloud) support event driven or periodic warm up (apache#4101)
[feature](cloud) support event driven or periodic warm up (apache#4101)
[feature](cloud) support event driven or periodic warm up (apache#4101)
[feature](cloud) support event driven or periodic warm up (apache#4101)
[feature](cloud) support event driven or periodic warm up (apache#4101)