{"id":1731,"date":"2019-12-16T01:00:00","date_gmt":"2019-12-16T01:00:00","guid":{"rendered":"https:\/\/www.javaadvent.com\/?p=1731"},"modified":"2019-12-15T07:49:05","modified_gmt":"2019-12-15T07:49:05","slug":"how-to-do-deep-learning-for-java","status":"publish","type":"post","link":"https:\/\/www.javaadvent.com\/2019\/12\/how-to-do-deep-learning-for-java.html","title":{"rendered":"How to do Deep Learning for Java?"},"content":{"rendered":"\n
\"\"\/<\/figure>\n\n\n\n

Introduction<\/h1>\n\n\n\n

Some time ago I came across this life-cycle management tool (or cloud service) called Valohai<\/a>\n and I was quite impressed by its user-interface and simplicity of \ndesign and layout. I had a good chat about the service at that time with\n one of the members of Valohai<\/a> and was given a demo. Previous to that I had written a simple pipeline using GNU Parallel<\/a>, JavaScript, Python and Bash \u2014 and another one purely using GNU Parallel<\/a>,\n and Bash. Also thought about replacing the moving parts with \nready-to-use task\/workflow management tools like Jenkins X, Jenkins \nPipeline, Concourse or Airflow but due to various reasons, I did not \nproceed with the idea.<\/p>\n\n\n\n

Coming back to our original conversation, I noticed a lot of the examples and docs on Valohai<\/a>\n were based on Python, R and the respective frameworks and libraries. \nThere was a lack of Java\/JVM based examples or docs. So I took this \nopportunity to do something about that.<\/p>\n\n\n\n

I was encouraged by Valohai<\/a> to implement something using the famous Java library called DL4J \u2014 Deep Learning for Java<\/a>.<\/p>\n\n\n\n

My initial experience with Valohai<\/a>\n already gave me a good impression after getting an understanding of its\n design, layout and workflow. That it was developer-friendly and the \nmakers already took into consideration various facets of both developer \nand infrastructure workflows. In our worlds, the latter is mostly run by\n DevOps or SysOps teams and we know the nuances and pain-points attached\n to it. You can find out more about its features from the Features section<\/a> of the site.<\/p>\n\n\n\n

Achtung!<\/strong>\n Just to let you know that from here onwards the post will be a bit more\n technical and may contain code snippets and mention of deep \nlearning\/machine learning and infrastructure-related terminologies.<\/p>\n\n\n\n

What do we need and how?<\/h1>\n\n\n\n

For\n any machine learning or deep learning project or initiative, these days\n the two important components (from a high-level perspective) are code \nthat will create and serve the model and infrastructure where this whole\n life-cycle will be executed.<\/p>\n\n\n\n

Of\n course, there are going to be steps and components needed before, \nduring and after the above, but to keep things simple let\u2019s say we need \ncode and infrastructure.<\/p>\n\n\n\n

Code<\/strong><\/p>\n\n\n\n

For code I have chosen a modified example using DL4J, it\u2019s an MNist project<\/a>\n with a training set of 60,000 images and test set of 10,000 images of \nhand-written digits. This dataset is available via the DL4J library \n(just like Keras<\/a> provides a stock of them). Look for the MnistDataSetIterator<\/a> under DatasetIterators<\/a> in the DL4J Cheatsheet<\/a> for further details on this particular dataset.<\/p>\n\n\n\n

Have a look at the source code we will be using before getting started, the main Java class is called org.deeplearning4j.feedforward.mnist.MLPMnistSingleLayerRunner<\/a>.<\/p>\n\n\n\n

Infrastructure<\/strong><\/p>\n\n\n\n

As it is obvious by now, we have decided to try out the Java example using Valohai<\/a> as our infrastructure to run our experiments (training and evaluation of the model). Valohai<\/a>\n recognizes git repositories and directly hooks into them and allows \nexecution of our code, irrespective of platform or language \u2014 we will \nsee how this works. This also means if you are a strong supporter of \nGitOps or Infrastructure-As-Code you will appreciate the workflow.<\/p>\n\n\n\n

For this we just need an account on Valohai<\/a>, we can avail a Free-tier account<\/a> and have access to several instances of various configurations when we sign up<\/a>. See Free-tier under Plans and Pricing<\/a> and the comparison chart for more details. For what we would like to do, the Free-tier is more than enough for now.<\/p>\n\n\n\n

Deep Learning for Java and Valohai<\/h1>\n\n\n\n

As\n we agreed, we\u2019re going to use these two technologies to achieve our \ngoal of training a single layer model and evaluating it, as well as \nseeing what the end-to-end experience is like on Valohai<\/a>.<\/p>\n\n\n\n

We\n will bundle the necessary build and run-time dependencies into the \nDocker image and use it to build our Java app, train a model and \nevaluate it on the Valohai<\/a> platform via a simple valohai.yaml<\/a> file which is placed in the root folder of the project repository.<\/p>\n\n\n\n

Deep Learning for Java: DL4J<\/h1>\n\n\n\n

The\n easy part is, we won\u2019t need to do much here, just build the jar and \ndownload the dataset into the Docker container. We have a pre-built \nDocker image that contains all the dependencies needed to build a Java \napp. We have pushed this image into Docker Hub<\/a>, you can find it by searching for dl4j-mnist-single-layer<\/a> (we will be using a specific tag as defined in the YAML file). We have chosen to use GraalVM 19.1.1<\/a> as our Java build and runtime for this project, and so it is embedded into the Docker image (see Dockerfile<\/a> for the definition of the Docker image). To learn more about GraalVM check out the resources at the official site of graalvm.org<\/a> and Awesome Graal<\/a>.<\/p>\n\n\n\n

Orchestration<\/strong><\/p>\n\n\n\n

When the uber jar is invoked from the command-line, we land into the MLPMnistSingleLayerRunner<\/code> class which directs us to the intended action depending on the parameters passed in:<\/p>\n\n\n\n

public static void main(String[] args) throws Exception {\n    MLPMnistSingleLayerRunner mlpMnistRunner = new MLPMnistSingleLayerRunner();\n\n    JCommander.newBuilder()\n            .addObject(mlpMnistRunner)\n            .build()\n            .parse(args);\n\n    mlpMnistRunner.execute();\n}<\/code><\/pre>\n\n\n\n
The parameters passed into the uber jar are received by this class and handled by the execute()<\/code> method.<\/p>\n\n\n\n
We can create a model via the --action train<\/code> parameter and evaluate the created model via the --action evaluate<\/code> parameter respectively passed to the Java app (uber jar).<\/p>\n\n\n\n
The main parts of the Java app that does this work can be found in the two Java classes mentioned in the sections below.<\/p>\n\n\n\n
Train a model<\/strong><\/p>\n\n\n\n
Can be invoked from the command-line via:<\/p>\n\n\n\n
.\/runMLPMnist.sh --action train --output-dir ${VH_OUTPUTS_DIR}\n\nor\n\njava -Djava.library.path=\"\"             \\\n     -jar target\/MLPMnist-1.0.0-bin.jar \\\n     --action train --output-dir ${VH_OUTPUTS_DIR}<\/code><\/pre>\n\n\n\n
This creates the model (when successful, at the end of the execution) by the name mlpmnist-single-layer.pb<\/code> in the folder specified by the --output-dir<\/code> passed in at the beginning of the execution. From the perspective of Valohai<\/a>, it should be placed into the ${VH_OUTPUTS_DIR} which is what we do (see valohai.yaml<\/a> file).<\/p>\n\n\n\n
For source code, see class MLPMNistSingleLayerTrain.java<\/a>.<\/p>\n\n\n\n
Evaluate a model<\/strong><\/p>\n\n\n\n
Can be invoked from the command-line via:<\/p>\n\n\n\n
.\/runMLPMnist.sh --action evaluate --input-dir ${VH_INPUTS_DIR}\/model\n\nor\n\njava -Djava.library.path=\"\"             \\\n     -jar target\/MLPMnist-1.0.0-bin.jar \\\n     --action evaluate --input-dir ${VH_INPUTS_DIR}\/model<\/code><\/pre>\n\n\n\n
This expects a model (created by the training step) by the name mlpmnist-single-layer.pb<\/code> to be present in the folder specified by the --input-dir<\/code> passed in when the app has been called.<\/p>\n\n\n\n
For source code, see class MLPMNistSingleLayerEvaluate.java<\/a>.<\/p>\n\n\n\n
I hope this short illustration makes it clear how the Java app that trains and evaluates the model works in general.<\/p>\n\n\n\n
That\u2019s all is needed of us, but feel free to play with the rest of the source<\/a> (along with the README.md<\/a>\n and bash scripts) and satisfy your curiosity and understanding on how \nthis is done! Further resources on how to use DL4J has been provided in \nthe Resources<\/strong> section at the end of the post.<\/p>\n\n\n\n
Valohai<\/h1>\n\n\n\n
Valohai<\/a>\n as a platform allows us to loosely couple our runtime environment, our \ncode, and our dataset, as you can see from the structure of the YAML \nfile below. That way the different components can evolve independently \nwithout impeding or being dependent on one another. Hence our Docker \ncontainer only has the build and runtime time components packed into it.\n At execution time we build the uber jar in the Docker container, upload\n it to some internal or external storage, and then via another execution\n step download the uber jar and dataset from storage (or another \nlocation) to run the training. This way the two execution steps are \ndecoupled; we can e.g. build the jar once and run hundreds of training \nsteps on the same jar. As the build and runtime environments should not \nchange that often we can cache them and the code, dataset and model \nsources can be made dynamically available during execution time.<\/p>\n\n\n\n
valohai.yaml<\/strong><\/a><\/p>\n\n\n\n
The heart of integrating our Java project with the Valohai<\/a> infrastructure is defining the steps of execution<\/a> of the steps in the valohai.yaml<\/a> file placed in the root of your project folder. Our valohai.yaml<\/a> looks like this:<\/p>\n\n\n
\n---\n\n- step:\n    name: Build-dl4j-mnist-single-layer-java-app\n    image: neomatrix369\/dl4j-mnist-single-layer:v0.5\n    command:\n      - cd ${VH_REPOSITORY_DIR}\n      - .\/buildUberJar.sh\n      - echo "~~~ Copying the build jar file into ${VH_OUTPUTS_DIR}"\n      - cp target\/MLPMnist-1.0.0-bin.jar ${VH_OUTPUTS_DIR}\/MLPMnist-1.0.0.jar\n      - ls -lash ${VH_OUTPUTS_DIR}\n    environment: aws-eu-west-1-g2-2xlarge\n- step:\n    name: Run-dl4j-mnist-single-layer-train-model\n    image: neomatrix369\/dl4j-mnist-single-layer:v0.5\n    command:\n      - echo "~~~ Unpack the MNist dataset into ${HOME} folder"\n      - tar xvzf ${VH_INPUTS_DIR}\/dataset\/mlp-mnist-dataset.tgz -C ${HOME}\n      - cd ${VH_REPOSITORY_DIR}\n      - echo "~~~ Copying the build jar file from ${VH_INPUTS_DIR} to current location"\n      - cp ${VH_INPUTS_DIR}\/dl4j-java-app\/MLPMnist-1.0.0.jar .\n      - echo "~~~ Run the DL4J app to train model based on the the MNist dataset"\n      - .\/runMLPMnist.sh {parameters}\n    inputs:\n      - name: dl4j-java-app\n        description: DL4J Java app file (jar) generated in the previous step 'Build-dl4j-mnist-single-layer-java-app'\n      - name: dataset\n        default: https:\/\/github.com\/neomatrix369\/awesome-ai-ml-dl\/releases\/download\/mnist-dataset-v0.1\/mlp-mnist-dataset.tgz\n        description: MNist dataset needed to train the model\n    parameters:\n      - name: --action\n        pass-as: '--action {v}'\n        type: string\n        default: train\n        description: Action to perform i.e. train or evaluate\n      - name: --output-dir\n        pass-as: '--output-dir {v}'\n        type: string\n        default: \/valohai\/outputs\/\n        description: Output directory where the model will be created, best to pick the Valohai output directory\n    environment: aws-eu-west-1-g2-2xlarge\n\n- step:\n    name: Run-dl4j-mnist-single-layer-evaluate-model\n    image: neomatrix369\/dl4j-mnist-single-layer:v0.5\n    command:\n      - cd ${VH_REPOSITORY_DIR}\n      - echo "~~~ Copying the build jar file from ${VH_INPUTS_DIR} to current location"\n      - cp ${VH_INPUTS_DIR}\/dl4j-java-app\/MLPMnist-1.0.0.jar .\n      - echo "~~~ Run the DL4J app to evaluate the trained MNist model"\n      - .\/runMLPMnist.sh {parameters}\n    inputs:\n      - name: dl4j-java-app\n        description: DL4J Java app file (jar) generated in the previous step 'Build-dl4j-mnist-single-layer-java-app'    \n      - name: model\n        description: Model file generated in the previous step 'Run-dl4j-mnist-single-layer-train-model'\n    parameters:\n      - name: --action\n        pass-as: '--action {v}'\n        type: string\n        default: evaluate\n        description: Action to perform i.e. train or evaluate\n      - name: --input-dir\n        pass-as: '--input-dir {v}'\n        type: string\n        default: \/valohai\/inputs\/model\n        description: Input directory where the model created by the previous step can be found created\n    environment: aws-eu-west-1-g2-2xlarge\n<\/pre><\/div>\n\n\n
Explanation of the step Build-dl4j-mnist-single-layer-java-app<\/strong><\/p>\n\n\n\n
From\n the YAML file, we can see that we define this step by first using the \nDocker image and then run the build script to build the uber jar. Our \ndocker image has the build environment dependencies setup (i.e. GraalVM \nJDK, Maven, etc\u2026) to build a Java app. We do not specify any inputs or \nparameters as this is the build step. Once the build will be successful \nwe want to copy the uber jar called MLPMnist-1.0.0-bin.jar<\/code> (original name) to the \/valohai\/outputs<\/code> folder (represented by ${VH_OUTPUTS_DIR}<\/code>).\n Everything within this folder automatically gets persisted within your \nproject’s storage, e.g. an AWS S3 bucket. Finally, we define our job to \nrun in the AWS environment.<\/p>\n\n\n\n
Note: The <\/em>Valohai<\/em><\/a>\n free tier does not have network access from inside the Docker container\n (this is disabled by default), please contact support to enable this \noption (I had to do the same), or else we cannot download our Maven and \nother dependencies during build time.<\/em><\/p>\n\n\n\n
Explanation of the step Run-dl4j-mnist-single-layer-train-model<\/strong><\/p>\n\n\n\n
The semantics of the definition is similar to the previous step except we specify two inputs one for the uber jar (MLPMnist-1.0.0.jar<\/code>) and the other for the dataset (to be unpacked into the${HOME}\/.deeplearning4j<\/code> folder). We will be passing the two parameters --action train<\/code> and --output-dir \/valohai\/outputs<\/code>. The model created from this step is collected into the \/valohai\/outputs\/model<\/code> folder (represented by ${VH_OUTPUTS_DIR}\/model<\/code>).<\/p>\n\n\n\n
Note: In\n the Input fields in the Executions tab of the Valohai Web UI, we can \nselect the outputs from previous executions by using the execution \nnumber i.e. <\/em>#1<\/em><\/code> or <\/em>#2<\/em><\/code>\n , in addition to using datum:\/\/ or http:\/\/ URLs. Typing in the few \nletters of the name of the file also helps search through the whole \nlist.<\/em><\/p>\n\n\n\n
Explanation of the step Run-dl4j-mnist-single-layer-evaluate-model<\/strong><\/p>\n\n\n\n
Again, this step is similar to the previous step, except that we will be passing in the two parameters --action evaluate<\/code> and --input-dir \/valohai\/inputs\/model<\/code>. Also, we have again specified two inputs:<\/code> sections defined in the YAML file called dl4j-java-app<\/code> and model<\/code> with no default<\/code>\n set for both of them. This will allow us to select the uber jar and the\n model we wish to evaluate – that was created by the step Run-dl4j-mnist-single-layer-train-model<\/strong>, using the web interface.<\/p>\n\n\n\n
Hope\n this explains the steps in the above definition file but if you require\n further help, please do not hesitate to look at the docs<\/a> and tutorials<\/a>.<\/p>\n\n\n\n
Once we have an account, we can sign-in<\/a> and continue with creating a project by the name mlpmnist-single-layer<\/code> and link the git repo https:\/\/github.com\/valohai\/mlpmnist-dl4j-example<\/a> to the project and save the project, have a quick look at the tutorials to see how to create a project using the web interface<\/a>.<\/p>\n\n\n\n
Now you can execute a step and see how it pans out!<\/p>\n\n\n\n
Building the DL4J Java app step<\/h2>\n\n\n\n
Go to the Executions<\/strong> tab in the web interface, either copy an existing or create a new execution using the [Create execution]<\/strong> button, all the necessary default options will be populated, select Step Build-dl4j-mnist-single-layer-java-app.<\/strong><\/p>\n\n\n\n
For Environment<\/em> I would select AWS eu-west-1 g2.2xlarge<\/em> and click on the [Create execution]<\/strong> button at the bottom of the page, to see the execution kick-off.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Training the model step<\/h2>\n\n\n\n
Go to the Executions<\/strong> tab in the web interface, and do the same as the previous step and select Step Run-dl4j-mnist-single-layer-train-model<\/strong>.  You will have to select the Java app (just type jar in the field) built  in the previous step, the dataset has already been pre-populated via  the valohai.yaml<\/strong> file:<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Click on [Create execution]<\/strong> to kick off this step.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
You will see the model summary fly by in the log console:<\/p>\n\n\n\n
[<--- snipped --->]\n11:17:05 =========================================================================\n11:17:05 LayerName (LayerType) nIn,nOut TotalParams ParamsShape\n11:17:05 =========================================================================\n11:17:05 layer0 (DenseLayer) 784,1000 785000 W:{784,1000}, b:{1,1000}\n11:17:05 layer1 (OutputLayer) 1000,10 10010 W:{1000,10}, b:{1,10}\n11:17:05 -------------------------------------------------------------------------\n11:17:05  Total Parameters: 795010\n11:17:05  Trainable Parameters: 795010\n11:17:05  Frozen Parameters: 0\n11:17:05 =========================================================================\n[<--- snipped --->]<\/code><\/pre>\n\n\n\n
The models created can be found under the Outputs<\/strong> sub-tab in the Executions<\/strong> main tab, during and at the end of the execution:<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
You might have noticed several artifacts in the Outputs<\/strong> sub-tab. That\u2019s because we save a checkpoint at the end of each epoch! Look out for these in the execution logs:<\/p>\n\n\n\n
[<--- snipped --->]\n11:17:14 o.d.o.l.CheckpointListener - Model checkpoint saved: epoch 0, iteration 469, path: \/valohai\/outputs\/checkpoint_0_MultiLayerNetwork.zip\n[<--- snipped --->]<\/code><\/pre>\n\n\n\n
The checkpoint zip contains the state of the model training at that point, saved in three of these files:<\/p>\n\n\n\n
configuration.json\ncoefficients.bin\nupdaterState.bin<\/code><\/pre>\n\n\n\n
Training the model > Metadata<\/strong><\/p>\n\n\n\n
You might have noticed these notations fly by in the execution logs:<\/p>\n\n\n\n
[<--- snipped --->]\n11:17:05 {\"epoch\": 0, \"iteration\": 0, \"score (loss function)\": 2.410047}\n11:17:07 {\"epoch\": 0, \"iteration\": 100, \"score (loss function)\": 0.613774}\n11:17:09 {\"epoch\": 0, \"iteration\": 200, \"score (loss function)\": 0.528494}\n11:17:11 {\"epoch\": 0, \"iteration\": 300, \"score (loss function)\": 0.400291}\n11:17:13 {\"epoch\": 0, \"iteration\": 400, \"score (loss function)\": 0.357800}\n11:17:14 o.d.o.l.CheckpointListener - Model checkpoint saved: epoch 0, iteration 469, path: \/valohai\/outputs\/checkpoint_0_MultiLayerNetwork.zip\n[<--- snipped --->]<\/code><\/pre>\n\n\n\n
These notations trigger Valohai<\/a>  to pickup these values (in JSON format) to be used to plot execution  metrics, which can be seen during and after the execution under the Metadata<\/strong> sub-tab in the Executions<\/strong> main tab:<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
We were able to do this by hooking a listener class (called ValohaiMetadataCreator<\/a>)  into the model, such that during training attention is passed on to  this listener class at the end of each iteration. In the case of this  class, we print the epoch count<\/strong>, iteration count<\/strong> and the score (the loss function value)<\/strong>, here is a code snippet from the class:<\/p>\n\n\n
\n    public void iterationDone(Model model, int iteration, int epoch) {\n        if (printIterations &lt;= 0)\n            printIterations = 1;\n        if (iteration % printIterations == 0) {\n            double score = model.score();\n            System.out.println(String.format(\n                    "{\\"epoch\\": %d, \\"iteration\\": %d, \\"score (loss function)\\": %f}",\n                    epoch,\n                    iteration,\n                    score)\n            );\n        }\n    }\n<\/pre><\/div>\n\n\n
Evaluating the model step<\/h2>\n\n\n\n
Once  the model has been successfully created via the previous step, we are  ready to evaluate it. We create a new execution just like we did  previously but this time select the Run-dl4j-mnist-single-layer-evaluate-model<\/strong> step. We will need to select the Java app (MLPMnist-1.0.0.jar<\/strong>) again and the created model (mlpmnist-single-layer.pb<\/strong>) before kicking off the execution (as shown below):<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
After selecting the desired model as input, click on the [Create execution]<\/strong> button. It is a quicker execution step than the previous one and we will see the following output:<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
The Evaluation Metrics<\/strong> and Confusion Matrix<\/strong> post model analysis will be displayed in the console logs.<\/p>\n\n\n\n
We can see our training activity has resulted in the model that is near 97%<\/strong> accurate based on the test dataset<\/em>.\n The confusion matrix helps point out the instances a digit has been \nincorrectly predicted as another digit. Maybe this is good feedback to \nthe creator of the model and maintainer of the dataset to do some \nfurther investigations.<\/p>\n\n\n\n
The question remains (and is outside the scope of this post) \u2014 how good is the model when faced with real-world data<\/em>?<\/p>\n\n\n\n
It\u2019s easy to install and get started with the CLI tool, see Command-line Usage<\/a>.<\/p>\n\n\n\n
If you haven\u2019t yet cloned the git repository then here\u2019s what to do:<\/p>\n\n\n\n
$ git clone https:\/\/github.com\/valohai\/mlpmnist-dl4j-example<\/a><\/pre>\n\n\n\n
We then need to link our Valohai<\/a>  project created via the web interface in the above section to the  project stored on our local machine (the one we just cloned). Run the  below commands to do that:<\/p>\n\n\n\n
$ cd mlpmnist-dl4j-example\n$ vh project --help   ### to see all the project-specific options we have for Valohai\n$ vh project link <\/code><\/pre>\n\n\n\n
You will be shown something like this:<\/p>\n\n\n\n
[  1] mlpmnist-single-layer\n...\nWhich project would you like to link with \/path\/to\/mlpmnist-dl4j-example?\nEnter [n] to create a new project.:<\/code><\/pre>\n\n\n\n
Select 1 (or the selection appropriate for you) and you should see this message:<\/p>\n\n\n\n
\ud83d\ude01 Success! Linked \/path\/to\/mlpmnist-dl4j-example to mlpmnist-single-layer.<\/code><\/p>\n\n\n\n
The quickest way to know of all the CLI options with the CLI tool is:<\/p>\n\n\n\n
$ vh \u2014 help<\/pre>\n\n\n\n
One more thing, before going any further ensure that your Valohai<\/a> project is in sync with the latest git project, by doing this:<\/p>\n\n\n\n
$ vh project fetch<\/pre>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
(on the top right side in your web interface, shown with the two-arrows-pointing-to-each-other icon).<\/p>\n\n\n\n
Now we can execute the steps from the CLI with:<\/p>\n\n\n\n
$ vh exec run Build-dl4j-mnist-single-layer-java-app<\/pre>\n\n\n\n
Once the execution is on, we can inspect and monitor it via:<\/p>\n\n\n\n
$ vh exec info\n$ vh exec logs\n$ vh exec watch<\/code><\/pre>\n\n\n\n
We can also see the above updates via the web interface at the same time.<\/p>\n\n\n\n
Further resources on how to go about with Valohai<\/a>\n has been provided in the Resources section at the end of the post, \nthere are a couple of blog posts on how to use the CLI tool, see [1]<\/a> | [2]<\/a>.<\/p>\n\n\n\n
Conclusion<\/h1>\n\n\n\n
As you have seen both DL4J<\/a> and Valohai<\/a>\n individually or combined are fairly easy to get started with. Further, \nwe can develop on the different components that make up our experiments \ni.e. build\/runtime environment, code, and dataset and integrate them \ninto an execution in a loosely coupled manner.<\/p>\n\n\n\n
The\n template examples used in this post are a good way to get started to \nbuild more complex projects. That you can use either the web interface \nor the CLI to get your job done with Valohai<\/a>! With the CLI you can also integrate it with your setup and scripts (or even with CRON or CI\/CD jobs).<\/p>\n\n\n\n
Also, it\u2019s clear that if I\u2019m working on an AI\/ML\/DL related project I don\u2019t need to concern myself with creating and maintaining<\/em> an end-to-end pipeline (which many others and I have had to do in the past) \u2014 thanks to the good work by the folks at Valohai<\/a>.<\/p>\n\n\n\n
Thanks to both <\/em>Skymind<\/em><\/a> (the startup behind <\/em>DL4J<\/em><\/a>, for creating, maintaining and keeping free) and <\/em>Valohai<\/em><\/a> for making this tool and cloud-service available for both free and commercial use.<\/em><\/p>\n\n\n\n
Please do let me know if this is helpful by dropping a line in the comments below or by tweeting at <\/em>@theNeomatrix369<\/em><\/a>, and I would also welcome feedback.<\/em><\/p>\n\n\n\n
Resources<\/h1>\n\n\n\n