Publicado el por aironman

Mandelbrot, Spark y UDF

export JAVA_TOOL_OPTIONS="-Djava.security.manager=allow -Dhadoop.security.token.service.use_ip=false" && \ export SPARK_LOCAL_IP=********* && \ export HADOOP_USER_NAME=$(whoami) && \ spark-shell --packages org.apache.arrow:arrow-memory:2.0.0,org.apache.arrow:arrow-vector:2.0.0 \ --conf "spark.hadoop.fs.defaultFS=file:///" import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions.{udf, lit} import org.apache.spark.sql.expressions.UserDefinedFunction object MandelbrotComparison { def main(args: Array[String]): Unit = { val spark = SparkSession.builder .appName("MandelbrotComparison") .getOrCreate() import spark.implicits._ // Configuración para optimizar el uso de memoria y partición spark.conf.set("spark.sql.inMemoryColumnarStorage.batchSize", "10000")…

Publicado el por aironman · 1 Comentario

How to create a langchain agent able to talk to a spark cluster.txt

Straight to the point, the code is so intuitive that I think it doesn't need much more. To say that I personally am very excited about this possibility, because the ability on the one hand to tell the spark cluster what to do, plus the possibility of having an agent with all the knowledge and…

Publicado el por aironman

First steps with Apache Spark 3.5.0 Delta Lake using scala.

https://docs.delta.io/latest/quick-start.html#create-a-table&language-scala first, install apache spark, i am osx user, so i will not recommend to use homebrew because it will not install third party libraries. I recommend to download from https://spark.apache.org Latest version is 3.5.0 at 28 nov 2023. Then, run spark-shell with delta lake support: ATTENTION, be sure about delta lake version, you must…