Load data in Spark from multiple known partitions

val paths = Seq(“path1”, “path2”, “path3”)
val data = spark.read.option(“basePath”, basePath).parquet(paths:_*)
This entry was posted in Uncategorized by swk. Bookmark the permalink.

About swk

I am a software developr, data scientist, computational linguist, teacher of computer science and above all a huge fan of LaTeX. I use LaTeX for everything, including things you never wanted to do with LaTeX. My latest love is lilypond, aka LaTeX for music. I'll post at irregular intervals about cool stuff, stupid hacks and annoying settings I want to remember for the future.