One main advantage of the Apache Spark is, it splits data into multiple partitions and executes operations on all partitions of data in parallel which allows us to complete the job faster.While working with partition data we often need to increase or decrease the partitions based on data distribution. Methods repartition and coalesce helps us to repartition.
Like this:
Like Loading...
Apache Hive 3 Changes in CDP Upgrade: Part-1
INTRODUCTION Cloudera Data Platform (CDP) is a cloud computing platform for businesses. It provides integrated and multifunctional self-service tools in order to analyze and centralize data. It brings security and governance. read more…
Share this:
Like this:
Continue Reading
Generate Sequential and Unique IDs in a Spark Dataframe
Apache Spark is an open source, general-purpose distributed computing engine used for processing and analyzing a large amount of data. Hence, adding sequential and unique IDs to a Spark Dataframe is not very straight forward, because of distributed nature of it.
Share this:
Like this:
Continue Reading
Spark Partitions with Coalesce and Repartition (hash, range, round robin)
One main advantage of the Apache Spark is, it splits data into multiple partitions and executes operations on all partitions of data in parallel which allows us to complete the job faster.While working with partition data we often need to increase or decrease the partitions based on data distribution. Methods repartition and coalesce helps us to repartition.
Share this:
Like this:
Continue Reading