Many times as a developer after changing code in spark jobs ,To test the changes inside databricks cluster we need to follow quite a number of steps create jar using sbt/maven create cluster with all required arn and put the created jar copy all dependencies to databricks cluster These are…

Many time when I work with kafka I feel tempted to use kafka to store the data but it should never be used as datastore instead we should use our data lake to store the topics. …

While working with any data pipeline projects most of times programmer deals with slowly changing dimension data . Here in this post I will try to jot down implementation of SCD Type 2 using apache spark.All these operations will be done in memory after reading your source and target data…