Data data and data everywhere. Everyone has their own’s data. Also, every organization has its huge cloud storage that is known as data-lake, and all sort of data is present in that data-lake. But, now it’s time to analyze the structure of your data lake. …

As we all know autoscaling is add processing power when needed, remove the idle instances when not needed. Why pay when you are not using it. In this blog, I will discuss some important custom autoscaling policies for your production-ready EMR cluster. AWS is happy to announce amazon managed EMR…

Delta lake comes with awesome features to overcome the outcomes of spark or any big data platform. So what outcomes does delta lake guarantees to support ACID transactions on Data Lake Schema Enforcement Time Travel Capabilities Delta-lake is safe and it is widely used in various production platforms. In this…

Monitoring spark application with time-series databases like influx, with help of spark performance metrics we can diagnose various issues in our spark application whether streaming or batch application Spark exposes metrics to various endpoint, we will be using graphite endpoints for exposing metrics ConsoleSink: Logs metrics information to the console. CSVSink…

Amazon MSK is being widely used, and it is a fully managed service for the streaming of data. MSK provides various out of the box monitoring capabilities like disk usage per broker, CPU usage per broker, Network TX(transmit) and RX(Receive) packet broker. …

Using spark for inserting and reading data from the Neo4j graph database