The problem of transforming data to deliver insights to business users hasn’t changed. However, the volume, velocity and variety of data now needing to be handled to achieve those goals has grown exponentially. As a data engineer, we can address that demand with highly scalable, distributed compute and storage, bringing together Apache Spark and the Hadoop Distributed File System (HDFS), now abstracted by many cloud providers and commonly known as our Data Lake or Lakehouse. These cloud native (opensource) resources give us the tools we need to fore fill the business requirements of data transformation, modelling and serving of any inbound datasets. With the ability to also create solid foundations for data science predictive workloads and artificial intelligence.

Using the mature product offering Azure Databricks, we will explore and together code data processing workloads in Python or Scala and Spark SQL to create rich, modelled outputs from dynamic, scalable executions. Once done, we’ll consider porting our Notebooks to other products such as Microsoft Fabric to deliver the same Apache Spark capabilities but wrapped as alternative serverless offerings.

As a new data engineer, or experienced professional looking to enhance your skills, attend this course and leave with everything you need to implement Apache Spark workloads in Azure.