#Spark intellij jar code

Depending on your version of IntelliJ, you may not have the second option as I mentioned in the screencast. Scala Breakpoints in IntelliJ Debugger Example IРІР‚в„ўll mention it in the resource section below as well. If you are entirely new to using IntelliJ for building Scala based Spark apps, you might wish to check out my previous tutorial on Scala Spark IntelliJ.

#Spark intellij jar code

But, feel free to translate the following to your own code right away as well. Pull the aforementioned repo if you want to take a slow approach to debug Scala in Spark. The ultimate goal will be to use your own code and environment. I know, I know, I caught you off guard their right? See Resources section below.Р”Р„nd hereРІР‚в„ўs a shocker, another requirement is IntelliJ. Portions of this tutorial require SBT and IРІР‚в„ўve provided a sample project available for you to pull from Github below. WeРІР‚в„ўll go through a few examples in this Scala Spark Debugging tutorial, but first, letРІР‚в„ўs get the requirements out of the way.

BreakpointsР’В are stored in IntelliJ (not in your applicationРІР‚в„ўs code).

A debugger is provided in IntelliJ and is responsible for stopping at breakpoints and displaying the current program state.

A breakpointР’В is a marker that you can set to specify where execution should pause when you are running your application.

WhatРІР‚в„ўs the difference between Breakpoints and Debuggers? SCala Breakpoints and DebuggersР”Р‡irst, letРІР‚в„ўs cover some background. It is assumed that you are already familiar with the concepts and value of debugging your Spark Scala code, but weРІР‚в„ўll quickly review a few key concepts before diving into the screencast examples. WeРІР‚в„ўll break the topic of debugging Scala based Spark programs into two sections:Р”Р„s youРІР‚в„ўll see in this tutorial there a few different options to choose from which depend on your Scala debug needs as well as if you are wishing to debug Spark in standalone mode vs debugging Spark job runningР’В on your cluster. These instructions and screencasts will hopefully allow you to start debugging Spark apps in IntelliJ and help me remember in the future. WeРІР‚в„ўll go through a few examples and utilize the occasional help of SBT.

So,Р’В in this tutorial, letРІР‚в„ўs cover debugging Scala based Spark programs in IntelliJР’В tutorial. Debugging with Scala code was easy, but when I moved to Spark things didnРІР‚в„ўt work as expected. Have you struggled to configure debugging in IntelliJ for your Spark programs? Yeah, me too.