Not 100% sure if this is necessary at this stage but eventually you’ll want to set SPARK_HOME in your bash profile &.Run databricks-connect configure and follow the instructions, filling in the configuration properties as written above.Org ID: # (don’t remember how I got this but it was part of the process).Cluster ID (see image below from the official docs).User token: get your personal access token with these instructions.You will need the following configuration properties: Renviron file ( open ~/.Renviron) and add JAVA_HOME='/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home' (replace the path with the output of /usr/libexec/java_home -v 1.8 if your path is different) Open your bash profile ( open ~/.bash_profile in the terminal) and add export JAVA_HOME=$(/usr/libexec/java_home -v 1.8) on a new line.To do so, either run the code below in the terminal OR (if you run into problems…)Įcho 'export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)' > $HOME/.bash_profileĮcho 'JAVA_HOME='$(/usr/libexec/java_home -v 1.8)'' > $HOME/.Renviron Add the JAVA_HOME variable to your bash profile and your.Run /usr/libexec/java_home -V in the terminal again to check that the install worked.Download Java 8 from (I had to register to download).

Note: the version number should read something like 1.8.*

Check to see if you have Java 8 by running /usr/libexec/java_home -V in the terminal.

To install, follow these instructions on StackOverflow or my tweaked version below.

Turns out, there is no escape from this obstacle if you want to use Spark from RStudio (at least with this Spark version). About a year ago, I gave up partway through a workshop because I couldn’t understand why I had to download Java 8 when I had Java 11.

Instal Hell is Others