You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
:: Setting up Databricks Connect with a specific version of Spark
conda create --name databricks-connect python=3.10 -y
conda activate databricks-connect
pip install databricks-connect==13.*
:: Using Python to get the site-packages directoryfor /f "delims="%%ain ('python -c "import site; print(site.getsitepackages()[-1])"') doset"_SITE=%%a":: Setting up environment variablessetSPARK_HOME=%_SITE%\pyspark
setPYSPARK_PYTHON=python
setPYSPARK_DRIVER_PYTHON=%PYSPARK_PYTHON%setPATH=%SPARK_HOME%\bin;%PATH%setPYTHONPATH=%SPARK_HOME%\python;%PYTHONPATH%setPYTHONPATH=%_SITE%;%SPARK_HOME%\python\lib\py4j-*.zip;%PYTHONPATH%echo"SPARK_HOME=%SPARK_HOME%;PYTHONPATH=%PYTHONPATH%;PATH=%PATH%":: Paste to the PyCharm run configuration environment
For the local Spark setup:
:: Assuming you have deactivated the previous environment:: Define the installation directory of your Spark version (modify according to your actual path)setSPARK_HOME=%USERPROFILE%\spark-3.4\spark-3.4.1-bin-hadoop3
setPYSPARK_PYTHON=python
setPYSPARK_DRIVER_PYTHON=%PYSPARK_PYTHON%setPATH=%SPARK_HOME%\bin;%PATH%setPYTHONPATH=%SPARK_HOME%\python;%PYTHONPATH%setPYTHONPATH=%SPARK_HOME%\python\lib\py4j-*.zip;%PYTHONPATH%echo"SPARK_HOME=%SPARK_HOME%;PYTHONPATH=%PYTHONPATH%;PATH=%PATH%":: Paste to the PyCharm run configuration environment
Please note:
This script assumes that you've installed Spark in the directory %USERPROFILE%\spark-3.4\spark-3.4.1-bin-hadoop3. You may need to adjust the path to match your Spark installation directory.
The %USERPROFILE% environment variable corresponds to the home directory of the current user. This is analogous to $HOME on Unix-like systems.
This is what I'm attempting to do. I'm using a Windows 10 laptop.