本地提交spark,Spark提交无法找到本地文件

叁歲伎倆 2022-08-30 11:16 227阅读 0赞

![Image 1][]

I've written a very simple python script for testing my spark streaming idea, and plan to run it on my local machine to mess around a little bit. Here is the command line:

spark-submit spark\_streaming.py localhost 9999

But the terminal threw me an error:

Error executing Jupyter command '': \[Errno 2\] No such file or directory

I have no idea why this would happen, and I'm sure the .py file does exist.

EDIT: there's no issue running it with python instead of spark-submit

And also, the lines added in the .bashrc file:

export PATH="/usr/local/spark/bin:$PATH"

export PYSPARK\_DRIVER\_PYTHON=jupyter

export PYSPARK\_DRIVER\_PYTHON\_OPTS='notebook'

export SPARK\_LOCAL\_IP=localhost

解决方案

Supposing you want to spark-submit to YARN a Python script located at /home/user/scripts/spark\_streaming.py, the correct syntax is as follows:

spark-submit --master yarn --deploy-mode client /home/user/scripts/spark\_streaming.py

You can interchange the ordering of the various flags, but the script itself must be at the end; if your script accepts arguments, they should follow the script name (e.g. see this example for calculating pi with 10 decimal digits).

For executing locally with, say, 2 cores, you should use --master local\[2\] - use --master local\[\*\] for all available local cores (no deploy-mode flag in both cases).

Check the docs for more info (although admittedly they are rather poor in pyspark demonstrations).

PS The mention of Jupyter, as well the path shown in your error message are extremely puzzling...

UPDATE: Seems that PYSPARK\_DRIVER\_PYTHON=jupyter messes up everything, funneling the execution through Jupyter (which is undesirable here, and it may explain the weird error message). Try modifying the environment variables in your .bashrc as follows: