Skip to content
This repository was archived by the owner on Jan 9, 2020. It is now read-only.

Commit ecfa6f2

Browse files
committed
Avoid formatting python path (when using --py-files) in kubernetes cluster mode for spark-submit
1 parent 79ffa72 commit ecfa6f2

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -682,10 +682,10 @@ object SparkSubmit extends CommandLineUtils {
682682
// explicitly sets `spark.submit.pyFiles` in his/her default properties file.
683683
sysProps.get("spark.submit.pyFiles").foreach { pyFiles =>
684684
val resolvedPyFiles = Utils.resolveURIs(pyFiles)
685-
val formattedPyFiles = if (!isYarnCluster && !isMesosCluster) {
685+
val formattedPyFiles = if (!isYarnCluster && !isMesosCluster && !isKubernetesCluster) {
686686
PythonRunner.formatPaths(resolvedPyFiles).mkString(",")
687687
} else {
688-
// Ignoring formatting python path in yarn and mesos cluster mode, these two modes
688+
// Ignoring formatting python path in yarn, mesos, and kubernetes cluster mode, these two modes
689689
// support dealing with remote python files, they could distribute and add python files
690690
// locally.
691691
resolvedPyFiles

0 commit comments

Comments
 (0)