Common Atlas Troubleshooting

TROUBLESHOOTING.md

Error seen in IntelliJ

@Override is not allowed when implementing interface
Diamond types are not supported at this language level

File --> Project Structure --> Modules --> Sources
Change the Language level to 8 - Lambdas

Incorrect JDK version for the project
Add this in the pom.xml

           <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>

Spark submit giving Class not found error
This will create a jar file of the project.

cd proj_dir
mvn package

Spark submit giving Class not found error Create one jar with all dependencies

           <plugin>
              <artifactId>maven-assembly-plugin</artifactId>
              <configuration>
                  <descriptorRefs>
                      <descriptorRef>jar-with-dependencies</descriptorRef>
                  </descriptorRefs>
                  <archive>
                      <manifest>
                          <mainClass></mainClass>
                      </manifest>
                  </archive>
              </configuration>
              <executions>
                  <execution>
                      <id>make-assembly</id>
                      <phase>package</phase>
                      <goals>
                          <goal>single</goal>
                      </goals>
                  </execution>
              </executions>
          </plugin>

How to submit spark job

/bin/spark-submit --class org.cse.JavaKafkaWordCount --master local[2] /Users/drehman/IdeaProjects/FbPageStream/target/fbpage-stream-1.0-SNAPSHOT-jar-with-dependencies.jar 

call(String) and call(T) clash
Change

 public Iterator<String> call(String x) {
                return Arrays.asList(SPACE.split(x)).iterator();
 }

to

  public Iterable<String> call(String x) {
                return Arrays.asList(SPACE.split(x));
  }

Turn off INFO level logging

import org.apache.log4j.Level;
import org.apache.log4j.Logger;

Logger.getLogger("org").setLevel(Level.OFF);
Logger.getLogger("akka").setLevel(Level.OFF);

Java Spark Cassandra connector

        <dependency>
            <groupId>com.datastax.spark</groupId>
            <artifactId>spark-cassandra-connector-java_2.10</artifactId>
            <version>1.5.1</version>
        </dependency>

Spark connector version difference between what is mentioned in the pom.xml and what is in the spark.conf

6/07/23 02:02:46 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 10.0.0.235): java.io.InvalidClassException: com.datastax.spark.connector.rdd.partitioner.CassandraPartition; local class incompatible: stream classdesc serialVersionUID = 147531139326522345, local class serialVersionUID = 7247106480529291035