jackcviers

import cats.MonadError
import cats.instances.either._
import cats.instances.future._
import cats.instances.try_._
import cats.syntax.applicativeError._
import cats.syntax.flatMap._

import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
import scala.concurrent.{Await, Future}

jackcviers

So the hacker news post said my comment was too long. Warning, long, opinionated post:

Scala dev for 10+ years here. Spark is weird. Databricks has a style guide that deliberately chooses not to use scala features that the rest of the community uses, and doesn't follow the same best practices around library and scala major version usage that the rest of the community uses [1]. It's no surprise that the project has trouble interoperating with libraries outside of the spark ecosystem, and is therefore a maintenance problem.

Spark's style and compatibility problems

Scala isn't a maintenance nightmare, but it does attract a lot of newcomers who dive in, don't stick within one of its many ecosystems, get confused, and generally leave a mess, and that is a direct result of the fact that scala is a multi-paradigm, relatively expressive language to the one(s) it is competing with and pulling developers from, and that those developers, for the large part, don't really want to change and think that Scala is just a