mikaelv

#!/bin/bash

# Script to merge all "GitHub Classroom: Sync Assignment" PRs across student repositories
# Usage: ./merge-sync-prs.sh [directory]
# If no directory is provided, uses current directory

set -e

# Use provided directory or current directory
SUBMISSIONS_DIR="${1:-.}"

mikaelv

create script in /root

vi /root/whitelist-domains.sh
# paste script below

# Check firewall rules are active
iptables -L FORWARD -n -v
iptables -t nat -L PREROUTING -n -v

mikaelv

Run without dynamic allocation to avoid inconsistent results

Look at the Spark UI, focus on the Stage that takes the most time

If the stage’s DAG is large, add a few .cache instructions to break it down

Once you find a bottleneck, reproduce it with a Performance IT test

Apply some Spark tricks:

• load only the data you need from parquet. Push down the filters and select only the columns you need.
• use aggregateByKey instead of groupByKey
• Avoid iterating many times, use partitionByKey and then use mapPartitions to fusion several operations
• Avoid shuffling as much as possible. One stage usually corresponds to a shuffling. The fewer stages you have the better. partitionByKey followed by mapPartitions can help: shuffle only once and then do what you have to do.
• cache RDDs or Datasets if you discover that the same stage is performed many times

mikaelv

  trait Foo[F[_]] {
    type Internal
    type Group[_, _]
    implicit val barInstance: Bar[Internal]
    implicit val groupAble: GroupAble[Group]

    def groupBy[T, K](ft: F[T])(f: T => K): Group[K, T]
    def getInternal[T](f: F[T]): Internal
  }