CustomMob.java

    public static boolean attemptSpawn(Entity entity) {
        Location loc = entity.getLocation();

        long now = System.currentTimeMillis();
        if (now - lastSpawnAttempt > 5000) {
            lastSpawnAttempt = now;
            for (CustomMob customMob : spawnableList) {
                float randChance = Util.RANDOM.nextFloat() * 100;
                if (customMob.spawnRate == 0 ||
                    now - customMob.lastSpawn < customMob.respawnRate) {
                    continue;
                }
                if (randChance > customMob.spawnRate || !customMob.canSpawn(loc)) {
                    continue;
                }

                boolean tooClose = false;
                for (Entity playerCheck : entity.getNearbyEntities(customMob.minPlayerSpawn,
                    256,
                    customMob.minPlayerSpawn)) {
                    if (playerCheck instanceof Player) {
                        tooClose = true;
                        break;
                    }
                }
                if (!tooClose) {
                    continue;
                }

                if (Wastelands.isWastelands(entity.getWorld())) {
                    if (Wastelands.isSpawnVicinity(entity.getLocation(), 64)) {
                        continue;
                    }
                }

                // Finally can spawn
                customMob.lastSpawn = now;
                customMob.spawn(loc);
                return true;
            }
        }
        return false;
    }

Listener.java

    @EventHandler(ignoreCancelled = true)
    public void onCreatureSpawn(CreatureSpawnEvent event) {
        Location loc = event.getLocation();
        if (Util.isTownWorld(loc.getWorld()) || event.getSpawnReason() != CreatureSpawnEvent.SpawnReason.NATURAL) {
            return;
        }
        if (event.getEntity() instanceof Monster) {
            if (CustomMob.attemptSpawn(event.getEntity())) {
                event.setCancelled(true);
            }
        }
    }

Why not just use Math.random() for random number generation?

You could move the randChance variable down below the first if statement. Probably doesn't make much difference, but if the first if statement evaluates to true, then the loop continues and it's a wasted calculation.

Is if (!tooClose) correct? Shouldn't it be reversed? o.O