Main.java

package tmp;

import java.util.Collections;
import java.util.Iterator;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;

public class Main {

	public static void main(String[] args) {
		Set<String> s = Collections.newSetFromMap(new ConcurrentHashMap<String, Boolean>());
		s.add("1");
		s.add("2");
		s.add("3");
		s.add("4");
		Iterator<String> it = s.iterator();
		System.out.println(it.next());
		s.remove("4");
		System.out.println(it.next());
		System.out.println(it.next());
		
	}

}

but you fixed the problem i ran into at my first attempt, i was not able to drop an connection into the map without creating an empty set first, so i choose to use the ConcurrentSkipListSet, as you can instantiate them without problems, your solution fixes this, in some way

ConcurrentHashMap<String, Set<Integer>> map = 
          new ConcurrentHashMap<String, Set<Integer>>();
map.put("con1", Collections.newSetFromMap(
          new ConcurrentHashMap<Integer, Boolean>()));
map.put("con2", Collections.newSetFromMap(
          new ConcurrentHashMap<Integer, Boolean>()));
Set<Integer> set1 = map.get("con2");
set1.add(1);
set1.add(2);
set1.add(3);
set1.add(4);
set1 = map.get("con2");
Set<Integer> set2 = map.get("con2");
Iterator<Integer> it2 = set2.iterator();
System.out.println(it2.next() + " map: " + map);
System.out.println(it2.next() + " map: " + map);
set1.remove(2);
System.out.println(it2.next() + " map: " + map);
System.out.println(it2.next() + " map: " + map);

looks fine for me :D with that the SkipListSet become history

but why the heck do i have to use Boolean there ???

We exchange a portion of wasted RAM for execution time therefore am i right ?

Ok, you can ignore my comment on the issue, github did not automatically subscribe me to this thread as it does with prs and issues.
I think pretty much every set(hashset and probably most others as well) is backed by a map <T, Boolean>, so it would not be additional memory.
So, are you ok with me pr'ing that commit?

as you said, every set is backed by a list or map ... the one it is created from of course XD .. but the default implementation for most sets is base of a non-concurrent safe list or hashmap, therefore i was curious if your aproach is properly CME safe and you proved me wrong :D thank you for teaching me a lesson