Skip to content

Instantly share code, notes, and snippets.

@jrudolph
Last active July 27, 2019 13:14
Show Gist options
  • Star 3 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save jrudolph/6552186 to your computer and use it in GitHub Desktop.
Save jrudolph/6552186 to your computer and use it in GitHub Desktop.
ListBuffer / List mutability fail
import scala.collection.mutable.ListBuffer
object ListBufferTest extends App {
new X().runTest()
}
class X extends Runnable {
var x: List[String] = List("", "")
def run() {
val threadId = Thread.currentThread().getId
println(s"Thread $threadId started")
while (true) {
val size = x.size
if (size != 2) println(f"${System.nanoTime()}%10d Thread: $threadId%2d size is $size%d")
Thread.sleep(1)
}
}
def runTest(): Unit = {
Seq.fill(50)(new Thread(this)).foreach(_.start())
println("Starting mutation")
while (true) {
val lb = new ListBuffer[String]
lb.+=("")
lb.+=("")
x = lb.toList
}
}
}
Thread 19 started
Thread 27 started
Thread 21 started
Thread 10 started
Thread 8 started
Thread 20 started
Thread 26 started
Thread 9 started
Thread 12 started
Thread 22 started
Thread 15 started
Thread 23 started
Thread 35 started
Thread 24 started
Thread 11 started
Thread 14 started
Thread 13 started
Thread 36 started
Thread 17 started
Thread 18 started
Thread 16 started
Thread 38 started
Thread 37 started
Thread 34 started
Thread 33 started
Thread 32 started
Thread 31 started
Thread 30 started
Thread 29 started
Thread 42 started
Thread 28 started
Thread 43 started
Thread 45 started
Thread 25 started
Thread 44 started
Thread 40 started
Thread 41 started
Thread 39 started
Thread 48 started
Thread 47 started
Thread 46 started
Thread 50 started
Thread 49 started
Thread 52 started
Thread 51 started
Thread 53 started
Thread 54 started
Thread 55 started
Thread 56 started
Starting mutation
Thread 57 started
76182900042609 Thread: 20 size is 1
76182899663916 Thread: 55 size is 1
76182900169880 Thread: 28 size is 1
76182900039041 Thread: 56 size is 1
76182900227595 Thread: 51 size is 1
76182900174803 Thread: 44 size is 1
76182900233685 Thread: 10 size is 1
76182900182129 Thread: 38 size is 1
76182900047492 Thread: 9 size is 1
76182899664207 Thread: 26 size is 1
76182900159549 Thread: 37 size is 1
76183001324166 Thread: 32 size is 1
76183001307880 Thread: 19 size is 1
76183001308082 Thread: 18 size is 1
76183001477762 Thread: 42 size is 1
76183001687972 Thread: 15 size is 1
76183001592480 Thread: 16 size is 1
76183054550012 Thread: 30 size is 1
76183147865429 Thread: 9 size is 1
76183147871973 Thread: 33 size is 1
76183147863188 Thread: 36 size is 1
76183147870613 Thread: 48 size is 1
76183225216315 Thread: 53 size is 1
76183225236539 Thread: 47 size is 1
76183225216317 Thread: 36 size is 1
76183225226867 Thread: 38 size is 1
76183440270289 Thread: 40 size is 1
76183736358984 Thread: 44 size is 1
76183772741699 Thread: 53 size is 1
76183772729575 Thread: 15 size is 1
76183772734699 Thread: 50 size is 1
76183772720198 Thread: 9 size is 1
76183772752927 Thread: 29 size is 1
76183780017018 Thread: 49 size is 1
76183995525941 Thread: 26 size is 1
76183995537013 Thread: 12 size is 1
76183995536995 Thread: 47 size is 1
76183995544795 Thread: 29 size is 1
76184070257285 Thread: 23 size is 1
76184204966458 Thread: 39 size is 1
76184205222561 Thread: 23 size is 1
76184205614475 Thread: 12 size is 1
76184205329783 Thread: 52 size is 1
76184205329080 Thread: 46 size is 1
76184204956682 Thread: 9 size is 1
76184205659702 Thread: 41 size is 1
76184205887909 Thread: 38 size is 1
76184205420312 Thread: 19 size is 1
76184205920225 Thread: 55 size is 1
76184204963788 Thread: 10 size is 1
76184205410299 Thread: 51 size is 1
76184205329154 Thread: 31 size is 1
76184205063133 Thread: 53 size is 1
76184205087921 Thread: 49 size is 1
76184205181487 Thread: 34 size is 1
76184204968248 Thread: 44 size is 1
76184205612416 Thread: 43 size is 1
76184205417808 Thread: 16 size is 1
76184204977072 Thread: 17 size is 1
76184355355982 Thread: 57 size is 1
76184380659279 Thread: 15 size is 1
76184505372060 Thread: 50 size is 1
76184505520948 Thread: 42 size is 1
76184505496872 Thread: 26 size is 1
76184505359275 Thread: 39 size is 1
76184505339999 Thread: 43 size is 1
76184505348026 Thread: 13 size is 1
76184505338428 Thread: 57 size is 1
76184505525632 Thread: 12 size is 1
76184505359191 Thread: 55 size is 1
76184528136891 Thread: 14 size is 1
76184528136899 Thread: 40 size is 1
76184695698888 Thread: 25 size is 1
76184858881189 Thread: 39 size is 1
76184858934828 Thread: 30 size is 1
76184858992123 Thread: 12 size is 1
76184858944946 Thread: 27 size is 1
76184859053092 Thread: 19 size is 1
76184858874368 Thread: 44 size is 1
76184858960528 Thread: 14 size is 1
76184858888525 Thread: 45 size is 1
76184859396870 Thread: 57 size is 1
76184859347617 Thread: 38 size is 1
76184859345531 Thread: 26 size is 1
76184859480958 Thread: 13 size is 1
76184858964017 Thread: 50 size is 1
76184858881722 Thread: 34 size is 1
76184858901695 Thread: 43 size is 1
76184975962728 Thread: 39 size is 1
76185033288317 Thread: 23 size is 1
76185140008914 Thread: 14 size is 1
76185140009508 Thread: 48 size is 1
76185140009121 Thread: 52 size is 1
76185140008387 Thread: 42 size is 1
76185226277225 Thread: 12 size is 1
76185238811668 Thread: 17 size is 1
76185238791600 Thread: 28 size is 1
76186051469814 Thread: 13 size is 1
76186532837737 Thread: 32 size is 1
76186952012101 Thread: 49 size is 1
76186952014544 Thread: 23 size is 1
76187060013313 Thread: 55 size is 1
76187060011998 Thread: 26 size is 1
76187263270307 Thread: 39 size is 1
76187443632503 Thread: 21 size is 1
76187719698937 Thread: 31 size is 1
76187749500076 Thread: 12 size is 1
76187785068791 Thread: 18 size is 1
76187785065852 Thread: 41 size is 1
76187996162484 Thread: 15 size is 1
76188010181979 Thread: 50 size is 1
76188021540430 Thread: 46 size is 1
76188021557448 Thread: 32 size is 1
76188356088859 Thread: 57 size is 1
76188376262502 Thread: 20 size is 1
76188377251717 Thread: 12 size is 1
76188377443464 Thread: 8 size is 1
76188377470522 Thread: 56 size is 1
76188377656454 Thread: 49 size is 1
76188377672130 Thread: 26 size is 1
76188377561861 Thread: 55 size is 1
76188377290140 Thread: 35 size is 1
76188378323170 Thread: 20 size is 1
76188377520355 Thread: 40 size is 1
76188377270222 Thread: 47 size is 1
76188377295204 Thread: 24 size is 1
76188377409624 Thread: 28 size is 1
76188377807433 Thread: 22 size is 1
76188377652883 Thread: 13 size is 1
76188377570763 Thread: 54 size is 1
76188377725460 Thread: 9 size is 1
76188377220068 Thread: 37 size is 1
76188377474385 Thread: 32 size is 1
76188377772749 Thread: 52 size is 1
76188377773196 Thread: 51 size is 1
76188377820276 Thread: 44 size is 1
76188452147554 Thread: 25 size is 1
76188642167003 Thread: 8 size is 1
76188729025008 Thread: 42 size is 1
76188856380411 Thread: 50 size is 1
76188856665314 Thread: 21 size is 1
76188856625018 Thread: 10 size is 1
76188856236848 Thread: 48 size is 1
76188856225581 Thread: 28 size is 1
76188856259002 Thread: 20 size is 1
76188856227599 Thread: 56 size is 1
76188856381790 Thread: 55 size is 1
76188856383197 Thread: 19 size is 1
76188856225610 Thread: 52 size is 1
76188856256976 Thread: 41 size is 1
76188856817475 Thread: 25 size is 1
76188856738336 Thread: 33 size is 1
76188856732765 Thread: 11 size is 1
76189028010631 Thread: 52 size is 1
76189028010715 Thread: 9 size is 1
76189055664409 Thread: 55 size is 1
76189089158162 Thread: 52 size is 1
76189549350604 Thread: 12 size is 1
76189549915766 Thread: 24 size is 1
76189549910058 Thread: 8 size is 1
76189550234811 Thread: 34 size is 1
76189550549363 Thread: 40 size is 1
76189549501448 Thread: 41 size is 1
76189549495902 Thread: 56 size is 1
76189549913447 Thread: 45 size is 1
76189550736182 Thread: 27 size is 1
76189549739007 Thread: 51 size is 1
76189549371779 Thread: 10 size is 1
76189549325080 Thread: 38 size is 1
76189549334962 Thread: 22 size is 1
76189549733379 Thread: 9 size is 1
76189550994887 Thread: 18 size is 1
76189549735619 Thread: 30 size is 1
76189551228555 Thread: 29 size is 1
76189550206236 Thread: 26 size is 1
76189549734098 Thread: 33 size is 1
76189549992592 Thread: 13 size is 1
76189549796000 Thread: 25 size is 1
76189551272173 Thread: 57 size is 1
76189551280120 Thread: 24 size is 1
76189550158280 Thread: 43 size is 1
76189550004152 Thread: 32 size is 1
76189550217012 Thread: 20 size is 1
76189550037162 Thread: 37 size is 1
76189551048278 Thread: 12 size is 1
76189549731252 Thread: 50 size is 1
76189550158319 Thread: 36 size is 1
76189550799159 Thread: 47 size is 1
76189549913452 Thread: 42 size is 1
76189550791117 Thread: 11 size is 1
76189550164145 Thread: 17 size is 1
76189550561657 Thread: 54 size is 1
76189550539231 Thread: 31 size is 1
76189550519820 Thread: 23 size is 1
76189550094306 Thread: 16 size is 1
76189550497524 Thread: 48 size is 1
76189550304738 Thread: 28 size is 1
76189550214324 Thread: 35 size is 1
76189550253049 Thread: 44 size is 1
76189549325062 Thread: 52 size is 1
76189659056948 Thread: 23 size is 1
76189853391602 Thread: 41 size is 1
76190047302012 Thread: 30 size is 1
76190130062478 Thread: 55 size is 1
76190171674186 Thread: 35 size is 1
76190209076245 Thread: 41 size is 1
76190251095310 Thread: 38 size is 1
76190355737115 Thread: 31 size is 1
76190524040530 Thread: 17 size is 1
76190632929867 Thread: 33 size is 1
76190684076491 Thread: 42 size is 1
76190901284568 Thread: 23 size is 1
76190901293362 Thread: 34 size is 1
76190901279379 Thread: 14 size is 1
76190904027777 Thread: 44 size is 1
76190904022142 Thread: 42 size is 1
76190910834666 Thread: 8 size is 1
76190910860446 Thread: 31 size is 1
76191476011244 Thread: 24 size is 1
76191592809369 Thread: 50 size is 1
76191654692729 Thread: 53 size is 1
76191654698240 Thread: 20 size is 1
76191654704327 Thread: 47 size is 1
76191797047390 Thread: 49 size is 1
76191859290855 Thread: 15 size is 1
76192057792233 Thread: 20 size is 1
76192108444460 Thread: 30 size is 1
76192108444091 Thread: 33 size is 1
76192113115327 Thread: 17 size is 1
76192324520155 Thread: 43 size is 1
76192324530774 Thread: 31 size is 1
76192324518660 Thread: 23 size is 1
76192370916245 Thread: 36 size is 1
76192556013536 Thread: 8 size is 1
76192556016998 Thread: 37 size is 1
76192616017947 Thread: 41 size is 1
76192665092727 Thread: 33 size is 1
76192665091605 Thread: 21 size is 1
76192675816178 Thread: 16 size is 1
76192805280864 Thread: 29 size is 1
76192884551843 Thread: 33 size is 1
76192893100068 Thread: 28 size is 1
76192935171308 Thread: 39 size is 1
76193049373304 Thread: 11 size is 1
76193049750077 Thread: 41 size is 1
76193049860408 Thread: 27 size is 1
76193050400686 Thread: 29 size is 1
76193049333597 Thread: 25 size is 1
76193050776059 Thread: 56 size is 1
76193049874672 Thread: 14 size is 1
76193049714114 Thread: 44 size is 1
76193049349455 Thread: 53 size is 1
76193049612094 Thread: 30 size is 1
76193049469757 Thread: 39 size is 1
76193049341218 Thread: 26 size is 1
76193049337996 Thread: 40 size is 1
76193051220459 Thread: 57 size is 1
76193050945877 Thread: 52 size is 1
76193051457772 Thread: 12 size is 1
76193049534073 Thread: 42 size is 1
76193051357931 Thread: 35 size is 1
76193049749787 Thread: 49 size is 1
76193051220670 Thread: 37 size is 1
76193050764989 Thread: 18 size is 1
76193051243987 Thread: 21 size is 1
76193050707773 Thread: 51 size is 1
76193051077516 Thread: 46 size is 1
76193051111395 Thread: 41 size is 1
76193050997791 Thread: 28 size is 1
76193051029176 Thread: 11 size is 1
76193050604689 Thread: 50 size is 1
76193050448836 Thread: 55 size is 1
76193050352376 Thread: 19 size is 1
76193050596868 Thread: 45 size is 1
76193050646756 Thread: 22 size is 1
76193050571876 Thread: 13 size is 1
76193050710086 Thread: 9 size is 1
76193050673763 Thread: 33 size is 1
76193050566219 Thread: 38 size is 1
76193049381903 Thread: 8 size is 1
76193050128107 Thread: 17 size is 1
76193050442374 Thread: 15 size is 1
76193050177538 Thread: 47 size is 1
76193050337495 Thread: 43 size is 1
76193050326500 Thread: 10 size is 1
76193050235930 Thread: 54 size is 1
76193050234183 Thread: 32 size is 1
76193050156190 Thread: 23 size is 1
76193050009130 Thread: 36 size is 1
76193049979117 Thread: 16 size is 1
76193049955803 Thread: 34 size is 1
76193049925485 Thread: 48 size is 1
76193057718281 Thread: 11 size is 1
76193049925407 Thread: 20 size is 1
76193049789227 Thread: 24 size is 1
76193061287542 Thread: 18 size is 1
76193061207499 Thread: 29 size is 1
76193061139060 Thread: 44 size is 1
76193061581279 Thread: 53 size is 1
76193061582704 Thread: 38 size is 1
76193061556795 Thread: 39 size is 1
76193061237332 Thread: 35 size is 1
76193061153620 Thread: 22 size is 1
76193062253740 Thread: 13 size is 1
76193062630279 Thread: 33 size is 1
76193062730328 Thread: 9 size is 1
76193062107440 Thread: 45 size is 1
76193062881463 Thread: 21 size is 1
76193062183162 Thread: 25 size is 1
76193062647641 Thread: 57 size is 1
76193063086607 Thread: 49 size is 1
76193063142090 Thread: 11 size is 1
76193063201536 Thread: 20 size is 1
76193063159192 Thread: 19 size is 1
76193063472958 Thread: 56 size is 1
76193063441579 Thread: 39 size is 1
76193063219810 Thread: 29 size is 1
76193063206927 Thread: 18 size is 1
76193063042381 Thread: 50 size is 1
76193063137220 Thread: 55 size is 1
76193062864497 Thread: 54 size is 1
76193062628778 Thread: 24 size is 1
76193062486362 Thread: 27 size is 1
76193062106844 Thread: 52 size is 1
@quelgar
Copy link

quelgar commented Sep 14, 2013

Um, isn't this programmer fail? You have absolutely no memory barriers. The Java memory model makes no guarantee that the writes you are making will be visible in the same order to other threads.

Almost certainly your hardware has multiple cores and some of the threads are running on a different core to the main thread. The hardware is making the write to x visible before the changes to the internal ListBuffer state, which is allowed absent a memory barrier.

@chris-martin
Copy link

If you write shared memory without locking, you're gonna have a bad time.

@joa
Copy link

joa commented Sep 14, 2013

If the List fields head and tail would be final, he would not need to worry about memory barriers. The example shows that publishing a List is unsafe although it is advertised as an immutable data structure.

@Mortimerp9
Copy link

The immutable list never gets modified as a new list is created every time x = lb.toList. Even if a list is immutable, if you store it in a mutable variable, there is mutability ... of the reference.

Rule of thumb: var and while and other constructions that rely on side effects (foreach, ...) are going to be trouble in Threads.

@scottcarey
Copy link

"The immutable list never gets modified as a new list is created every time x = lb.toList"

look at the source code, it does not create a new list when toList is called, and fields are mutated. The point is that mutation is being used internally to build a supposedly "immutable" data structure, leading to the side effects of mutation being observable. If it did not use mutation, this could not happen, no synchronization primitives needed, since the JVM can guarantee that an object is 'fully formed' after it is constructed from the perspective of all threads.

@dlandis
Copy link

dlandis commented Dec 11, 2013

I can't reproduce this on my computer. size is always 2 after waiting a while. Does it depend on a particular os/vm/architecture?

@simbo1905
Copy link

@Blaisorblade
Copy link

@dlandis:

I can't reproduce this on my computer. size is always 2 after waiting a while. Does it depend on a particular os/vm/architecture?

Sure it depends — the debate is on what meanings are possible, but nobody claims x.size being 2 is forbidden, only that some valid optimizations make it not 2. Your JVM isn't forced to do those optimizations (for instance, https://issues.scala-lang.org/browse/SI-7838?focusedCommentId=75218#comment-75218 explains why other breakage does not happen).

Answering to @szeiger in scala/collection-strawman#49 (comment):

I'm late to the party because I've been on vacation for a week but I just wanted to point out that I have yet to see an otherwise correct program that gets broken by unsafe publishing of Lists. The snippet mentioned by @scottcarey shows a new failure mode created by the lack of safe publishing but without any synchronization or making x volatile, there is no guarantee that changes would be observed on other threads, so the code would still be meaningless.

We're talking of https://gist.github.com/jrudolph/6552186, right? Writes to final fields from constructors are visible to other threads.

The only question is what happens with writes to X.x, but according to the JMM author Jeremy Manson, we're guaranteed reads from it will always see some value that was actually written to it. In this example, if lb.toList returned a properly immutable list, X.x.size isn't allowed to be 1 according to the JMM.* Quoting that blog post:

Java has a special guarantee for fields that are 32-bit or fewer and object references — a thread will always see some value that was assigned to that field. As a result of this, a thread reading hash will either see 0 or the calculated value. Edited to add: I just noticed that this was misleading. An object reference will point to the right object, but the contents of the object aren't guaranteed to be correct unless the object is immutable. Please see my series on immutability.

*In fact, the JMM as formally stated is broken, especially for racy programs, and it does not model what JMMs are allowed to do, and I'm not sure anybody came up with a replacement yet. But I understand Jeremy Manson's posts are still accurate.

@szeiger
Copy link

szeiger commented Apr 3, 2017

@Blaisorblade Yes, I agree. Without safe publication you can observe x in an invalid state, which is clearly a bug. However, even with safe publication you are not guaranteed to observe any reassignment tox from another thread at all, so you already have a concurrency bug and any fix for the latter will also fix the former.

@NthPortal
Copy link

@szeiger There are situations where valid code may see a List in an invalid state. For example:

class Message(val text: String) {
  private[this] var _words = null;

  def words: String = {
    var res = _words
    if (words == null) {
      res = words.split("""\s""").toList
      _words = res
    }
    res
  }
}

The above class only computes the words in a message if needed, similarly to how String computes its hashCode() (as seen in Jeremy Manson's post which @Blaisorblade mentioned). The result of the computation is not guaranteed to be seen by other threads, and if it is not, the result is recomputed.

The problem arises because List is not actually immutable, so a thread might see the result of a computation from another thread in an invalid state. If List was properly immutable, the class would be guaranteed to work properly.

@plokhotnyuk
Copy link

Now with Scala 2.13.0 all lists have size == 2:

Thread 11 started
Thread 10 started
Thread 34 started
Thread 25 started
Thread 22 started
Thread 23 started
Thread 17 started
Thread 16 started
Thread 18 started
Thread 14 started
Thread 15 started
Thread 19 started
Thread 13 started
Thread 35 started
Thread 12 started
Thread 33 started
Thread 21 started
Thread 28 started
Thread 24 started
Thread 20 started
Thread 30 started
Thread 31 started
Thread 27 started
Thread 29 started
Thread 26 started
Thread 36 started
Thread 32 started
Thread 37 started
Thread 38 started
Thread 39 started
Thread 40 started
Starting mutation
Thread 55 started
Thread 54 started
Thread 53 started
Thread 56 started
Thread 57 started
Thread 58 started
Thread 52 started
Thread 41 started
Thread 51 started
Thread 50 started
Thread 49 started
Thread 48 started
Thread 47 started
Thread 46 started
Thread 45 started
Thread 44 started
Thread 43 started
Thread 42 started
Thread 59 started

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment