Duplicate ManyToMany - Duplicate key Error

episode.xml

    <!-- Many To Many -->
    <many-to-many target-entity="App\Entity\AnimeTag" field="tags" inversed-by="episodes">
      <cascade>
        <cascade-persist />
      </cascade>
      <join-table name="episodes_tags">
        <join-columns>
          <join-column name="episode_uuid" referenced-column-name="uuid" />
        </join-columns>
        <inverse-join-columns>
          <join-column name="tag_uuid" referenced-column-name="uuid" />
        </inverse-join-columns>
      </join-table>
    </many-to-many>

fix.php

<?php
  /**
   * @throws Exception
   */
  public function __invoke() : void
  {
    // get parsing urls
    $batchSize = 25;
    /** @var ParseUrl $url */
    foreach ($urls as $url) {
      $this->parseUrlContent($url);
      if (0 === $this->totalParsed % $batchSize) {
        $this->parseUrlRepository->flushWithClear();
      }
    }
    $this->parseUrlRepository->flushWithClear();
  }

  /**
   * @param ParseUrl $url
   *
   * @throws EntityIsNotAnimeTagException
   * @throws Exception
   */
  private function parseUrlContent(ParseUrl $url) : void
  {
    $tagsCollection = $this->createTags($this->parseHtmlPage($url));
    $this->createEpisodeWithDetails($anime, $tagsCollection, $parsedPageData);
    $this->totalParsed++;
  }

  /**
   * @param Anime              $anime
   * @param AnimeTagCollection $animeTagCollection
   * @param array              $parsedPageData
   *
   * @throws EntityIsNotAnimeTagException
   * @throws Exception
   */
  private function createEpisodeWithDetails(
    Anime $anime,
    AnimeTagCollection $animeTagCollection,
    array $parsedPageData
  ) : void {
  // collect data from parsedPageData, required data to create entities
    try {
      $episode = $this->episodeRepository->single($episodeFilter);
    } catch (EpisodeNotFoundException $exception) {
      /** @var Episode $lastEpisode */
      $episode = $this->episodeFactory->create('some_data');
      if (false !== $this->getPersistedEntityFromUnitOfWork(Episode::class, 'name', $episode->name()->value())) {
        $episode = $this->getPersistedEntityFromUnitOfWork(Episode::class, 'slug', $episode->name()->value());
      }
    }

    foreach ($animeTagCollection as $tag) {
      $details->addTag($tag);
    }
    $this->episodeRepository->persist($episode);
  }

  /**
   * @param array $parsedPageData
   *
   * @return AnimeTagCollection
   * @throws EntityIsNotAnimeTagException
   * @throws Exception
   */
  private function createTags(array $parsedPageData) : AnimeTagCollection
  {
    $tags = ['some_tags', 1,2,3,4];
    $tagsCollection = new AnimeTagCollection();
    foreach ($tags as $tagData) {
      try {
        $tag = $this->animeTagRepository->single($tagFilter);
      } catch (AnimeTagNotFoundException $exception) {
        $tag = $this->animeTagFactory->create(\ucfirst($tagName));
        if (false !== $this->getPersistedEntityFromUnitOfWork(AnimeTag::class, 'slug', $slug)) {
          $tag = $this->getPersistedEntityFromUnitOfWork(AnimeTag::class, 'slug', $slug);
        }
        $this->animeTagRepository->persist($tag);
      }
      $tagsCollection->add($tag);
    }

    return $tagsCollection;
  }

  /**
   * @param string $class
   * @param string $methodName
   * @param string $value
   *
   * @return mixed
   */
  private function getPersistedEntityFromUnitOfWork(string $class, string $methodName, string $value)
  {
    foreach ($this->entityManager->getUnitOfWork()->getScheduledEntityInsertions() as $scheduledEntityInsertion) {
      if (
        $class !== \get_class($scheduledEntityInsertion) ||
        $value !== $scheduledEntityInsertion->$methodName()->value()
      ) {
        continue;
      }

      return $scheduledEntityInsertion;
    }

    return false;
  }

info.md

Error Description: during batch processing there are many entites which may have common data. In our example different parse url pages have same tag entities, but until we flush them to DB they have been created in exception handler every time and persiste once again. Sample code is very simplified. For exmaple Avatar movie has tag sci-fi and next one entitiy GameOfThornes also has sci-fi tag, during flush sci-fi tag flushed twice because was twice persisted. UnitOfWork, uses $this->entityIdentifiers[spl_object_hash($entity)]; for cached, persisted entites, that is why entity was persisted twice.

Salvation: if no record in DB, catch exception and create entity, but before persisting it, check if it exists in UnitOfWork as shown in example.

Additional info: https://stackoverflow.com/questions/24120435/doctrine-manytomany-on-same-entity-duplicate-entry-error https://stackoverflow.com/questions/25405144/not-inserting-duplicate-items-into-mysql-db-doctrine-and-symfony2 https://stackoverflow.com/questions/21712710/how-to-avoid-duplicate-entries-in-a-many-to-many-relationship-with-doctrine https://stackoverflow.com/questions/44649514/why-does-doctrine-try-to-duplicate-a-many-to-many-relationship-even-though-i-che

not_working.php

<?php

if (
false === $this->persistedTags->contains($tag) &&
true=== ( \Doctrine\ORM\UnitOfWork::STATE_MANAGED === $this->entityManager->getUnitOfWork()->getEntityState($tag)) &&
false === $this->entityManager->getUnitOfWork()->isScheduledForInsert($tag) &&
false === $this->entityManager->getUnitOfWork()->isInIdentityMap($tag)
) {
                          
// if we create custom cache we can not be sure UnitOfWork has entity
// if we do checks like STATE / isScheduledForInsert or isInIdentityMap, we can not be sure how many and what avlues are  repsisted

tag.xml

    <many-to-many target-entity="App\Entity\EpisodeDetails" field="episodes"
                  mapped-by="tags"
    />