This gist is a reconstruction of a failing test in CakePHP version 4.0.5. This test passes in 4.0.4

TestCommand.php

<?php
declare(strict_types=1);

namespace App\Command;

use Cake\Console\Arguments;
use Cake\Command\Command;
use Cake\Console\ConsoleIo;

/**
 * Test command.
 */
class TestCommand extends Command
{
    /**
     * When performing "bin/cake test first" it will normal output "First".
     * When performing "bin/cake test second" it will error output "Second".
     *
     * @param \Cake\Console\Arguments $args The command arguments.
     * @param \Cake\Console\ConsoleIo $io The console io
     * @return null|void|int The exit code or null for success
     */
    public function execute(Arguments $args, ConsoleIo $io)
    {
        $inc = $args->getArgumentAt(0);
        if ($inc === "first") {
            $io->out('First');
            return Command::CODE_SUCCESS;
        } elseif ($inc === 'second') {
            $io->err('Second');
            return Command::CODE_ERROR;
        }
    }
}

TestCommandTest.php

<?php
declare(strict_types=1);

namespace App\Test\TestCase\Command;

use Cake\TestSuite\ConsoleIntegrationTestTrait;
use Cake\TestSuite\TestCase;

/**
 * App\Command\TestCommand Test Case
 *
 * @uses \App\Command\TestCommand
 */
class TestCommandTest extends TestCase
{
    use ConsoleIntegrationTestTrait;

    /**
     * setUp method
     *
     * @return void
     */
    public function setUp(): void
    {
        parent::setUp();
        $this->useCommandRunner();
    }

    /**
     * Test execute method
     *
     * This test method will fail as the assertOutputEmpty is not empty. It contains the output from the first execution
     *
     * @return void
     */
    public function testExecute(): void
    {
        // First input - Prints "first" to normal output. No error output
        $this->exec('test first');
        $this->assertErrorEmpty();
        $this->assertOutputContains('First');

        // Second input - Prints "second" to error output. No normal output
        $this->exec('test second');
        $this->assertOutputEmpty(); // FAILS: Output is not empty. Actual output: "First". Seems it stores output from previous execution
        $this->assertErrorContains('Second');
    }
}

Commit that causes issue: cakephp/cakephp@d2694e2

Previously it created a new instance of ConsoleOutput no matter what. Now it only does it the first execution. The following executions it retains the ConsoleOutput that exists.

Yes, but you are not using it as intended. See ticket in core.