A "Pokemon" - including the species, level, battle stats, moves, etc - are represented within the game as a "struct" or "object". In a programming language, that would look like this:
struct Pokemon {
Species: u32, // This is a 32-bit integer that can't be negative
Nickname: String,
Level: u8,
IvAttack: u8,
// IVs...
EvAttack: u8,
// EVs...
// rest of the properties of a Pokemon...
}While the game is running, that object is stored in binary form and sits in the computer (in this case, the game console) memory. But it can be "serialized" into a well-known format in order to be transfered elsewhere. Think of serialization like the act of writing something. I'm having thoughts in my mind right now, and I'm serializing them into a a well-known format (in this case, English text) so that others who know how to read English can read it. When programs want to transfer things to other programs, they usually do so in a similar way, using an intermediate form analagous to English.
Hold that thought, we'll revisit it in a moment.
As of today, we know the following transfers are possible. Arrows are significant and show the directionality of transfer that is possible.
1. Home <--- Go (Mobile)
2. Home <--- Bank (3DS)
3. Home <--- Let's Go Pikachu/Eevee (Switch)
4. Home <--> Sword/Shield (Switch)
Let's elaborate on #4 - bidirectional transfer between Home and Sword/Shield. Watching videos on this transfer being performed online (as I have neither game), the transfer works like this:
- User launches the game Pokemon Home
- User selects another Pokemon game that has save data stored on the currently running console (in this case Sword/Shield)
- Home directly reads/writes the save data of that game in order to transfer the Pokemon to/from that game and Home
Going back to our point about serialization before - there is no "intermediate" representation of a Pokemon that both Home and Sword/Shield agree to in this case. Instead, Home in essence "uses Dream Eater" on the Sword/Shield save file; instead of two people communicating via English, this would be akin to one of the people (ie, Home) reading the other person's (Sword/Shield) mind and sending thoughts to them telepathically. In this analogy, only Home has telepathic powers; Sword/Shield do not, and are not even aware of Home using telepathic powers on them.
As for the topic of whether or not Home and the remakes of Diamond/Pearl (I'll call them BDSP from here on out) can have compatibility:
If the developers opt for the "Dream Eater" method instead of the "English" method: Home will need an update to read BDSP's mind. BDSP requires no action.
In order for the "English" method to be possble: BDSP will need an update to communicate with Home's cloud server. Whether or not this is feasible depends on how the Home game communicates with the Home server, as BDSP would also be communicating with the Home server. In my opinion, this would be a cross-development team effort that would take lots of collaboration and testing, and would probably be too much extra logic to include in the BDSP games. BDSP are not unique in this respect, and this would all apply to any past Pokemon game wanting to go down this route. In summary, the development overhead of this solution probably precludes it from happening.
With the "English" method excluded for the above reasons, it's clear why the "Dream Eater" method is chosen - the BDSP dev team doesn't have to be involved, and the Home dev team just reads their save file spec and adds logic to Home to manipulate it.
That brings us to the save file spec itself. I'd seen on Twitter someone talking about BDSP being written in C# (Unity) while Home was written in C++ (Game Freak internal toolkit), and that the BDSP save file was a C# object dump. It's hard to confirm whether or not this is true without some more research, but let's take it at face value for now.
In theory, adding support to Home for reading BDSP's save file should be possible. Assuming past Pokemon games were not written in Unity and instead used the Game Freak internal toolkit, this was probably not that hard to implement as there was shared code that could be reused. With BDSP, the object format would be quite different and a library to parse and manipulate it would need to be added to Home. This is far from impossible, but it would not be trivial. In the worst case, the Home devs would need to write a BDSP save file editor from scratch, assuming they can't reuse anything or interoperate with C++ at all. I kind of find this unlikely, as here's a big piece of info we've left out so far: Unity is a C++ engine, and C# is only used for scripting (ie, automating) it. There's likely nothing C#-specific going on in the BDSP save file that would hold Home back from being able to manipulate it.
In summary, the BDSP save file is probably just different enough of a change that it's taking the Home devs a long time to implement "Dream Eater" properly, but it's not impossible.
I may update this at some point with extra info about properties that may be present on BDSP Pokemon that Home does not know about. Think of RBY vs GSC, where there were new moves introduced, so a Charizard with Hidden Power (for example) couldn't be transferred from Silver -> Blue.