The fly in this soup is, that switching the segmentation registers is actually far more expensive than changing CR3, so an operating system written this way would be quite slow.
This is why the segment registers are rigged in every modern OS to just be the entire valid address space. In fact, on amd64, CS/SS/DS/ES are all hard-coded to be this value. FS and GS are usually abused to do other tricky things - on Windows NT, FS is used as a pointer to the current thread.