Well, I once saw someone running full-length N scale passenger cars on 9.75". I'm not one of those people who think that 50' HO cars need 22" and SD40s need 30". In fact, if I knew 80' cars would run reliably on sharp curves, I might actually do it. I don't mind too much when I see Lionel equipment - on their bigger curves, anyway. I also don't mind too much seeing Marklin HO on 14" or Z on <6", because I know they can take it. I wouldn't go that tight myself. I'd fear for the survival of my equipment.
Your nolix looks like a helix, but with open track on some of it or most of it. The more you open up your helix and scenic it, the larger the layout will feel. The helix will have many levels to it. If I were building a nolix, I'd use the left side of your layout. From the bottom at 5 feet to the top at 14 feet and a bit gives you around 8 feet and a bit. If you did an oval there, you could have a run of around 20 feet per loop, so three loops would be 60 feet and that would give you 15 inches of seperation at 2 inches for every 8 feet of track. If you raised the rail heading into the nolix by an inch, you'd get a 16 inch seperation between decks. I'd then do the same as mine and have the first one hidden. The second on third loops going up would be visible. I have the third loop a bit inside the second loop, with a "river" in between them. The nolix area would be moutainous with perhaps two peaks one shorter than the other. On the second and third track I would have them visible except when they disappear behind the very back of the nolix.
Yeah, Rick... disisme seems to have a different idea of a nolix from what you do. As he said, it's now a helix/nolix combination, but look at his first plan. He is working on the assumption that the nolix part is once-around. Maybe he has to, given the very limited space for turnarounds. He's got a tight radius as it is. The helix hanging in the air is a bit wacky. I know there's got to be a better solution, but yours just won't work in HO. You can't have multiple concentric loops on a shelf accessible from one side. In N, it would be a great layout.
I wasn't sure if it was HO or N, he could make the benchwork wider on the left and narrower on the right. He could also add another deck underneath for staging, even in HO. In N there is tons of space, I agree in HO it is much much tighter.
Staging! I'd forgotten about it completely! Some sort of staging, or at least an interchange, to connect with the "rest of the world" would really help the plan.