A rasterization like method might be a smarter approach. Break the scene down into tetrahedrons and rasterize them one at a time onto a 3D pixel grid. Originally I dismissed this idea because I was thinking of rasterizing them into vector like 3D graphics to then be rendered again by the final regular camera, because such a method would involve costly boolean operations.
However if it's done as a true analogue to rasterization onto a 2D screen, then that'd be avoided. The 512x512x512 or so grid would have a color value and a depth value for each voxel. When rasterizing a new tetrahedron onto the grid, it'd do a depth check per voxel and only change the color if the new object is in front of the old object.
This method would be relatively fast for simple scenes and I imagine the resolution would be limited more by memory than processing power.
A hybrid approach could potentially work even better. Drop the grid's resolution compared to the previous method and don't record color per pixel. Instead record object ID per pixel. Then only when the final 2D image of the 3D image is being calculated would the textures and lighting be calculated. That way it's a per pixel effect on the final TV image and the initial grid's resolution would only matter for depth testing. Changing its resolution would only affect the borders between the 3D images of the 4D objects which while important, isn't that important.
This hypothetical 4D game is coming along nicely!
On paper it does seem like something I could potentially manage, so maybe I should figure out the 4D gameplay possibilities? Would this just be a little sandbox for the player to explore and observe a truly 4D world, or could it offer fun gameplay that isn't just mimicking what players are already used to?
The above example about a Doom style shooter would be almost like a isometric top down game in its basic form. Without motion it'd literally feel like playing a game where you're a satellite or something moving to chase people down. The world looping back around on itself like if it was a super small planet wouldn't really bring much to the table either. With motion and a maze like setting it could be really weird. Occlusion would probably become the big game element as it'd look like enemies disappearing into other objects. You would have to move to make them reappear again. Maybe some sort of x ray vision might help with this to stop it from being too confusing.
From a mechanics point of view, on the 3D image of the 4D scene you'd see the shape representing the cover/wall and the sphere representing the enemy moving towards that first shape. When it gets there if it is closer to you (is less dense with its texture than the other shape's texture) then it cuts into the other shape and stays visible. However if it is farther away from you in 4D, then the other shape cuts into it and the sphere stops being visible once it's fully inside. To make it come outside and thus become visible again, you'd push the left joystick in the opposite direction of where the wall shape is compared to the center of the 3D image. This would move the camera in 4D which would result in pushing everything on the 3D image screen in the opposite direction of the joystick. However, the 3D representations of 4D objects would move faster the closer they are to the camera, so the wall obstructing the enemy would have its 3D image move out of the way revealing the image of the enemy again. You'd also want to push the right joystick in the opposite direction so that the enemy and wall's images stay centered on the 3D screen.
So that gameplay wouldn't be too bad. Sure it'd be a simple 4D shooter but it'd be a good way to push the player to move about the world.
How about a race game? The track would be 3D in the 4D world. If we have the same camera setup as before with the infinite ground and leveled camera, that'd look like the 3D image box with the top half sky color, the bottom half ground color, and a pyramid rising from the bottom through the ground color to the exact center, assuming the camera was aligned with the track. Going forward more or less means expanding the 3D image as if you were zooming in on it. The 4D car could turn left/right and 4D +/- and so the racetrack could as well. In the 3D image the track turning left would look like the pyramid twisting its top left and then extending it off the image. Turning the camera/car left would just drag the whole 3D image right (some distortion but not based off 4D distance). Keeping the car on the road would just mean keeping the center bottom of the 3D image within the base of the road's pyramid. Doesn't sound that fun to me and would be pretty comparable to just flying in a 3D game. Sure you could spice it up with visuals along the track, other cars, and maybe even some jumps, but that wouldn't have a large impact on how it actually plays.
What about a 4D platformer? Again using the example scene of infinite flat ground, but this time with a pit right in front of the camera. The 3D image of the 4D scene would be the color of sky for the top half and color of ground for the bottom half, but within the ground would be an image of a shape for the pit. It'd be kinda hard to tell it's a pit and not some other object on the ground unless it was fully shaded in. If the player jumps, the entire 3D image moves down with closer objects moving faster, and then moves back up as the player lands. Moving forward makes the whole 3D image expand from the center with closer objects expanding faster. So if you moved forward without jumping, the image of the pit would just move down and off the screen as you reach it. If you jump and move forward, the pit also moves down off the screen but doesn't return when you land. A player avatar doesn't really simplify this since without shadows and a solid understanding of 4D depth, it's just as abstract. IE the 3D image of the player could be right next to the 3D image of the pit yet in the 4D space could be very far apart.
Also a 4D platformer would be really hard to design. In 2D, a hole/death zone must be jumped over no matter what since by default the hole blocks player progression. In 3D, the player can walk around the hole. This is why 3D platformers can't really do holes and instead focus on the platform itself being a safe space. In 4D it'd be even more extreme where you essentially have to make everywhere a kill zone and just a few dots of safe space. Would be very hard to design without it just feeling like jumping from one safe space to another. Plus it wouldn't feel that unique? Sure it'd require setting up the correct camera angle, but once that's done it'd just be a linear jump.
So a simple shooting game seems like it might actually be the best option. First level could just be a big empty plane, while additional levels would put more and more around you until you're inside a 4D building.
However if it's done as a true analogue to rasterization onto a 2D screen, then that'd be avoided. The 512x512x512 or so grid would have a color value and a depth value for each voxel. When rasterizing a new tetrahedron onto the grid, it'd do a depth check per voxel and only change the color if the new object is in front of the old object.
This method would be relatively fast for simple scenes and I imagine the resolution would be limited more by memory than processing power.
A hybrid approach could potentially work even better. Drop the grid's resolution compared to the previous method and don't record color per pixel. Instead record object ID per pixel. Then only when the final 2D image of the 3D image is being calculated would the textures and lighting be calculated. That way it's a per pixel effect on the final TV image and the initial grid's resolution would only matter for depth testing. Changing its resolution would only affect the borders between the 3D images of the 4D objects which while important, isn't that important.
This hypothetical 4D game is coming along nicely!
On paper it does seem like something I could potentially manage, so maybe I should figure out the 4D gameplay possibilities? Would this just be a little sandbox for the player to explore and observe a truly 4D world, or could it offer fun gameplay that isn't just mimicking what players are already used to?
The above example about a Doom style shooter would be almost like a isometric top down game in its basic form. Without motion it'd literally feel like playing a game where you're a satellite or something moving to chase people down. The world looping back around on itself like if it was a super small planet wouldn't really bring much to the table either. With motion and a maze like setting it could be really weird. Occlusion would probably become the big game element as it'd look like enemies disappearing into other objects. You would have to move to make them reappear again. Maybe some sort of x ray vision might help with this to stop it from being too confusing.
From a mechanics point of view, on the 3D image of the 4D scene you'd see the shape representing the cover/wall and the sphere representing the enemy moving towards that first shape. When it gets there if it is closer to you (is less dense with its texture than the other shape's texture) then it cuts into the other shape and stays visible. However if it is farther away from you in 4D, then the other shape cuts into it and the sphere stops being visible once it's fully inside. To make it come outside and thus become visible again, you'd push the left joystick in the opposite direction of where the wall shape is compared to the center of the 3D image. This would move the camera in 4D which would result in pushing everything on the 3D image screen in the opposite direction of the joystick. However, the 3D representations of 4D objects would move faster the closer they are to the camera, so the wall obstructing the enemy would have its 3D image move out of the way revealing the image of the enemy again. You'd also want to push the right joystick in the opposite direction so that the enemy and wall's images stay centered on the 3D screen.
So that gameplay wouldn't be too bad. Sure it'd be a simple 4D shooter but it'd be a good way to push the player to move about the world.
How about a race game? The track would be 3D in the 4D world. If we have the same camera setup as before with the infinite ground and leveled camera, that'd look like the 3D image box with the top half sky color, the bottom half ground color, and a pyramid rising from the bottom through the ground color to the exact center, assuming the camera was aligned with the track. Going forward more or less means expanding the 3D image as if you were zooming in on it. The 4D car could turn left/right and 4D +/- and so the racetrack could as well. In the 3D image the track turning left would look like the pyramid twisting its top left and then extending it off the image. Turning the camera/car left would just drag the whole 3D image right (some distortion but not based off 4D distance). Keeping the car on the road would just mean keeping the center bottom of the 3D image within the base of the road's pyramid. Doesn't sound that fun to me and would be pretty comparable to just flying in a 3D game. Sure you could spice it up with visuals along the track, other cars, and maybe even some jumps, but that wouldn't have a large impact on how it actually plays.
What about a 4D platformer? Again using the example scene of infinite flat ground, but this time with a pit right in front of the camera. The 3D image of the 4D scene would be the color of sky for the top half and color of ground for the bottom half, but within the ground would be an image of a shape for the pit. It'd be kinda hard to tell it's a pit and not some other object on the ground unless it was fully shaded in. If the player jumps, the entire 3D image moves down with closer objects moving faster, and then moves back up as the player lands. Moving forward makes the whole 3D image expand from the center with closer objects expanding faster. So if you moved forward without jumping, the image of the pit would just move down and off the screen as you reach it. If you jump and move forward, the pit also moves down off the screen but doesn't return when you land. A player avatar doesn't really simplify this since without shadows and a solid understanding of 4D depth, it's just as abstract. IE the 3D image of the player could be right next to the 3D image of the pit yet in the 4D space could be very far apart.
Also a 4D platformer would be really hard to design. In 2D, a hole/death zone must be jumped over no matter what since by default the hole blocks player progression. In 3D, the player can walk around the hole. This is why 3D platformers can't really do holes and instead focus on the platform itself being a safe space. In 4D it'd be even more extreme where you essentially have to make everywhere a kill zone and just a few dots of safe space. Would be very hard to design without it just feeling like jumping from one safe space to another. Plus it wouldn't feel that unique? Sure it'd require setting up the correct camera angle, but once that's done it'd just be a linear jump.
So a simple shooting game seems like it might actually be the best option. First level could just be a big empty plane, while additional levels would put more and more around you until you're inside a 4D building.
