That sounds really cool!I guess a more a more updated/accurate answer here would be:
How scratch are you talking? Open source libraries when possible or as scratch as possible to help you learn?
Graphics: using the SDL library to work with OpenGL.
But I still think everything else will be pretty from scratch.
SDL is a lot better than GLUT, which is super outdated.
Edit: ive made more progress in half an hour with SDL than I did over a day with GLUT