Musical instruments and video games have more in common than you may expect. Ultimately, both define a possibility space. In the case of a game, this can be an environment for a player avatar to navigate, while the possibility space of a musical instrument is its notes, range, timbre, and ornamental capabilities. In both cases there are a limited set of possibilities. Musicians are simply exploring the constraints of a system in a similar way to someone playing a game.
Given this relationship, one may question whether it is possible to teach someone to play a musical instrument by presenting it as a game. In some regards, I think the answer to this question is yes. Many games present their player with a mechanical challenge, demanding precise button inputs. At first, the player will be bad at the task but, over time, these things are get easier.
A good example of a game like this is Pilotwings Resort, a flight simulator that challenges the player to fly an aircraft, following guide dots and rings in the environment, scoring you on accuracy.
The process of learning to play this game is resembles a beginner learning the fingerings for a song on an instrument. Initially, hitting all of the guides is extremely difficult, just like fingering notes feels hard at first. But, over time we learn the controls of the game, or the fingerings of an instrument, and the errors reduce.
However, just knowing the notes is not enough. There are many other details like ornamentation and articulation that go into playing well, which can be called 'musicality'. While it may be possible to introduce the basics of this in a game, perhaps through a 'style' system awarding extra points, there is a problem. That is, music is a subjective art.
Such a game would have to make a value judgement of the player. This is pretty much impossible with music as what is considered 'good' is to a large extent a matter of taste. Effectively, 'good music' is a voting system based on the subjective experience of listeners. If enough people think it's good, then it's good. Tastes in music also change over time.
Consequently, while I think a game can teach the mechanical techniques that go into playing music, I don't believe that they can universally teach musicality. With this in mind, I have some ideas regarding how this could be applied.
Before someone can learn music, they need to develop an interest in it. I think this is one way in which games featuring music can be beneficial. There are numerous games which feature rhythm, and the well known Legend of Zelda: Ocarina of Time does a similar thing with melody. However, a general problem with the examples that currently exist is they either focus on only a single aspect of music, trivialise the task, or both. OoT focuses only on playing the notes, ignoring rhythm, ornamentation, and the breath control required to play a real ocarina.
I would really like to see games of this nature evolve to teach music in a less constrained way, at least covering both rhythm and melody. One way to approach that would be to introduce the player to rhythm and pitch separately at first, then combine them later in the game. Working at the basic level that would be needed to appeal to a general audience, I don't think that 'music is subjective' would be a blocker in this case.
A game could also analyse what the player is doing, and provide feedback which makes mistakes obvious and intuitive. An example of how that may be done is to listen to the player and play a note in unison when they are in tune—essentially a simulation of sympathetic resonance. This could also be applied to other things, such as rhythm.
Something that games definitely can help with in teaching music are technical practice tools, things like 'clap the shown rhythm to a metronome', 'identify the following intervals', or 'play the following melody'. Many things like this already exist, but all of the ones I have experience with are so dry to be fatiguing to use. This isn't a good thing, as it discourages a learner from using them.
I'm unsure exactly how this could be solved, but something simple like varying between exercise types could help. When learning organically, it is uncommon to focus so tightly on only one thing. The game The Witness could be a source of inspiration: it's mainly about solving line puzzles, which by itself would also be dry. However, the game mixes this up a lot, with the art in the environment providing a respite.
A game can introduce a player to new music, and help them practice it by breaking it down for them, or gradually increasing the tempo over time. A game could also score a player on technical accuracy of a performance, using something like a MIDI keyboard and scoring the player similarly to Pilotwings as mentioned above. The exercises could be created by a human musician and be constrained to a single genre, thus could avoid the 'subjective art' issue to some extent. This, however, would still present a very constrained point of view.
I expect that this sort of automated tutorial has limited value beyond the absolute basics, as even with crafted excises, it will still be far more limited than just studying a bunch of performances.