Teaching musical instruments with video games

Musical instruments and video games have more in common than you may expect. Ultimately, both define a possibility space. In the case of a game, this can be an environment for a player avatar to navigate, while the possibility space of a musical instrument is its notes, range, timbre, and ornamental capabilities. In both cases there are a limited set of possibilities. Musicians are simply exploring the constraints of a system in a similar way to someone playing a game.

Given this relationship, one may question whether it is possible to teach someone to play a musical instrument by presenting it as a game. In some regards, I think the answer to this question is yes. Many games present their player with a mechanical challenge, demanding precise button inputs. At first, the player will be bad at the task but, over time, these things are automated by the subconscious—a clear example being Pilotwings Resort, a flight simulator that challenges the player to fly an aircraft, following guide dots in the environment.

This process somewhat resembles a beginner learning the fingerings for a song on an instrument. Initially, this is extremely difficult. The finger movements are alien to anything done in day to day life. But, over time, these tasks are automated by the subconscious and the errors reduce. However, just knowing the notes is not enough. There are many other details like ornamentation and variations in articulation that go into playing well, which can be called 'musicality'.

While it may be possible to introduce the basics of this in a game, perhaps through a 'style' system awarding extra points, there is a problem. That is, music is a subjective art. This is a big difference from many educational 'sandbox' games, as most are based on hard rules. Kerbal Space Program is a good example of this. The game teaches space flight intuitively, through experimentation and failure, ultimately directing the player towards effective solutions.

Such a system requires the game to make a value judgement of the player. This is pretty much impossible with music as what is considered 'good' is to a large extent a matter of taste. Effectively, 'good music' is a voting system based on the subjective experience of listeners. If enough people think it's good, then it's good. However, tastes have changed over time and will keep changing. Even at a single time, different people find different things enjoyable, and that isn't going to change.

Consequently, while I think a game can teach the mechanical techniques that go into playing music, I don't believe that they can universally teach musicality. Any attempt to do so will always be a poor representation of reality. With this in mind, I have some ideas regarding how this could be applied.

Practical applications

Before someone can learn music, they need to develop an interest in it. I think this is one way in which games featuring music can be beneficial. There are numerous games which feature rhythm, and the well known Legend of Zelda: Ocarina of Time does a similar thing with melody. However, a general problem with the examples that currently exist is they either focus on only a single aspect of music, trivialise the task, or both. OoT focuses only on playing the notes, ignoring rhythm, ornamentation, and the breath control required to play a real ocarina.

It would be good to see something like this which targets music in a much less constrained way, at least covering both rhythm and melody simultaneously. One way to do this is iterative learning, introducing the player to rhythm and pitch separately at first, then combining them later in the game. Working at the basic level that would be needed to be understandable to a general audience, I don't think 'music is subjective' would be a blocker in this case.

Something that games definitely can help with in teaching music are technical practice tools, things like 'clap the shown rhythm to a metronome', 'identify the following intervals', or 'play the following melody'. Many things like this already exist, but all of the ones I have experience with are so dry to be fatiguing to use. This isn't a good thing, as it discourages a learner from using them.

I'm unsure exactly how this could be solved, but something simple like varying between exercise types could help. When learning organically, it is uncommon to focus so tightly on only one thing. The game The Witness could be a source of inspiration: it's mainly about solving line puzzles, which by itself would also be dry. However, the game mixes this up a lot, with the art in the environment providing a respite.

It would be possible to have a game score a player on technical accuracy of a performance, using something like a MIDI keyboard and scoring the player similarly to Pilotwings as mentioned above. The exercises could be created by a human musician and be constrained to a single genre, thus could avoid the 'subjective art' issue to some extent. This, however, would still present a very constrained point of view.

I expect that this sort of automated tutorial has limited value beyond the absolute basics, as even with crafted excises, it will still be far more limited than just studying a bunch of performances. Another way of approaching this problem is less 'game' and more 'teaching aid'. Musical instruments have no opinion of how they should be played. This is a good thing, as it enables a skilled player to fully explore the possibilities if the instrument. Yet it creates traps, as many of the essential techniques are not intuitive.

Game designers go out of their way to make sure that this never happens, and it would be possible to offer some of those benefits while staying 'hands off' of what the player wants to play. The idea would be to analyse what the player is doing, and provide feedback which makes mistakes obvious and intuitive. An example of how that may be done is to listen to the player and play a note in unison when they are in tune—essentially a simulation of sympathetic resonance. This could also be applied to other things, such as rhythm.

Thoughts on...

Article Headings