Visualization in music / sound collections: an introduction
Existing library and archival catalogs base their functionality and interface design on a metadata-centered approach: textual descriptions of intellectual works and of their containers are used for querying and indexing purposes.
In a music or audio collection, there is plenty of metadata to be used for describing and querying sources. These metadata can be the name of a composer, the title of a work, the instrument(s) of the composition, the date of creation, details of item publication or distribution, tonality, and other characteristics. This information is already available in library or archival catalogs and provides a significant gateway to knowledge. However, today we have the privilege of having additional and more advanced options, that enable access to the content itself beyond descriptions of works and their publications: in other words, an interest in content retrieval is gradually developing and the distinction between data and metadata is growing larger. Content-based retrieval as an "important retrieval method for multimedia data, which uses the low-level features (automatically) extracted from the data as the indexes to match with queries" [1] is an important element of this growing interest in accessing the content itself.
In music and sound, the field of music information retrieval (MIR) has a significant role to play. MIR analyses musical content (symbolic or audio), and develops algorithms for music feature extraction and music classification. These approaches provide mechanisms that enable automatic description of content and they can be used for enhancing music searching experiences. Other benefits of MIR include automatic music transcription (i.e. generating symbolic notation out of audio), score following (synchronization between scores and audio), instrument recognition, similarity research / pattern matching, and others.
At the same time, metadata do not have the answer for all searching needs. There are users, who don't look for specific musical works, therefore information about the "composer" or "title" is not helpful in these cases. Instead, there may be queries on work-agnostic, morphological or compositional characteristics, like the use of modes, or the exploitation and repetition of musical motives. Moreover, there may be users who are looking for works with a given time signature or a large variety in tempo markings. The possible queries can be actually endless. In May / June 2018 I conducted a short survey with German and American music librarians, aiming at collecting such types of questions. Here a small selection of them:
"I want to retrieve string quartets with a strong contrapuntal texture"
"I want a work for clarinet, harder than the Brahms sonatas but easier than Stravinsky’s three pieces for clarinet solo"
"I need some dense music"
"I am looking for light, easy-going works that don’t last longer than 4 minutes"
"I need music excerpts of high energy to accompany an intense and active video scene"
"I need something pensive / metallic / angsty / etc."
It is obvious that such questions cannot be answered by existing metadata, and there is need for content information that should be able to help answer them. Such information could include compositional characteristics, sound properties, or psychoacoustic attributes.
Parallel to the developments in information retrieval, there are visualization techniques that could also fulfill many of the aforementioned needs. As information visualization is being established and is gaining in importance, its contribution to exploring, analyzing and describing complex systems is growing larger. New insights are born and music material can benefit from this progress.
An interdisciplinary approach of information visualization and music information retrieval can provide a state of the art solution for generating new interfaces, that could facilitate browsing of music / audio collections and could help answer complex questions, like the ones described above.
References
[1] "What Is Content-Based Retrieval | IGI Global." (n.d.) Accessed February 17, 2021. https://www.igi-global.com/dictionary/content-based-retrieval/5591.
