ROBOTIC MUSICIAN SHIMON

Unit of Assessment (UoA): Mathematics & Statistics – Mathematics

   1. Summary of the impact 

A parsimonious high-order multivariate Markov chain model (HOMMCM) enabled the robotic musician Shimon, the first robot that composes and plays music using A.I. and deep learning, to simultaneously generate musical sequences for correlated features, such as pitch contour, note density, and rhythmic complexity plus look back at many notes in a timely manner. This parsimonious HOMMCM was adopted as the generative methodology for Shimon to create its own music and have real-time collaborations with human players for the Shimon and Friends concert series with a performance in more than 10 major cities in US, Europe, Asia and Australia.

   2. Underpinning research

Development of the Research

Markov chains models are widely used for studying categorical data sequences which occur frequently in many real-world applications. But conventional Markov chain models for s categorical data sequences of m possible states (e.g. data point) have O(mˢ)states. The number of parameters increases exponentially with respect to the value of s. This large number of parameters creates a major difficulty in using such a model directly. In 2001, a parsimonious multivariate Markov chain model was proposed [1] for capturing both the intra-and inter-transition probabilities among the sequences and the number of parameters of the model is only O(s²m²). Using our extended results of the classical Perron-Frobenius theorem for Markov chains, an efficient method based on Linear Programming (LP) was developed for estimating the model parameters.

For categorical data sequences, there are applications for high-order Markov chain models. The conventional model for an nth-order Markov chain of m possible states has O((m-1)mⁿ) parameters which grow exponentially with respect to the order n. This large number of parameters makes it infeasible to apply the model directly. In 2002, a parsimonious high-order Markov chain model was proposed [2] and the number of model parameters is only O(nm²). An efficient estimation method based on LP was developed for solving the model parameters. The model was then applied successfully to different types of data [2].

Following the success of the parsimonious high-dimensional Markov chain models in [1-2], the idea was further generalized to the case of the high-order multivariate Markov chain model. In a conventional nth order multivariate Markov chain model of s chains, and each chain has m possible states, the number of states is O(mⁿˢ) which grows exponentially in n and s. In 2006, a parsimonious HOMMCM was proposed [3] for multiple categorical data sequences whose number of parameters is O(ns²m²), which grows linearly in n. The proposed model requires significantly fewer parameters than the conventional one and makes real-time applications possible. Efficient estimation methods were developed for solving the model parameters and the classical Perron-Frobenius theorem for Markov chains was generalized to the setting of the parsimonious HOMMCM.

The parsimonious HOMMCMs are both effective (small number of parameters) and efficient (fast parameter estimation methods) in capturing the correlations among multiple sequences and their past states. These nice properties make the models excellent choices for real-time prediction in many complex real-world problems such as music generation in robotic musician Shimon, the first robot that composes and plays music using A.I. and Deep Learning [J].

Key Research Staff Involved

The key research staff for the project was Prof. Wai-Ki Ching and Prof. Michael K. Ng (experts in matrix computations and mathematical modelling). The key research findings resulted from the M. Phil. and Ph.D. studies of Eric S. Fung at HKU supervised by Ching and Ng. The model was then adopted by Mason Bretan (Georgia Institute of Technology) and his belonging robotic musician group for robotic musician Shimon.

   3. References to the research

1. [2002] Ching, W., Fung, E., & Ng, M. (2002). A multivariate Markov chain model for categorical data sequences and its applications in demand predictions. IMA Journal of Management Mathematics, 13, 187-199. https://doi.org/10.1093/imaman/13.3.187 [Citations in Google Scholar: 101,  by 8 Feb. 2019]

   2. [2004] Ching, W., Fung, E., & Ng, M. (2004). Higher-order Markov chain models for categorical data sequences. Naval Research Logistics, 51, 557-574. https://doi.org/10.1002/nav.20017 [Citations in Google Scholar: 69, by 8 Feb. 2019]

   3. [2008] Ching, W., Ng, M., & Fung, E. (2008). Higher-order multivariate Markov chains and their applications. Linear Algebra and Its Applications, 428 (2-3), 492-507. https://doi.org/10.1016/j.laa.2007.05.021 [Citations in Google Scholar: 92, by 8 Feb. 2019]

   4. Details of the impact

The impacts are behavior and practice: enhanced planning and joint optimization capabilities based on the research have resulted in superior algorithm design for robotic musicianship and automatic music generation by robotic musicians with consideration of morphological constraints in the decision-making process, higher-level musical semantics, minimized risk of collision and number of dropped notes. The beneficiaries are robotic musicians, especially Shimon, the first robot that composes and plays music using A.I. and Deep Learning with an anthropomorphic design to interact with players.

From research to impact

Mason Bretan, a PhD graduate in Music from Georgia Institute of Technology developed a generative jazz improvisation system [A] in his study period (2012-2017). The system, including the mathematical theory behind it, was designed and developed based off of the work by Ching in the period 2001-2008 at HKU [1-3], and subsequently adopted by Mason and his belonging robotic musician group, forming the basis of the jazz improvisation system for the Shimon and Friends concert series [pp.77-78, A]. Mason and Gil Weinberg’s Robotic Musicianship Group use the parsimonious HOMMCM to autonomously generate the semantic contours (including the semantic observation sequences of: pitch contour, note density, and rhythmic complexity, for every single set of contours) simultaneously in a timely manner [p.72, A].

The  ability  to  simultaneously  generate  the  musical sequences  for  all  features,  while taking into consideration physical constraints of the systems, in a timely manner, using parsimonious HOMMCM is crucial to robotic musicianship and Shimon’s embodied generative music. Firstly, the state-space is likely to vary considerably depending on the robotic platform. For platforms with enormous state-space, predicting the musical sequence is infeasible if the musical sequence is to be found in a timely manner [pp.70-71, A]. Secondly, the features are correlated in some way, without capturing these relationships in a multivariate method, these correlations would be lost [p.72, A]. Thirdly, a look back of many notes is required in order to effectively capture musical semantics; however, this is infeasible for extremely large state-space problems [p.114, A] but solved by HOMMCM.

Parsimonious HOMMCM offers robotic musician and Shimon higher level musical semantics, minimized risk of collision and number of dropped notes. Mason reported that the real robotic platform is now able to generate note sequence the robot is capable of performing [p.73, A]. The jazz improvisation system, designed and developed based on parsimonious HOMMCM, significantly reduces the number of parameters. For example, in a conventional 4th-order HOMMCM of 6 chains and 6 states, the number of parameters is 624= 4.74x1018 and it will reduce to 4x62x62=5.18x103 when using the parsimonious model. The most powerful super-computer, Sunway TaihuLight, can only conduct 9.3x1016 operations per second [C]. Thus it is not possible for Shimon to generate musical note sequences (and therefore public performances) without the help of the parsimonious HOMMCM if it is to be played in a real-time manner.

Nature and extent of the impact

The greatest impact of HOMMCM has been on the automatic music generation by a robotic musician at the semantic level. Traditional machine musicianship generates notes without considering physical constraints and often results in a collision and dropped notes [p.2, A]. Also, note-level methods of music generation, which is classical and widely used, may not be feasible for an integrated optimization method [p.10, A].

The parsimonious HOMMCM in [3] has given robotic musicians, especially Shimon, a single methodology for (i) simultaneously handling and generating sequences for correlated features, such as pitch contour, note density, and rhythmic complexity, physical configuration of robotic musician and the note that can be played in the configuration; (ii) looking back of many notes; (iii) reducing the number of parameters; and (iv) estimating the parameters in a timely manner.

The parsimonious HOMMCM serves as the generative methodology for Shimon to create its own music and generate all of its solos for the Shimon and Friends concert series [pp. 132, A]  performance in more than 10 major cities in US, Europe, Asia and Australia [B], and performance fee for a Shimon show is around $25K [D]. Shimon regularly amazes audience around the world and has performed with human musicians in dozens of concerts and festivals from Moogfest in North Carolina [I], to the 25th Anniversary Celebration of the Americans with Disabilities Act in Washington [E], to the Shanghai International Interactive Arts Festival, China [F], and Robotronica in Brisbane, Australia [G], with over 18,000 visitors enjoying its performance. The performances were also broadcasted through online channels [E,H,I], with a total of over 150,000 views.

   5. Sources to corroborate the impact 

[A]   [2017] [The Ph.D. Thesis shows the adoption of the parsimonious HOMMCM in Shimon] Mason Bretan (2017). Towards an embodied musical mind: Generative algorithms for robotic musicians, Ph.D. Thesis, School of Music, Georgia Institute of Technology, https://smartech.gatech.edu/handle/1853/58630

[B]    [2015] [The webpage for Shimon and friends showing the activities and performance of Shimon]  https://www.shimonrobot.com/

[C]    [2017] [Sunway TaihuLight]

https://www.top500.org/lists/2017/11/

[D]   [2018] [The performance fee of Shimon]

https://hkumath.hku.hk/~wkc/fee.pdf

[E]    [2017] [Robot Music - featuring Shimon, the robotic marimba player, the 25th Anniversary Celebration of the Americans with Disabilities Act in Washington. Over 57,000 views since 09/09/2015]

https://www.youtube.com/watch?v=l9OUbqWHOSk

[F]    [2015] [China Shanghai International Arts Festival: Shimon Robot & Friends: Musical Robots & Cyborgs from Room 100. The Organizer claimed the festival benefits 4 millions people 07/09/2015]

http://www.artsbird.com/ATP/Front/jyh/itemDetail.htm?programId=141&lang=en

[G]   [2015] [Shimon Robot & Friends - Musical Robots and Cyborgs, Brisbane, Australia. Over 18,000 visitors in 2015]

http://www.robotronica.qut.edu.au/performances/shimon.php

[H]   [2017] [The robot Shimon composes and performs his first deep learning driven piece. Over 65,000 views since 03/05/2017]

https://www.youtube.com/watch?v=j82nYLOnKtM

[I]     [2016] [Shimon the robot jamming with a human marimba player at Moogfest, North Carolina. More than 10,000 attendants in the event. Over 36,000 views since 22/05/2016]

https://www.youtube.com/watch?v=U7jf3MYL3Xg

[J]     [2017] [Shimon, the first robot that composes and plays music using A.I. and Deep Learning]  

https://gadgets.ndtv.com/science/news/shimon-robot-music-ai-compose-play-1714230