Cinema + Machine Learning
Artificial intelligence (AI) made its first appearance in cinema with Fritz Lang’s German silent movie “Metropolis”, released in 1927 (Tomlinson, 2018). It is regarded as one of the greatest and most influential films of all time, ranking in the Top 100 Sight & Sound's critics' poll of the British Film Institute (The 100 Greatest Films of All Time, 2021). Since then, AI has been a recurring topic in many more science fiction films and television series. Although its presence is well known on screen, is it as present behind the scenes? In recent years, Machine Learning (ML) has been used in the cinema industry to help and accelerate certain film production processes.
ML's impact on film industry
Film Distribution
Before the creation of a movie, film producers need to be strategic and decide on which movie is worth producing. Such investment is a big financial risk and with the success of a script being uncertain, it puts even more importance on the decision process. This decision process can take days or weeks of research and deliberation (Chow, 2020) and so, many researchers have seen this as an opportunity to explore prediction models in machine learning to reduce the time delay. According to Lee et al. (2018), researchers in the past (e.g., Du et al., 2014; Sharda & Delen, 2006; Zhang et al., 2009) have put their focus on the objective of introducing new machine learning algorithms and testing their performances, which proved to result in high-level prediction accuracy. However, Lee et al. (2018) argue that “their efforts to improve the models’ prediction power have been limited only to the modification of the algorithms, rather than finding meaningful features that might be critical to anticipate the success” (p. 577). Their own research focused on building an ensemble model with one new feature added, and using seven machine learning algorithms: adaptive tree boosting, gradient tree boosting, linear discriminant, logistic regression, neural networks, random forests, and support vector classifier (Lee et al., 2018). In the end, their model has predicted movie success with an accuracy of 58.5%, a higher percentage than previous studies (Lee et al., 2018). As of 2020, the two most high-profile predictive AI tools to have been launched are Scriptbook and Cinelytic (Chow, 2020).
AI tools can have a positive and negative impact on both the cultural and industrial facets of cinema according to Chow (2020). From an optimistic perspective, the prediction model allows the producers to save time on the low-level tasks required to assess a movie's potential (e.g., research and gathering data). In this way there are no important decisions made according to subjective assessments or someone’s limited experience, and although the tool provides a more accurate prediction about the performance of a film, the final decision is still up to the human team.
On the other hand, the negative aspects of AI tools have are more impactful. First, an automated decision-making process can be a threat to creativity and diversity. New and creative screenplays that would be assessed as high financial risks could be overlooked. If the datasets being used to train the algorithms are biased, this could be detrimental to current efforts aimed at dismantling structural inequalities in film. As Chow (2020) explains, “stories and character relations that deviate from a predominantly white, conservative, heteronormative patriarchal worldview would be assigned a low probability of success based on historical trends” (p. 203). If screenplays follow templates of successful films of the past, the films that are funded will become homogenized, much to the detriment of viewers and film culture.
Film Production
The impact of machine learning doesn’t stop at the distribution of film. In 2016, the companies 20th Century Fox and IBM Research developed together the very first movie trailer with AI (Smith, 2016) for a horror movie. They used various machine learning techniques including audio visual segmentation, audio emotion recognition, audio and visual sentiment analysis and they built an audio-visual and a statistical model (Smith et al., 2017). Additionnally, in a recent research three American animation studios have successfully improved the animation and visual effects of three movies using a new machine learning denoising render during the production process (see Figure 1) (Dahlberg et al., 2019).
Figure 1: Production examples from Toy Story 4 (left), Ralph Breaks the Internet (middle) and Bumblebee (right). Image taken from Machine-learning Denoising in Feature Film Production (Dahlberg et al., 2019).
