What does the economy look like? When we think about financial markets, what do we see? Scrolling ticker tapes, price graphs, and tables with numbers? Demonstrating activists and police in riot gear? Skyscrapers and men in suits?

In the M-Scopes project, we delve into the theme of visualization and multimodal signification of finance and economy in the news media. In this first blog entry, we start to make sense of the complex field by introducing some key encounters.

Where language, image and other resources meet meaning and perception

Multimodal or multisemiotic is, simply put, any presentation or act of communication – printed page, speech, video, interactive infograph etc. – that draws from two or more semiotic resources, for example language and visual imagery. Following this definition, practically all communication is multimodal. In fact, even language itself can be considered inherently multimodal. It realizes either as writing, bringing along dimensions of typography and layout, or as speech, inextricable from prosody, intonation, gesture, and so forth.

In a single act of communication, semiotic resources can have manifold roles. For example, text may guide and constrain the interpretation of a picture or graph – or vice versa. Picture and text may work as equal components, which together produce “a revelation” (Olson 1994). A picture may express something that the language cannot (yet) convey: this was the case with anatomical and botanical illustrations before the development of specialized technical language and terminology within these fields (see Matthiessen 2006).

Scholars in multimodal discourse analysis say that where two or more semiotic systems meet, an expansion of meaning takes place. Such semantic expansion is based on differences in the perception and processing of semiotic systems, and on their specialized communicative potentials or semantic orientations. In visual imagery, the whole is perceived before individual components; a linguistic text, by contrast, typically unfolds as a chain and is processed sequentially. In addition, different semiotic resources are specialized in expressing different types of meanings, such as category difference or difference of degree (see Lemke 1998, O’Halloran 2008).

The division of semantic labour between different semiotic systems varies according to genre, situation, and communicative purpose. It has also changed over time. What is intriguing is that sometimes these changes have profound implications for ways of reasoning. Olson (1994) records how this happened in the 11^th and 12^th centuries, when illustration as a key signifying resource in multimodal (biblical) texts started to lose ground in relation to written text – “which came, increasingly, to be seen as the primary conveyor of meaning. And with the emphasis on text came (…) the greater concern with the logical form of a text, with single, clear lines of argument and universal unambiguous meanings of terms. This linearity was also reflected in the development of a single linear perspective in art.”

Semiotic systems are also continuously shaped through interaction with other systems, i.e. through intersemiosis. When statistical methods and graphs gained ground, language was forced to find ways to express new kinds of relations, processes and entities, pushing new vocabulary and, also, new grammar. It has been argued, for example, that the very familiar grammatical phenomenon of ”nominalization”, where a process (‘to move’) is expressed as a noun (‘movement’), was motivated by the need to express meanings of mathematical symbolism and scientific imagery through language (O’Halloran 2008).

Where multimodal signification meets the economy

The M-Scopes project focuses on the (multisemiotic) signification of economic phenomena and mechanisms in the news. How do the news media deploy different semiotic systems in creating accounts and narratives of finance and economy? How do the new multimodal communication tools enable and influence the signification of financial and economic data?

The motivation for choosing the economy as a research theme is clear. We think that citizens and decision makers need to understand complex economic phenomena better – in order to be able to ask better questions, present better critique, make better decisions. And we believe that professional news journalism plays a central role in tackling this challenge.

From a cultural political economy perspective, we see the economy as a political and cultural achievement and an object of continuous discursive struggle. The media plays an important part in reproducing, challenging, and (de-)legitimizing prevailing ideologies and understandings of the economy and financial practices. Meaning work in the media is thus inherently political and linked with power.

The aim of the project is to help both researchers (including ourselves) and journalists to better understand the ways in which different semiotic resources can combine and interact to make meaning – to produce coherent narratives, tensions and contradictions, in-depth analyses and interesting insights – in the context of finance and economy. The project thus explores the potential of different semiotic resources and their combinations in the articulation, analysis and presentation of complex economic phenomena and mechanisms.

Where financial news meets new technologies

Over the course of time, technological innovations (e.g. printing press, electricity, telegraph, telephone, radio, television, personal computer, internet, WWW, development of mobile technologies) have altered both our communication systems and the structures and mechanisms of industrial production and commerce. Also, there might be seen a tendency of increased frequency of these innovations.

During the past few decades, information and communication technologies, along with prevailing trends in economic policies, have inarguably made a fundamental impact on the global economy. Digital technology and the Internet have influenced whole systems of production chains, storage and logistics, and the fields of marketing and consumption. The effects can be seen not only in the production and consumption of commodities but, following Matthiessen (2006), also in the production and trading of meaning, and the material conditions for semiosis. New technologies have made possible the development of new ways of meaning, and also the “development of new ways of buying and selling meaning, and thus making it the key kind of commodity of the ‘information age’” (Matthiessen 2006).

High-speed computational tools and automated trading systems have become part of the transformation of global financial markets by offering millisecond quick-fire tools to trade securities in global markets. Quant (i.e. quantitative) trading and analysis and high frequency trading (HFT) are part of the contemporary financial intermediation. The quants, being skilled statisticians, mathematicians and physicists, are tracking patterns in trading behavior and creating models to predict future market movements relying on complex algorithms and differential calculus. It has been assessed that quant trading may have generally been improving market quality (see Foresight Working Paper 2011), reducing dealing costs and improving liquidity but, it has also brought along concerns of self-reinforcing feedback loops (e.g. May 2010 Flash Crash).

Emphasis on speed and real-time processing has also created a demand for real-time reporting in web-based journalism. Nevertheless, apart from speed and immediacy, the aspects of depth, density and perceivable entities are required in representing multi-stratal and multi-dimensional financial topics and phenomena. The Internet as a database that offers the possibility of collecting massive quantities of information, requires advanced tools to give the information visual form. The possibilities and challenges of interactivity, hypertextuality, self organizing map algorithms and other means of assessing, rendering and representing statistics and massive data sets are increasingly important for professionals if they are to contribute meaningfully to the wider discourses of economics and finance.

We also see that it is important to evaluate whether ubiquitous visual communication affects the way issues in financial journalism are framed, structured and contextualized. One of our objectives is to explore whether new digital tools offer advanced narrative performativity or, to what extent, they merely create astonishing effects.

Where the media meets the public

New technologies have encouraged manifold experiments in multimodal signification, not all of which have been successful, as Gert K. Nielsen points out in his blog entry The Death of Datavisualization in the News. His main point is that if visualizations or multisemiotic presentations lack true journalistic input and insight, the public is not well served.

Throughout the project we will emphasize that professional journalism is – and should also be in future – a business of building and telling stories, providing new insights and perspectives, making bits and pieces of data mean something and feel like something. On a more political note, within a democratic society, journalism should be a business of serving the public. Furthermore, from the perspective of the public, understanding economic processes, being able to take and express a stand on them, can be considered a basic right of citizenship.

It seems to us that now is a good moment to jump on the visualization wagon – without needing to re-invent the wheel; to assess recent developments and adopt good practices, making use of lessons learned by pioneers of multimodal and visual journalism worldwide. Ultimately we need to tackle the really big questions: What do we want to see? What do we need to see?

Reference list:

Foresight, Government Office for Science, UK. 2011. Working Paper: The Future of Computer Trading in Financial Markets.  http://www.bis.gov.uk/assets/bispartners/foresight/docs/computer-trading/11-1276-the-future-of-computer-trading-in-financial-markets.pdf

Lemke, J.L. (1998) ‘Multiplying Meaning: Visual and Verbal Semiotics in Scientific Text’, in J.R. Martin and R. Veel (eds). Reading Science: Critical and Functional Perspectives on Discourses of Science , pp. 87–113. London: Routledge.

Matthiessen, Christian. 2006.The multimodal page: a systemic functional exploration. In Bowcher Wendy and Terry Royce (eds.), New Directions in Multimodal Discourse Analysis. Hillsdale, NJ: Lawrence Erlbaum Associates, 1-62.

O’Halloran Kay L. 2008. Systemic-functional-multimodal discourse analysis (SF-MDA): constructing ideational meaning using language and visual imagery. Visual Communication 7(4): 443-475.

Olson, David R. 1994. The world on paper. Cambridge: Cambridge University Press.

Related websites:

The 2010 Flash Crash: What Caused It and How to Prevent the Next One, by Peter Cohan, Daily Finance. http://www.dailyfinance.com/2010/08/18/the-2010-flash-crash-what-caused-it-and-how-to-prevent-the-next/

Quant trading: How mathematicians rule the markets, by Richard Anderson, BBC News. http://www.bbc.co.uk/news/business-14631547

AutomatedTrader.net. http://www.automatedtrader.net/

High Frequency Trading News, Insight and Resources. http://www.highfrequencytraders.com/