Frame jumps and mixed modalities: Reading and/as interface

Interface design pioneers Douglas Engelbart and Ivan Sutherland strove to include hands, feet, body movements and orientation to the screen as part of the basic computational apparatus. They also relied on the visual aspects of graphical interface as a crucial feature of this embodied experience. Since then, the tactile aspects of interface have intensified, and the structures that organize the graphical user interface remain essential to our use of digital environments. Multiple types of media are now embedded in screen-based environments, posing challenges for reading across very disparate frameworks and modalities (media types). I refer to this process as ‘frame-jumping’, the challenge of moving from one visual and cognitive frame to another, while trying to position ourselves as readers/viewers within the multi-media environment.