Scene perception is a dynamic activity. As one views a natural scene, attention and the eyes are directed serially to objects and agents of interest. Memory is required to span brief disruptions in scene input as the eyes are shifted from one object to the next. And memory is required to accumulate visual information from attended scene regions. I will discuss research examining the use of visual memory to construct representations of scenes and to establish continuity across eye movements. Our findings show that both visual short-term memory (VSTM) and long-term memory (LTM) contribute to the online representations of natural scenes, that visual LTM representations of scenes are highly robust and resistant to decay, that scene spatial structure plays an important role in knitting together visual information acquired from different scene regions, and that a key function of VSTM during scene viewing is to establish the correspondence between objects visible on separate fixations.