How to Make Rear Projections for a Live Show That Is Happening in Less Than 12 Hours

As the title suggests, we found ourselves in a bit of a time crunch. After pulling an all-nighter (and navigating a generally hellish week proceeding it) to swap both the engine and the transmission out of my car, we were tasked with designing and executing programmed lights for a 20-minute performance to be held in the Counter Office.

With about 30 minutes of rest, the first thing that came to mind to tackle the rear projections was to utilize stable diffusion AI to generate frames for a video. The only problem was I hadn't ever done that previously.

Nonetheless, I researched Deforum, an extension for stable defusion that generates videos from text to img prompts. Simple on face value (funny that we can say that nowadays, lol), but there were a whole swathe of features I was very impressed with and found extremely useful while working with it.

What initially struck me was the ability to program camera movements in both 2D and 3D space, which allowed for some very complex camera movements even when applied to very abstract prompts.

I was able to achieve some pretty interesting sequences through manipulating these values (and with some help from plenty of online sources):

What you might notice in the above sequence is distinct shifts in which the subject matter, while abstract, seems to shift from one thing to another. This is achieved through keyframing prompts. You begin your sequence with an initial image or (init) as well as an initial prompt. From there you can keyframe additional prompts to be fed to the computer at certain frames. This allows for the image to "animate" beyond the AI recycling the same prompt over and over again while you manipulate the camera (which lends to pretty mediocre results).

I am nowhere near an expert after about 8 or so hours of fiddling, but I can surely say I am extremely interested in what these tools have to offer for my workflow, especially in this particular context where abstract uncanny imagery is a viable final product.