The Grand Finale: Adventures in the World of Chroma Keying

The Final Project    

For my final project of the year, I was tasked to explore the process of chroma keying, which is the technique of removing a background from footage to place said footage over another background. This is typically done with the use of a green screen, which I did use. I worked with my friend Mike Webster (his blog has been linked) to film some very basic footage in front of a green screen, and then used the Keylight plugin in After Effects to remove the background and place it over a background image. After getting the footage, we both edited it in our own separate way. Given the closeness to the end of the year, there was not much time to polish the final product, although I did create a very rough version that demonstrates the process to some degree.

The Project and Pre-Production

My own project involved me walking up to greet a friend, walking away, and tripping/exploding in a blaze of glory. It was laid over a background of a calm looking field, mostly because said background was free and from a website that was not full of malware. I began the process by creating a very simple storyboard, attached below, to plan out the simple action of the scene. Afterward, I went and downloaded a few free green screen images to practice the chroma keying process (you can see one of my attempts, the Chernobyl Cat, below). This practice was nice, but not entirely representative of real chroma keying, given how the background of the subject (the cat) was one perfectly solid block of color. This luxury was not afforded to me in the actual project, due to inconsistencies in the lighting of the green screen as well as the camera being somewhat out of focus. At any rate, the next step was to film the footage that would be overlaid on the background.

The Storyboard. Not much, but it's something.

A smaller version of the background image we used.

It's a cat in Chernobyl. I don't know where I was going with this.

Production and Post-Production

After acquiring a camera, Mike and I filmed the scene (not a tremendous task, given its simplicity), and imported the raw footage into After Effects. Using a plugin called Keylight, I took a sample of the green screen's coloration and modified the Clip White, Clip Black, and Screen Gain parameters until the green screen was completely transparent while Mike and I were completely opaque, with a somewhat transparent outline surrounding us. I also modified the Screen Shrink/Grow and Screen Softness parameters to soften the edge between the foreground and background. Finally, I added some assets from the Action Essentials footage pack to create the explosion, and used a sound clip from SoundBible.com to put the finishing touch on. You can see the raw and final footage below. Had I had more time, I would have also added color and lighting correction to the foreground to make it match the background image, and also would have refined the edge of the foreground footage to make the blend between the foreground and background even smoother. Most of the issues came from the somewhat shoddy camera work we did, as the camera was not focused well, nor was the green screen lit particularly well. There was only so much I could have done in post-production to fix said issues, although I did what I could to make the final result as presentable as possible. I also would have better refined the Action Essentials assets to feather out the edges of the explosion and smoke, but again, time constraints prevented me from doing so.

The Concepts Involved and Lessons Learned

Through this exercise, I learned the basics of the chroma keying process, including setting up a green screen, taking the foreground footage, and using Keylight to render the green screen transparent. In addition, I learned more about the process of planning live-action shooting (however simplistic said shooting may have been) and how to deal with complications in the shooting process (read: it took us about a week to get access to the studio and a camera). I also got a slight review in overlaying assets for special effects, although it was not the main focus. This was a nice refresher in special effects editing, which I haven't done much of since the end of my sophomore year. Knowledge of chroma keying is instrumental for a majority of special effects in movies, so knowing the process, even if I likely won't end up using it much myself, makes it much easier to understand and appreciate the work and thought processes involved. Although the end result wasn't that great, I believe that, with more time for refinement through color correction and edge transparency as well better raw footage, using the same method would have yielded a presentable result.

A Reflection of this Semester

My main projects this semester were this, and the Goodnight Moon animation detailed in my last post. Both involved a break from 3D work in Maya and had a heavy focus on After Effects use. Both were not ridiculously difficult in terms of technique, so I didn't have too much difficulty with any of them. Maintaining focus on Goodnight Moon was at times difficult, due to the large amounts of work involved in animating almost every segment of the book in Photoshop as well as syncing it with the voiceover and sound effects, but I did manage to finish it without any major complaints. You can see the final version below.

This project, although quick, was very information-dense. However, this information (at least the pieces that I learned) was fairly easy to take in and apply to my work. Like I said, my main qualm with this project wasn't the work with the actual post-production editing, it was with the initial shooting of the footage. I don't have much of a background in video, so I just went with what Mike said and hoped for the best. Considering the fact that I only learned about this skill a week ago and only had two or three days to put this together, I'm not too upset about it.

Unrelated to the two major projects this semester, I also received an honor at the eMagine Media Festival for my character model, Kopf, that I made in the first semester. I was within the top 5 of 9 submitted, and while I did not place in the top 3, after seeing the winners I was perfectly fine with not doing so. The work put into all of the pieces at the festival awed me, from web design to typography to documentary to character modeling. I was humbled to be considered among the nominees, and am greatly appreciative of the opportunity I had to partake in the festival. You can see several rotations of the model with varying render qualities below.

I will close on a broader reflection of my involvement with e-Comm as a whole. This will be my last year with the program, due to issues with class schedules and simply a lessened zeal for the material. I believe that I can apply what I have learned about accessibility and the basic principles of design to almost anything I do. I also greatly enjoyed learning about more niche elements of animation, like lighting and rendering in mental ray and designing characters in Maya (even if my work was somewhat rough). However, I do not believe at this point that I want to make a career out of this field. I simply lack the zeal for it I once did, and while I still greatly appreciate what I have learned, I just don't think that continuing with the program would be the best use of my time in high school. Now that I've learned the basics of things like organizing projects, creating accessible designs, and operating software like Photoshop and After Effects, I don't feel that deeper involvement with the program will help me more than other academic focuses. I am grateful for the friends I've made and instruction I've received, but I believe that it is best for me to leave the e-Communications program in order to further my new goals for life.

Goodnight Moon Animation: It's For Da Kids

My latest project has been to animate a reading of the board book Goodnight Moon so that my class can include it in a compilation of similar projects to send to an area children's hospital. Given how this is a rather humanitarian gesture, I decided I should probably treat it with some air of seriousness. Because my ability to produce hand-drawn art is subpar at best (read: I can barely do stickmen) and there is not near enough time to animate a 3D environment for my book, I resorted to a Photoshop animation compiled in After Effects. I'll go more in-depth on the production process below, and also include storyboards, rough animations, and all sorts of other fun resource-y stuff.

Pre-Production

As mentioned before, I am animating the book Goodnight Moon in Photoshop and After Effects (with audio capture in Final Cut Pro and edited in Audacity, since I have no idea how Final Cut works). The planning and pre-production phase mainly consisted of me rereading the book and getting down the proper rhythm to read it at, while also scanning in the book's pages and storyboarding the changes I would be making to each page. For example, on the page reading "And there were three little bears, sitting on chairs," I storyboarded it so that the bears would open their eyes as I read. I scanned the book's pages in using our fancy-schmancy Epson scanner (which makes a whole lot of cool sounds) so that I could later alter them. I've included the three storyboard pages I created below.

My storyboards (Observe my lack of drawing skills)

As you can see, for my storyboards, I listed the text being read, the planned animation for the page, any audio clips I would need to find, and then drew a really, REALLY rough sketch of the altered element. After this process was complete, I moved on to the actual creation.

Production

I began the process of animating the pages in Photoshop by importing the scanned pairs of pages and changing the canvas and image sizes so that only one page was showing. I then selected the actual pages (while scanning them in, the scanner also included extra empty space) with the Quick Selection Tool and created a new layer by copying them, hiding the background layer under them. After that, the process began to vary. I made a lot of small alterations using one very simple process: the content-aware fill. Since I was working with pre-drawn images that I couldn't really add on to, my only real option was to play with what was already there. To do this, I would make a selection of the element on the page that would be animated (for example, the cow that jumps over the moon) with the Quick Selection Tool and create a new layer by copy from that selection. I then hid the new layer, selected the original background layer, and covered the area encompassing the element with a marquee. After that, I used the content-aware fill to fill in the background layer and unhid the copied layer. This gave me a replica of the element that I could alter the position and opacity of. I also had to do this with some text containers, since they were in positions that would interfere with the animation or were too difficult to see. For some pages, like the one with the little toy house, I made further alterations (like the dark windows). To do that, I just duplicated the copied layer to make a second copy of the element. Finally, for some pages, I just panned across the background, which just involved a position keyframe in Photoshop. After I made the animations in Photoshop, I exported them as .mp4 files, and imported them into After Effects.

Post-Production (Unfinished as of now)

In After Effects, I began the process of synchronizing the audio with the animation. In order to do this, I had to cut some clips short (I made them all with a little bit of extra time), and also had to do a Time Stretch on some clips to make the animation synch better. The process was pretty menial, since most of it consisted of me waiting for the RAM Preview to catch up to where I was (After Effects doesn't play audio with the default preview, and RAM Preview always starts from the beginning). I'm still working on synchronizing some scenes, improving the audio, and finding good sound effects, but you can see the work I've done so far in the rough (read: REALLY REALLY ROUGH) animation below.

I'll keep updating this post as I make more edits, but that's all I have for now.

EDIT: I'll probably end up putting this is Post-Production after I finish the project entirely and finalize this blog post, but I figure I should update my current progress. I've found sound effects for most of the elements, but still need to find a good "plop" sound for the mush. I synchronized most of the sound effects pretty well, but I'm still toying around trying to make it the best it can be. I also sped up the balloon in the first scene, since it was moving much slower than it does later on, and added in the text for the first page. Since the text for the first page was on an entirely separate page, I had to grab the text from that separate page, grab a black text bubble from another page, and then overlay those two on top of the first scene. I've also been working on getting rid of the book's crease in Photoshop, mainly through the content-aware fill. On pages where there's nothing but white space or solid colors behind the crease, it's been very easy. On other pages where things like the fireplace or wood basket are in the way, it has been a trek to keep the content-aware fill from obliterating whatever object was in the way. I used the Brush tool to fill in areas of the crease that content-aware fill couldn't, taking a color sample from a near area and trying to free-hand the design to look as believable as possible. (Also, content-aware fill hates the patterns on the rug and wood basket. I had to redo all of that by hand with the Brush tool. It wasn't fun.)

Items still on the agenda include audio edits to my voiceover and finding a few elusive sound effects, as well as finishing getting rid of the crease. I'm also going to try to darken the rabbit's room after he says "Goodnight light," probably through an adjustment layer with low Brightness. I'll see how it works and whether or not I want to include it, since I won't really know how well it will work until I do it. I'll finalize this post whenever I finish/a new deadline approaches.

The Notebook Stopmotion (A.K.A Still Frames Go In, Video Comes Out)

A normal scene.

Kopf has been set aside for the moment, as my new focus is on a group stopmotion project. I was placed with a group of 5 other students with the task of creating an animation incorporating stopmotion in some form. My group decided to fuse together three different styles: live-action, stopmotion, and 2D computer animation. The story focuses around three objects that come alive when their owner leaves the room: a highlighter, a sticky-note pad, and a staple-eater. They fall notebook and transported to a doodle world, which they then explore and try to escape from. It ends with them successfully escaping and rejoicing before the owner returns and everything returns to reality.

Pre-Production

In order to create this, we had to compromise and combine parts of our own individual ideas. We had each come up with our own plan and concept, so we had a communal session and included the aspects that we found most interesting/practical. We combined our interests (including my fascination with googly eyes) to decide on the premise: a bunch of office supplies going on an adventure in an alternate dimension. We then created a storyboard for the whole thing, took a few rough shots to determine which angles would work best, and created an animated storyboard with scene timings that we agreed on.

Production

The actual shooting was rather painful, as stopmotion is apt to be. We carefully moved the objects a slight amount each frame to simulate motion, and then compiled them all in After Effects. We also took the live-action shots and compiled them, as well. The doodle world animators worked in Photoshop and After Effects, creating a black-and-white world inside of the notebook that was rendered out and combined with the stopmotion and live-action during compilation.

 

                                               Post-Production

An abnormal scene.

For the post-production phase, we looked at all of the shots we took and made edits to the exposure, gamma correction, and color curves to make each shot look as close to the others as possible (since we had to shoot in a dynamic environment, there was variations in lighting that needed to be fixed). We also checked through the footage and extended a few frames to make the animation smoother (for example, when the notepad stands up on the highlighter). The doodle world animators also put the finishing touches on their segments.

 

 Reflections

During this project, I learned how to better operate and set up cameras due to the importance of the angles and perspective in the shooting process. I also learned how to sequence frames in After Effects by using a Time Stretch and a Sequencer script (which is incredibly convenient). I also developed a better idea of the group planning process, and how to compromise with ideas and create goals and deadlines.

After the great escape.

If I were to do this project again, then I would definitely pick a more stable shooting environment. It wasn't until fairly late into the shooting process that the flaws became evident, but there is noticeable shadow movement in some shots. There was also the issue of inconsistent lighting, which could be controlled in a more stable area. I would definitely use the same planning process though, as the storyboarding and creation of an animated storyboard with scene timings helped A LOT.

This project definitely helped to develop my thinking regarding pre-production planning, as well as indicating the necessity of a controlled shooting environment. Even though there are some noticeable flubs so far, I still don't think it looks too bad considering the fact that it was some of our first times doing it. If nothing else, at least the doodle world is pretty cool. The animated storyboard and regular storyboards are below. Some other shots are below as well, so feel free to take a look.

Why yes, that is a conga line.

Buddies 4 Lyfe.

Part 1

Part 2

Part 3

This is one of our early drafts of the project. We compiled all of the frames and video and added sound effects and a background song. Yes, the background song is elevator music. (We were running out of time until the deadline.) There were still some edits to be made, mostly involving light and color correction and small edits to sound effects, but this was pretty close to a final product that I wouldn't feel bad about showing.



This is our final edit. We added some more color and lighting correction and some changes to sound (such as a new, zipping sound when they get taken into the notebook, a non-copyrighted alert sound, and better synchronization with the video). We also added an intro and some credits.

You can read my other group members' posts by following these links:

Maddie Culotta (Stopmotion/Live action)

Emma Johnson (Stopmotion/Live action)

Makella Child (Doodle World)

Nathan Goldberg (Credits/Intro)

Brock Townsend (Doodle World)

The Perils of Waiting Until the Last Day to Render Things (aka This Project in Retrospect)

As I said in the previous post, I had to render out my final rotation using the Maya Software instead of mental ray, hence the rather sketchy looking video. I have discovered that it actually does render out in mental ray, it just takes forever and makes it possible to roast marshmallows over the heat vents. At any rate, I will get it rendered out with mental ray. Until then, you will have to settle with this:

Sorry it's a bit underwhelming. At any rate, I'll use this to discuss faults I have with the final project:

The skinning still is pretty flawed in his mid-section, which I am still attempting to find a plug-in to fix.

Some topography around his armpits and near the corners of his mouth is pretty sketchy; there are a few wavy edges that I would like to fix.

I still need to find a good setting for the amount of photons, so that it will stop displaying those circles on the ground when I render with mental ray.

The skinning is by far the worst issue, though, and the other two are fairly minor. Overall, I am very happy with how this project turned out, given how it was my first time really trying to do something entirely on my own in Maya. Once I iron out the kinks, I think I will try to make an animation with Kopf (I would like to try to animate him next quarter, but I am still considering other options/ my inherent idiocy at using the software).

No matter what I decide, I'll find another project to do next quarter. I'll also probably post a blog after I fix (or at the very least attempt to fix) the issues I discussed after I fix them.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Edit: I have now successfully rendered it out in mental ray (WOOHOO). Here is the final result:

He looks much, MUCH better when he actually has decent lighting on him. Something I realized that I haven't really addressed was my rationale behind creating Kopf the way I created him. I wanted to make a comical, cartoony character, but wasn't sure exactly what to make. I ended up seeing an Easter Island head during a Google image search, so I decided at that point to make something with a human body and an Easter Island head. I ended up dropping the human body idea, and creating a more exaggerated buff upper body with dinky little legs. I at first wanted to put him in a suit, but made a separate body model and separate clothing models, which resulted in problems when I began rigging him. Therefore, he runs around naked now. Once I had my idea, I started watching Lynda tutorials and applied what I saw to my own character.

Honestly, I found the most fun aspect of this process to be the lighting and rendering. Even though it takes forever to render and check your progress, watching the scene go from flat and colorless to deep and vibrant is kind of fun. I also think the lighting was the best part of this scene, because there are a few issues with the model and rig that I still need to work out.

During the class critique, the largest criticism I got was that the camera rotated too fast for a character demonstration. This is easily fixable, and it's really just an issue I overlooked in my haste. People actually liked the circles that resulted from a lack of photons, so I may keep them. When I re-render it with better camera motion (and possibly without circles), I will make yet another edit to this post with the final product.

Finally, I supposed I should cover what exactly I learned through this process. I learned how to model characters (and the fact that it involves a butt-ton of extruding, edge loops, and sculpting) and how to rig characters (with a skeleton/joints, IK/FK handles, and constraints), and how to render with global illumination in mental ray (WOOHOO INDIRECT LIGHTING). I already kind of sort of knew the basics of shading and lighting, but this also served as a good refresher on those. The biggest issue I had was weighting the skin of my character, since the fine-tuning takes FOREVER. If I had to redo something, I would probably redo the modelling process, since there were a few areas with really sketchy edge loops that I could have fixed by putting off the smoothing process until I had more basic topography done. Overall, though, apart from the skin weighting, I am rather proud of this project. For my first foray into character modeling, I believe I put forth a worthy effort.

Project Weeks 5/6: Lighting and Rendering

Even though I promised to make two different blogs for the lighting and the rendering, they really kind of go hand in hand, so I'm going to condense them into this single post.

Lighting is the process of adding lights to the scene (imagine that). I used a modified three point lighting system, which involved placing a point light, two fill lights, and a back light. The point, or main, light is in front of and slightly above Kopf, and is shining down and casting a shadow behind him. It is the only of the 4 lights to emit diffuse and specular. My two fill lights are located on Kopf's sides, and help to make the shadows on his body less intense so as not to obscure detail. The back light is directly behind Kopf and shines forward to help him stand out more against the background and add depth.

I then added a simple camera, placed its pivot point inside of Kopf around his mid-section, and set the camera to rotate 360 degrees around him over a span of 120 frames (about 5 seconds).

My lights from the back, showing the point light furthest away, the back light closest, and the fill lights in between

I attempted to render out in mental ray, a plugin which using raytracing to simulate natural light. However, given how OS X Mavericks seems to hate mental ray (and Maya in general), I was not able to batch render out my full rotation. I did, however, get this pretty sweet still shot.

He looks pretty nice when his spine isn't wrapping around itself

I used an additional setting in mental ray called global illumination to help with indirect bounce lighting, which is why you can see the tinge of red from the floor shining onto his lower body. However, this also created the issue of the red circles on the floor. The circles are created when the amount of photons being cast by a light are too small to accurately bounce light off of an area, but the Mac was already about to catch fire while rendering it, so I couldn't really push it further.

You may be asking: Where is the final rotation? Well, it's coming in the next post, where I will summarize my experience and point out areas that I still need to work on. It won't initially be rendered in mental ray, but I will find some way to get that done later. It will instead be in the (rather disappointing looking) Maya Software.

Project Week 4: Character Shading

As promised, here is the next update (this time only half a week late!). Shading took much less time than rigging, simply because there is much less work involved in the process.

Shading is just the process of creating the texture and color on your model. Given how lambert1 is rarely the most enthralling shader, I had to create a new shader, add a bump map to that shader, and then attach the shader to my character using a UV map.

At first, I created a Blinn shader, since I prefer working with Blinn (I think they display bump maps better, but that is mostly opinion). I then just altered the color to make it more stone-like, and changed the specular settings so that the reflections and eccentricity would not reflect much light.

Shader settings

After that, I added a bump map to create texture, so that Kopf actually looks like he's made out of rock instead of clay. I used a fractal bump map to make the texture kind of random, but still coherent, and set the bump value fairly low since the Easter Island heads are actually pretty smooth.

Fractal settings

Bump map settings

The final step involved mapping to shader to my character using UVs. Although the color applied itself without a UV map, bump maps cannot be applied without them. All I did was use the Automatic Projection tool, since I didn't have to worry lining up UV shells due to the fact that the shader was just a constant color. I had considered using a seamless rock image to make him actually look like stone, but the UV map was just too complicated to fix within my limited timeframe.

Model with shader

Overall, the result was pretty good. I may still alter the shader or bump map after I add lighting and start rendering, but this is a pretty decent start.

Project Week 3: Character Rigging

First off: I am a week behind at this point. Rigging is very, very difficult, and takes a REALLY LONG TIME. I'm still not even done, but I'll elaborate on that later on.

Rigging is, in a nutshell, the process of creating a manipulable skeletal system and control system, and then attaching that skeleton and system to the model. Creating the actual skeleton is as simple as tracing the outline of a skeleton (foot, ankle, knee, hips, spine, clavicle, shoulder, elbow, wrist, fingers, and neck are my major joints) with the Joint tool, and creating IK handles which make the skeleton move realistically when certain joints are moved (for example, the knees bend when you duck the hips down).

Creating the control panel is much more difficult. Using constraints to make joints move and rotate or track NURBS curves is basically all that you're doing, but keeping a clean hierarchy can be a puzzle at times. I used position constraints for things that would move like the feet and hips, orient constraints for things that would rotate like spine joints and wrists, and pole vector constraints for things that would bend in unique ways, like elbows and knees.

Attaching the skeleton and control panel to the skin is similarly difficult. Using the Smooth Bind tool, Maya will do a basic (read: pretty awful) job of making the model move with the skeleton. After smooth skinning it, you have to check the influence certain joints have on different areas of the skin, to make sure that joints aren't moving more than they should be (so that your chest doesn't cave in when you push the elbow closer to the body). Painting the skin weights manually takes FOREVER, so I am currently attempting to find a plug-in to assist with the process. (I'll include a link to the plug-in if I find one in an edit.)

So, here are the results of my labor:

Skeleton and control panel with transparent model

Skeleton and control panel sans model

I don't think wrists bend that way

DEAR GOD MAN GET THAT LOOKED AT

As you can see, I'm not entirely done yet. There's still some things to clean up, like overlapping wrist skin and that gaping crater in his chest that happens when he leans back. But that will all be allayed at a later date when I'm polishing, the brunt of the work is mostly over. Now, I get to begin shading the model. Given how I am horrendously backed up, I will work tirelessly over the weekend to catch up, and will attempt to get the next post up by next Wednesday.

Wish me luck. I will sincerely need it.