Anime Studio 6 Auto Lip Sync

We'll be posting a full video tutorial on how we do our lip sync for the series (not hard but does take some time to get right) but in the meantime I thought I'd comment on the "automatic" lip sync function in Anime Studio 6.

The bottom line: it's practically worthless. Other than perhaps using it for a word or two it won't work for anything you have any pride in (and even then you're better off just using the respond to .WAV function).

It was clear this was the case from even a casual test of the feature, but in order to be fair I did some scientific tests and here are the results for those interested in the details (for those of you who don't understand lip sync, just understand it's not worth using).

I took a .WAV clip from the web (from the movie "As Good As It Gets") as being pretty typical of the sound files some hobbyist might work with. Better audio might yield better results but since it's based on typed text I kind of doubt it -- that is to say, if you type the text in, the auto sync should be generating phonemes on that text and matching it up to the audio, not generating phonemes from the audio. But perhaps it's doing both or even worse, using the audio only to generate the phonemes and then the text to try and line things up (rather than the other way around as Papagayo does, although it does it by asking you to line things up manually).

For comparison I put the text in Papagayo and manually lined up the generated phonemes. PG does an excellent job at generating the appropriate phonemes from the typed text. You can (and indeed, must) at times change what you type to match up to the audio you are hearing -- a character might say "all" instead of "oil" but as long as what you type is what it sounds like you are in good shape. In this case the lines are simple and clear enough to understand "How do you write women so well? "I think of a man and take away reason and accountability" so I did not have to change anything I typed.

For fair purposes I treated this as one character -- one thing the auto lip sync in AS doesn't do is allow for multiple characters with one audio file, unlike Papagayo, but it would be unfair to penalize it for that since that's not what it's intended to do. Then I took the XSheet from PG and matched it up with the keys generated by AS (reading the saved anme file). Here are the actual stats.

PG generated 56 phonemes and AS generated 89. It might be thought that AS generated a lot of dupes but that was not really the case. However, what was very obvious was that AS misread the track on the majority of words.

Example: "How". PG broke it down into two phonemes, "etc". and "O" and this is visibly accurate. AS took the same word and made it into four phonemes -- "etc", "rest", "MBP" and "F", which isn't even close (and looks very odd, as do most of these misreads). "do" was etc and U in PG, "E" and "O" in AS (not awful but no where near as convincing). They agreed upon "you" as "U" but "write" came out as "etc', "AI" , "etc" in PG (looks fine) and the very odd "U", "WQ", "U", "L" in AS. One more in detail: "women" is (in PG) "WQ", "AI", "MBP", "AI" and "etc" (as always, spot on) while AS didn't even come close with the bizarre "E", "L", "F", "MBP" and "F" (well, they did get ONE right).

And so it goes, on and on. The AS auto sync has the mouth opening and closing correctly most of the time (but not even that can be taken for granted) but the phonemes it generates are so bizarre and off that I have no idea how it is coming up with them.

I have tried the auto sync with much better audio files -- such as the ones we use in our weekly series, which are digital audio recorded at better then CD quality -- and the results were equally off. So I don't think the quality of the audio is the problem here.

To be fair, there isn't anything on the market that can auto sync properly -- and there are some programs that purport to do just that. One, indeed, costs a lot of money and does an even worse job than AS. I don't think this is Mike's fault (although I do wish there were an option to force AS to use the phonemes as generated by PG and match those up with the audio. I suspect that would be much better but that's just a guess. If I have time someday I may try programming this.)

I'd be glad to supply the files for anyone who wants to compare and/or try this experiment for themselves, but for me it is compelling -- AS auto lip sync isn't ready for prime time or much of anything else.

That shouldn't deter most folks seeking excellent lip sync -- Papagayo (the freeware that Mike wrote to work in conjunction with Anime Studio) is easy to use and does a superb job as noted above. Just like in all other forms of animation, you can't do it automagically but if everyone could do this they wouldn't pay us the big bucks .

Disarming things

One last addendum about the arm tutorial posted a few weeks ago -- this quick video will show you a shortcut that makes the whole process even easier (you ought to be able to create an arm and rig it from scratch in less than five minutes)

Second Arm Tip Followup from Mike Kelley on Vimeo.

Preventing Arm Artifacts

Today we're going to show you a very short tutorial on how to get rid of possible line artifacts you might get using the arm bending technique demonstrated a few weeks back.

A followup tip on the arm bend tutorial from Mike Kelley on Vimeo.

Next up: an even faster way of doing the arm bending technique.

Secret AS Tricks -- Keyboard Shortcuts

You lucky folks following this blog will have access to tricks, addons and lessons that even AS veterans don't know -- and will be posted no where else (since I no longer am welcomed on the Lost Marble website).

First up: secret tool shortcut tips. Nearly every AS vet knows of the _tool.list.txt file that exists in the scripts/tool directory. In this file you can add or edit the existing shortcut tips to any tool in your arsenal (and we'll be providing some killer tools you can't get anywhere in the months to come).

But what most don't know is that you are not limited to just the alpha and numeric keys A-Z and 0-9 to assigning shortcuts. You can also use the odd keys such as brackets ([]), semi-colon (;) comma, period and backslash -- in short any single key press (not shift keys) that you have.

This opens up at least 11 more keys for you to use as shortcuts -- which along with reassigning the keys you don't need should be more than enough to shortcut your way to any tool you use often. If you use a device like the Shuttle Pro (which I highly recommend) you can go further and assign these keystrokes to the buttons there, making construction and animation even easier (we'll talk in detail about this in another entry).

In the meantime -- enjoy editing away and setting up AS exactly how you want it!

Arms and the Man

One of the things I really like about Anime Studio is the way it deals with it's bone rigs. They are very straightforward (unlike ToonBoom -- don't get me started on pegs) and generally very easy to setup and use.

One thing that is a little difficult, however, is controlling the bend of a limb. The good news is there is a fairly easy and extremely effective way around this, and it's an approach that many animators use.

Rather than try and describe it in words, I thought I'd post a video tutorial (my first, and done in one take although with an edit here or there) to demonstrate how to set this up. While it might look a little involved trust me that once you've done it once or twice the whole thing takes less than five minutes to do. And then you're left with a very effective looking arm or leg.

Thanks to Genete for his bone rig and Vern for explaining to me how to use it (and possibly modifying it). If you want to download it you can do it here:

Genete's Rig

Arm Tutorial from Mike Kelley on Vimeo.

It's Rigged

Animators often talk about how their characters are "rigged". Essentially, this means how the parts of the character they are creating fit together using a system of "rigging", like the rigging of sails on a ship.

Character rigging comes from 3D animation, in which the parts of the characters are manipulated by "rigs" of external control devices, usually referred to as "bones" much as the bones control the human body. Move an arm bone in such a rig and the arm moves (hopefully naturally).

It's much less common to talk about rigging a 2D character, but certain animation systems allow such a control system. In Anime Studio (which we use for all our animation) there is such a bone system almost identical to those in 3D programs (although, obviously, it will only move and deform in 2 dimensions).

To use this effectively we need to create our character parts so they can be manipulated properly and the above image is the way that we do it. It's not the only way, nor is it necessarily the best, but it works for us.

Each of the major body parts is put on a separate layer. Once again, this isn't the only way to do it, but it does mean that parts such as the arms can be moved in front or back of other parts, like the head, for different effects. If they were all on the same layer this would be extremely difficult.

There are other things to note: this character is drawn in 3/4 view, as all all of our characters. This kind of "American Dad/Family Guy" style basically mimics the "cheat" that an actor in a play does as they turn slightly to open their body up to the audience while at the same time interacting with other actors on stage. Next time you watch a play notice that this 3/4 position is the one they maintain at almost all times (the old stage adage is "never turn your back on an audience -- they might start to throw things" :>)

It's also worth noting that the downstage arm is finished straight across at the top -- this allows it to rotate freely within the shoulder while not showing any bad edges. The upstage arm is finished because the shoulder part will be partially hidden by the body.

We'll talk about the actual bones and how they control things in a future post.

Pan 3D in 2D

Woods Pan from Mike Kelley on Vimeo.

Last night on the Anime Studio forum someone mentioned doing a 3D type pan and so I quickly created this example (in less than five minutes). It's a simple technique and easy to do in almost any animation program, yet one that eluded animators for decades.

Traditionally animation was "filmed" by taking a single frame (or 2 or more depending on the movement and the desired effect) with a camera from a flat stand that held the artwork. This was relatively easy but didn't permit very many camera type effects. One could zoom the camera in by moving it close to the artwork, or (if the artwork was large enough) pan around on it by moving either the camera or the artwork up or down, but as everything was at one depth this didn't closely approximate the Real World at all.

In the Real World life doesn't exist on one plane. If you look out the window of a train you'll see that objects that are closer to you move faster than objects further away. Telephone poles zoom by, while distant mountains slowly come and go. Celestial objects, like the moon and the stars, seem to be fixed and not move laterally at all. The same is true if you are heading directly towards or away from something -- the further it is the slower the change in relative size will be.

Walt Disney himself figured out you could approximate this effect in animation by using a camera which focused on more than one plane, and have artwork on several planes that moved at different speeds. Thus whether you were panning or zooming in or out the artwork would could change according to it's position relative to the camera. Thus the multi-plane camera was born.

It was famously used in such films as Pinocchio and Peter Pan. It eventually became too expensive (separate exposures were needed for each element so that shots sometimes took days to complete) and was used less and less until it was finally obsoleted by digital technology.

In the digital world it's very straightforward to achieve this effect, and most animation software includes it. In Anime Studio you simply assign a "depth" or Z value to each layer and this will automagically make camera movements, pans and zooms, displace the layers correctly (the only real trick is experimenting with the values to get the effect you want). In essence the layers are arranged in a 3D space, even though the camera itself is 2D. In this regard it's almost a perfect analogy to Disney's ground breaking camera.