Masking Anime Studio Tips

  • Published on

  • View

  • Download

Embed Size (px)


  • 7/30/2019 Masking Anime Studio Tips


    Anime Studio ForumSkip to content

    Search Search

    Advanced search

    Board indexAnime Studio DiscussionTips & TechniquesChange font sizePrint view


    Masking explained... I hope!

    Moderators:Fahim, selgin, Agent_Smith, Tori, Belgarath, slowtiger

    Post a reply

    Search this topic Search

    24 posts Page 1 of2 1, 2

    Masking explained... I hope!

    by heyvern Mon Oct 15, 2007 3:05 pm
  • 7/30/2019 Masking Anime Studio Tips


    First things first:

    Masking looks incorrect in the preview when you use it with nested layers. It must be rendered to see it properly. This is what made me thinkthat it was confusing at first. If you don't trust the preview and only look at a render than masking should make more sense.


    A mask is just like an alpha channel in Photoshop. Adding isn't the same as... "adding" as you might think of it. A mask is a black or white"alpha channel".

    This has nothing to do with the color of the shapes used in the mask layer. Black and white is just a term used that is related to alphachannels. When a layer has a filled shape it is the "transparency" that determines its masking or whether that layer is "black" or "white" in themasking alpha channel.

    The really confusing bit is Reveal all and Hide all. These are opposites. The mask "starts out" in the opposite way.

    Think of it this way:

    Black hideswhite reveals

    Adding to a mask creates white, subtracting from a mask creates black It isn't really like taking away or adding to a mask at all... it is basedon black or white pixels. Subtracting only "adds" black. Adding only "adds" white.

    This has its origins from the mathematical concept of "adding" and "subtracting" the grayscale or RGB values of pixels. Each pixel has anumber value from 0 to 255. Adding those numbers makes a pixel "brighter" subtracting wil make it "darker".

    In the version of photoshop I started with that was the only way to combine masks by using Add or Subtract or Multiply from one alphachannel to another.

    The hide and reveal with white and black alpha channel concept relates to the following group layer masking properties:

    Reveal allis a totally "white" mask revealing everything. It is as if there is no mask. Adding a layer to it is adding "white" to "white" which meansnothing happens... unless you use "clear the mask and add to it". This "erases" the mask and "starts over". If you "clear" the mask you aremaking it all black. If the group mask property is set to "reveal all" and you clear it you basically have turned it into "Hide all".

    Subtracting from Reveal all creates a "blank" spot. A black hole through the mask. remember black hides white shows or reveals and Reveal

  • 7/30/2019 Masking Anime Studio Tips


    All is all white, unless of course you have "cleared it" now it is reversed. if not then the mask is all white so subtracting creates a black orhidden spot.

    I can see how this could get confusing...

    Hide AllNow the mask is "turned on" or completely black and hides everything. Adding to the mask will create a "white hole" through the black.

    remember when you "add" you are creating a white spot that makes things visible. If you use subtract this is just "adding black" to the blackmask... nothing happens.

    clear the mask and add this layer to it works differently now. Clearing the mask makes it all black and then adds that layer... but.. how itlooks depends on layer order. If this layer is above another mask layer it will override that layers masking (see below about layer order).

    Layer Order!!!The ORDER of the masking layer is also a factor. Layers higher in the stacking order take precedence over layers below it.

    This is critical for making complex masks. It allows you to stack elements under mask layers with different mask layers... a lot of this

    requires experimentation to see how it all works. Eventually it starts to make sense.

    Making a mask layer invisible will also turn off its effect on the mask... obviously.


    This is the idea you start with. It will get very complicated when you start to add multiple group layers with different masking to this setup.You can create a complex masking situation in a group layer with multiple layers all doing different things, then set the masking of theGROUP layer itself to mask layers in it's parent layer which has DIFFERENT masking set.

    You can create amazing stuff if you get the hang of it.

    In the sample below the character on the right pours a can of red paint over the character on the left. This done with several group layers andduplicated layers to create the perfect mask to produce this effect. This would be an example of a "complex" mask because the character isduplicated in "red" and a mask is used to reveal the red character... but also the strokes had to be duplicated on another layer since they areALSO part of the mask... etc etc etc... very complicated but it works and I know WHY it works.

    click to view animation

  • 7/30/2019 Masking Anime Studio Tips


    -------------------------------------------------Additional info moved here from later post-------------------------------------------------

    Sample file used below:

    Masking in AS "in general" is based on the transparency of the layer or shapes on a layer NOT ON COLOR VALUE!

    What this means is, if you have a soft edge fill... then you get a "fuzzy" edged mask based on the "transparency" or "falloff" of the soft edgeof that fill.

    BUT if you use a black to white gradient... that has no effect on masking. The shape is filled but it is filled with black to white. I think this isincorrect. I think it should be an option in masking to use the color values. I think you should be able to create a black to white gradient anduse it as a gradient mask but for now this will not work.

    Now on to that other gradient example in your file which I think should ALSO work but doesn't... the one with a 100% transparent color forone of the colors in the gradient... This ABSOLUTELY SHOULD WORK but it does not.

    I think in both of these situations masking is based on the "filled-ness" of the shape. If the shape has a fill PERIOD it is 100% filled and anytransparency other than soft edge effect is ignored. So even though you have a transparent "end" on the gradient... the shape is still "filled" sono gradient mask. I did EXTENSIVE testing of this and absolutely no combination of layers or effects even with separate styles will createthe proper gradient mask.

    I believe this is INCORRECT behavior but unfortunately this is how it works.

    There is however a "trick" you can use to make this work:
  • 7/30/2019 Masking Anime Studio Tips


    Simple gradient mask:

    Simple version:Soft edge does work as a mask. The trick then is to use a soft edged fill to create a "pseudo" gradient mask on a separate layer. You create asoft edge fill that is very large with a large edge radius for the soft edge and use the "edge" as the gradient mask. You would then have 2mask layers.

    Complex Gradient mask:

    Complex version with 3 mask layers:There are problems with the simple version since the large shape will mask EVERYTHING not just a specific bound area like a box with agradient. So to do anything more complex will require breaking this up into several layers and using a combination of masking settings.

    I will try to explain how the complex version in the sample file works:

    The main group is set to hide all. The "mask" is now "black" hiding everything.
  • 7/30/2019 Masking Anime Studio Tips


    Layer 1 ---> the color shape layer to be maskedMask 1 ----> Add to mask but keep invisible(Adding is "white" so now the shapes are revealed in the layer above it but the mask layer is invisible. We have added white areas to theoriginal black mask)

    Mask 2 ----> Subtract from mask(this layer is soft edge shape to create the gradient. Subtracting removes from the the whole mask. The mask started out black, then we added

    white areas, now we add a black area which is the soft edge fill.)

    Mask 3 ----> Add to mask but keep invisibleThis layer "masks" the soft edge layer. The last mask confines the soft edge layer shape to a specific shape but has no effect on the first masklayer above it because it is ADDING. The first mask layer is also adding... adding is creating "white" so adding white to white = white... sono change in the masking of the first mask layer. Mask 2 however is SUBTRACTED which is black. So now the third mask layer isADDING WHITE only to the second mask. All of this is working based on layer order.)

    EDIT:That last bit for the 3rd mask layer seems... strange. You would think that "adding" to the 2nd mask layer would eliminate it... what is really

    happening is the 3rd mask is MASKING the 2nd mask. The end result is the actual mask that is masking Layer 1 at the top. As I said it isconfusing. Once you "understand it" it gets easier. Took me ages to get the hang of it.

    You can get more sophisticated with this by putting these "complex" masks inside ANOTHER group layer to further modify the mask.Remember a group layer can have its own set of masking with in it that is completely separate from the parent group.

    You can even have groups within groups within groups. Remember though.. if you do THAT you can't rely on the preview you must renderto see the results.

    Remember that none of this has anything to do with color values! It has to do with the "filled-ness" (made up term) of shapes in thelayer.

    Using black and white to refer to the values of a mask is just to put it into alpha channel terms.


    -vernLast edited by heyvern on Mon May 11, 2009 9:27 am, edited 6 times in total.
  • 7/30/2019 Masking Anime Studio Tips


    heyvernPosts: 5857Joined: Fri Sep 02, 2005 4:49 am


    byjorgy Mon Oct 22, 2007 12:07 am


    Been a long time, huh? The last I was on here was when the 2007 Foota contest was starting. Time flies!?

    Anyway, I've been trying to get some masks working with an alpha.

    In my example, located here:, I have a masking group with blue that's supposed to show through,and then two bars on the right hand side where I've done gradients - going from white to alpha and white to black. However, it doesn't haveany effect and the two bars appear the same as the blob!

    Let me know if that doesn't make any sense and I need to clarify.



    p.s. Is it also possible to create a png image with an alpha, and use that as a mask?

  • 7/30/2019 Masking Anime Studio Tips


    Posts: 779Joined: Sun Sep 05, 2004 8:01 pmLocation: Colorado, USA


    by heyvern Mon Oct 22, 2007 4:49 am

    -----------------------------------------------------------------EDIT: The information in this post was moved to the top-----------------------------------------------------------------

    -vernLast edited by heyvern on Thu Feb 07, 2008 8:39 am, edited 3 times in total.

    heyvernPosts: 5857Joined: Fri Sep 02, 2005 4:49 am


    by heyvern Mon Oct 22, 2007 5:02 am

    p.s. Is it also possible to create a png image with an alpha, and use that as a mask?

    Yes! Absolutely!

    This is another alternative to using multiple layers for a gradient mask.

    Remember that the alpha or transparency of a PNG file effects the transparency of that layer directly. Using it as a mask will use the
  • 7/30/2019 Masking Anime Studio Tips


    transparency of the PNG to "hide" or "reveal" the masked layers above it.

    Once again I believe that the color values themselves should be an OPTION at least when doing masking even with an image. In mostapplications with masking black and white values are used to determine masking.

    Until that happens we do what we can to work around it.


    heyvernPosts: 5857

    Joined: Fri Sep 02, 2005 4:49 am


    by cribble Mon Oct 22, 2007 11:43 am

    Someone needs to sticky this. This post is fantastic, co...


View more >