Week five is the first week of the XNA 4.0 workshop where I'll step away from doing an overview of the "textbook" chapters, and instead focus on the weekly topics themselves.
In Week five we take a look at XNA 4.0's Audio/Video features. If you DO want to read the chapters in the "textbook", check out chapters 7 and 12. But in truth, you'll find everything you need to know below, or in the external resources provided at the bottom.
XNA's Audio/Video support can be broken up in to a few smaller categories:
- Playing 2D Audio
- Playing 3D Audio
- Accessing Media
- Recording Audio
- Playing Video
Playing 2D Audio
The first of the four categories above "Playing 2D Audio" can itself be broken up into two categories:
- Custom Audio Playback
- Simple Audio Playback
Custom Audio Playback
Prior to XNA 3.0 the only method for Audio playback was the Custom Audio Playback method. This method entails using the XACT Framework (Cross-Platform Audio Creation Tool) to combine wave files into sounds and then bind sounds to cues to be triggered at run-time. The benefit to the XACT method of creating sounds is it cleanly separates the job of audio designer/composer from programmer. Engineers are freed from needing to know what sounds to play and when. All they need to do is use the in-game API to trigger cues, and whatever nifty effects have been created by the audio engineer will play. Additionally, using the XACT method sounds can be cleanly grouped into categories, with volume control and other settings being configurable at a categorical level. This is great for allowing users to modify the ambient sound volume, or music volume, while leaving the sound effect volume alone.
The main downside to the XACT method is it doesn't go through the traditional content pipeline XNA developers are used to. The XACT tool instead creates project files which reference external wave files, and then these XACT project files are added as a resource to the XNA Content Pipeline. This necessarily changes the way sounds are visualized in the Content folder, and also how they're accessed in game. To access the sound cues, programmers are required to instantiate bank objects, and then make regular calls to the AudioEngine class which is responsible for processing audio data.
The main classes involved in the Custom Audio Playback method are:
All of the above classes can be found in the Microsoft.Xna.Framework.Audio namespace.
Simple Audio Playback
Beginning with XNA 3.0 Microsoft introduced the SoundEffect class. SoundEffect is an in-game asset type similar to Texture, SpriteFont, etc... which maps one-to-one with an on-disk sound file. SoundEffect objects can be associated with sound files of different types, including WAV, WMA, or MP3.
Unlike with the XACT Framework, there's no good way to create randomization over which sound will play when you call SoundEffect.Play, there's no way to make it cross-fade between multiple sounds, and there's no good way to make multiple waves play at the same time, or with time delays. This level of customization requires the use of the XACT Framework.
What you do get with the SoundEffect class is the ability load an audio file with one line of code, and play it with another. It's quick, easy, and follows the usual method of getting assets into the game.
In addition to just being able to "fire and forget" sound effects using the SoundEffect class, engineers can also get access to a SoundEffectInstance object of an associated SoundEffect, and then pause, play, or resume playback while being able to adjust things like pan, pitch, and volume.
For 90% of games out there the Sound Effect API will work just fine. Also note that the XACT Framework isn't available for Windows Phone 7. So if your plan is to hit the Windows Phone Marketplace, SoundEffect and its instance class are your only options.
In addition to the SoundEffectInstance class which was introduced in XNA 3.0, Microsoft introduced the DynamicSoundEffectInstance class in XNA 4.0. This class allows you to work directly with audio buffers to enable the dynamic creation of audio at run-time, and can also be used to implement streaming of audio from disk.
The main classes involved in the Simple Audio Playback method are:
All of the above classes can be found in the Microsoft.Xna.Framework.Audio namespace.
Playing 3D Audio
Whether you're using custom sounds or the simple sound effect API, there will come a time when you may want to work with 3D (positional) sound. Positional sound works the same as normal audio, except that the volume will rise/lower and interpolate between speakers based on the origin of a sound relative to the listener. This allows you to easily create effects where audio sounds as though it's coming from the left or right of you. I have never tried using the 3D Sound API with more than two speakers, so I'm not sure how well it works with 5.1 and 7.1 speaker setups.
To use 3D audio with XACT, simply call Cue.Apply3D, passing in an emitter and listener. Similarly, to use 3D audio with the SoundEffect API, call SoundEffectInstance.Apply3D.
The main classes involved in 3D Audio are:
- AudioEmitter (used to set data for the origin)
- AudioListener (used to set data for the listener)
All of the above classes can be found in the Microsoft.Xna.Framework.Audio namespace.
Up until this point I've mostly been assuming engineers will be accessing their audio through the Content Pipeline, using either XACT projects or sound files which are loaded at run-time and then played. However, beginning with XNA 3.0, when Microsoft added (and subsequently removed) support for the Zune, they also added access to your system's media library through the MediaLibrary class. This includes all the songs, albums, cover art, and play lists in your audio collection, and also the pictures and videos (more on this later) in your Libraries.
These assets are all accessed using intuitively named classes in the Microsoft.Xna.Framework.Media namespace in combination with the MediaPlayer class, another addition in XNA 3.0.
To use the MediaPlayer class just call one of the static functions such as MediaPlayer.Play and pass in the Song you want. You can filter the song you want by using the MediaLibrary.Songs collection, or if you want more control you can first access the albums in your library through the MediaLibrary.Albums property, etc...
With XNA 4.0 also comes a nifty new method on the Song class called Song.FromUri. This method allows you to play a song from the web or from file, but does not work on the Xbox 360. It also doesn't support spaces in the path name, so make sure that doesn't happen or you'll get an exception.
The main classes involved in accessing audio from the media library are:
- Album, AlbumCollection
- Artist, ArtistCollection
- Genre, GenreCollection
- PlayList, PlayListCollection
- Song, SongCollection
Being able to play audio is all well and good, but we also need to be able to record or stream microphone audio. Beginning with XNA 4.0 Microsoft has added the Microphone class to the Microsoft.Xna.Framework.Audio namespace.
Using the Microphone.Default or Microphone.All properties developers can gain access to one or more Microphone objects hooked up to their Phone, PC, or Xbox 360. Once you have a reference to the Microphone, it's as easy as calling Microphone.Start and Microphone.Stop in order to record audio. Then, using the Microphone.GetData method (in response to BufferReady events) developers can gather the audio as a byte array to use in combination with the DynamicSoundEffectInstance class discussed previously to play back the audio.
There are of course other properties on the Microphone class to help you identify important information such as the SampleRate, and BufferDuration, and to determine whether it's a headset or not, but you get the basic idea. Play, queue buffers, stop.
As an aside I should note that recording audio is meant to record audio for use in a game. If you just want to use a microphone to communicate with your friends over Xbox Live, that comes for free.
If you're taking advantage of the GamerServices API you can call LocalNetworkGamer.EnableSendVoice to allow audio to be transmitted. With the addition of several read-only properties such as NetworkGamer.HasVoice, NetworkGamer.IsTalking, and NetworkGamer.IsMutedByLocalUser, developers can add interface elements to indicate who can/is talking and whether they're muted.
And finally we come to playing video. Starting with XNA 3.1 Microsoft introduced the VideoPlayer class, located in Microsoft.XNa.Framework.Media. This class allows you to play videos (obviously).
To use it you simply:
- Construct an instance of the VideoPlayer class
- Call VideoPlayer.Play(video)
Then each frame call VideoPlayer.GetTexture and it will return, sequentially, the frames of the video as a Texture2D object. This is great because it means it can be used with SpriteBatch to draw your video as an animated sprite, or applied to a 3D quad to be rotated, stretched, and then projected into any 3D universe. Great for adding video to signs, on soda vending machines, etc...
Initially the only way to gain access to a Video was through the Media Library (discussed previously), but in XNA 4.0 Microsoft added Video as an asset type, allowing you to Content.Load any WMV file via the Content Pipeline.
AppHub Education Catalog
When the XNA 4.0 Workshop first began back in January I did a survey of the available XNA books. At the time, there were very few XNA 4.0 books available, and even among the 3.0 books there weren't many that I felt covered the topics in sufficient depth and in the correct order. After careful consideration I ultimately chose XNA Game Studio 3.0 Unleashed as the "textbook" for this workshop.
Unfortunately, the book covers XNA 3.0, not 4.0 which has made it difficult for people following the workshop to get projects functioning, as there were several breaking changes between XNA 3.1. Additionally, after eight months, the AppHub has come a long way in providing additional resources for those wanting to learn XNA 4.0.
So my plan moving forward is to, in the weekly workshop introduction, identify the topics we'll be covering for the week, point to a few resources on the AppHub which may help, identify some documentation pages in the MSDN Library, and then wherever necessary write additional tutorials to supplement the information already available. Also, I will indicate which chapters in the textbook we'll be looking at for those who purchased the book and would like to follow along that way.
What this ultimately means is it doesn't matter which XNA books you have, or if you have none at all. Hopefully by addressing the topics, documentation, and tutorials already available I can make the workshop more accessible to everyone.
This week in the XNA 4.0 workshop we're covering 2D graphics using the SpriteBatch class. For those that used XNA 3.1 or earlier, and especially those that have never used DirectX, the new arguments to SpriteBatch.Begin can be a bit off-putting.
When using the SpriteBatch class there are really just four functions that'll you'll use most of the time. These are
The first two are fairly obvious. Any time you want to draw a sprite or a string you call the corresponding function. The latter two need some explanation.
XNA is built on DirectX which is a hardware accelerated API for manipulating 3D primitives. This means that everything we do with XNA is really in 3D using triangles. Drawing sprites is no exception. Every time you call Draw or DrawString it's really just queuing a texture to be drawn on a 3D quad rendered in pre-transformed screen space.
Because XNA/DirectX is really 3D, the API has to do some initial setup with the Graphics Pipeline to make sure things are drawn with the appearance of 2D. In XNA 3.1 and prior this was done with shader code that was called from SpriteBatch during the drawing process. When that happened depended on which sort mode you were using. But ultimately your textures, combined with specific device states, would be drawn to the screen in a series of pre-transformed quads.
In XNA 3.1 you had a little bit of control over this process, but not much. When you called Begin you could pass it a number of arguments depending on which overload you chose. The most complicated overload looked like this:
Begin(SpriteBlendMode, SpriteSortMode, SaveStateMode, Matrix)
In the above overload, SpriteBlendMode, SpriteSortMode, and SaveStateMode were all just simple enumerations.
SpriteBlendMode was used to determine how sprites you drew were combined with data already in the target buffer. The options were either SpriteBlendMode.Additive, SpriteBlendMode.AlphaBlend, or SpriteBlendMode.None.
SaveStateMode was used to determine whether or not device states would be reset after calling Begin/End on the SpriteBatch class. The options were SaveStateMode.None and SaveStateMode.SaveState. The default was None, and would often cause confusion for people when switching back and forth between drawing with SpriteBatch and without in the same draw pass. Suddenly, people's 3D models would draw without respect for the depth buffer and all of their alpha blending states would be reset.
SpriteSortMode was used to determine in what order your sprites were drawn. The possible values were: BackToFront, Deferred, FrontToBack, Immediate, Texture.
In deferred mode all of your draw calls were deferred, and nothing was actually flushed to the hardware until End was called. Once End was called, the device had all of its render states set and all of your sprites were drawn to the target surface in the same order as they were "drawn". Of course, if you drew your sprites in something other than a painter's algorithm, or if you didn't intelligently split up your sprites by texture, it could result in undesirable layering or poorer performance, especially if you were rendering sprite-based particles.
The BackToFront and FrontToBack sort modes were designed to handle the painter's algorithm. That is, by passing a z-value in with your Draw calls you had more control over the order in which sprites were drawn. In these modes, it first sorted all your sprites by the z-value, ascending or descending, and then drew them. So it didn't matter the order in which you called Draw.
The Texture mode is/was for performance. If you drew from several different texture sheets within the same Begin/End pair, each switch from one texture to another would cause a flush and the device had to make a unique DrawPrimitive DirectX call. This is bad m'kay. By using the Texture mode it first sorted all of your sprites by texture, and then called them with as few DrawPrimitive calls as possible. Note, if all your sprites are/were from the same (or relatively few) texture sheet(s), then Texture sorting is useless, and you're better off using BackToFront or just Deferred.
The Immediate mode is interesting. With Immediate sort mode the device state was initialized as soon as you called Begin, and then each time you called Draw the quad was immediately pushed to the target surface. This is bad performance, but gave you control over your sprite rendering in a way none of the previous sort modes did. This is because you could call Begin, then manually make changes to the GraphicsDevice, such as changing blending values, filtering modes, depth buffer flags, etc... and THEN draw your quads. The problem was, now all of those quads were being drawn individually.
Enter XNA 4.0. With XNA 4.0, Microsoft acknowledged a need to give greater control over sprite drawing, while still maintaining performance. So the SpriteBatch.Begin method has a different set of arguments. The corresponding overload now looks like this:
Begin(SpriteSortMode, BlendState, SamplerState, DepthStencilState,
RasterizerState, Effect, Matrix)
Notice that gone are most of the enumerations. While there is still an enumeration for SpriteSortMode, BlendState, SamplerState, DepthStencilState, RasterizerState, and even Effect are all fully realized objects. This allows you to set properties on the device at the time you call Begin, and then have those states be set for the duration of the Begin/End pair. So you can set certain properties, set your Sort mode to Defferred, and gain the benefit of both custom settings and batched rendering.
If you're like most people, when you see that set of object parameters your first instinct is to go..."uh, how do I set all those." Fortunately for us, Microsoft provided static fields on each of those classes which grants access to instances of the most common scenarios you'll need when rendering with SpriteBatch.
- For BlendState, there are static fields for Additive, AlphaBlend, NonPremultiplied, and Opaque Blend States.
- For SamplerState there are several static fields, but you'll most often use the default, which is LinearClamp.
- For DepthStencilState there are Default, DepthRead, and None. Interestingly enough, Default is pre-configured with the default states you'll want set if you want to use the stencil buffer in your 2D rendering. In truth, None is actually the "default" setting.
- And last, RasterizerState includes CullClockwise, CullCounterClockwise, and CullNone. The default is CullCounterClockwise.
You'll also notice from the above prototype the addition of the Effect parameter. This allows you to pass in your own custom effect, so you could even do your 2D drawing in 3D if you wanted to. It helps to have something to start from though if you're going to provide your own custom effect. That's why Microsoft has released the Source Code to the SpriteEffect, which can be downloaded from the AppHub.
If at the end of the day you just want to draw in 2D and don't need any of the options you can just call the simpler version of the Begin method, or use the above but pass in null for as many arguments as you like. Since those are objects and not enumerations, it's fine to just let the SpriteBatch class determine what the most appropriate paramter is.
So you see, with the transition from XNA 3.1 to XNA 4.0 Microsoft has given users a lot more control over how they do 2D rendering with the SpriteBatch class. By messing with the different state objects you can make your sprite blend with the surface in new and unique ways, have it draw to the stencil buffer, ignore (or not) the depth buffer, or even have sprites wrap within the target rectangle. Because you can also provide your own effect it's now also possible to do things such as having your sprites cast shadows, or react in unique ways to custom lighting, all without sacrificing performance.
Welcome to Week 4 of the XNA 4.0 Winter Workshop. Over the next week, ending Sunday, July 31st we'll be working together as a cohort to master the knowledge, both of the XNA API as well as game programming in general, to work with 2D Graphics. As you read my little overview below, please remember the goal of these guided workshops is both to point people in the right direction, but mostly to create a community of people working through the information together. It's through this community that questions are asked and answered and real knowledge is obtained. I can't regurgitate what's in the textbook here in my blog, nor can I provided an entire math curriculum for learning vectors, matrices, and trig. But I CAN
most assuredly answer well thought-out and narrowly defined questions; as can all of the other folks following along. So remember, read the text, and then hit the forums
to be part of the community.
This week's reading material is chapters 9 and 10 from our textbook.
Chapter 9 - 2D Basics
The first chapter of the week covers a few different topics related to 2D game programming with XNA. First, it begins with an overview of Sprite Batches. In games, a sprite is generally an actor with a set of animations, but can also be a projectile, a piece of background or foreground environment art, or even the entire background itself. In truth, a sprite can be anything you draw on the screen. Because XNA is really a 3D engine, it draws all of our sprites as a series of 2D quads (two triangles together to form a rectangle). Because sending quads to the video card individually would be time consuming, XNA "batches" them together, and attempts to submit them to the video card all at once. If all of your sprites are pulled from a single texture, it can actually accomplish this quite nicely.
Within chapter 9 it spends some time talking about the properties of a SpriteBatch. In specific it looks at the Blend Modes, the Sort Modes, and the Save State Modes. How these are used has changed slightly between XNA 3.1 and 4.0. I'll make an additional post on that later.
After that, chapter 9 shows a few demos on how to use the things above to create loading screens.
Chapter 9 concludes with a section on drawing text with XNA. When we're in Windows, OS X, or some other operating system, there is a text engine which takes font data from a file in the form of strokes and is able to reproduce the characters of the font on screen at any size desired. This is because these font files are vector based, similar to a postscript or pdf file, while most of the images we see on computers and in games are raster based, that is, they're just a series of pixels. Because our Zune, Windows Phone, and Xbox 360 don't have font engines, it's necessary for us to render our fonts to a texture first, and then use portions of that texture like a sprite sheet in order to draw text to the screen. This is done by creating a .spritefont resource which is then turned into a texture during the build phase of our game by the content pipeline.
Chapter 10 - 2D Effects
The second chapter of the week builds on to the first by looking at Cel Animation, the process of creating animation by flipping rapidly through different cels of a sprite sheet. A demo is made with a class called CalAnimationManager. After that, the chapter looks at how you can manipulate the way sprites (and text) are drawn on the screen by providing different values to the SpriteBatch.Begin and SpriteBatch.Draw methods. These include rotating and scaling sprites, as well tinting the color, or changing the blend mode so that sprites will have alpha or additive blending. Additive blending is then used, in combination with the CelAnimationManager, to create an explosion special effect.
Note, as with Chapter 9, the way in which we set Blend Modes, etc... has changed slightly from XNA 3.1 to 4.0. I'll provide a follow-up to this post that covers those topics individually.
Below is a list of supplemental reading that might help this week's reading make more sense to you. I am likely to add to this list over the course of the week (or even the workshop) and will make an announcement when I've added additional reading or resources.
XNA 4.0 Source Code
Source code for Chapters 9 and 10 coming later this week.
Back in January I began the XNA 4.0 Winter Workshop with the goal of aiding people in their journey to learn XNA. The goal was for it to be a guided workshop, where each student takes responsibility for reading the material introduced each week and learning the information, with me (or other tutors) identifying the topics to be learned and where to learn that information - starting with the chosen textbook. Because everyone was to be working from the same source material, students should have found it easier to get answers to the specific questions they had.
Unfortunately, real-life issues, both personal and professional made it impossible for me to uphold my end of the bargain. While I had provided an outline of what topics to cover each week, I wasn't able to post study notes, resource lists, etc... and so after about the third week participation ended for most people.
Summer is here. While I am busy writing curriculum for the high school students at DigiPen, I'm not in class every day so my schedule is considerably more flexible. With that in mind, I'd like to give the workshop another go and make myself available to all those out there who want to either pick up where we left off, or use this as an opportunity to start learning XNA 4.0 from the beginning.
Before we move on with week 4, I want to quickly provide an overview of the topics we've covered so far and provide rationale for putting them in that order.
Week 1 - The first week of the XNA 4.0 Workshop was focused on learning exactly what XNA 4.0 is, as well as how to get it installed on your computer. Week 1's reading material was chapters 1-3 in our textbook, "Microsoft XAN Game Studio 3.0". As we're using XNA 4.0, there were some small changes that people needed to be made aware of. In specific, I posted an article on time stepping and the Main Game Loop, which was part of the information covered in the text. You can find the complete Week 1 post here: XNA 4.0 Workshop - Week 1
Week 2 - The second week of the workshop jumped right into 3D mathematics and computer graphics. This seems a bit extreme for most people, but the reason I did this is because while 2D game development is mathematicslly simpler, all modern games are inherently 3D, even the ones that look 2D, because they're being developed with a 3D API on 3D hardware. I wanted the readers to understand that when we finally got to the chapters on 2D graphics, primarily for performance reasons; as well as simply pulling back the curtains.
I think it was in this Week that I started to lose people. The reason for this is that for many, the hardest part of working with 3D graphics is the mathematics required; specifically trigonometry & linear algebra. There is a very specific process for taking 3D geometry in model space, and converting it into a 2D image in screen space. This process involves a set of tranformations most often done with matrices, and applied to the geometry's primitives - in this case vertices (points with extra data). It is impracticle for me to try and provide a single tutorial or outline on how to perform the math for this process. But, there are a number of great resources out there if you do a Google Search. The goal of Week 2 is for you to get an understanding of the 3D mathematics required, and to bring your questions to our official workshop forums for answers. I'll continue to collect a list of resources for Week 2 and put them on the XNA 4.0 Workshop - Week 2 post.
Week 3 - The third week of the workshop focused on a few different topics related to user input. In specific, it addressed the different input devices available for use with XNA 4.0, as well as how to create a simple wrapper class and then make a service out of it. The weekly reading ended with creating a camera and then a split screen which demonstrated moving around in a world. If you got lost in Week 2, Week 3 was only going to be harder. For a look back, see the XNA 4.0 Workshop - Week 3 post.
And that brings us up to where we are now. In week 4, we're going to take a step back from 3D and focus on 2D. This will be easier for many people, as the classes are more straight forward and there's no complicated matrix mathematics that needs to be done. After reading over the material for Week 4 you may find it beneficial to go back and read over weeks 2 and 3 again. Also, I'm going to make an attempt at providing many more, smaller tutorials related to the weekly topics. That way I can push them out quicker, and you all can see the simplest cases possible when trying to understand the topics.
For those that would like a reminder as to what topics we'll be covering in the XNA 4.0 Workshop, take a look at the XNA 4.0 Workshop Schedule & Topics.
If you're just now joining us or are returning to the workshop I say Welcome! I'll provide the weekly post for Week 4 as soon as I can.