The Encoder Preset popup is found in the middle of the Broadcast Window:
Wirecast offers several presets by default.
See Broadcast Settings in the Good Presentations chapter for details on selecting a good encoder.
QuickTime Settings
In its simplest form, you need to choose the minimum connection speed your viewers (or you) will have. If you or any of your viewers will have a modem, you should choose one of the modem. If all of you have at least DSL or cable, then you can choose a higher speed.
Motion simply means whether your broadcast will have a lot of motion in it or not. If you're doing an interview, then most likely its going to be Low Motion. If you are broadcasting jugglers, High Motion is your ticket.
Modem
For use when you or your viewers have only a modem connection.
DSL / Cable
When you and all your viewers have at least a DSL or Cable connection.
LAN
For use in a company/school environment when all of your viewers are on the same LAN as you.
If you are going to only Record To Disk and your viewers will download the file from your server, then you may choose to encode for "LAN" as well. You can do this because your viewers are not trying to download the file in real-time.
H.264 or MPEG-4?
If you can ensure your viewers will have QuickTime 7 or greater, you should use H.264 encoding as you'll see higher quality for the same bit-rate (amount of data). It is worth noting that H.264 will consume significantly more CPU than MPEG-4, so if you find Wirecast is dropping frames, you might want to throttle back the settings you use.