Some tips here:
FFdshow with H264/x264 on AVI container.
I can't find much info, most people use x264 CLI with a front-end like
meGUI for x264 encodes. Normally what I do is edit the video (I use premiere but vdub is fine, too), encode the edited version in a lossless format, then import it into meGUI and use one of the presets with minor (if any) tweaks. Anyway good luck.