transcode: Fix filename usage for output format detection

If movs faststart option is set, the muxer will try to reopen the
output file using the path passed to avformat_alloc_output_context2
instead of the one passed to avio_open.