How to compress this "mpeg nightmare scene" ?

Hi,

I have a scene with lof of movement and small details. Whenever I try to encode it to mpeg, I got a lot of mpeg artifacts around the edges of all shapes.

Here is a screenshot of the original scène in DV format (the big cristals and small lights move with the wind):


I also put a small animated part (1 second) in original DV format here: scene.avi

Here is the settings I tried:

CBR: 8000 KBPS
VBR: 7000 - 9200 kbps
CCE 7000-9200 kbps with 5 passes

So how can I get a good quality version of this scene ?

You should give a try to a high-bitrate quantization matrix.
If you use the default one, high-motioned and high-detailed scenes
will keep being overcompressed, and the encoder will just
pad the video stream with "zeroes" in order to achieve the 8000/9200 kbps bitrate.

Did you use an AviSynth script, or just dump the AVI into CCE? What were your other CCE settings? Or also give us a second of the output so we can see what you did and judge the output. What kind of audio are you using? I'm trying to judge how much you can lift the max bitrate.

Edit: I just had a look at the sample. Even the source DV AVI has noise and other garbage artifacts around all the lights and colored stuff. There isn't a high enough bitrate for the DV, which uses many more bits than is allowed for DVD video. There's no way to make that part look presentable without some serious filtering.

Are you limited to MPEG2 for DVD ? or can you use other formats?

I tried TMPGEnc Plus in constant quality mode at 85 (this is a decent quality setting for constant quality encoding), a max bitrate of 40000 (to make sure there was sufficient headroom), a min bitrate of 0, using the High Profile & High Level (HP@HL) profile (you have to use a high level profile to use high bitrates). The output file averaged around 20,000 kbps and was nearly constant bitrate, but looked very close to the DV AVI. The point being -- you're gonna need a lot of bitrate for a shot like that.

==> My Target is DVD.

For CCE, I don't use any avisynth script, I put directly the AVI file.

For the Matrix, where can I find a good one for Mpeg-2 ?

Here is my settings for CCE and Tmpg Xpress







If I need avisynth, what filters/settings do I need to use to improve the quality of the compressed video ?

In my opinion, a good quantisation matrix (one that retains more detail) produces worse results than the lousy default matrix CCE uses. In a couple of CQ tests, setting the max bitrate at 9500, a good matrix gave me an average quant of 22, and one that retains even less detail than CCE's default one gave me an average quant of 11. For the matrix used, neither result was very good, although the lower quality one looked a little bit better. Your problem is the max bitrate restriction of DVD. Your video will always look like crap after being converted to DVD video. It doesn't even look all that good as DV AVI. Like I said earlier, filtering might improve it. One thing to consider is deinterlacing it to make it progressive. Progressive video compresses a heck of a lot better than does interlaced video. But it also produces artifacts of its own.

Thanks for these infos, and what kind of filtering I can try ?

I already suggested one (deinterlacing). You might also try spatial and/or temporal cleaning/smoothing. If it were I, I'd probably deinterlace it. Others might handle it differently.

You could try another mpeg encoder HC is a good choice. You could also try the filters in tmpgenc but I don't know about the quality. However avisynth is supposed to have very good plugins but I don't do enough filtering to offer a recommendation.

thanks.

For the deinterlacing, after encoding can I insert this clip between two interlaved videos ??

Like this=> interlaced + progressive + interlaced ?

Originally Posted by cd090580

For the deinterlacing, after encoding can I insert this clip between two interlaved videos ??

Like this=> interlaced + progressive + interlaced ?

Yes, I join separate M2Vs during authoring. Muxman allows for the addition of multiple clips to make one larger video. Another way to join them is to open them all in DGIndex and File->Save Project and Demux Video. And I'm sure there are other ways to join them.

If that difficult-to-compress part is only one small part of an overall much easier to compress video, that's a good way to handle it. Separate it out and filter it and compress it separately, joining it with the rest of the footage later on. Be careful to keep the correct field order when separating them. By the way, most PAL video, even if from a progressive source, is encoded as interlaced.

Yes, this video is just a part of a whole one.
I tried the deinterlacing and got a huge improvement in quality.

For now I used the following avisynth script:

LeakKernelDeint(order=0, threshold=5, sharp=true, twoway=false, map=false, linked=false, debug=false, forceCPU=0)
Convolution3D(preset="movieHQ")

What other filters can I eventually add in my script ?

If you want to improve that section you could use a better deinterlacer that leaves less noise and jaggies which "eats up" more bitrate (bob it then you have to 1/2 the frames to make it pal compliant)

TempGaussMC_Beta1()
SelectEven()
FFT3DFilter()
Deen()

This is a ss your script, being fed to the encoder, notice the remenants of deinterlacing


And a ss of the above script, being fed to the encoder


Feel free to adjust the settings, or use more or less filtering as the tradeoff is loss of detail and oversmoothing. Once you've optimized the script, the 2nd part is adjusting the mpeg2 encoder settings. You want that low pass filter enabled as in your ss to attenuate the noise (for normal encodes you should disable it, because it makes the image too soft, and less detail), and a LOW bitrate matrix might help (although this is maxing out MPEG2 for DVD specs, it is to low for this scene). Also you want a higher quantizer characteristic in CCE (more flat) for this scene, because you want a flatter image and don't want to preserve more detail (MPEG2 doesn't provide good enough compression for this scene, so you want to attenuate all the noise), so shift the slider to the right, then encode with progressive settings

The alternative approach, if you want to keep it interlaced (much smoother playback, but more noise and artifacts) is to separate the fields, clean them with filters, then weave it back together and encode with interlaced settings

AssumeBFF()
SeparateFields()
fft3dfilter()
Deen()
Weave()

Thanks,

I tried this filter TempGaussMC_Beta1() but it always crash with an unknown exception in avisynth. I have installed the required plugins.
DOes the fact of having several version (MV_Tools v1 and v2) can cause such crashs ?

Can you post the full error message? (preview the script in vdub, for example)

The most common cause of crashing for that filter is having several removegrain.dll versions - try getting rid of the removegrainsse2, sse3 versions, and keep only the removegrain.dll

mvtools.dll and mvtools2.dll can coexist, but if you can't debug it, try moving them to another folder and loading the plugins manually (ie. clean out your plugins folder instead of having dozens of autoloading .dll's)

EDIT: I just saw your specs, and 1GB memory might be too low. I should also warn you the filter is very slow and CPU intensive. Make sure your CPU isn't overheating either (check temps with coretemp or realtemp).

Try adding this to the beginning of the script (or maybe 256), check your taskmanager to see how much memory your system is using:

SetMemoryMax(512)

I remove removegrain extra dlls and now it's work although it's very slow, about 1h30-2h for 30 secs of video

yep the one thing about filtering is that it is slow. I would also recommend procoder (only used 2.0 but very clear and not blocky output). However procoder 2.0 can be much slower the CCE.

I use TmpgEnc Xpress, it gives better resulmts than cce.

Can I use the trial versions of these softwares ? Did they put a logo on the video with the trial versions ?

If you're going to be doing some testing runs to see what encoder & settings are better for this case, I strongly suggest you make a lossless intermediate e.g. lagarith yv12, huffyuv etc... which was encoded from the .avs (especially tempgaussmc_beta1 , plus whatever filters you think feed a "good" image to the encoder) - So you don't waste any time - it will take minute or two instead of an hour for each run if you do this.

From some quick testing, even with various matices and low pass filters, you tend to get ringing/artifacting around the lights which is a scintilating mess on playback. If you used that script or something similar, the image being fed to the encoder is clean and noise-free, so I think the limitation here , of course, is MPEG2 @ DVD bitrates. I tried CCE, HCEnc, Mainconcept Reference, neither did very well. HCenc did the worst in this case (too much noise/artifacting).

Maybe lordsmurf or someone else can suggest some magical settings or matrices to use for this scene?

I don't think you're going to find anything that will do a great job on that clip. MPEG encoding is mostly about not including portions of the picture that don't change from frame to frame, and using motion vectors to mark 8x8 blocks of pixels that move from one location to another between frames. Just about everything in that shot is moving incoherently which means both of those techniques will be ineffective.

The video sample you provided does not agree with me. There seems to be some issues with that source video you provided for D/L.

I have a scene with lof of movement and small details. Whenever I try to encode it to mpeg, I got a lot of mpeg artifacts around the edges of all shapes.

The artifacts are comming from the original avi source you provided. That is why you are having trouble with your all your encoding trials. You are duplicating the issues and mistaking it for a first-time issue. I call it chizeling around the edges but most refer to it as ringing, I guess. Those are already in the .avi you posted, but again, they are being duplicated and most probable, being over-exagerated.

I would rather ask:

1. how *exactly* did you obtain the video sample: ie, cap card or other means
2. where does the actual video: ie, an old hi-8 on tape; or, hi-8 -> vcr tape -> cap card[codec ?]; etc.
3. maybe the source origin is a vhs tape. I don't know.

. . .

The reason I mention these things is because your vid sample has flaws in it. There are great signs of macro blocks (dv or mpeg encoding errors) and these seem to point to the poor job it did at compressing/archiving it in the .avi you provided, that is, assuming that you cap-ed it (but didn't do anything to it) and cut a 1 sec piece of it and posted here for D/L.

But, something doesn't flow right with me..it seems that you did something to the video sample already and then posted it here. I mean, it seems (or has the feel of) an avi->xvid and then to dv look about it. I don't know. But that's what the vid comes across to me as. Sorry. Don't mean anything negative. I mean. it could just be a cheap cam that was used and just has that look I just mentioned.

. . .

If you can describe futher details of the origin of your main source, and then how you are obtaining it, that could help us provide better suggestions, etc.

. . .

If the source still originates from a hi-8 cam tape, perhaps reviewing an alternative way of transfering it could help provide better results.

else, if the source still originates from a vhs tape (ie, hi-8 -> vcr) again, perhaps reviewing an alternative way of capturing it to your HDD it could help provide better results. For example, I would elimiate any DV codec as the archival, and go straight to a huffy codec instead. You want to get as raw (untouched/tainted) video as possible, and only a raw codec can do this, within reason.

. . .

Personally, if it were my home videos, I would not use DV as the archiving format. DV (like MPEG) suffers from similar encoding problems, only it is not as pronounced as MPEG is. But in this case, it seems to be very pronouncing.

. . .

Last commenting notes..

The one thing that is missing in these home videos (assuming that they are) is 'grain'. There is none!

My humble advice, don't throw away your hi-8 cams. They are the last living breed of grain, that last definitive in true video detail. All these AVC and MPEG cams are destroying the true art of Videography.

-vhelp 5080

The AVI the OP uploaded is way over sharpened. I wonder if the OP or the camcorder did this?

Yes. I'd like to know what camcorder (make/model/year) was used in this video, and for format of the recording, such as 8mm; hi-8 or other. This can give us an origin to start with.

And then, how was it obtained. Was it through a cap card; maybe one of those dv cams that accepts 8mm through one of the cartridges-gizmos or something; or I don't know.

Now that I think about it, I think that one of my *old* camcorders, an 8mm by Samsung produces sluggishy videos like this, I think. I'd have to go back to it and re-review it. It has 4 dead pixels, but only because it used to be on for 12 hours a day. But, its a noisy video when in low light. You can see the pixels jump around like static noise.

But, I once saw a hi-8 demo (I think it was on the doom9 site last year) and I D/L'ed the clip, and it had a very nice grain to the picture. And it compressed very well..mpeg, AVC, h264 .. didn't matter. I never did get the make / model of that cam. Well, I don't remember. All I remember was that it had a couple of kids shooting a watergun on a porch. It was a very cute video. Oh, and the person wanted to remove the grain--can you believe it? Anyway. Not even my latest cam (well, its old now, a Sony TRV-22 cam) can shoot that kind of video. Theres just no grain in it. I guess that's what DV (and MPEG) cams destroy: the natural beauty or grain-ness.

-vhelp 5081

I think the DCT ringing noise in the OP's video is normal for DV compression given the nature of the video. I made an image of whitish spots with oversharpening dark halos on a similar flat tan background then encoded with Cidocida. Here are crops from the original image (left) and the DV compressed AVI (right) enlarge 4x with a nearest neighbor filter:



The ringing noise is very similar to that found in the OPs video: