# How to Enhance Video Quality with AI on a GPU: Upscaling, Frame Interpolation, and Pipeline Setup

> AI Release · @ai_releaseorg · https://ai-release.org/guides/kak-uluchshit-kachestvo-video-neyrosetyu-na-gpu-apskeyl_en.html

## TL;DR

- **Time to complete:** 45–60 minutes of setup, plus render time that you measure on your own 10-second sample before committing to a full render.
- **Difficulty:** Intermediate
- **You need:** FFmpeg + ffprobe in `PATH`, Python 3 with a virtual environment, a CUDA-enabled PyTorch build, Real-ESRGAN (upscaler), Practical-RIFE or `rife-ncnn-vulkan` (frame interpolation), an NVIDIA GPU, and free disk equal to roughly 3× your source size.
- **Decide the order before you start.** Upscale-then-interpolate and interpolate-then-upscale give different GPU loads and different results on fast motion. The decision table is in Step 2.
- **Never run upscaling and interpolation inside one pass.** Two tools, two frame folders, two checkpoints — that is what makes the pipeline debuggable when a step fails halfway.
- **The silent killer is frame rate.** Extracting with `-vsync 0` from a variable-frame-rate (VFR) source breaks all frame math downstream, because VFR means the file does not hold a fixed number of frames per second.

## Requirements and setup
![Illustration: Requirements and setup](https://ai-release.org/guides/img/kak-uluchshit-kachestvo-video-neyrosetyu-na-gpu-apskeyl-en-1.jpg?v=1795121352)


**What to do:** Install the tools and record the exact build strings before touching any video.

**Why:** Every failure later in this guide is diagnosed by comparing what the tool *says it is* with what you assumed. GPU encoders, CUDA wheels and model flags are all build-specific, so the first thing you need is a baseline.

- **OS:** Windows 10/11, Linux, or macOS. Windows paths below use `C:\video\`; Linux/macOS users substitute `/opt/video/`.
- **GPU:** NVIDIA card with a current driver. Confirm with `nvidia-smi`. AMD/Intel users should use the Vulkan builds of the same tools — the CLI flags differ, so check the repository README.
- **FFmpeg:** a build that includes `libx264`/`libx265` and, ideally, NVENC. Confirm the encoder list before planning an encode.
- **Python:** 3.x in a virtual environment, plus PyTorch matching your CUDA version.
- **Disk:** PNG frame sequences are large. Rule of thumb: a 1920×1080 PNG frame is roughly 6 MB, so a 10-second 30 fps clip is about 300 frames ≈ 1.8 GB for the input frames alone, and the 2× upscaled set is roughly 4× that per frame. Measure once with `du -sh` / `dir` before rendering an hour of footage.

```bash
ffmpeg -version | head -n 3
ffmpeg -hide_banner -encoders | grep -i nvenc
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
```

On Windows PowerShell the second and third lines become:

```powershell
ffmpeg -hide_banner -encoders | Select-String nvenc
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv
```

**How to check:** `torch.cuda.is_available()` must print `True`. The encoder grep must list at least `h264_nvenc` and `hevc_nvenc`. If the encoder list is empty, you can still finish the pipeline — use `libx265` in Step 2 — but the encode will run on the CPU.

Checklist before Step 1:

- [ ] `ffmpeg -version` prints a build string (record it in your notes)
- [ ] `nvidia-smi` shows the GPU and driver version
- [ ] `torch.cuda.is_available()` returns `True`
- [ ] `python inference_realesrgan_video.py --help` prints a flag list
- [ ] `python inference_video.py --help` prints a flag list
- [ ] At least 3× the source file size is free on the target drive

Exact version numbers, download URLs and flag names change between releases — check the official repository READMEs for the versions you install.

## Step 1. Profile the source and fix the target parameters

**What to do:** Measure the input and write down four numbers: width, height, frame rate, frame count. Then decide the scale factor and the target frame rate.

**Why:** Upscaling multiplies pixels by the square of the factor (2× scale = 4× the pixels), and interpolation multiplies frame count by the multiplier. Both numbers must be known *before* rendering, because they determine disk usage, VRAM pressure and how long the job will take.

```bash
ffprobe -v error -select_streams v:0 \
  -show_entries stream=width,height,r_frame_rate,avg_frame_rate,nb_frames,codec_name \
  -of default=noprint_wrappers=1 input.mp4
```

Expected output for a typical source:

```
codec_name=h264
width=1280
height=720
r_frame_rate=30/1
avg_frame_rate=30/1
nb_frames=300
```

**Example:** `r_frame_rate=30/1` and `nb_frames=300` means 10 seconds of video. Target `-s 2` upscale → 2560×1440. Target `--multi=2` interpolation → 600 frames at 60 fps, still 10 seconds. Budget for 300 input PNGs and 600 output PNGs.

If `r_frame_rate` and `avg_frame_rate` differ (for example `30000/1001` vs `24000/1001`), the file is variable frame rate. Normalise it before anything else:

```bash
ffmpeg -i input.mp4 -vsync cfr -r 30 -c:v libx264 -crf 16 -an input_cfr.mp4
```

`-vsync cfr` forces constant frame rate, which is what makes "300 frames = 10 seconds" true.

If you only need a fragment of a long recording, cut it first — trimming with a GUI wrapper is covered in [this FFmpeg GUI trimming guide](https://ai-release.org/guides/kak-obrezat-video-bez-poteri-kachestva-v-ffmpeg-gui-na.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide). Cutting before upscaling saves hours, because everything downstream scales with duration.

**How to check:** Re-run `ffprobe` on `input_cfr.mp4`. `r_frame_rate` and `avg_frame_rate` must now be identical. Write the four numbers into a `notes.txt` next to the project folder — you will compare them against the output in Step 3.

## Step 2. Run the GPU pipeline: extract, upscale, interpolate, re-encode
![Illustration: Step 2. Run the GPU pipeline: extract, upscale, interpolate, re-encode](https://ai-release.org/guides/img/kak-uluchshit-kachestvo-video-neyrosetyu-na-gpu-apskeyl-en-2.jpg?v=1795121359)


**What to do:** Run four discrete stages. Each stage writes to its own folder, so a crash never forces you to restart from the beginning.

**Why:** Frame sequences are the only intermediate format every tool agrees on. Real-ESRGAN and RIFE both consume and produce image folders or short clips, and neither knows about your audio track or container — FFmpeg reattaches those at the end.

**Stage A — extract frames losslessly.**

```bash
mkdir -p work/in work/up work/out
ffmpeg -i input_cfr.mp4 -vsync 0 work/in/%06d.png
```

`-vsync 0` prevents FFmpeg from duplicating or dropping frames during extraction. Duplicated frames poison motion estimation in Stage C: RIFE will interpolate between two identical frames and produce nothing.

**Stage B — upscale on the GPU.**

```bash
python inference_realesrgan_video.py \
  -i work/in -o work/up \
  -n realesr-animevideov3 -s 2 \
  --tile 256
```

`-s 2` is the scale factor; `--tile` splits each frame into tiles so VRAM fits. Flag names differ across releases — confirm with `--help` if a flag is rejected.

**Stage C — interpolate.**

```bash
python inference_video.py --multi=2 --video=input_cfr.mp4 --output=work/interp
```

Or, in the frame-folder workflow: `rife-ncnn-vulkan -i work/up -o work/out -m <model> -n 2`. `--multi=2` doubles the frame count; `--multi=4` quadruples it. Check the repository README for available model names.

**Decision table — which stage runs first:**

| Order | GPU cost | Use when |
|---|---|---|
| Interpolate → upscale | Motion estimation runs on small frames; more frames to upscale | VRAM is the bottleneck, or the source is low-resolution |
| Upscale → interpolate | Motion estimation runs on sharp, high-resolution frames; fewer frames to interpolate | Output sharpness matters more than VRAM, e.g. fast camera pans |

Settle this with a measurement, not with an opinion: render a **10-second** sample both ways and time each stage.

```bash
time python inference_realesrgan_video.py -i work/in -o work/up -n realesr-animevideov3 -s 2
```

Record the wall-clock number and the frames-per-second line the tool prints. Multiply by your real duration to get the full render estimate.

**Stage D — reassemble and encode.**

```bash
ffmpeg -framerate 60 -i work/out/%06d.png \
  -i input_cfr.mp4 \
  -map 0:v:0 -map 1:a? \
  -c:v hevc_nvenc -preset p5 -rc vbr -cq 23 -pix_fmt yuv420p \
  -c:a copy -shortest output_1440p60.mp4
```

`-framerate 60` tells FFmpeg the input sequence is 60 fps — get this wrong and the video plays at double speed or half speed. `-map 1:a?` copies the original audio if one exists; the `?` makes a missing audio track non-fatal.

If `hevc_nvenc` is unavailable, swap in `-c:v libx265 -crf 20 -preset medium`.

**How to check:** Count the frames. Expected output count = input frame count × multiplier.

```bash
ls work/out | wc -l          # Linux/macOS
(Get-ChildItem work\out).Count   # PowerShell
```

**Example:** 300 input frames × 2 = 600. If you see 599 or 601, one frame was dropped or duplicated at a stage boundary — re-run that stage on the affected 10-second chunk rather than on the whole file.

## Step 3. Verify output quality and reproducibility
![Illustration: Step 3. Verify output quality and reproducibility](https://ai-release.org/guides/img/kak-uluchshit-kachestvo-video-neyrosetyu-na-gpu-apskeyl-en-3.jpg?v=1795121364)


**What to do:** Confirm the container metadata, compare the upscaled result against a reference, and freeze the exact commands you used into a script.

**Why:** A file that plays is not the same as a file that is correct. Metadata catches the frame-rate and duration mistakes; a similarity metric catches a stage that silently degraded the image; a script catches the fact that you will not remember the flags next week.

```bash
ffprobe -v error -select_streams v:0 \
  -show_entries stream=width,height,r_frame_rate,nb_frames,duration \
  -of default=noprint_wrappers=1 output_1440p60.mp4
```

**Example success output:**

```
width=2560
height=1440
r_frame_rate=60/1
nb_frames=600
duration=10.000000
```

Duration must match the source, width/height must be exactly the scale factor × source, and `nb_frames` must equal the number you calculated in Step 2.

Next, compare against a reference. Upscale one frame with your chosen settings, then measure similarity between the final output and a baseline encode at the same resolution:

```bash
ffmpeg -i output_1440p60.mp4 -i baseline_1440p60.mp4 -lavfi ssim -f null -
```

The last line reports `SSIM ... All:<value>`. Write the number down, change one flag, re-measure: this is the only reliable way to tell whether `--tile 256` or a different interpolation model actually helped. Treat any single measurement without a fixed baseline as meaningless — the same discipline described in [this guide to testing and evaluating AI models](https://ai-release.org/guides/kak-testirovat-i-otsenivat-kachestvo-ii-modeley-gayd-po.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

Finally, inspect the frame where interpolation usually fails — the fastest pan or the biggest cut. Extract it and look at it:

```bash
ffmpeg -ss 00:00:05 -i output_1440p60.mp4 -frames:v 1 check_frame.png
```

**How to check:** `check_frame.png` must open at 2560×1440 with no ghosting halo around moving edges. Ghosting means the motion model could not resolve the displacement — split the clip at the cut and process the segments separately.

Save the whole run as `run.sh` with the four commands from Step 2, and commit it next to the output. That is your reproducibility record.

## Troubleshooting

**Problem. Encoding fails immediately with an "unknown encoder" message.**

- Error: `Unknown encoder 'hevc_nvenc'`
- Cause: your FFmpeg build was compiled without NVENC support, so the GPU encoder simply does not exist in this binary.
- Solution: confirm with `ffmpeg -hide_banner -encoders | grep -i nvenc`. If nothing is listed, either install a build with NVENC or fall back to CPU encoding: `-c:v libx265 -crf 20 -preset medium`. The pipeline output is identical; only the encode time changes.

**Problem. The upscaler starts, then stops with a CUDA assertion.**

- Error: `AssertionError: Torch not compiled with CUDA enabled`
- Cause: you installed the CPU-only PyTorch wheel, so `torch.cuda.is_available()` returns `False` and every GPU call falls through.
- Solution: verify first, then reinstall the CUDA build into the same virtual environment:
  ```bash
  python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
  ```
  If it prints `False`, reinstall PyTorch from the index URL matching your CUDA version (check the official PyTorch install selector for the exact command), then re-run the check until it prints `True`.

**Problem. Upscaling dies partway through a long clip.**

- Error: `CUDA out of memory. Tried to allocate <N> GiB`
- Cause: a single frame at your scale factor does not fit alongside the model weights in VRAM. Long clips make this worse because the process holds memory across frames.
- Solution: lower the tile size (`--tile 128` instead of `--tile 256`), and process the source in 10-second chunks with a shell loop, running `ffmpeg -ss <start> -t 10 -i input_cfr.mp4 ...` for each chunk and concatenating the results afterwards. Confirm the flag name with `--help` — it differs between releases.

## Summary and next steps

You now have a working four-stage pipeline — probe, extract, upscale, interpolate, re-encode — with a measurement step that tells you which stage order is cheaper on your hardware. From here, two directions: automate the chunk loop into a script that processes a whole folder, or move on to generating the source clips programmatically with [Remotion-based AI video automation](https://ai-release.org/guides/kak-sozdavat-video-s-pomoschyu-ii-i-remotion-gayd-po-av.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

## FAQ

**Do I need a GPU to upscale video with a neural network?**
No, but the difference is decisive. Real-ESRGAN and RIFE both run on CPU if PyTorch is installed without CUDA, and both become impractically slow for anything longer than a few seconds. Use the CPU path only to validate the pipeline on a 10-second sample.

**Should I upscale first or interpolate first?**
Run a 10-second sample both ways and compare wall-clock time and output quality. Interpolating first keeps motion estimation on small frames and suits tight VRAM; upscaling first gives the motion model sharp input and suits fast pans. The decision table in Step 2 lists the trade-off.

**Why does my upscaled video play at the wrong speed?**
The `-framerate` value in the reassembly command does not match the real frame rate of your sequence. If you interpolated `--multi=2` on a 30 fps source, the sequence is 60 fps and `-framerate 60` is mandatory. Verify with `ffprobe` on the output — `duration` must match the source.

**Does frame interpolation fix motion blur or a stuttering source?**
No. Interpolation synthesises intermediate frames between existing ones; it cannot recover detail that was never captured, and it amplifies artefacts on VFR sources. Normalise to constant frame rate first, as shown in Step 1, and expect ghosting around cuts — split the clip at each cut instead of interpolating across it.
