← All guides AI Release
RU Subscribe to @ai_release1

How to Generate Music and Sound Effects Locally with AI: Stable Audio 3 and MCP Servers

Updated: 19.10.2026 · AI Release · @ai_release1
How to Generate Music and Sound Effects Locally with AI: Stable Audio 3 and MCP Servers

TL;DR

Requirements and setup

Illustration: Requirements and setup
Illustration: Requirements and setup

Checklist before you start:

If you have never run a model locally before, start with How to run AI models locally on your PC: a beginner's guide — the virtualenv and weights-caching habits it describes are the same ones used here.

Step 1. Create an isolated environment and download the model weights

What to do: create a project folder, a virtual environment, install dependencies, and pull the checkpoint into a local models/ directory.

Why: a virtual environment (venv) is a private Python installation for one project, so a broken dependency here cannot break your system Python. Downloading weights into a project-local folder instead of the global Hugging Face cache makes it trivial to delete 10 GB later.

mkdir -p ~/local-audio && cd ~/local-audio
python3 -m venv .venv
source .venv/bin/activate          # Windows PowerShell: .\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
pip install "mcp[cli]" torch torchaudio soundfile numpy

Example: on a CPU-only machine the PyTorch install downloads the CPU wheel automatically; on an NVIDIA machine, replace torch with the CUDA wheel command from the PyTorch install matrix.

Download the weights with the Hugging Face CLI, substituting the real repository ID from the model card:

pip install huggingface_hub
huggingface-cli download <org>/<stable-audio-checkpoint> --local-dir ./models/stable-audio

How to check: confirm the imports and the download.

python -c "import torch, soundfile, mcp; print(torch.__version__, 'cuda', torch.cuda.is_available())"
du -sh ./models/stable-audio

Expected output: a PyTorch version string followed by cuda True or cuda False, and a size line for the weights folder. If cuda prints False on a machine with an NVIDIA GPU, you installed the CPU wheel — reinstall PyTorch with the CUDA build.

Step 2. Wrap the model in an MCP server

Illustration: Step 2. Wrap the model in an MCP server
Illustration: Step 2. Wrap the model in an MCP server

What to do: split the code into two files — engine.py (loads the model, renders audio) and server.py (exposes it as an MCP tool) — then register the server in your client's config.

Why: MCP (Model Context Protocol) is an open JSON-RPC protocol that lets an LLM client discover and call external tools. Instead of pasting code into a chat, you describe a function once; the client then calls generate_music(prompt="...") on its own. Keeping the model loading in engine.py means the same code serves both the MCP server and a plain CLI test.

engine.py:

import os, torch

MODEL_DIR = os.environ.get("AUDIO_MODEL_DIR", "./models/stable-audio")
_model = None

def _load():
    global _model
    if _model is None:
        # Replace this block with the loader from the model card.
        # It must return an object exposing a text-to-audio call.
        _model = load_pipeline(MODEL_DIR)   # noqa: F821
    return _model

def render(prompt: str, seconds: float = 30.0):
    pipe = _load()
    result = pipe(prompt=prompt, duration=seconds)  # argument names per model card
    return result.audio, result.sample_rate

server.py:

import os, uuid, soundfile as sf
from mcp.server.fastmcp import FastMCP
from engine import render

OUT_DIR = os.environ.get("AUDIO_OUT_DIR", "./out")
os.makedirs(OUT_DIR, exist_ok=True)

mcp = FastMCP("local-audio")

@mcp.tool()
def generate_music(prompt: str, seconds: float = 30.0) -> str:
    """Generate a music or SFX clip from a text prompt. Returns the .wav path."""
    audio, sample_rate = render(prompt, seconds)
    path = os.path.join(OUT_DIR, f"{uuid.uuid4().hex}.wav")
    sf.write(path, audio, sample_rate)
    return path

if __name__ == "__main__":
    mcp.run(transport="stdio")   # stdio = client talks to the process over stdin/stdout

Register it in the client config. For a stdio server the client launches the process itself, so command must point at the virtualenv interpreter, not the system one:

{
  "mcpServers": {
    "local-audio": {
      "command": "/home/<user>/local-audio/.venv/bin/python",
      "args": ["/home/<user>/local-audio/server.py"],
      "env": {
        "AUDIO_MODEL_DIR": "/home/<user>/local-audio/models/stable-audio",
        "AUDIO_OUT_DIR": "/home/<user>/local-audio/out"
      }
    }
  }
}

On Windows the path is C:\\Users\\<user>\\local-audio\\.venv\\Scripts\\python.exe.

How to check: start the server manually first. It should print nothing and sit waiting — that is correct for stdio.

source .venv/bin/activate
python server.py
# ... nothing printed, cursor hangs → press Ctrl+C

Then confirm the tool is discoverable with the SDK's dev runner (mcp dev server.py, which opens the MCP Inspector) — the exact CLI flags for your SDK version should be checked against the MCP Python SDK documentation.

Step 3. Generate a track and a sound effect, then verify the file

Illustration: Step 3. Generate a track and a sound effect, then verify the file
Illustration: Step 3. Generate a track and a sound effect, then verify the file

What to do: run one music prompt and one short SFX prompt through the same render() function, then inspect the produced .wav with ffprobe.

Why: ffprobe reads the file header without playing it, so it proves the audio is a real, decodable stream with the duration you asked for — not an empty buffer that was written to disk.

python - <<'PY'
from engine import render
import soundfile as sf, uuid
for prompt, secs in [
    ("warm analog synth pad, slow attack, 90 bpm", 20),
    ("short metallic door slam with reverb tail", 3),
]:
    audio, sr = render(prompt, secs)
    name = f"out/{uuid.uuid4().hex[:8]}.wav"
    sf.write(name, audio, sr)
    print(name, secs)
PY

Example output:

out/3f9a12bc.wav 20
out/a71c04de.wav 3

How to check:

ffprobe -v error -show_entries stream=sample_rate,channels \
        -show_entries format=duration -of default=noprint_wrappers=1 out/3f9a12bc.wav

Expected output is three lines — sample_rate=..., channels=..., duration=20.000000. The sample_rate and channels values must match what the model card specifies for that checkpoint; the duration must equal the number of seconds you requested. Play the file (ffplay out/3f9a12bc.wav) and listen for a clean start and no digital clipping.

Now call the same tool from your MCP client and ask for something like "generate a 5-second rain-on-a-window sound effect". The client should return the path of a new .wav inside AUDIO_OUT_DIR — that round trip is the final proof the whole chain works.

Troubleshooting

Problem. The MCP client shows the server as disconnected immediately after launch.

/home/<user>/local-audio/.venv/bin/python -c "import mcp; print('ok')"

If that prints ok, restart the client so it re-reads the config.

Problem. Generation starts, then dies partway through a long clip.

CUDA_VISIBLE_DEVICES="" python -c "import torch; print(torch.cuda.is_available())"

This must print False. CPU rendering is slower but has no VRAM ceiling.

Problem. The server process exits the moment the client connects.

cd ~/local-audio && source .venv/bin/activate && python server.py

Fix whatever traceback appears, confirm the process now hangs silently, and reconnect the client.

Summary and next steps

You now have a local text-to-audio pipeline that writes real .wav files and is callable as an MCP tool from any compatible client. Next, add a second tool such as list_generated_files() or an AUDIO_OUT_DIR cleanup routine, and pin your dependency versions in requirements.txt so the environment is reproducible.

FAQ

Do I need a GPU to run Stable Audio locally?

No. CPU rendering works; it is just slower. A CUDA-capable NVIDIA card reduces generation time, and the required VRAM depends on the checkpoint — check the model card for the exact figure.

What is an MCP server in one sentence?

It is a small local program that exposes functions (here: "generate audio from a prompt") over the Model Context Protocol so an LLM client can discover and call them without you copying code into the chat.

Where do the generated files go?

Into the directory set by the AUDIO_OUT_DIR environment variable, which defaults to ./out relative to the working directory of the server process. If the files "disappear", check that variable in your client config.

Can I generate sound effects and music with the same tool?

Yes. Both are text-to-audio tasks, so one generate_music(prompt, seconds) tool covers them — the difference is only the prompt and the requested duration.

🤖 AI summary

Related guides

How to run AI models locally on your PC: a beginner's guide

Пошаговое руководство по установке и запуску локальных нейросетей на домашнем ПК без облачных сервисов.

How to Use GPT-6.1 Sol via API: Complete Guide

Полное руководство по подключению и использованию GPT-6.1 Sol через API: от получения ключа до примеров кода.

How to Run an AI Model Locally on Android: Apps and Setup

Инструкция по запуску LLM на смартфоне с Android: выбор приложений, установка моделей и ограничения по памяти.