Transforming LLM Efficiency: A DSP-Inspired Semantic Vocoder Approach

Hey everyone,

I’ve been experimenting with bridging Digital Signal Processing (DSP) multirate theory and discrete text generation. I wanted to share a PyTorch reference architecture I've been working on, mostly to get feedback from folks who work on hierarchical modeling or continuous-to-discrete latent spaces.

The premise: Standard dense LLMs treat text generation as a flat sequence. Predicting the "e" in "the" costs the same attention compute as calculating the crux of a logical argument. In audio (TTS), it is standard to decouple this: a model generates a slow-rate continuous signal (e.g., a mel-spectrogram), and a high-rate vocoder (e.g., WaveNet) synthesizes the discrete audio samples.

I built a dual-rate architecture to do exactly this for BPE tokens.

How it works:

1. The Planner (Slow Rate): A sentence-level autoregressive transformer. It operates on highly compressed semantic embeddings (via a frozen SentenceTransformer) and predicts the next sentence's continuous embedding.
2. The Vocoder (Fast Rate): An autoregressive GPT that handles high-frequency local grammar. It uses a banded sliding-window causal mask to restrict attention strictly to the local context.
3. The Bridge: The slow-rate semantic timeline is step-repeated (upsampled) to exactly match the discrete BPE token boundaries. A late-stage adapter uses cross-attention to calculate a "delta" probability distribution, adjusting the base GPT's logits right before the softmax: Logits_final = Logits_base + softplus(alpha) * Logits_delta.

The Results : Testing on TinyStories, the decoupled architecture converges much faster and deeper than a standard unconditioned baseline GPT of equivalent size (validation loss hit 0.61 vs the baseline's 2.37 at similar steps).

However, I hit some interesting architectural bottlenecks that I've documented in the repo:

• Conditioning Over-Reliance: The adapter transmits the semantic signal too efficiently. The base GPT gets lazy and starts using the 384D semantic vector as a hash-key for the sentence rather than learning robust local grammar. Even with 15% Semantic Dropout (Classifier-Free Guidance), the Top-1 accuracy sits artificially high (~85%), which can lead to exposure bias and repetitive loops during greedy decoding.
• Hardware vs. Logical Complexity: Logically, the attention splits to O((N/C)^2) for the planner and O(N*W) for the vocoder. However, since my reference implementation uses standard PyTorch boolean masking for the sliding window, it still allocates the full NxN matrix under the hood. True VRAM savings will require swapping the base blocks for FlashAttention-2 block-sparse masks.

This is an exploratory proof-of-concept, not a SOTA claim against monolithic multi-billion parameter models. But I think the residual logit delta and the continuous-to-discrete phase alignment provide an interesting alternative to standard prefix-tuning or deep cross-attention.

Code & Architecture Diagrams: https://github.com/eladwf/topdown-semantic-vocoder

Would love to hear if anyone has successfully stabilized similar hierarchical text models, or has suggestions for applying more aggressive continuous noise injection to fix the vocoder's exposure bias!