The research introduces MHA2MLA, a novel fine-tuning framework designed to adapt existing MHA-based language models to the more efficient MLA architecture. MLA achieves economical inference by compressing the key-value (KV) cache. MHA2MLA employs partial RoPE and low-rank approximation techniques to minimize performance degradation during the adaptation. Experiments demonstrate that MHA2MLA, requiring only a fraction of the original training data, significantly reduces KV cache size while preserving performance in commonsense reasoning and long-context tasks. The study further shows MHA2MLA is compatible with quantization techniques, offering compound efficiency gains. Ablation studies explore different RoPE removal strategies and SVD methods to optimize performance.