Benzodiazepines are widely administered drugs to treat anxiety and insomnia. In addition to tolerance development and abuse liability, their chronic use may cause cognitive impairment and increase the risk for dementia. However, the mechanism by which benzodiazepines might contribute to persistent cognitive decline remains unknown. Here we report that diazepam, a widely prescribed benzodiazepine, impairs the structural plasticity of dendritic spines, causing cognitive impairment in mice. Diazepam induces these deficits via the mitochondrial 18 kDa translocator protein (TSPO), rather than classical γ-aminobutyric acid type A receptors, which alters microglial morphology, and phagocytosis of synaptic material. Collectively, our findings demonstrate a mechanism by which TSPO ligands alter synaptic plasticity and, as a consequence, cause cognitive impairment.