Electroencephalography (EEG) is vital for brain-computer interfaces (BCIs) due to its non-invasive approach and high temporal resolution data capabilities, amid challenges such as data scarcity and the need for extensive labeling. Significant inter-individual variability in EEG signals further limits model generalization. Concurrently, the use of self-supervised pre-training, particularly through masked modeling, is gaining traction in time series analysis to mitigate labeling costs. Although this method involves reconstructing masked signal from unmasked series, random masking can disrupt critical temporal variations, complicating effective representation learning. We thus introduce SSL-MEMI, a novel self-supervised contrastive learning framework for masked EEG motor imagery modeling, integrating Domain Adaptive Alignment (DAA) and Multi-View Temporal-spatial Attention module (MTSA) to effectively handle EEG variability. This framework utilizes manifold-based masking to reconstruct original sequences from masked series, thereby enhancing classification accuracy. When tested on the BCI Competition IV and High Gamma datasets, SSL-MEMI outperforms existing methods, achieving top accuracies and demonstrating superior domain adaptation through reduced Global A-distance scores. This study advances EEG classification and indicates broader applications for self-supervised learning in biomedical signal processing. The source code is available at https://github.com/KunKun-Zhang/SSL-MEMI.git.