One of the assumption in EFA and PCA regards the fact that items should be measured on a continuous scale.
In my opinion in this case you need to compute the tetrachoric correlation coefficients and then use this as input matrix for the factor analysis.
Alternatively, you can use also a multiple correspondence analysis, item response theory or confirmatory factor analysis employing an estimator suitable for categorical and dichotomous variables.
You don't mention whether it is your manifest or latent variables that are binary, but applying latent variable models to either is fine. Bartholomew et al. (Book Latent Variable Models and Factor Analysis: A Unified Approach
) show show how the models relate to the common factor analysis used assuming metric manifest and latent variables. When the manifest variables are binary, as is the case in much education research, the models are often called item response theory (or item response models). As Gianmaria states, you can also use correspondence analysis and other methods. These don't assume any latent variables and were developed in large part under the group name Gifi (for Galton's servant) (https://books.google.com/books/about/Nonlinear_Multivariate_Analysis.html?id=XkamAAAAIAAJ&source=kp_cover). That is a pretty major change though in how you are conceptualizing your data, so I am assuming that you want to keep with the latent variable models.
There is nothing problematic with estimating a latent variable model with binary indicator as long as you use the correct estimator (WLSMV or DWLS). The first is used by Mplus, the second is used by R / lavaan.
As I've encountered (and finally solved) the same issue, here are other useful references for this case:
Bandalos, D. L. (2014). Relative Performance of Categorical Diagonally Weighted Least Squares and Robust Maximum Likelihood Estimation. Structural Equation Modeling: A Multidisciplinary Journal, 21(1), 102–116. https://doi.org/10.1080/10705511.2014.859510
Flora, D. B., & Curran, P. J. (2004). An Empirical Evaluation of Alternative Methods of Estimation for Confirmatory Factor Analysis With Ordinal Data. Psychological Methods, 9(4), 466–491. https://doi.org/10.1037/1082-989X.9.4.466
Muthén, B. (1978). Contributions to factor analysis of dichotomous variables. Psychometrika, 43(4), 551–560.
Revuelta, J., Maydeu-Olivares, A., & Ximénez, C. (2020). Factor Analysis for Nominal (First Choice) Data. Structural Equation Modeling: A Multidisciplinary Journal, 27(5), 781–797. https://doi.org/10.1080/10705511.2019.1668276
Savalei, V., Bonett, D. G., & Bentler, P. M. (2015). CFA with binary variables in small samples: A comparison of two methods. Frontiers in Psychology, 5. https://doi.org/10.3389/fpsyg.2014.01515