It is possible that the high percentage of mitochondrial RNA UMIs in your 10x Genomics Multiome data could be due to a number of factors. Some possibilities include:
Contamination of the sample with mitochondrial RNA: This could occur if the sample preparation or handling process was not strictly controlled to minimize the introduction of contaminants.
High levels of mitochondrial transcription: It is possible that the cells in your sample have a higher-than-average level of transcription from the mitochondria, which could lead to an increased proportion of mitochondrial RNA in the sequencing data.
Inefficient capture of nuclear RNA: The method you described for isolating nuclei (based on the TST protocol) involves capturing nuclei using magnetic beads coated with a protein called Neutravidin. If the efficiency of this capture step is low, it could lead to a higher proportion of non-nuclear RNA (such as mitochondrial RNA) in the final sample.
It is worth noting that the ATAC data you mentioned looks good, which suggests that the nuclei were successfully isolated and that the high percentage of mitochondrial RNA may not be due to contamination or poor sample preparation. It may be worth further investigating the possibility of high levels of mitochondrial transcription in your cells, or seeking the advice of a molecular biology expert to help identify the cause of the high mitochondrial RNA UMIs.