On the evaluation of IVD assay, le verification of linearity, precision, LoD, LoQ and bias are crucial. In the verification test of linearity and precision it is not necessary to use reference materials. To estimating the bias, yes, but it is possible to use samples from patients to verifying LoD and LoQ?
To estimate the LoD and LoQ you need to know the analytical signal (e.g. peak area), known concentration / amount of analyte producing this signal and the noise / background of the signal (in the same units as the analytical signal). So, you need to know the concentration / mass of the analyte and thus you should use SRM (standard reference material). This should be in possibly similar concentration range and form to your analysed sample.
http://www.iupac.org/publications/analytical_compendium/Cha18sec437.pdf
ftp://ftp.nist.gov/pub/media/currie-docs/Currie_181.pdf
https://ec.europa.eu/jrc/sites/default/files/eur22756en.pdf