you can look for cell attachment, short term (24-48 hr) toxicity using a kit like WST or alamar blue. Over longer term, for example a week, you could look at cell proliferation perhaps using DNA content.
There are quite a few different assays you can do and a lot will depend on the purpose of the material. One main thing to realise is sometimes apoptosis (programmed cell death) can take time, so only looking for viability after 24 hours is not long enough.
To expand on Martin's excellent point about apoptosis, there are many types of indicators of what a substance may do or not do to cells as far as biocompatibility. Apoptosis, necrosis, do cells attach, do they change shape? (cell morphology). do they change the cell metabolism even if they do not kill the cell quickly? and of course cell culture is very important place to start but 3D cultures and in vivo also have their place ultimately too.
Also key is not only if the substance is toxic or not, but is it stable or changing over time? we are seeing this now with long-term implants and so while toxicity or not of course is very important, I think we also need to try to understand what the body (cells) are doing with the material-- (ignoring it, breaking it down, changing it?)
Attachment, proliferation (XTT, DNA content, WST1) and cytotoxicity (e.g.LDH-release) are prerequisites. If you want to characterize the material further gene expression studies of markergenes (monolayer cells vs. cells on biomatrial) for your intended cell type would be a fine thing to evaluate not only obvious phenotypic changes but also fine changes in expressions patterns.
Depending on your cell type, the timeframe can be significantly different. I was working with osteoblasts and biomaterials and we usually monitored our cells over 21 days. Thats the timeframe until they have fully matured.
What structure has your material? is it porous? is it likely that the cells will invade the material. you could get really pretty pics with SEM to show this.
is it resorbable? In this case i would check on metabolic products of resorption.
The approach for selecting a biomaterial for medical application is a little different. It starts with the application in mind and based on the the nature and the duration of contact with body, different in vitro and in vivo tests are done. They are cytotoxicity - direct and indirect (we use L929 fibroblast cells) , sensitization (to study allergenicity) , systemic toxicity (to study toxic leachants, extracts are injected iv into mouse and the effect is noted), sub chronic toxicity , genotixicity (using AMES tests or in vivo tests) , tissue response to implantation( tissue of final application is preferred), haemocompatibility (materials for blood contacting devices such as vascular grafts, heart valves etc), chronic toxicity, carcinogenicity, reproductive/developmental toxicity, and study on biodegradation.
International substandard ISO 10993 (parts 1-20) gives all the details. In our setup, for a preliminary screening cytotoxicity (with direct contact and with extracts (indirect contact) and haemolysis test are done initially and if the material is good further tests based on the above criteria are done.
One important point to remember is a 'biomaterial' used in the body needs to be both 'fit' and 'suitable' - for the given area where the particular 'biomaterial' in question will be used.