Considering that the objective of the decision support system is to provide diagnosis based on images of dental region, it can be thought of as a supervised classifier, taking the image (its pixels) as inputs and suggesting certain steps and how strongly they are suggested.
Step 1. Collect real images of dental patients and corresponding diagnosis (the steps taken) by a real dentist and label images accordingly. Images may be processed using certain sub-steps depending on specific objective.
Step 2. Split data set into training and testing data and train classifier on training data. Most effective classifier in this case would be a Convolutional Neural Network. For implementations, please see Keras and TensorFlow.
Step 3. Evaluate performance of the classifier by using testing data and and fine-tune the model to reduce error and increase accuracy.
Please note that the problem definition/objective will affect the aforementioned steps. For example, a system for detection of location of cavities in dental images may require a You Only Look Once network.
Its great idea. As Ahmed mentioned you can start with passion in image processing by collecting raw data followed by processing of the same. Then based on data you can create classification data set as a diseased or not as label. Then slowly you can make still more interpretation and build decision support system. Its really exhaustive and challenging work that requires knowledge about dentistry . Blooming trend is multi disciplinary research. All the Best.