I want to develop a Markov decision process Model where risk is factor in model formulation.
Hello Kizito.
Given your question, I think your doubt lies on the double stochastic layer of a HMM.
...The first layer (hidden) is just an ordinary Markov Chain. Its outputs in time (transient states) will be used as probabilities for its emissions in time (visible states).
One possible approach is to use "risk" as probabilities for the emissions according to your initial setup.
Next, I leave some useful links so you can go further.
A good page to understand the basics (also they have some code): http://www.shokhirev.com/nikolai/abc/alg/hmm/hmm.html
A good book with practical examples: https://books.google.fr/books?id=LDDzvCsdVs8C&printsec=frontcover&dq=hmm+book&hl=fr&sa=X&redir_esc=y#v=onepage&q=hmm%20book&f=false
A good way to create your models and use then in practice: R packages: "HMM", hmm.discn, HiddenMarkov