For example, I have a regression model like E=aC+Ɛ. in the regression, Ɛ is the error of using the equation to calculate E from C. a= constant factor for estimating E from C. I want to incorporate this regression model into Joint probability concept to derive the Joint PDF of E and C given that I have few data points of E and C.
I think you should look the book “Bayesian Data Analysis” (2nd or 3rd edition) of Guelman et al.. It is really easy to follow and nice examples are discussed. You have a whole part in regression models that include Hierarchical and Generalize linear models. Please check it up the book’s homepage:
http://www.stat.columbia.edu/~gelman/book/
When you say “I have a regression model like E=aC+Ɛ.”, is C a random variable also? Or you mean E|c = a*c+ Ɛ, because this last exemple is just a case of a simple linear regression.
If E and C are normal-distributed variables, and *not* parameters (it's a huge difference!), then this is the simplest case of linear regression. Please remember that with bayesian modeling you are concerned with parameter distributions, and not variable distributions.
In your case, the parameter in question is a.
Ɛ is just the error term, and can be derived from the distribution of a. So, in your case, there is no 'joint parameter distribution': there is just the pdf of a.
Try to model a's distribution starting from your datapoints of E and C, and look at its HDI. You will then know how uncertain you are about your estimate of a.
Your estimate a=.303, which I presume you derived with a simple linear model, is a point estimate and represents the most probable value for a. Using "point estimates", however, is a frequentist approach. The very reason for using a bayesian model is getting away from point estimates and having a clearer view of the amount of uncertainty of estimates (so to speak - and if you have just "a few data points", uncertainty is probably high).
In line with Rui Borges, I suggest you to check out an introductory book on bayesian statistics. I never read Gelman's, so I suggest you Kruschke's "Doing Bayesian Data Analysis", which is a true pleasure to read and which explains very clearly the difference between modeling variables and parameters distributions.
If my understanding of your question is correct, a construction of the joint distribution of E and C can be implemented with copulas, Alternatively, you may consider the regression parameters a and Ɛ as random variables; thus, the joint distribution a and Ɛ can be modelled through copulas.
Your question here is pretty similar with the story in soil mechanics, for the paired shear strength variables, such as cohesion and friction angle. They are regression parameters of the Mohr-Coulomb strength criterion, so they are correlated in physics. The uncertainties between tau (shear stress; i.e., E here) and sigma (normal stress; i.e., C here) can be described with a copula successfully, as described in my recent publication