I'm trying to optimize a power system that has PV and Wind energy, Unfortunately I couldn't find any paper that could have a formula for the cost of PV and Wind generation.
You can refer to paper published in Elsevier journals like (Energy, Energy & Building, Energy Conversion Management, Renewable Energy, Renewable Energy & Sustainable Energy Reviews). These journals have many papers related to hybrid combination of PV & Wind.
Few more papers related to HOMER tool can be useful for you...!!
In my new book, I have used EIA's data to derive the life cycle costs of energy for a range of technologies. I also devoted a chapter showing the formula and the applications of LCOE to economic evaluation and policy setting.
Data Preface: Energy Investments: An adaptive approach to profiti...
There are various types of cost that renewable energies are dealt with it. the most famous three of which are: Net Present Cost (NPC) or Net Present Value (NPV), Levelized Cost Of Energy (COE), and Total Annualized Cost (TAC). the firs one relates to long-term economic feasibility of a project, the second one is the price of producing each kWh energy by your renewable system, and the third one is all cost in one year. According to this, first have look at doi:10.1016/j.solener.2011.10.016 to find the best criterion, then search in google scholar with (your cost criterion-for example NPC ) plus (your renewable resources-for example PV).