really its a decision about whether your test measures what you want it to measure. So you need to have a very good idea of the putative construct, there may be many other scales attempting to measure this as well.
Strauss, M. E., & Smith, G. T. (2009). Construct validity: advances in theory and methodology. Annual Review Of Clinical Psychology, 5, 1-25. doi: 10.1146/annurev.clinpsy.032408.153639
and the classic:
Campbell, D. T., & Fiske, D. W. (1959). Convergent and divergent validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
MTMM is complicated but is the best way. However, before investing in such a hude endeavor I would establish inter-item reliability (Cronbach's), then establish discriminate validity with individual difference measures that are "like" the one you created by just using correlations.