I have customized a new algorithm which is giving better result than the basic one. In terms of t test results ,it is giving a p value of 0.10. If I take 10% confidence interval it is marginally significant. My question is how to decide the confidence interval?
It depends on the field of research and what is generally accepted by your peers. In biomedical sciences, the common threshold for rejecting the tested hypothesis is 5%, that is, all hypotheses within the 95% confidence interval are judged are "statistically compatible" with the observed data. Given this convention, your data is not yet sufficient to conclude that your algorithm is better (although the performance in the observed sample may be better, as you say).
You can use your data to calculate a sample size that will be large enough to have a good chance to give a p-value smaller than 0.05 - then you may possibly repeat the experiment and use the new data to demonstrate the superiority of your algorithm with an acceptably high confidence. All you need is an (gu)estimate for the mean difference d and the standard deviation s. The required sample size per group giving you an (approximate) 80% chance of getting a p-value < 0.05 is the n = 16*s/d. I suggest being a bit conservative in estimating d and s from your sample. More about this in DOI:10.1002/9780470377963.
I have tested it on 5 different test systems. Out of 5, 4 are giving a p value of less than 0.05. In one test case it is giving 0.10. Can I conclude that it new algorithm gives significantly better values than basic one?
Are these 5 different test systems a random sample from a statistical population of test systems?
If yes, then you have a sample of n=5. I don't know what the readout is, if it is quantitative or just qualitative. In the simplest case you can just say that one is better than the other, in which case your data is binary. Let's call the "event" here that your algorithm is better. The p-value would then be the probability from a binomial test, p = 2 * 0.5^5 = 0.0625, what is not sufficient to conclude that your algorithm performs better on similar test systems.
If no, then each test system is standing on its own. In this case you did a series of test which can be considered a screening of systems (for which your test is better). You would then use the individual p-values, but you should control the family-wise error rate (e.g. using the Bonferroni-Holm correction). This may lead to the conclusion that your algorithm is better for some system(s).