I have modified a cryptographic algorithm, and now I would like to analyze it. Can anyone suggest a best tool that can be used? (If possible open software).
There are not really automated analysis tools, as the domain of crypto algorithm analysis is huge. There are some symbolic analysis tools for specific problems. For educational tool that may help you see cryptool: https://www.cryptool.org/en/
If the algorithm can be written in Matlab, it is easy to test the strength of the encryption algorithm using images. The histogram analysis, adjacent pixel correlation analysis, mean value analysis and key space analysis. can be obtained with the encrypted image. Also we can obtain encryption quality, encryption speed with different images of same and different sizes. Then the results can be compared with any other encryption algorithm. It requires that the algorithms be written in Matlab.
To start with we must consider a standard encryption algorithm such as AES as a bench mark. AES has been used world-wide as an encryption standard that has not been reported being cracked by any adversary till date. Now we can compare the results obtained in AES as well as the encryption algorithm under consideration in terms of the above analysis. We can say that the analysis shows that the algorithm is able to achieve such desirable characteristics similar to AES. These analysis are only qualitative and not quantitative.
Maybe, the best way to analyse your algorithm is to submit it to be analysed by the community. Given the high complexity involved, until I know, there are no automated software able to analyse.
On the other hand, if you just want to analyse some specific metric on it (e.g.: cipher speed), you can use, as Paul said, an well known algorithm as a parameter.
There is no general tool for analyzing the strength of. any cryptographic algorithm
There are some tools for analyzing some properties of those algorithms, but they depend on the type of algorithm.
To analyze the properties of the S-boxes in a block cipher I suggest the SET software.
To analyze the statistical properties of the output of a stream cipher I suggest the statistical test package proposed as standard by the NIST. You must carefully interpret the results.
The NIST battery only studies the randomness of the output, but not the possible correlations between the output and an element of the internal state
E. J. M. Capó, O. J. Cuellar, C. M. L. Pérez and G. S. Gómez, "Evaluation of input — output statistical dependence PRNGs by SAC," 2016 International Conference on Software Process Improvement (CIMPS), 2016, pp. 1-6, doi: 10.1109/CIMPS.2016.7802810.