In KNIME, there are nodes available for generating fingerprints (choose a proper one) and creating similarity matrix (choose a proper measure of similarity), which can be clustered (choose a proper algorithm).
Hi for ongeneric series of molecules for 3D QSAR , generate conformations of the dataset and aligned all the ligand then generate atom-distance or surface-based matrix to represent the 3D shape of the molecule. In final step calculate the similarity index by using Tanimoto or Manhatten approach