Lookup jakstab, votum, and ROSE. Not sure if they are geared to be included in a malware detection engine. You could use them for experimentation to tune your technique. But for a production engine you would likely need to write something on your own.