I found that an organ contains an enzyme that can deacetylate N-acetylserotonin, but I cannot identify it. Does anyone have suggestions for identifying the enzyme?
If you have access to a lab stocked with standard molecular biology equipment, my suggestion would be a zymogram.
Prepare a small amount of tissue homogenate and separate the total protein content by SDS-PAGE, using a relevant concentration of N-acetylserotonin in the gel matrix. Wash the gel to elute the SDS and attempt to refold the proteins, and then the gel can be stained in anything that will bind to/detect the secondary amines in N-acetylserotonin (I would try ninhydrin, but you can find a list of many here: https://www.chemistry.mcmaster.ca/adronov/resources/Stains_for_Developing_TLC_Plates.pdf). You should get a zone of clearing where the band(s) responsible for the activity is located in the gel. From here, you can excise the band(s) and perform an in-gel tryptic digest. The resulting peptides can be identified by LC-MS and matched to proteomic databases to identify the protein(s). Many core facilities or MS-proteomics companies can provide this as a service.
In case the enzyme is irreversibly denatured by SDS-PAGE in the method suggested above, you could attempt to isolate the enzyme by purification on chromatography columns, using your assay to locate the fractions containing enzyme. It is difficult to say how challenging such an approach will be as you know nothing about the abundance of the enzyme in crude extracts. However, if you can get to a stage where you have a single band on a gel, or a band that is the only one whose intensity across column fractions clearly correlates with enzyme activity, then you can again use fragmentation and sequencing approach.
In addition to the excellent suggestions above, look up the substrate in BRENDA (https://www.brenda-enzymes.org/). A quick search shows at least 5 potential candidates. By cross-referencing the organism, you may be able to determine the enzyme's sequence and size.
N-acetyl serotonin appears to be the substrate for several reactions (EC numbers 2.1.1.4, 2.1.1.68, 2.4.1.17, 2.8.2.9, 3.5.1.76).
However, there is a possibility that your enzyme is actually 2.3.1.5 or 2.3.1.87, assuming it "prefers" deacetylation.
If you look up the EC numbers, you can find a lot of info, including links to relevant papers and links to uniprot, where you can find sequences and sometimes subcellular localization.