I've overcome sensor measurement noise by an analog low-pass filter and filtering out the measurement noise. We built a sixth order bessel low-pass filter. You can get some help on how to design low-pass filters from Texas Instrument data sheets (see the link below). Then, design and simulate the filter in modelling softwares such as Multisim. Fine tune the components and then build the circuit. Let me know, if you are not familiar with Multisim. I might be able to help. http://www.ti.com.cn/cn/lit/an/sloa049b/sloa049b.pdf
After looking at the power spectral density of the noise, you can also include discrete-time filter in DSP. A guideline is http://nl.mathworks.com/discovery/filter-design.html.
If the PSD of the noise is reasonably high near the frequency band where you want to measure structural vibration response, then it is better to investigate how to place the piezoelectric sensor.
What you are describing, likely is the so called pyro-electric effect, i.e. that the piezoelement is stressed and reacts from thermal loads.
Pyro-electric signals usually add a low frequency rumble that can be mitigated using a RC circuit as a High Pass filter after the piezocharge signal has been ampified. Usual accelerometers use a HP of 0.25 to 3 Hz depending on how they are used, while seismic accelerometers intended for lab use may go as low as 0.05 Hz.
This very much is standard business, e.g. for piezoelectric accelerometers and is well described in the below links.
http://www.pcb.com/TechSupport/tech_gen
http://www.pcb.com/TechSupport/tech_signal
Last, if what you measure on is linear, you can suppress noise by averaging your signals.
Before we purchased a dedicated data acquisition system PAQ (http://www.ec-systems.pl/en/paq-16000d.html) I used in the past charge amplifiers and simple conditioning systems with high-pass filter.
The key issue is also the electrical capacity of both the transducer and cable - the bigger it is, the more noise it can generate in the signal.
In laboratory conditions, you can also try to shield the sensor with aluminum foil.
What is your shaker/exciter? PZT or you are using "enviroment excitation" for Operational Modal Analisys (OMA)?
I belive that you can to obtain very good results using only a correct window, according to your signal excitation. The "average number" its very usefull to elimite the non-correlated (noise) measures. Also, you can perform a short investigation to define the better estimator (H1, H2 ou Hv).
All this has an impact on measurement quality.
Using PZT as actuator/exciter, I had a lot of problems due to the contact between piezoelectric ceramic (PZT) and the plate, but using a correct adhesive it possible to improve around 90% (in my case).