I need a bit more information. Is this a recombinant protein that you have expressed and purified? Is this a collection of membrane proteins from a cell homogenate or cell lysate that you wish to concentrate? What are your downstream requirements... what do you want to do with the membrane protein? Do you need it to be active?

If you have detergent present, it should help prevent the membrane protein from sticking to the ultrafiltration membrane. However, it could be that the protein lacks sufficient solubility in the detergent solution you are using to allow it to be concentrated, so that it is precipitating. A different pH, a change in salt concentration, a change of detergent, a different concentration of the detergent, or the addition of a stabilizing agent could help. Stabilizing agents include polyols such as glycerol or trehalose, or a ligand of the protein.

Thaks for your answers. To add more details:

I have recombinant protein expressed and purified via His tag on Ni-NTA column. I'm able to purify this particular protein (it is in the solution of  1% of octyl glucoside), but the concentration is very low, cca nM (the protein is labeled with GFP so the concentration is roughly determined using FCS). I'm trying to reconstitute it into the model vesicles (GUVs). This step is quite inefficient, so I thought the concentration of my protein solution might help. The protein consist of transmembrane domain, GFP and purification tags (His and Strep) as the intracellular domain and a one helix at the extracellular side.

First, the concentration is quite low. The CMC of OG is 0.73% and it is usually recommended to use concentration about 10-times higher than CMC. I would go at least to 3 or 5% (although it's best to try, which is best.

The micelle has size of 8 kDa, so if you use the standard 10 kDa cut-off, you wash it out. Do you have the detergent also in the washing buffer?

How about octyl-thio-glucoside? That should have better properties than OG.

Another way to concentrate the protein is to rebind it to a Ni2+ resin and elute it in a sharp band using a single step at high imidazole concentration.

Did you try to reduce the surface of membrane adding a detergent?

Not knowing the size or properties of your target membrane protein other than it has a single TM domain, is his-tagged, and is fused with GFP… I have a few ideas/comments one might consider.

I have often used TRITON X-114 PHASE PARTITIONING to enrich for membrane associated protein (lipidated and integral membrane proteins), but this may not be a good suggestion for you. Typically to enrich the membrane protein using this method it may require more than 1 TM domain, especially if your target protein is fused to a soluble protein (like you have done). With a large soluble region, some proteins with only a single TM domain can be pulled from the detergent phase into the aqueous phase during partitioning. Also, one typically precipitates the proteins with acetone or methanol to rid them of Triton X-114.

Do you know if your recombinant protein is being overexpressed at a high amount and/or efficiently binding your column?  Is the protein tolerated well in your expression system or being forced into inclusion bodies or degraded? This could be the source of low concentration upon elution, and may only require a few changes in your purification protocol to obtain higher or suitable yields for your downstream applications.

For your consideration, I have attached a Current Protocols in Microbiology article that I wrote a few years ago. It describes the Triton X-114 protocol among other techniques.