If you want the protein to stay inside, dialysis membrane with MWCO less than or equal to 30,000 Da should be satisfactory.
If you want the protein to pass through the membrane, I don't think there is any such dialysis membrane that will allow the protein to pass through at a reasonable rate. There is ultrafiltration membrane with a MWCO of 300,000 Da that would probably allow the protein through (assuming it's monomeric), but it might not be very efficient at doing so.
For in-vitro drug release studies involving a protein with a molecular weight of 117 kDa, you would typically use a dialysis bag with a Molecular Weight Cut-Off (MWCO) slightly lower than the protein's molecular weight. For a 117 kDa protein, an MWCO in the range of 100 kDa to 120 kDa is often recommended. This allows the retention of the protein while enabling the diffusion of smaller molecules.
Besides using membranes to separate the released protein from the formulation, which might not be very efficient, even with a 300 kDa MWCO ultrafiltration membrane, another approach to consider is centrifugation. If the formulation can be pelleted in a centrifuge or ultracentrifuge, leaving the released protein in the supernatant, you could measure the protein concentration in the supernatant to measure its release.