I don't know exactly what you've meant. See the screenshot above. Done in NX 12. Left is a Solid Punch with front side pierced, right a dimple. Both have on top a Cutout. The 4 "Hem Flanges" are Standard SB Flanges with Bend Angle 179° (180° will not work). On these edges Hem Flange did NOT work. So I think I can confirm. Maybe SB Flange with 179.999° could be a workaround.
Currently Hem is not supported on deform feature edges. You can emulate a hem using SB flange - use a bend radius of 0.004mm.
And as stated above, 180deg is not supported here. You could use two SB Flanges at 90deg each to achieve the perfect 180deg.
First flange: Bend Radius 0.004, Length 0.002
second Flange: Bend RAdius 0.004, Length as desired.
Now if you meant to run a hem along the highlighted edhe in the picture below? That is not yet supported. Please raise an enhancement request if that's what you need.
There is an alternate solution here. It depends on:
In NX12, you can locally convert the rest of the part back so it will unfold. Take care in your internal cutout that the hems do not intersect when flattened.
I just noticed that you want the hem around the inside cutout too. Do you have material that can do that without splitting?