There is a OOTB handler 'epm-display-form'.. can we achieve the same behavior (open/display form) using ITK call in custom handler..
I need to display form based on some conditions thus can't use OOTB handler itself
You can't do that. Although it's the ITK handler that appears to display the form, display form (which is GUI work) is never done in ITK (ITK does not have GUI library). That would never work - consider where handler runs in 4-tier environment....
It's not the handler itself, but actually the presence of the handler EPM-display-form in the workflow that prompts the Teamcenter client to display form based on handler argument at appropriate times.
Your best option is to use conditional tasks, based on conditions that you wanted to use to display form, to route workflow to different workflow tasks using different handler configuration of EPM-display-form.