You probably need to open port 1433? Or your SA (infodba?) username/password are not correct. You can click the "?" on any of the TEM screens for more information about the fields.
Note: The data directory is where you want the db data files to reside on SERVERDEFAULT and is not TC_DATA.
You can also manually install the Teamcenter database (see Install Guide, Part II, Ch 5, "Create an SQL Server database").
NX 11.0.1.mp01 | SW 2016 | TcUA 11.2.3
I would give a look at the HTTPsocket configuration of your MS SQL Server instance as stated in the documentation.