Installing a a datapack that adds a new custom dimension and running a command such as

/execute in <dimension> run tp @p ~ ~ ~

results in the error "Unknown dimension '<dimension>'". However, that'll successfully teleport the player in both singleplayer and a Vanilla 1.16.1 server.

I've attached a minimal datapack that adds the dimension "custom:test" for easy testing.