How to return a custom data type in a plugin action?

That appears to me to be what you’ve defined in the Fields section, and that’s an input for the action. When defining Returned values, you should see only Bubble native types.

If you are really really sure that you need to define a custom return type, there is a sophisticated hack, but bear in mind that it is not sanctioned by Bubble and is not for the uninitiated. I would strongly urge you to at least get one normal server-side action working and understand it first.