There are controls of course. The container validates that the request is coming from a properly installed gadget, and poorly behaving gadgets can be rate limited or shut off if necessary.
You can also pass certain headers which are awfully useful. For example...
Which would let you do authenticated cross-domain requests.Authorization: OAuth realm="http://sp.example.com/",
(This assumes of course that the gadget can securely store a token for later use. Gadgets can store data securely using the OpenSocial APIs, but since the user at the browser ultimately has full control over the client side environment, this is effectively the same as a client without a very secure secret. A server side signing solution is needed if you want to to beyond more than simple scenarios involving a user looking at their own data. We're already using OAuth with an empty token to let gadgets talk to their home servers securely, so adding this won't be difficult.)