I checked and it seems that HTTP referrer can be easily forged.
Even if you have it configured to only accept API requests from A.com,
If someone is familiar with the program, he/she can forge the HTTP referrer to make it look like the API request is coming from A.com, even though he/she is accessing from B.com.
It seems that the API can be used fraudulently in that way.