If I understand correctly, this is a long-standing and known issue. See this thread. In my experience, it happens regardless of the number of items remaining in the list. (I first encountered it years ago in the context of a shopping cart stored as a list of cart items directly on the user.)
Basically, as a best practice, a reference to a thing should always be explicitly removed from list fields before deleting the thing from the database.