There’s probably two things you should worry about.
Small older phone screens (375 px width) that can squash text etc, and tablets in landscape mode
Tablets have gotten larger over time, and with certain devices having 13 inch screens (which is basically a small laptop screen) You may need to have some sizing changes on some elements. Certain UI components optimized for phones can also just feel weird on larger screens. Go download an app not designed for tablets (instagram I believe does not have good tablet support) and see how awkward it can get
So to answer your question, if you want tablets to be involved in your mobile app, then I would not ignore it. If your app has pretty locked down width, and you’re okay with white space on either side of your main content area, then you can just work around that. BUT…
With a lot of computer manufactures moving towards ARM processors, the gap between mobile, desktop, and web will get closer and closer. If I was you, I would account for responsiveness. It will be more painful to have to go back and turn everything responsive if you change your mind