The next/previous links are core WordPress functionality and don’t provide the default functionality of linking within a category only.
The core calls are in the single.php template, which can be customized using a child theme.
Edit the core previous_post_link() and next_post_link() calls.
Hiding the links is even simpler with a bit of CSS: