It's definitely tricky with server side pagination.
Since the built-in filters aren't fully compatible, you could create your own filter options & apply them to your query (note for SQL, the filters need to be applied to the total count query & the table query):
Here's another example (but note the risks of disabling prepared statements)