Is this OK?
This will not damage the device.
Note, however, that the output pin may become high level if it is open.
I imagine that the switch time in your case is short, so there shouldn't be a problem.
However, whether this brief open state will affect the operation of the device or not depends on the usage conditions.
Therefore, check the effect of this open state under your actual usage conditions.