For some reason the arduino code reverses the speed direction and creates an endless wrong calculation loop.
It seems fixable by reuploading the code, which makes me wonder it might be the raspberry pi pico that's causing this.
Good learning that in the future something weird happens, just re-upload the code.