Slowing down and speeding up can both be insults.
They can both be misconstrued.
When you don't go the same pace as your neighbor:
he may think his own pace must be faulty,
or he may think that YOU think his own pace must be faulty,
this brings up feelings of inadequacy, guilt, or anger.