If you tune the springs at the tension at 4 Volts and run at 12 volts the brushes will float/bounce, arcing and accelerating the wear rate.
What you did was find the optimum for a free running motor at 4 volts. To run at higher RPM (higher voltage) you need more spring pressure.
So you would need to run your experiment at 12 volts/race voltage, which is not advised usually due to destruction of the windings or Comm.
Hence it is best to track tune with various spring pressures which give you the best lap times or best life etc.
Everything I was taught in Engineering school I had already learned from slot cars....