We should have simple performance tests in OSRM for each service. It's probably enough to have just one test case and execute it many times, outputting runtime then.

Goal is to catch major performance regressions and automatically have numbers to reason about changes. Not sure if the tests should contribute to failing test suites, maybe with very high ceilings. Variability will be high based on execution environment (hello travisci).

The tests can be done in Cucumber. A very early version is https://github.com/Project-OSRM/osrm-backend/blob/develop/features/stress/launch.feature by @emiltin but this only executes many times, not looks at actual performance.