The compiledPlugins map used to be a global variable, which can be
problematic as we move to independent test suites, since those test
suites run in the same process space, the global variable could point to
now deleted plugin versions/paths in separate suites, which would make
tests fail with random errors.
To avoid this, the map is now scoped to the test suite, and a new copy
is created lazily if used by the test suite.