Many of the language communities that I interact with now are fond of asserting that their particular community does a wonderful job of testing, documentation, or both. Most languages come with very good testing frameworks, and have libraries (or modules, gems, or crates) for even more. (Going forward I’m just going to use the term library, mentally replace with the term used in your language.)
Most of my experience with newer languages and their emphasis on testing involves Ruby and Rust, with a little Java and JavaScript on the side. These languages claim that testing is part of the culture of the language. The supporters of these languages point to testing support in the language. They point to documentation on testing the libraries associated with the language.
Some History
The proponents of some of these languages use their testing culture as one of the arguments why they are better than the languages that came before. To be fair, back when I was a C programmer, most of us did not do automated testing. The same was true when I was a Fortran programmer, and when I programmed in Forth.
One day, I began programming with a language that was very different than those languages. It did have a strong suite of tests for the language, although most of us users never saw it. The same framework that supported the language tests was available to library authors. The community encouraged library authors to provide a unit test suite with the library. The tooling around the package management system made certain the tests were run as part of installation to be certain the library would work in your environment.
A System to Learn From
The language was, of course, Perl 5. The CPAN repository stores the publicly accessible modules. If you were to browse around the modules, you will find that almost every one has a sizable test suite. Many of these tests are built on the venerable TAP system for tests. If Perl’s testing stopped there, other languages could shrug and move on.
However, the Perl 5 community was not finished, yet. Perl 5 was supported on a number of OSes and a large number of Perl 5 versions are always supported and in use. So, some volunteers in the Perl community built the CPAN-Testers system. Using processing time donated by the community and various companies, this service tests the modules on CPAN against every supported version of Perl 5, on every supported OS. Any person that wants to can set up a machine to run the testing code. Some people set up dedicated machines that run many different versions of Perl 5. Others use downtime on their machines to run the tests.
The result is a huge compatibility matrix that is available to help people who want to know if a module is supported on their OS/Perl version combination. As an added feature, when a module author uploads a new version of a module to CPAN, the system emails the user reporting whether or not the module passed its test suite across all of the supported Perl versions and OSes. It’s up to the author to decide if it is worth the effort needed to fix any incompatibilities.
But, once again, the Perl community would not leave well-enough alone. Even with good testing tools, it can be hard to get developers to add the testing if they don’t see a benefit. After evangelizing writing good test suites for a while, some volunteers in the community built the CPANTS system to provide a bit of gamification for good attributes the community would like to see in a module. They call their measure Kwalitee. It’s not the same as quality, but it’s kind of similar.
Other Languages
I would really like to see other languages learn some lessons from the Perl 5 ecosystem. A system that does basically continuous testing on all modules for all supported language versions on all supported OSes is an amazing resource. Not all module have OS dependencies, but some do. Being able to tell which language versions will and won’t work with your library allows you to explicitly decide when you want to drop support for an older version.
I know many people will scoff at the possibility of learning anything from Perl in any form. That would be a mistake. Perl 5 (the language and the community) is battle-hardened. Like a grizzled old soldier it has a lot to teach the young-uns. Not all of its lessons will apply to all languages, but it is pretty silly not to learn from the experience..