Many of my customers are in an industry with a huge C++ code base and it's all under active development. Safety certification requirements are onerous and lead-times for development are long: many are now experimenting with C++17 and C++20 is on the long-term horizon but not yet a requirement. Because of the safety certification requirements and the fact that the expected lifecycle of the software is the order of decades after their products have been released, changing any lines of their code for any reason is always risky. Lives can be at stake.
But this is a multi-billion-dollar industry. If you're working on scripting a little browser "app" for a phone things may be different.
Is there a lot of manual work for getting the new certificate? E.g. is human rewiewing the code? If not, someone should build CI pipeline for the certification process.
Hundreds of hours of manual testing. I don't have to do safety certificates, but my code gets 500 hours of manual testing (I'm not allowed to give real numbers, these numbers are close enough) - they find enough critical can't ship issues where the fix is risky enough to start all over that we typically are doing 2500 hours of manual testing. on every release.
We have a large automated test suite that runs on every build and takes hours. The problem with automated tests is they only verify situations you thought of work the way you think they should, while human testers find slight variations of setup that you wouldn't think matter until they do. Human tests also find cases where the way you expect things to work don't make sense in the real world.
Wait until you find out about the cat test. It found a failure mode no human had thought of. No amount of the developer claiming a test like that was not fair was enough to invalidate the results. No actual cats were harmed but treats may have been given.
Do you have more context? I'm having trouble googling what you're referencing.
Simulate a cat walking on the keyboard to handle weird inputs?
Isn't that just fuzzing? I thought maybe there was a specific thing called the cat test.