Until sometime back, fuzz testing was pretty much manual operation. Passing random data as input to check how target system reacts is one effective way to identify if the system is having flaws that may go unnoticed and creep their way into release models.
But how much of data is enough to test the system's intended functionality? Can there be a sort of data left out that could make the system act in a bizarre way? Fuzz testing conventionally have limitations of which, constrained dataset for testing the model is a major challenge. More importantly, with growing complications of a contemporary system, quality data matters even more.
With growing features in automotives, the major challenge pertains to testing such high end models to greater extent such that root cause of each fault can be identified effectively. This is ascertained to be possible only when there are large number of dynamic yet relevant test cases performed on the system and test results are closely observed or tested in iterative way to reach the exact cause of identified trouble. Building such large amount of test cases covering diverse features and parameters relevant to an automotive and that too in a short time, let alone in real-time i.e., on field can be a really tough job.
Thanks to AI, Fuzz testing has evolved and become intelligent enough to test a system in a more efficient manner. AI helps in solving the aforementioned problem of datasets. We all know, how difficult it can be to get large number of data especially test cases no amount how much of permutation and combination one can put in. AI on the other hand with the help two fantastic neural networks - General Adversarial Network (GAN), also known as Generator and a Discriminator network, are able to generate synthetic data that are not superficial and very much the same as any real test case data.
While I will be covering Generator and Discriminator separately in another blog, just for quick understanding, consider a tug of war between Generator and Discriminator and their objective as a whole system is to be powerful enough to retain the balance! While Generator generates synthetic data, Discriminator acts like the Devil's advocate and checks the accuracy of these synthetic data with that of limited real time data. The synthetic data is refined iteratively to match the real - data until it becomes difficult to distinguish between real data and synthetic data, as if they succeeded Turing's test! (Wait for another blog entry for this😏)
Once an abundance of such test data are generated which are like real data in terms of their features and values, these are basically sanitized data. And these data need to be used for training the AI powered Fuzzer model.
Just like any other training process of an AI model, the sanitized data is provided as input to the fuzzer algorithm and based on various features, labels, hyperparameters, and training iterations, the algorithm is shaped into a test case prediction model.
Now this smart fuzzer with ample of training is able to make really smart and fantastic predictions and classifies unlabeled data as passed or failed.
But the story doesn't end here. Its true power is measured only with its action in real time scenario. This fuzzer in field generates Fuzzing test cases for CAN bus based messages in an automotive. Just so you know, Controller Area Network (CAN) is used for communicating information throughout a modern vehicle.
For every test case, the fuzzer traces back to determine what passed and what failed and collects this information for further prediction. Basically, the Fuzzer evolves with dynamic test cases and results in order to identify newer faults and their root cause.
Sounds amazing! Right?
But what can go wrong if the training data is messed up? Or the test data is tweaked just a little to confuse the model? Or queried enough to determine the training data and features used?Or tested too much to reveal the AI model in the Fuzzer, as a whole?
Fuzzer can go crazy and bias the results. For safety critical features, the model can cause some serious trouble.
So before deploying it on the field the Fuzz test model needs some rigorous testing. Not just for data and models, but other strategic measures to prevent it from failing its designated purpose.
Can you think of some strategic ways to secure the Fuzz Testing Model?
Reference:
https://www.escrypt.com/en/news-events/ai-enhanced-fuzz-testing-can-bus