Data and ML testing as a service - Hypervector

Long-time lurker and listener who's finally decided to start work on his own side project - so hi! I'm a data and machine learning engineer from Glasgow, Scotland - and I've began trying to validate an idea I've had whilst working in my field.

Shipping data-intensive and machine learning-enabled applications and services to production is infamously challenging at almost every stage of the process. My experience working in the field suggests to me that there's a potential requirement for useful tooling to be built to provide a means of automatically testing and validating services that rely heavily on data input, in a similar way to other continuous integration and deployment pipelines.

Take a machine learning model deployed as a microservice that is tasked with predicting a price output based on some number of (potentially high dimensional) input features. At present, the build-deploy process for such an artefact might rely on a data scientist or engineer defining a handful of exemplar input cases with an expected price output for each. This can be a difficult process to gain confidence in as selection of these examples are often taken from the training or model test data cases, which can potentially introduce bias ("it's worked for every case we've seen so far!"), plus the maintenance of these production-facing tests as the model develops can be an overhead that gets neglected.

Furthermore, the input space to a machine learning model can be extremely broad for the case of high-dimensional input spaces - and getting "test coverage" in the same way as more traditional unit or integration tests have is more of a statistical than rigorously-defined problem space.

Hypervector (http://hypervector.io) looks to address this problem by introducing smart on-demand testing for data-intensive applications and microservices. At its core, it provides generated payloads of valid data features created via a user-defined scope. The user builds a configuration that descibes the input space of their feature vector under test, and Hypervector automatically pushes a raft of test cases that exercise the model input space over a statistically defined set of bounds and parameters - totally independent of the machine learning experimentation and development process. Imagine Postman's Newman feature with random data, but with a specific focus on generating feature vector inputs to test data services with often empirically-developed features like ML artefacts.

I'm trying to take a market-first approach to this, and haven't wrote a line of code as yet (other than the home-made tools I've put together over the years to try and address this problem in my day job). My first aim is to get feedback from the landing page I've thrown together from both domain experts (so ML engineers, data scientists) and the start-up community more generally (which is why I'm posting it here too!), and hopefully begin speaking to folks about whether this would be a useful tool to build or not.

Any and all feedback highly appreciated.