Add API to batch test model
Provide an API that allows one to send a labeled set of utterances to a model, batch test the model with this set, and return the statistical results, like entity precision and recall, and overall percentage, etc.
Phil
shared this idea
Hello,
You can do this with the batch testing panel. Please see: https://docs.microsoft.com/en-us/azure/cognitive-services/luis/luis-how-to-batch-test.
Thank,s
Luke
9 comments
-
Thanks everyone for your patience. This feature is currently in the works. Keep checking this page for updates:
https://docs.microsoft.com/en-us/azure/cognitive-services/luis/whats-new
-
Guillermo Majchrzak
commented
Agree this has not been resolved, we are currentlly trying to have some kind of regression testing, the use of the GetPrediction method will come with a very high cost. The UI implemented is really useful but without an API not really useful on a CD/CI environment. even more, you cant even download the result to compare side by side
-
Anonymous
commented
This marked resolved however I don’t see a testing api in the lastest release?
The current batch panel has a hard limit of how many entries it can hold. Couple with the ability of the author api able to push new intents but no real way to test
(without going through the Microsoft interface) -
Nathan
commented
Hi Luke,
I wouldn't really regard this as a solution to the OP. The batch testing panel has its uses, but doesn't provide any easily injestable or summarisable summary statistics (precision/recall/F1/etc). We could (and have) reverse engineer the underlying web api the batch testing panel uses and calculate those ourselves. Given it isn't a published API, we couldn't rely on it for production use. As other commenters have mentioned, this really needs to be an official API so that it can be used as a part of training pipelines.
-
Petr Vavro
commented
Luke. Please change the status, as we need an API to do the batch test (I. E. As part od release pipeline in Azure Devops)
-
Anonymous
commented
Luke, we know you can do batch testing from the LUIS portal. The request is to add REST API support for batch testing to LUIS authoring API.
-
Feroz Ahmad
commented
A very important feature for developers that they can leverage for faster experimentation.
-
François
commented
Definitely a KEY feature to be able to monitor progress made through trainings. Key and URGENT too.
-
Bobby Reynolds
commented
We would really love the ability to programmatically batch-test changes to our LUIS model as part of our CI/CD workflow.
