Mikyla Writing Feedback Accuracy

If we submit the same pigeon story to mikyla three times, the response are noticable different. Is it acceptable? How to explain the difference?

Here are the detailed feedback screenshots.

1st	2nd	3rd

Mikyla is actually an OpenAI ChatGPT-4-powered agent. This AI tirelessly provides personalized writing coaching to each child, assisting parents and teachers but not replacing the essential roles they play in kids’ education. Kids’ handwritten essays are often messy and difficult to read, requiring some guesswork even from humans. Therefore, the differences in AI responses should be acceptable.

Comparison of Three AI Feedback Responses

#

Overall Grade:

#

• First Feedback: Great Work!
• Second Feedback: Great Work!
• Third Feedback: Great Work!

Essay Summary:

#

• First Feedback: The essay talks about pigeons traveling to different cities and the duration of their flights.
• Second Feedback: The essay talks about pigeons, some of which died while others are traveling to different cities like New York and Singapore.
• Third Feedback: The essay describes pigeons traveling to New York and Singapore, highlighting the excitement and differences in travel times.

Detailed Feedback:

#

• First Feedback: Praises the creativity and suggests improvements in spelling and punctuation.
• Second Feedback: Highlights imagination and provides advice on spelling and sentence structure.
• Third Feedback: Commends the interesting story and suggests focusing on spelling and punctuation.

Spelling Mistakes Identified:

#

Writing Correction:

#

• Overall Summary: The writing corrections provided by all three feedbacks are quite similar, as they all focus on correcting spelling mistakes, capitalization errors, and improving sentence clarity. The first and third feedbacks are almost identical in their corrections, both changing “died” to “dead,” “New york” to “New York,” and “singapore” to “Singapore.” The second feedback also makes similar corrections but includes an additional mistake (“dutch” to “Dutch”) not mentioned in the other two feedbacks. Overall, while there are minor differences in the specific corrections listed, the general approach to improving the text is consistent across all three feedbacks.

Analysis of Acceptability

#

1. Algorithm Variability:

   • Different AI assessments may prioritize various aspects of writing. Some might emphasize spelling and grammar corrections, while others focus more on providing feedback on structure and creativity. This variability is normal and acceptable as long as the core feedback helps the student improve. This variance mirrors the differences found in human evaluations, making the feedback feel more personalized and human-like.

2. Comprehensive Feedback:

   • The differences in spelling corrections suggest that using all feedbacks together can provide a more comprehensive review. One feedback might catch errors that the other missed.

3. Encouragement and Constructive Feedback:

   • All feedbacks provide a balance of encouragement and constructive feedback, which is crucial for young learners. The slight differences in wording and emphasis are acceptable as they align with the overall goal of improvement and motivation.

4. Contextual Understanding:

   • The differences might also stem from the context or specific instructions given to the AI when generating the feedback. Ensuring that the context provided to the AI is clear and specific can help in generating more consistent feedback.

Conclusion

#

The differences in the feedback from the three AI-generated responses are acceptable as they complement each other, providing a broader range of corrections and suggestions. The core purpose of encouraging the child and providing constructive feedback is maintained in all, which is the most important aspect.