Use of AI for Marking

Practical AI In the Classroom Series

By Miroslav Belik

How can AI be used for Marking and Grading

One of the most daunting tasks for teachers, particularly when faced with a large class and identical topics, is the marking or grading of reports and essays. This task is widely recognized as not only tedious but also incredibly time-consuming, often requiring additional hours of work outside the classroom, typically conducted at home. The pursuit of efficiency and consistency in this process led to the exploration of various tools, with ChatGPT emerging as the superior option due to its ability to deliver the most consistent results, unlike alternatives such as Geminin.

The value of using AI in this context cannot be overstated. Imagine the possibility of automating the grading process, where AI could instantly evaluate assignments. This idea has led me towards the development of a specific prompt tailored for a TAFE TPC English unit, which has shown promising results and possesses the potential to be adapted to a wide range of marking and grading scenarios across different subjects. On average Australian teachers spend 4.9 hours per week marking, so this would lead to a significant time saving. https://www.acer.org/au/discover/article/snapshot-of-australian-teachers-and-principals

The aim was to create a prompt that not only simplified the marking process but also aligns closely with the standards and expectations set by educators. By doing so, this approach can significantly reduce the hours spent by teachers on marking, allowing them to redirect their focus towards more interactive and enriching educational activities with their students.

This practical application of AI represents a step towards using AI in educational settings, offering a practical solution to one of the most labour-intensive aspects of teaching.

How does it work

The prompt was developed over approximately 8 hours. It may seem like a long time but I just could not get it to work properly until I arrived at this version, I received inconsistent grades that often diverged from the teacher's marks. The prompt is truncated or compressed, as I don't use full sentences and use words I want the tool to pay particular attention to. I also introduced a consideration section to the prompt, due to the somewhat unusual marking criteria, allowing it to distinguish between different levels of marks. This final step made the prompt work constantly, i think due to its humanising effect.

Considerations:?remember you can assign a 0, 2, 4 mark if you think its between the other marks, not many people are perfect, if you think its 3.5 for example make it a 4 or give the half mark.?

Additionally, I included guidance that full marks should be reserved, and that not everyone is perfect, to prevent it from consistently awarding full marks. If any part of this is unclear, please let me know, and I'll attempt to clarify the rationale.

The Prompt

You: relaxed English teacher, degree in English, specialised skills assessing English essays in TPC course ?

Need: evaluate forthcoming essay using criteria provided below ?

Format: perform detailed assessment, assign a score out of 25 based on criteria convert this mark out of 50 ?

Musts: Australian English ?

Considerations:?remember you can assign a 0, 2, 4 mark if you think its between the other marks, not many people are perfect, if you think its 3.5 for example make it a 4 or give the half mark.?

Don'ts: use my info, realm, delve, voila, embark, game-changer, transformative, embrace, tapestry, unlock, unleash ?

Audience: TAFE student, teacher Example: use a simple table ?

Steps: Before commencing the marking, please pause to gather your thoughts, and then proceed methodically through the task. I will supply the marking criteria below and then wait for the actual essay to be sent to you. ?

Following the submission of the marking criteria and the essay, the output generated not only provided the scores but also included comments and a summarising statement. This feature allows teachers to quickly review the AI's assessment and use it as constructive feedback for students, streamlining the feedback process.

Based on the provided essay, "Fast Fashion, Helpful or Harmful?" and the marking criteria, let's proceed with the evaluation:

1. Determines Topic and Scope of Research

• Score: 4.5
• Justification: The essay identifies the research topic clearly and discusses the scope of research with detailed information from a wide range of sources, including both primary and secondary. The bibliography includes multiple sources, demonstrating extensive research. However, some references lack a detailed analysis of their credibility, slightly reducing the score.

2. Drafts Text Considering Structure, Language, Register, and Tone

• Score: 4
• Justification: The essay is well-structured and presents information coherently with appropriate academic language, tone, and register. The mechanics of language are generally well applied, with minor issues that do not significantly impact readability or clarity.

3. Organises Information and Integrates Research Material

• Score: 4.5
• Justification: Demonstrates a strong integration of relevant research material to develop and support the thesis. The essay makes consistent and sustained reference to a variety of sources to explain and support its conclusions, with slight room for deeper integration in some areas.

4. Evaluates Information Considering Relevance, Reliability, and Credibility

• Score: 4
• Justification: The essay uses a range of reliable and credible sources, with annotations that analyse the credibility and relevance of sources. However, there could be a more detailed examination of some sources to fully ensure their validity.

5. Finalises and Presents Text in a Digital Format

• Score: 4.5
• Justification: The final version of the essay meets all assessment requirements with a consistently and accurately applied referencing style. There are minor discrepancies or areas for improvement in ensuring the uniformity of formatting or in the completeness of annotations.

Overall Score and Feedback

• Total Score: 21.5/25
• Converted Score: 43/50
• Overall Feedback: The essay effectively addresses the negative impacts of fast fashion on the environment and society, demonstrating extensive research and a strong argument. The structure and language are mostly well-executed, with thorough integration of research material. There's room for improvement in deeper analysis and evaluation of sources and minor refinements in language mechanics and formatting. Excellent work overall, with detailed insights into the detrimental effects of fast fashion, supported by a wide range of evidence. The argument is persuasive, making a compelling case for the necessity of reform in the fashion industry.

Review of Methodology

This approach can be adopted by educators, but it necessitates the verification of AI-generated marks to ensure accuracy and alignment with educational standards. For instance, if an AI system grades 50 papers, teachers might need to manually check a representative 10% sample, verifying the bottom, middle, and top performers, to confirm the alignment of AI assessments with human judgment. If discrepancies are found, adjustments can be made manually to ensure the grading remains consistent and fair, while maintaining the original rank order of students from lowest to highest scores. This method provides a balance between leveraging AI efficiency and maintaining educational integrity.

To ensure the accuracy and fairness of AI-generated marks, the verification process could involve the following steps:

• Selection of Samples: Randomly select a 10% sample of papers across all performance levels (low, medium, high) graded by AI.
• Manual Review: A human marker manually reviews these selected papers to compare AI-generated marks against traditional marking standards.
• Analysis of Discrepancies: Identify any deviations between AI and human assessments, paying close attention to criteria met or missed by AI.
• Adjustment of Marks: If necessary, adjust the marks manually to correct any inaccuracies found during the review. This might involve changing the marks of some or all of the papers.
• Consistency Check: Ensure that the rank order of students (from lowest to highest scores) remains unchanged after adjustments.
• Feedback Loop: Incorporate findings from the manual review back into the AI's learning model to improve accuracy for future grading.

If the rank order is not maintained after manual adjustments, the following steps should be taken:

• Re-evaluation: Carefully review the adjusted papers to understand the cause of rank order disruption.
• Further Adjustments: Make additional adjustments if necessary, focusing on maintaining academic integrity and fairness.
• Documentation: Record reasons for significant adjustments to maintain transparency.
• Quality Control: Engage another educator for a secondary review if discrepancies persist.
• AI Calibration: Use insights from discrepancies to refine AI grading algorithms for future accuracy.

If unaligned results are obtained, the outcomes of this review could be used to adjust the prompt for future use.

Benefits

• Time Efficiency: Utilizing AI for grading significantly saves time, allowing educators to focus on other critical teaching tasks.
• Plagiarism Checks: AI can simultaneously check for plagiarism, enhancing academic integrity.
• AI Integration: Offers potential for integrating AI in educational processes, paving the way for innovative teaching and learning methods.
• Student Review: Students can utilise the same prompt to evaluate their work and identify areas of improvement. This self-assessment process enables them to enhance their understanding and performance.

Negatives

• Accuracy Concerns: There's a risk of losing student trust due to potential inaccuracies in AI marking.
• Personal Touch: AI may not fully capture some the unique student responses that might be add a few marks, possibly overlooking creative or critical thinking aspects.

Final Thoughts

The development of the prompt and its subsequent outcomes required careful consideration. Achieving consistency in the AI's grading process was challenging, needing several hours of adjustments to the prompt. The aim was to align the AI's evaluations with the expectations and standards of human grading.

The generate output was highly accurate for this example as the AI's calculated score closely matched the average mark given by a group of 50 teachers, which was 22 out of 25. The AI's score of 21.5, when rounding up, demonstrated a significant level of consistency with the human teachers' evaluations, underscoring the potential for AI to reliably assist in educational assessments once the necessary considerations are meticulously incorporated into its programming.

The generate output was highly accurate for this example as the AI's calculated score closely matched the average mark given by a group of 50 teachers, which was 22 out of 25.

The exploration of AI in grading and marking, reveals its promising potential to streamline the assessment process, offering time efficiency and consistency that aligns closely with human judgment. However, educators need to be careful during prompt development to ensure marking criteria alignment and the need for ongoing refinement to ensure that AI grading not only matches but enhances educational standards. As technology evolves, the role of AI in education could expand, provided its application is approached in a way that maintains integrity and personal touch of traditional grading and marking.