Using chatGPT to teach computational thinking

?

Background?

I saw a recent paper from an MIT professor that found the following?

The professor divided a class into three groups and gave them a programming task to solve in the Fortran language, which none of them knew. One group was allowed to use ChatGPT to solve the problem, the second group was told to use Meta’s Code Llama large language model (LLM), and the third group could only use Google (search). The group that used ChatGPT, predictably, solved the problem quickest, while it took the second group longer to solve it. It took the group using Google even longer, because they had to break the task down into components.Then, the students were tested on how they solved the problem from memory, and the tables turned. The ChatGPT group “remembered nothing, and they all failed,”? Meanwhile, half of the Code Llama group passed the test. The group that used Google? Every student passed.

The conclusion drawn was

?Working hard and struggling is actually an important way of learning. When you’re given an answer, you’re not struggling and you’re not learning. And when you get more of a complex problem, it’s tedious to go back to the beginning of a large language model and troubleshoot it and integrate it.” In contrast, breaking the problem into components allows you to use an LLM to work on small aspects, as opposed to trying to use the model for an entire project, he says. “These skills, of how to break down the problem, are critical to learn.”

However .. I think ..

There is an obvious flaw in this which the professor seems to have missed namely - you can use chatGPT (or any LLM) for teaching computational thinking. In fact, thats what we are doing in the the jigsaw methodology? for teaching AI to non developers.

I have intentionally used chatGPT itself for my thought process below - to demonstrate how flawed this thinking is.???

What is Computational thinking??

Computational thinking is a problem-solving process that involves a set of mental skills and practices used in computer science, but it can be applied to various fields beyond just programming. It is a way of approaching problems in a structured and logical manner, similar to how computers solve problems. The core aspects of computational thinking include:

Decomposition: Breaking down complex problems into smaller, more manageable parts. This allows for easier understanding and tackling of each component one at a time.

Pattern Recognition: Identifying patterns or similarities within problems. Recognizing these patterns can help in making the problem-solving process more efficient by applying known solutions to new problems.

Abstraction: Focusing on the important information only and ignoring irrelevant details. This involves creating generalizations or models that can represent complex systems without the need to understand every detail.

Algorithm Design: Developing step-by-step instructions or rules to solve a problem. Algorithms can be implemented in computer programs or even followed manually to arrive at a solution.

Debugging: Testing and refining solutions, identifying errors or inefficiencies, and fixing them iteratively.

Applying computational thinking to reach machine learning and deep learning to non developers??

The problem statement I am considering is: How do I teach machine learning and deep learning to non developers(domain experts)?

Let's consider the problem decomposition process

The traditional steps for machine learning are??

?

1. Problem Definition

2. Data Collection

3. Data Preprocessing

4. Feature Engineering (for traditional ML)

5. Model Selection

6. Model Training

7. Model Validation

8. Model Evaluation

9. Model Optimization

10. Deployment

11. Monitoring and Maintenance

12. Model Interpretability (Optional)

13. Retraining and Feedback Loop

?

Now, for the domain expert, I reframe and decompose this problem in terms of the model building and model evaluation steps.? Why? Because from a domain experts perspective it makes sense to see the end in mind and decompose backwards from model evaluation metrics,?I used the question/prompt to decompose the overall problem as - "how do I relate each step of machine learning to the bigger problem(of model evaluation)" ?

1. Problem Definition

Impact on Evaluation: The problem definition determines the evaluation metrics used. For example:

Classification Problems: Metrics like accuracy, precision, recall, F1-score, AUC-ROC are crucial.

Regression Problems: Metrics like RMSE, MAE, R2 are relevant.

Without a clear problem definition, it’s challenging to select appropriate metrics for evaluating the model's performance.

2. Data Collection

Impact on Evaluation: The quality and quantity of the data directly affect model evaluation.

High-Quality Data: Well-curated data ensures that the model has the potential to perform well on unseen data, leading to better evaluation scores.

Biased or Noisy Data: If the collected data is biased or noisy, it will likely degrade performance, and evaluation metrics will reflect poor generalization.

Insufficient Data: Inadequate data may lead to underfitting, impacting metrics like accuracy or precision during evaluation.

3. Data Preprocessing

Impact on Evaluation: Proper data preprocessing is essential for model performance, and it ensures that evaluation results are meaningful.

Data Cleaning: Reduces noise, improving accuracy and other metrics.

Normalization/Scaling: Without proper scaling, certain algorithms (like SVMs, neural networks) perform poorly, resulting in suboptimal evaluation scores.

Data Splitting: Proper train-validation-test splitting ensures that evaluation metrics reflect generalization to unseen data. Poor splitting can lead to overfitting, skewing evaluation results.

4. Feature Engineering

Impact on Evaluation: Feature selection and transformation affect the model’s ability to learn patterns in the data.

Good Feature Engineering: Helps the model capture useful information, improving performance metrics like accuracy or R2.

Poor Feature Selection: Using irrelevant or redundant features can lead to overfitting or underfitting, which negatively impacts evaluation metrics such as precision, recall, or RMSE.

(For deep learning, feature engineering is minimal, but ensuring high-quality input features is still critical for proper model evaluation.)

5. Model Selection

Impact on Evaluation: The choice of algorithm affects the evaluation metrics directly.

Algorithm Appropriateness: Selecting the right algorithm for the task leads to better evaluation metrics. For example, using decision trees for structured data or convolutional neural networks (CNNs) for image data improves accuracy, precision, or F1-score.

Algorithm Mismatch: Choosing an algorithm that’s not well-suited for the problem will likely result in poor performance during evaluation (e.g., low accuracy, high error).

6. Model Training

Impact on Evaluation: The training process determines how well the model fits the data, which is reflected in evaluation metrics.

Good Training: Optimized hyperparameters, learning rate, and proper training result in a model that generalizes well, improving metrics like accuracy or R2.

Overfitting/Underfitting: If the model overfits (memorizes training data) or underfits (fails to capture patterns), evaluation metrics will show poor generalization, with validation accuracy much lower than training accuracy or high validation error.

Hyperparameter Tuning: The right combination of hyperparameters leads to better performance during evaluation by balancing bias and variance.

7. Model Validation

Impact on Evaluation: Model validation helps ensure that the model will generalize to unseen data, making evaluation metrics more reliable.

Cross-Validation: Techniques like k-fold cross-validation provide more robust performance estimates, which will be more accurate during model evaluation.

Avoiding Overfitting: Regular validation helps to stop training at the right time, improving metrics like F1-score or RMSE during final evaluation.

8. Model Evaluation

Direct Impact: This is the stage where the model's performance is quantitatively assessed using predefined metrics. The results depend on how well each of the previous steps was executed.

Evaluation on the test set ensures that the model's performance (accuracy, precision, recall, etc.) is unbiased and reflects real-world scenarios.

9. Model Optimization

Impact on Evaluation: Optimization techniques improve the model's ability to generalize, which is reflected in the evaluation metrics.

Regularization (L1/L2): Reduces overfitting, leading to better generalization and higher evaluation scores (accuracy, precision, recall).

Early Stopping: Prevents overfitting, ensuring that evaluation metrics reflect true model performance on unseen data.

Ensembling: Improves evaluation metrics like accuracy, precision, and recall by combining predictions from multiple models.

10. Deployment

Impact on Evaluation: While deployment is typically after model evaluation, the real-world performance of the deployed model (as monitored by online evaluation) may influence future offline evaluations.

Real-World Feedback: If the model performs poorly in production (i.e., real-world predictions deviate from offline evaluations), it might signal issues with the initial evaluation metrics or the data distribution mismatch.

11. Monitoring and Maintenance

Impact on Evaluation: Continuous monitoring helps track model drift, which indicates a change in evaluation metrics over time.

Model Drift: Degradation in evaluation metrics like accuracy or F1-score in production can signal the need to retrain or update the model to align it with current data distributions.

12. Model Interpretability

Impact on Evaluation: Interpretability allows for understanding which features contribute most to model performance, helping to refine the model and improve evaluation scores.

Understanding Model Decisions: Techniques like SHAP and LIME help ensure that high evaluation scores are meaningful and that the model is not learning spurious correlations.

13. Retraining and Feedback Loop

Impact on Evaluation: As more data becomes available, retraining the model improves evaluation metrics by incorporating updated patterns in the data.

Continuous Improvement: Regular retraining and fine-tuning ensure the model’s evaluation metrics (e.g., accuracy, precision) remain high as the data evolves.

Conclusion

Its very easy to teach and implement computational thinking using chatGPT (or any other LLM)? as above. The tool does not matter. If anything, chatGPT is a far richer pedagogical tool in conjunction with computational thinking I am disappointed in the MIT paper for missing something so obvious.? We are working on exactly this with the jigsaw methodology.