ChatGPT, far to the future

tl;dr; In terms of Software Engineering and creative works, both ChatGPT and Bard still have a long way to go.

ChatGPT and Bard (C&B) have recently gained popularity with a hype that they one day can potentially replace for humans in various fields, particularly in coding and software development. To test this hypothesis, I conducted several tests using C&B to see if they can provide solutions for technical questions.

I selected problem A in a CodeForce contest under DIV 3, which is generally targeted towards beginner coders. The problem itself is fairly straightforward.

The solution to the problem is quite simple: you can scan all videos from 1 to n to determine which ones you can play. To watch the i-th video, you need to skip i-1 videos, which leaves you with t-i+1 seconds to watch. From the list of available videos, choose the one with the highest b value.

void solve() 
	int n, t;
	std::cin >> n;
	std::cin >> t;
	std::vector<int> a(n);
	std::vector<int> b(n);
	for (int i = 0; i != n; ++i) {		
		std::cin >> a[i];
	}
	for (int i = 0; i != n; ++i) {
		std::cin >> b[i];
	}
	int max_i = -2;
	for (int i = 0; i != n; ++i) {
		if (t - i >= a[i] && (max_i < 0|| b[i] > b[max_i])) {
			max_i = i;
		}
	}
	std::cout << max_i + 1 << std::endl;
}{        

This problem can be easily solved by a 7th-grade English student at VNUK++.

I posted the question to both ChatGPT and Bard to see how they would answer it. Unfortunately, both of them provided incorrect solutions that required watching all videos from 1 to n.

Note that you can get different answers with different try with ChatGPT and Bard.

It was evident that both ChatGPT and Bard misunderstood the question, as they did not consider the possibility of skipping videos at the cost of 1 second. To clarify the question, I provided additional information.

Despite providing additional information, both ChatGPT and Bard still failed to include the skip logic in the code, resulting in similar incorrect answers. To assist C&B in understanding the problem better, I provided more detailed information about the skip logic.

As observed, ChatGPT produced an entirely incorrect solution, suggesting a lack of control. In contrast, Bard produced a similar solution as before but with an additional random variable. Further attempts to improve their solutions proved futile, as it seemed impossible to teach them to comprehend the skip logic.

Conclusion: Although C&B can be helpful in many cases, including assisting me in editing this post, their ability to comprehend logic, even simple logic, is limited. Hence, the day when C&B or AI could replace humans in creative or knowledge-intensive work seems distant. C&B can undoubtedly aid us in completing tasks, but they lack the capability to take charge of the work entirely.