QnA: AI for your Testing Career
Jason Arbon
??♂? CEO, Checkie.AI | test.ai, Google, Microsoft, Chrome, Search, “Automating the World"
There were too many questions to get to during my session at #TestMuConf 2024 Conference presentation, that we couldn't get through them all. I've answered all the questions and they might be useful to folks who havent seen the presentation yet.
Q: What are the key indicators that an AI tool is enhancing your testing efficiency and supporting your career growth?
Efficiency:
Career Growth:
Q: What’s one key strategy for using AI to advance a testing career?
Become a "ChatGPT Expert" - learn to leverage AI tools like ChatGPT effectively for testing tasks. On almost every task, no matter how small, start work on it with a prompt–ask the AI for help or thoughts or feedback on everything you do.? You will quickly learn what GTP can help with and how to leverage it best.
Note: It is unlikely a testing career will advance today without being an expert in this particular area–it is now as important as using Google.com to find answers and context for work.
Q: Can you share some specific examples of how AI in testing has influenced career success or promotion opportunities for professionals in this field?
Personally, I became a CEO in the process of learning how to apply AI to software testing.
Many folks that worked with me at test.ai ended up moving to positions that were title and pay promotions, and new opportunities specifically related to their AI experience.
My buddy Kevin has taken a new job with the title “AI/ML SDET” given his experience with AI
If you look on linked in you will see many other examples and these are just the early days.
Q: How can you translate your current skills in AI Testing world?
Also Q: Can you provide examples of successful career paths that have been enhanced by AI expertise in testing?
If you already have strong ‘critical thinking’ skills, they will prove very useful when testing AI or AI-based systems. That said, AI will require even more critical thinking skills that most testers have developed.
In many ways, traditional approaches to testing will cause confusion and might hinder a transition to AI testing.? Concepts such as test cases, oracles, test cases, don’t transfer well. Generally best to consider how to ensure quality with an AI-first approach with a fresh mind.?
Frankly, its also just as much about simply using new technology to perform your work better faster and cheaper–if you don’t leverage it you will likely soon face ‘negative’ career impacts.
Q: What are some common challenges testers face when integrating AI into their testing processes, and how can they overcome these challenges?
Q: What skills should testers focus on to stay relevant with AI in testing?
Q: How do we test a product with 10x demand when we're fired?
Difficult, but realize you can always work for free and be helpful.
Q: Can you please suggest a Codeless Automation AI tool for testing?
Q: Please suggest codeless automation AI tools if possible
I have used these in the past, I’ve even built a couple myself, but I don’t recommend most of these types of tools.? I also don’t use any today or frankly even much in talking to folks about them these days so I can’t give a recommendation.??
Also worth mentioning that most vendors are adding AI? features to the existing product, but I worry a lot about the new complexity of mixing old and new technologies and runtimes.? It might look like a cool/easy demo, but the real issue with all these tools is debugging and fixing things when they break.
Plug: You may think I'm just biased, but I see it as putting my money where my mouth and understanding :) :Checkie.AI ( https://checkie.ai ) is ‘codeless’ but does the testing autonomously, avoiding many of the issues above.
Q: How can AI in testing contribute to faster career advancement or promotions, especially for those just starting their careers?
I actually don’t recommend folks start a career in testing these days. Same with software engineering. There are a lot of unknowns, volatility, and frankly AI is probably far more useful to experienced folks than it is for helping new folks learn and catch up.? But, if you want it strongly enough, folks that ignore this advice will probably do fine ??
For folks new to testing, I’d suggest ignoring most of the ‘legacy’ and ‘traditional’ testing methods, books, and classes.? They aren’t as useful in the world of AI.? Instead, new testers should focus on understanding quality (not methods) and quickly focus on these topics instead:
<Same answers as above>
Q: How can networking within AI communities and attending AI-focused events benefit a tester’s career progression?
Learning, staying current on tech and adoption trends will help folks apply AI at their own job.
Also, being visible to other companies/opportunities opens up new career opportunities.
Q: What AI test Tool have you used in your shared examples? Thanks
I’ve been showing test results from the AI testing bots at Checkie.AI (https://checkie.ai). Checkie.AI is different from most tools in that it works completely autonomously to deliver a basic test coverage across a broad array of types of testing.? Checkie.AI only needs access to the websites’ URL to start testing.? “The simplest and smartest way to add AI to your testing.” :)
Q: How can we introduce AI testing in an organization where there is a lot of tools/sites which use legacy programming
It depends a lot on the organization, tools and culture.That said, most folks are slowly introducing ‘coPilots’ into their existing workflows.? Generally it's a good idea not to pitch replacing the existing work–rather try new AI-first approaches on new and even low-risk areas to get results quickly, and evaluate how useful AI testing can be in the?
Q: How important are AI certifications for career advancement in testing?
Some of the new AI certifications from places like Amazon, Google, etc are a great forcing function to learn the tech, and demonstrate to the world that you are investing time learning AI, and that you understand it.? I think it's a great idea given the newness of AI.??
Q: What should testers train in and upskill to learn in AI, automation, and machine learning to keep their skills at the forefront of the market?
<Same answers as above>
Q: What challenges might testers face when incorporating AI in testing, and how can overcoming these lead to career advancement??
Lots of challenges! It's not easy.??
When trying to use it for test automation, here is a list of issues i’ve seen and run into myself: https://www.dhirubhai.net/posts/jasonarbon_chatgpt-sucks-for-test-automation-activity-7117765005030133760-ScSv/
Ironically, when testers use AI to generate more test code–that will often just mean more test code to maintain.? Of course the AI can help fix/update the code, but its not well known what the ROI of that combination of change will be.
Manual testers frankly just need to worry about their ego.? The AI is pretty darn good at testing, even if only using a chatbot.? Testers that embrace AI will be better testers, finding better bugs, better reporting, and better/faster test case creation, and will be super-testers, ripe for promotion.
From a test management perspective: your direct reports can be spooked by your interest in AI as they can worry about their job security, justified or not. I’d also suggest AI experimentation be kept quiet, and only celebrate and make it visible when there is a ‘win’--there is risk in setting expectations too high. But, an AI win will help secure and should get visibility up to the CTO in most companies, and that will help with promotion.
Q: Is prompt engineering the most crucial skill to pick up (or build on) these days for testers?
Yes.? If you study everything else but lack great prompting skills, you won’t get much dont.? Great prompting skills alone will help testers unlock the potential of LLMs in most testing activities. Here is a post on the most relevant prompting techniques for testers: https://jarbon.medium.com/prompting-for-testers-d3828bbde318
Q: Are there any certifications available for learning Claude.ai?
<Answer From SearchGPT>
Claude AI Certifications and Training
Claude AI, developed by Anthropic, offers several certifications and training programs to help users understand and utilize its capabilities effectively. Here are some key options:
Great Learning Academy: Offers a free course titled "Introduction to Claude" which covers the basics of AI assistants, Claude's capabilities, prompt engineering, and hands-on usage. The course includes a certificate of completion (
领英推荐
MyGreatLearning (https://www.mygreatlearning.com/academy/learn-for-free/courses/introduction-to-claude)
).?
Q: How easy is it to maintain AI scripts. what kind of effort is needed when there is high fluctuation in coding?
Great question–very difficult.? Even if/when the promises of AI breaking less often, when it does break it is difficult to debug, and even more difficult to ‘fix’ versus traditional test code (e.g. JAVA/Selenium, etc.)
Worse, when AI is added to existing tools or code, things become even more complex to debug and ‘step’ the code as execution crosses the runtime boundaries of traditional procedural code and ‘neural nets’.
Q: What is the best way to test performance of AI models as responses change every time you give a prompt??
Human judgment is ‘best’, but slow and expensive.
Some folks are using regexes or searching for substrings, but really the most robust was is to verify the response with, and AI. Preferably a different AI/LLM.? AI can handle the ambiguity and variability better than traditional code/testing.
Q: You said LLMs are only tested well “in bulk”, and any individual anecdote doesn't prove that the LLMs are not working good. But usually people use it this way (e.g. chatbots). Isn't that an unrealistic argument??
People do lots of things that aren’t useful or efficient or effective :)? I recognize that many folks are doing this, which is why i’m sharing this information.? Most will ignore it, but worth a try.
A good way to start to understand how to ‘test in bulk’ is understanding how search is tested, it is very similar: https://en.wikipedia.org/wiki/Discounted_cumulative_gain ? And, also look at Hugging face, which is the current standard of ‘testing’ LLMs, note they use scores from large suites of test prompts.? https://huggingface.co/
Q: Can you suggest how we might cut through the marketing hype and tell if something is AI??
The best way to tell is to learn about AI yourself–then you can see through most of the hype and fake AI marketing.
That said, with the rise of ChatGPT–it is now easy for most marketing to be accurate these days–it's just a simple prompt, and children can do this.?
Really, now the question is simply whether the tool and their use of AI is ‘useful’ or not.?
Q: Don't you think any kind of testing (even asking silly questions) is good for exploring what are the limitations of an LLM??
Yes, i’ve even tried copy pasting random binary sequences, and empty strings (found a bug with OpenAI that way, lol. https://jarbon.medium.com/gpt-prompt-bug-94322a96c574).
The problem arises when naive folks think they have discovered a ‘bug’. There are things that LLMs just can’t be expected to do because of their design/architecture/data/training.
Q: What do you think that in future we will have a label that "it was tested by AI" similar to "generated by AI"? Human work will be more expensive and businesses can choose something in the middle??
I think it's a fun idea, butI don't think it will matter.? Much like how people don't care what metals were used to manufacture their automobile, or even whether their phone is ARM or Intel, what matters from a label perspective is just how useful/appealing the software is.?
That said, the efficiency gains should be significant for most software, so the reality is that if software isn’t generated with the help of AI–it just won’t be relevant or will be too expensive in the near future–no label needed.
Q: What kind of learning would you recommend regarding AI for someone moving from testing to team leading?
Learning to use AI to summarize test results, review your communications, and asking AI to help you discover what is ‘missing’ in the testing across the team. LLMs are pretty darn good at helping with those issues.
Also, learn yourself, and get your team to focus on:
?
Q: What are the best strategies to learn the appropriate deep/critical thinking skills that make testers better able to work better with AI and automation??
Great question.? This is a great resource. https://www.criticalthinking.org/pages/defining-critical-thinking/766
I would also ignore the shallow, and often just marketing material, versions circulating in the testing community.?
Q: Given enough context, could an AI help find bottlenecks in the software lifecycle process??
Yes, probably for the more basic obvious stuff.? I haven’t tried anything of that scope, but at the same time, the definitions and even verification of any improvements are probably difficult.? Beyond my paygrade :)
Q: What is the implementation cost for introducing AI tools in the mix, if using available solutions from the market?
It varies widely, and the market hasn’t quite settled on pricing in the world of AI yet. But a good metric to think of when buying and selling is pricing around 20% of the total savings / value.? If the price is higher than that, make sure to compare, but it is very contextual, maybe it comes with 24/7 support, or speed not cost is your prime motivation, etc. Hope that is directionally helpful.
Q: I get AI testing websites / writing automation overall by just pointing to it. However, how would it deal with A/B testing that we do before pushing out features??
The good news is that AI can dynamically create tests for whichever A/B flight it happens to be interacting with.
Often folks stage their A or B in different environments, or have URL parameters to force a particular flight.? So the bots can just be pointed to the right environment or pass the flight parameter via the URL.?
If the A/B flight staging is more difficult than that..I suggest folks fix that first ;)
Q: How can testers avoid the common pitfalls of AI tools that generate excessive test cases & instead focus on value-driven automation??
Well, if not dupes, I’d recommend folks question whether they really don't need those tests–are they really excessive and low ROI :)?
A big problem is AI generating nonsensical, dupe, or even invalid test cases (assertion is incorrect).? This is particularly a problem in unit-test generation–I think the statistic is that 60% of generated tests need human review just to check for correctness.
In general, I’d recommend folks run most all tests they have. It is far less expensive than explaining why you chose not to run a particular test–expensive to your career in addition tot he business.
Q: would the AI test scripts also be similar to manual test scripts in the sense that they can wear out too?
I hope I understand the question.? I have personally deprecated thousands of manual tests because they were invalid,. Outdated, or very low ROI.?
It really depends on the ‘AI’/tool.? Most AI tooling today is very much like manual test development and does suffer from these same problems.? Especially the ones where you can craft manual test cases or test step definitions one at a time. Looks like a cool demo, but will suffer from the same maintenance and ‘wear out’ issues as the product evolves.
Smarter AI/tooling will generate the test coverage dynamically so it is never out of date, always relevant, and even automatically tests for the new features even if the human didn’t know about them ??Plug: Checkie.AI operates like this: https://checkie.ai?
Q: What is the top-most certification to learn AI??
I don’t know.? I don’t do certifications myself.???
Google and LinkedIn have great AI certification programs.
In terms of AI and Testing certification, I know Tariq worked on one with Brightest.? I offered to help review it, but I think they never got back to me, so I can’t vouch for it other than I know Tariq knows his stuff :)? https://brightest.org/en/certifications/AiU-Certified-GenAI-Assisted-Test-Engineer/?
Q: Can you give some good sources to get educated on basic and advanced AI Testing?
I can’t recommend any books on the topic yet..
I’d follow AI in testing folks on LinkedIn, some good context there.? E.g. Tariq King (https://www.dhirubhai.net/in/tariqking/) , Me (I’ve written a ton of posts on this topic: over the past year, all of wish are still relevant: https://www.dhirubhai.net/in/jasonarbon/),?
Most all the testing vendors have fun AI demos to give you a taste/tease
Lambda Test’s Testu conference have lots of free webinars and session on these topics (https://www.lambdatest.com/testmuconf-2024/)
There are half and full day workshops at StarWest and other conferences soon as well: https://starwest.techwell.com/program/tutorials/harnessing-generative-ai-software-testing-real-world-guide-starwest-2024
Keep up on AI news on twitter
Thanks to everyone for the Questions and hope the answers might be useful to folks
--Jason Arbon
Testing Philosopher | SDET | System Architect | Developer - I help companies deliver quality
2 个月I enjoyed your talk at TestMu and appreciate you answering these questions as well. I found your advice on testing ai insightful. I also think the skills you’ve emphasized for testers in the age of ai are important. I’m less convinced about how soon autonomous testing agents will be making a major impact, at least in the contexts I mostly work in - but you’re closer to that edge then I am. Thanks for sharing your thoughts.