Testers Discuss AI

Thanks to Marcel Veselka , Juraj ?abka for a fun conversation that spanned AI, and the future of testing.

makeITfun YouTube vid @ https://www.youtube.com/watch?v=qUa0hdKtlNE

Marcel

Okay, so hi Jason, can you hear me? Can you see me? Welcome to our show. Yes, and thank you. Okay, so let's start with a very quick introduction. I'll do my best and then hopefully you will help me to fix it because I believe there will be a lot of gaps or misunderstanding.

So you, Jason Arbon, are the ex-Microsoft ex-Google ex-Applause tester. But last seven years you are an entrepreneur in testing. May I say that in that way? I don't know. Yep. And I would be interested, but we could discuss later whether you are still tester or developer of testing tools.

Jason Arbon

I write tests every day and like literally every day, I write tests and testing things. Okay. and writing testing code to test code to test code. You are also author of book or co -author of book, right?

Mixed

How Google test books. Two books, okay. That's a self -help book. Okay, self -help book. Another book on how to grind coffee. Really? No. No, it's a joke. A good one. You're a scientist. So another thing is that you are also co -chair of in the association for artificial intelligence for software testing.

Marcel

I don't know whether this is still alive or how it goes. I was refreshing the page or exploring the page last days and it looks very same like I saw it a couple of years ago. So that would be maybe also interesting topic to discuss later.

Sure. And I don't know anything about your hobbies if there are any. Maybe testing is not sure I do either. But okay. So that's it. What I prepared and hopefully you could enhance and improve and and fix what I said.

Jason Arbon

Okay. I learned from previous ones that I'm not supposed to answer until you've declared yourself finished with the list to be discussed. So are you ready? Yeah. Okay. You got it right. Okay. As my as my family calls me called ex-Microsoft ex-Google.

I'm basically like a serial quitter. Just when my family thinks i have got a decent job, I quit it. That's basically the story like kind of literally my family sees it. I like the term serial quitter.

Yeah. Yeah. And in the serial entrepreneur is don't want to go back to where I quit. What else can I possibly do? So yeah, I would say the interesting, maybe interesting thing to people about those....

... I got into testing because of some projects in college and I was testing the stuff that was supposed to be, maybe it's not very politically correct, but stuff that was supposed to work. I did some projects for missile stuff for Lockheed Martin back when I was in school and it made me really think about code quality, like very seriously code quality, like the stuff like satellite communciations and navigation, you want to make sure that the code's good.

And so I became kind of obsessed about kind of realized how dangerous code is during that senior project with Lockheed. And so I went down the testing path. For many people it's pretty accidental, but for me it was pretty deliberate.

And I thought it was also I'll add, while we are geeking out with test people here is I think it's the hardest problem on the planet. Like it's actually more difficult than a lot of nuclear physics research, string theory, quality or testing.

Yeah, I'm talking about the code quality or testing or well, all of that. How do you know that the software, it means it's the requirements even, right? How do you know requirements are correct? How do you know that the software operates correctly?

How do you know that it operates within parameters, within expected contexts? Yeah, we're geeking out forever. But software powers our lives so much more than when I graduated college, right? Like it's now like our daily lives are run by software, we're on software right now talking.

And I think it's also the stakes have gone up much, much higher than I ever even expected. Like it's more the world runs on software. So we need to make sure the stuff works. But that was a very, very technical.

Marcel

It could be very technical. So you can easily to judge whether it works or not. But if you look on the from the human perspective, it might be very subjective. Whether this is good or not. Yep, then the subjective stuff is, from a business perspective, almost more important.

And then, I would argue one thing is that the more technical you are on software testing, the harder the problem is. Like, the more that you think it's a miracle, the stuff works at the end of the day.

Especially when you have all the entanglement between multiple systems these days. Like the complexity we built into these things is insane. If you wanted to build software to be unreliable, you would probably be doing what we're doing today.

Mixed

In terms of just training systems and client server, blend software, and now AI. Okay, yeah, yeah. We will come to AI soon, we will come to AI soon. We don't have to keep it in a structure. And so, I was asking about the hobby.

Marcel

Do you have any hobbies or the testing is your hobby as well?

Jason Arbon

Testing is as actually a hobby like like on Saturday night at 1 in the morning I am usually or Sunday morning or whatever. I'm usually writing some test code Which is embarrassing, but I do like to go to National parks with my kids like we drove through seven or eight national parks in the last 12 months, you know everything from you know freezing winter, you know hikes to Over the Olympics down into you know that literally I wanted to go we went to Vegas and and death Valley In the heat and summer and my kids asked me to leave them in the desert So I I shouldn't probably talk about this publicly, but they want to be left alone Yeah, of course I did so But they well they took pictures of themselves on laying on the ground on the desert sand reaching for water Like that was there fun, but we went through the high of one of the highest peaks in the United States in the lowest peaks in last 12 months.

Marcel

I was called that kind of a hobby, but oh, but I don't other than that really not much Okay, cool. So nature traveling and maybe hiking in nature would be the Yeah Okay, I see okay, let's move on now to the hot seat questions Yeah, I will let you write it to run it.

Juraj

Yeah, we prepared some question. Well, a hot seat question that you have to choose maybe One option or two short answer from your side that we can start really quickly And the first one is: "will manual testing still be here in 10 years."

Jason Arbon

No Well, we have the fall I had a follow question then in 20 years, but I have don't need to ask you 20 years old That's the sad truth, that's the sad truth. Oh no. AI or human? Just in general, those two words.

Yeah, that's in general. AI or human. That's not a tough one for me because like my profile setting, which everyone really follows, there's always been blending human and machines. So literally like the midpoint.

But between the two, I would actually choose AI. I had to make a forced Boolean cast. Okay.

Juraj

Yep. Okay, tester or developer?

Jason Arbon

Tester. Oh. I don't know if it's me you're talking about or just in general, but 100% tester.

Developers, development is actually pretty boring. You just build what other people tell you to build and then hope it works and people just send bugs your way all day. It's a pretty lame job actually.

Juraj

Interesting.

I expected a lot of like feedbacks under the video. Very good. Yeah, why do you think I've been quiet the past year? Okay, corporate or startup?

Jason Arbon

Oh, 100% startup. Yeah, 99% startup. Okay, good. But every large corporate was a startup, so kind of the ultimate goal of every startup in a weird way, so it's entangled.

But yeah. Okay. And what was the last time that you created or reported a defect? To somebody other than myself. I'm actually being serious, like I'm not being... Yeah, it was a couple months ago, I filed a bug with OpenAI on their API.

And they... I wrote an actual blog post on it. It was kind of fun because I wrote... I had the AI... Like, I found the bug. I'll talk about it or not, but probably not. But, details. But I had the... I was sort of writing up the description for it, right?

Like why they should fix it and why it's a problem. And I just DM'd the API guy on Twitter or whatever. But I had GPT write the bug report and the justification. And ithe AI actually had a point in there that I didn't think about, and it was far more articulate than me.

And it was the first time, I think, maybe in the history of the planet that I'm aware of, where... Where the AI actually technically kind of filed the bug against itself. And then the humans decided to fix it.

So the humans are actually now in the realm of the kind of the AI in a way. That's kind of... Yeah, that was the first... That was the last probably external bug I filed of interest.

Juraj

Okay, good. Microsoft or Google?

Jason Arbon

It's a Boolean, right? I mean...

Juraj

Yeah, sorry for this question, but we have to do...

Jason Arbon

No, this is good. Sorry, and then we're asking today in... I was trying to integrate over time. Okay. And I will additional one probably last one that ChatGPT or Bard?.

ChatGPT.

Juraj

Okay. Yeah. That's it. It looks that I have to prepare next time better because you are absolutely okay that it looks that are not really hot seats so that I have to improve it for the next time.

Jason Arbon

Well, it's just I have no shame, no dignity. Now it's the time to move on because I over the next 15 minutes if you think of some new ones write some more other ones down I'm happy to answer those are kind of fun.

Marcel

Quick questions.

So I was doing some research to make it sound cool. And I found that you quit. Google in 2011 December 2011.

Jason Arbon

Oh, you found that?

Marcel

It was in your LinkedIn profile. It's not so complicated. But there is a follow up question.

Jason Arbon

Is that correct or not? Was that correct? Yeah, actually it's crazier than that. I'll give you the tease. The real tease is this. So I started a startup-- I was working at Bing back when it was MSNSearch just converted to Bing I think at the time.

Back in the day. It was a small team. But we were actually funny thing. We're using AI and Google wasn't using AI at that point. That's kind of funny. Anyway, inside baseball. But the yeah, I created a personalized web search thing because I was so frustrated that my dad would get the same search results that I did.

That was ridiculous. And my dad was searching for like I would search for say "bush", right? And I wanted like Bush the band. And instead of got Bush the president and my dad is a gardener. He wanted like bushes-- things with leaves.

And so I thought I want to solve that problem so that web search was actually personalized for everybody back in the filter bubble kind of days. Yeah. And then I actually funny thing I still own that IP that kind of core IP, what it matters, but but the financial crisis hit and it's a long story.

They got an entrepreneurial story, but I ended up going to Google and working on. I didn't talk about it publicly. I worked on the search. I had a search team on the search team or at Google, which is very.

Marcel

Was the was the James Whittaker your boss or not?

Jason Arbon

Yeah, for part of the time and actually, yeah, yeah. I mean, absolutely for most of the time. Yeah, but I also had a team that was not in testing, like it was just in the search team was a weird arrangement

Marcel

because there was in October 2011, there was a conference.

Google Test Automation Conference. I don't know what you remember. You think and I found one funny video and I would like to share it. 13 seconds or so with you. And then I would like to discuss that.

Jason Arbon

I think I know what this is. I mean, is this, was I wearing, was it a onesie or a bikini? Let's see. I would work history videos. Yeah. We all do.

Marcel

That is the right, this should be the entire screen.

Jason Arbon

Is this the "test is dead" thing?

Marcel

Okay. I'm guessing right, but I would like to share it with other, other audience. This is the video.

Jason Arbon

I shared an office with James for a long time. I actually used his, when I was at Microsoft, his college kids were interns on my team.

Marcel

Yeah, so now I stop it. Hopefully now you can't see that, right? Yeah, I can't see it, correct, yeah. Okay, so now we can continue. So this is one video from the conference and Google had this keynote about testing is dead on this time.

Jason Arbon

Yeah.

Marcel

And now we are 13 years from there. And not too much in that area.

Jason Arbon

You're wrong, actually, like, so this is what, oh, I'm saying you're wrong, I have an opinion. Usually people like to bring that up in the sense that, "oh, I'm still here".

Number one, the best thing ever, if there's only one line answer to that, it's called survivor bias. If you're familiar with the survivor bias thing? Uh -huh, uh -huh. Like, for everybody who doesn't know, it's basically, I don't remember, the basic of the story is that there's one story of survivor bias, which is like in one of the wars, people like to have wars, I guess, but in one of the wars, the airplanes that came back were all shot up, right?

And so they realized they were shot up like in certain parts of the plane. So they made, they were thinking about making those parts of the plane stronger because they were getting shot up. The problem was, those were the holes that were okay to get shot up in, because the people can't.

If you got shot at the other places, you didn't come back. So the fact that you, that plane survived, or that a tester in this mapping, a tester has survived today, doesn't mean that a bunch of other people are not testers any longer.

It also, by the way, doesn't account for all the testers that should be around us. And look at us, are we all spring chickens?

Marcel

Probably not. Really?

Jason Arbon

Look at the conferences, look at the talks. James, in those old videos, James looks like a...

a spring chicken then. So the key, my key point is that the test is that is interesting. I used to go on walks with James as a, as a, as a, as a shallow copy of the talks that Zuckerberg would give people around the Facebook campus to try to think that we're cool.

But we talk about this thing, these meta kind of problems all time. And a lot of that stuff came from the core notion that talk, it's like a whole thing, but like improvements in engineering process and CI/CD and observability and all these types of things.

..were going to take up a lot of good improved software, by the way, without testing help, it's just going to make software better. But it's also going to like relieve a lot of the burden on testing because these things are coming into existence and they were, you know, they were newer back then.

So the idea is when I propose on this, I haven't really said it much, but I see it a lot of time. People think this is that that oh, it's, you know, it's not dead. If you look at how much. more software there is just the growth in market cap of Microsoft and Google alone

Marcel

Mm-hmm.

Jason Arbon

If they kept the same percentage of testers. There should be hundreds or millions more testers like. We should do the math There should be millions more testers than there are instead a lot of us are aging in place And and it's funny because the teams that don't need those testers where test is dead See, this is why I can't get on the internet guys you Ah, Thanks. But here's the thing: the thing is that all those teams should have testers and they don't anymore.

They're not there, but the teams that want humans, that need humans because they haven't adopted all that latest technology, still need human beings. So I think it's a huge survivor bias thing and Like literally just look at the math percentage of people that are doing testing It testing grows the CAGR the growth year-over-year as an entrepreneur, total money spent on software testers and software testing services, is like growing it like I think it's an 11% Rate this year.

It was like 20% maybe like six years ago. You know these numbers, guess what the overall growth in in tech is and just big Over 20% still

Marcel

I think the metric which says 20 to 40 percent of all the spending in IT igoes to quality assurance.

Jason Arbon

Okay, it's changing what and how we are doing this quality we're numbers based on which report they're doing so like Sometimes they account they count developer activities that are related to testing and most developers actually spend probably like 40% of time testing locally They could change

Marcel

My point is that this trend is not did not change for a couple of years when you ask or check the reports and analysis it used to be 20.

Jason Arbon

No, no used to be 20. This would be depending on the, depending on Gartner or whoever you're listening to, it was like between 20 and 23% about four or five years ago. I had to make a pitch deck and get through VC analysis and all that stuff.

And, and that number's been dropping about one and a half, 2% a year for the last like,

Marcel

Okay, this is interesting. If you have any source for that, I would like to see that because I felt like this is all for the longer time.

Jason Arbon

Yeah. So then there's, and if you look at the reports, they also anyway, they mess with the numbers because it's a product. Yeah. The data is a product. And so like, here's it. Here's the best one. I'll give it a question back to you.

Like these quick challenges. So we've seen all the garden reports, right? That like, guess what? 70 something percent of testers are using AI today. Yes. And this is about before ChatGPT, by the way, too.

Marcel

Yeah, that I that's I think it's bullshit and we could see such a bullshit on many places because if you look on the reports five Sometimes you could see the small letters they ask about are you using or planning to use these tools?

Jason Arbon

They're all scammed. Yeah Because then they want to because then all the companies have to pay them to get the data on how people are planning to use AI

Marcel

and the key point you set in that in that topic was that they data are they product If they want to improve their product, they need to be slightly influenced by the vendors who are trying to sell something to the market.

Jason Arbon

Everyone they're selling stuff to the vendors who are selling stuff to people and the people don't want to, people also self report strangely because they don't want to say that they're not up to the cutting edge. Like so they'll don't make stuff up to like they lie to themselves.

Those numbers are very cagey at best but the ones that have been consistent and persistent over time It's the percentage of spend on QA is actually been dropping and if I'm trying to raise money I shouldn't be telling the world that but it's just the reality.

Marcel

I'll ask you the last question in this section about the testing is dead and future and so on. So what do you think what what what is going on out there and where we will end up or you said in 10 years there will be humans in test or there won't be manual testing manual testing is not here anymore in 10 years you said that in that question.

Jason Arbon

I have whole snarky ways to answer that.

Marcel

Yeah, I know so so what if you would now try if you would now have the chance to fix it or improve or give some more details what will happen in next 10 years.

What do you....

Jason Arbon

I think everyone has the bias like it's in this question in general in our space is thoroughly unexplored because people, they just start the question and thinking about themselves as if " Well How do I maintain my job? "

"How do I survive?" and then they think about answering the question. So there's an incredibly huge strong bias toward that we will always need people.

Marcel

I'm biased with that your opinion. Do you think that you don't you don't know because you are too influenced by your

Jason Arbon

Kind of, I think people will always assume that, and we can get to it like we what kind of triggered us to maybe reconnect it was was um recent for this was you know I'm actually building some stuff for human testers to try to help them be better testers like I'm trying to help. I'm there's evidence that I'm trying to help these humans that don't really want to be helped. But when it comes to the are there any manual testers in the future, the reality is that people are gonna rename themselves anyway.

I actually think that's them So when you do the query manual testers, they're actually because the manual testers themselves... which is makes me breaks my heart, because I'm a manual tester. If you look at the Google book I worked on I'm a manual testing manager lead.

I don't care what title was but that's what was in the book and Max your title at Google was social engineer But I got in trouble because I also wasn't even checking any code in because that was that wasn't a hard problem So anyway Um So one is the testers are redefining themselves.

They think it's an ugly word I think manual testing is an ugly word and I think it's a it's an art form if done well and ironically I sound like the like a Bach or a Bolton but that it's very quickly from there It's not even really big deal to me it's but it's I think it's an art form when done really really well It's the hard it's the hardest problem.

It will be the last thing to be automated is the great human tester. Well, I think that will happen though. I absolutely do I think like we're like since if we talked like a year ago, in the last year, it does it's starting to protein folding now. Like protein folding I don't know like Microsoft there was some, I don't know if it's marketware, but like you know Microsoft had some headline yesterday that or two days ago that their AI has figured out a new chemical to replace lithium and electrical batteries And I'll add the best one was came out just two days ago from the CodiumAI guys.

Marcel

I don't know if you've seen this Yeah, I saw you share something but I didn't follow the details yet

Jason Arbon

That's what the paper does just let me tell you and then then we'll come back to the original question I think it's context to think about . So I thought that what they're kind of working on would eventually happen.

I thought it might be like next year even not this year, and I'm like an optimist. But what it does is it generates code. But think about this it's back to that test is dead too context. It generates the code to solve some problem, right?

It can create classes and functions, whatever. But rather than just like with today with coPilot, right, with Microsoft CoalPilot or something, it'll help you complete functions and stuff like that.

What it does is it takes the problem and then it creates code to solve the problem, some mathematical, some IO function, right? The cool crazy thing is what their paper does is brilliant. I think it's totally underrated right now is that it generates several different possible implementations.

It works with the LLMs to come up with different implementations of the code, right? The problem with every test you're talking about AI is like, I didn't give it very much information and it gave me a very poor answer, which is like how humans work too.

It just shows, reveals the sophistication of the user really. But in this case, the AI interacts with itself and its own state and it has regressively iterates on itself. So it'll create permutations of itself, almost like a genetic algorithm, but it's not really.

But like, and then tests, it'll generate the test for that. Those I/O functions, each individual modular part. And then it will take the one that passes the tests that it generated. And if it doesn't pass the tests, it goes back and generates more permutations until it generates tests for those that finally pass.

Does that eventually go awry? Yeah, but that's shift left software testing. In fact, the test is dead. Let me talk about humans and process and training and getting the developers to care and testers and SDETs to get further up and shift left way up into development.

This is as the code is being conceived and generated before it's persisted the disk, right? It's all permutated and tested. That's the most. It's our dream as testers. We thought this should happen ideally, right?

But it will never happen. But it's happening today. There's a paper written on it, Codium guys are legit. And it happened just in the last couple of days.

Marcel

What will happen then to the testers, if I could say that?

Jason Arbon

OK, I think, who knows what the time frame is. It's not 20 years hough.

Marcel

I appreciate that.

Jason Arbon

Actually, there's some good curves. I've done some little bit of work and presentations on this, recycling other people's actual intellectual work.

But you can kind of predict how it's been the compute capacity of the planet in terms of machine compute has been growing exponentially on a very, very predictable curve, regardless of Moore's law stuff.

And now it's just it's just keeping advancing and advancing. So with different chip architectures and software. And now, how fast is an LLM that now does things that we couldn't have even done before?

Didn't make was able to dream.

Marcel

So, yeah. But you see that now is a lot of opinions that AI is not like, you know, removing or replacing people. And especially the manual testers, because, you know, a lot of like testing experts still, you know, you know, trying to or sharing the, you know, the opinion that.

You know that we don't need to more, you know, test automation or test engineers because it will be enough to have some manual testers, maybe from offshore, give them some AI and you will have like super powerful superman that I don't know if it's really true.

Jason Arbon

Let's let's be like painfully real for two seconds. Okay. look at the credentials. Now I know that, you know, that's a, I don't come from Ivy League school. Okay. Don't I'm a university of Utah. So I'm not at the top of this list.

So I'm not biased in that sense. Look at the credentials and not even by credentials. I don't mean like school credentials, what they've done, like what they've actually built or accomplished or with their track-record in predicting the future is and see and you stack those up and the people that think things are changing very quickly.

and then the ones that that think it's just going to be the same that also have a bias in that fight and that argument and that thinking. It's very clear to me what the answer is, but it's okay if people want to believe that I'm okay with that.

To be clear, I'm fine with it. I'm not going to fight I I've tried to argue a little bit goes back to the philosophical stuff a little bit of testing but if people want to believe that it's okay.

There's this best quote I read the other day, which was that he said like no matter what you do and you tell people like there's always going to be like I don't know what it is 3% of people that are flat earthers that they just are.

I guess just it's and guess what? They're very vocal. The people that think the earth is round are usually pretty quiet. And they're also launching stuff into space. Like, so you kind of start to ignore the flat earthers and I apologize for offending the flat earthers that are also testers that may be a large Venn diagram overlap.

But no, I think, but I think the jobs will be taken away. The real story is actually pretty good for if you're a great manual tester, you're the last one to be replaced. Because all the energy, money and progress has been in replacing software engineering.

Even the thing I just mentioned with podium, right? And Microsoft's coPilot, Google has their own, their own code gen stuff that's fantastic and provably so.

Marcel

Plenty of such a tool for for for the code code general.

Jason Arbon

So so guess who's gonna a be losing jobs more dramatically?

It's gonna be the developers. Will it eventually get to the testers? Yeah. It's already starting to get to the SDETs, the API testers with the stuff we're talking about from Codium. But it's gonna encroach.

And here's the beautiful thing. Bear with me. You'll appreciates this. its all a lead up to this. I do believe that in 10 years, you will need very few. I think by the way, most software will be written for robots, not for humans to click.

That's embarrassing. Stonehenge-like right. Most stuff will be written for other robots. So they can do their own thing by themselves without humans. But the last person to go will be a human that says ship or not ship.

That may be the Eng director. But right before that will be the tester and before that will be all the engineers. I believe that's the way it's going to come. So we'll have a lot of warning in the testing world.

But I do think it's kind of happening. But when that even gets close to being that, the dynamics of the economy and world politics and security in general even is going to be, these things are technology integrated into , like back to my time at Lockheed, like in weapon systems and you know political game theory and things like that.

It's it's going to get more dangerous than like the last thing we'll be worried about by the time By the time most software testers lose their job, we'll have many other things to worry about. Or, and probably though, also matched with a lot of other cool things to do.

But nobody knows. If anyone says they know, they don't know. That's the biggest thing. The people that say that you can't get rid of the manual tester, they actually can't know that. Like nobody can know that.

They claim they know that. So, obviously-

Marcel

So let's see how it'll end up. It looks like it's a bit blurry in it. In a future. Maybe, as I said, I'm trying to fit it into one hour at least. So let's now move on.

You're supposed to just go with this. Yeah, good. And the next one would be, next one would be, you are building some new cool stuff and I would like to see how it works. And we discussed it before during the preparation that we will do some quick demo.

So would you like to share me my screen and let's play with that or? Yes, sir. You will do the magic. Share me that the future. Add it. Yeah, no problem. No problem. If it doesn't work, I will share my screen because I have installed your plug -in.

Jason Arbon

Hold on. You know that. That would break my heart. That would break my heart. All this work I did over the holidays. Oh, no, that's not it. Okay, hold on. I know what to do. All right, now we're gonna pretend like that just took like a second or something.

So this is what you were talking about. To be clear, I've actually worked on a bunch of other stuff. I was inspired for a quick context for this. It was Richard Bradshaw over at Ministry of Test, like a month and a half ago.

He, now independent or he's doing something on his own. now--he's doing another gig now. But he was complaining to be very clear. This is my apology for the bias that people think I have. He was saying, hey, these AI vendors are all working on automation.

How about helping testers? How about helping exploratory manual testers? I see a couple of people complained about that. So I reached out to him, like, what do you want? He gave me a one -liner, which wasn't a spec.

But he said, I am a manual tester by trade, and I love it, and I do it all the time. I have a hobby. Let's see if we can figure something out. So I did the most obvious thing, and just started sharing it out a little bit on the LinkedIn world.

But so here's the Chrome extension, right? Because everybody has a browser. You install the Chrome extension, and then let's see if it works. OK, starting to work. We can still see. You can't see the extension?

Marcel

No, no, I could see the green. Yeah.

Jason Arbon

Can you see the robot heads and stuff? No, no, no, no, no, no, you need to share the entire screen. Otherwise,

it will only share the tab. So yeah, I try to share the entire screen

Marcel

or maybe no, I can do that.

Mixed

I could try to embed the tab.

Marcel

Stop sharing like a window where there are no other steps so we can see only the window and then share the window. OK, yes, sir. Yes, sir. Thank you, sir. See. It's like practicing to share the screen.

Jason Arbon

Oh, oh, you know what? Forget that. If we flash something dangerous, life goes on. We have a little. We could cut it out. Marcel--the pictures you sent me a couple of weeks ago--ou can just edit it out. All right. See Google?

Marcel

Yeah. Yeah. Now we can see all of everything. Almost what do you have in your fridge?

Jason Arbon

Yeah, you can actually look at the hairs on my head. Yeah, those are my code windows. Very good. That's going to be boring.

So let's see. OK, so we're going to go here. So we launched the extension, on say Google. since everyone knows what that is. You saw it was kind of scanning the elements and the page and stuff.

Marcel

Mm hmm.

Jason Arbon

What you've seen here is across the top, we've got a bunch of little robots. We're robot heads. The idea is that the testers testing Google .com or some other website and they want assistance, right?

They want the AI to help them with their testing. This is what people have asked. And so this is an experiment to see if people care, if it's useful and whatever. So we're trying to make it as super easy as possible.

It also helps trying to solve the problem of all of the strange, the feigned helplessness sometimes on LinkedIn and Twitter of people asking, you know, very doing very poor prompting, I would say. So imagine basically prompting.

And it's in context. So you don't have to copy paste and not to think about the prompt. It just kind of does it I'm just saying that's an it's an LLM. It's actually two different LLM. So that's another story.

So you've got different bots. So this one this bot does breaking thing He tries to break your website comes up with ideas that would probably break the website, right? Not the happy path. This guy will look for bugs, right?

It won't go through them all but he'll look for problems in the page Crazy just comes up the crazy test cases. They're kind of fun from a nerd perspective like oh, I never thought of that Edge was something that people specifically asked for because they want edge cases....

They want they want crazy test cases But ones that would be reasonable they find a bug with it. Did they go? Oh, that's an edge case But they would still file it right thing ridiculous, but it's still kind of on the edge So we'll see here is this is what the edge bot says about the Google homepage So the idea is to give you some in context with we didn't we only clicked at once, right?

That's all we did AI does all the other stuff You can see here and I'm working on the formatting because you're getting a live version of this. So it says searching for an extremely long query. So it's telling the tester, hey, try a really long query, right?

Some newbie testers, right? Or some freshies may not think about doing that, right? And then you think, well, how long should it be, right? How many characters is that? It gives you a quick copy paste string.

You can just copy paste in to make a quick trial long query. And it tells you in here too, it gives you hints on like what you should look for, right? Like any slowdown or issues in processing the long search and in response times or crashes, right?

So it gives basic kind of information of like here, oh, and it also, yeah, let me see that second, but it'll tell you like what's, you know, some ideas of what's crazy to do. Another one was like, I thought was interesting, but I had never thought of myself.

I wrote a quick article about this, but clicking, right? So it says click the store link multiple times rapidly. And I was going like, okay, that's kind of dorky maybe even me, right? I've been around the block of it and I used to work on Google.

And I'm like, that's a little weird, but the funny thing is I clicked it a few times and it was way less respond... Like it only responds to the last click it seems like, but then I've used sourcing the code too as a nerd just and Google's got, their homepage used to be very vanilla.

It's got like all this obfuscated JavaScript and custom event handlers and tracking and stuff like that in it now. And so it's always that click is actually handled by a bunch of JavaScript convoluted code and third party.

I mean, a third party, but you know, pings back to the server and stuff like that. So clicking it multiple times could cause some issues. It's kind of an interesting one. But you can go down here, search for a non -existent URL.

Like people, when they're usually searching, they're searching for something they know, right? This should be there, but think about the opposite of that. So that may give, again, testers some thoughts.

And then lastly, this was a fun one was this thumb. It says, you know, search for like this query, right? Which is a quick snippet of CSS, but it was hilariously nerdy funny to me as a test nerd is that I like, I just, just hit run.

The first time I hit run on this bot, right, on the Google homepage, it sent that back. And guess what happened? My whole extension went red in the background. And I was like, going, I was searching through the code looking for the string red, you know, I couldn't find it except the substrings of other strings.

And so I looked for like variations of quoted red. It was because I wasn't escaping the very code in the extension. And so it messed up my CSS of my extension. So it found a bug in itself, which is kind of fun.

And you can go on and on and on. And there's just pages, these things that are interesting. So that's, that's the basics of the extension I talked about.

Marcel

And lastly, it might be next stage with this, because it looks like it could catch some simple technical bugs.

Marcel

Do you have some idea how to develop it further or how to evolve this.

Jason Arbon

No, none. I'm kidding. Yeah. So what, when is this little thing here? One is, you can also type in whatever you want here. So like, I don't know, ignore links and be really edgy.

Do you know what I mean? And guess what it does? It modifies. And now you can, like you were saying, you want to be more edgy. You want to find weirder things or focus on images or a search box or, or, or timing and things, you can just type it in there and it will use that to influence the response.

So that's, that runs today in the new version that I haven't pushed. There's another tab here, which if you really, you're asking, it's a whole 'nother thing here. You can imagine where this is going, but you can create a test plan here.

So I want to focus on monetizable things or just look for bugs or happy paths. Right. And then you come in here and you hit play and it will generate test cases. And when you click these little nodes, they'll go back 90% of your work now, but I don't want to risk it, but you click that and it will come back and generate more and more and more granular and more permutations of test cases.

So you can very quickly generate hundreds or thousands of tests, but in the areas that you want them to be generated. And again, you know, testers, like you were, you were responding to, are saying, like, just when they see the screenshot, they're like, Oh, well, that'll do the simple stuff.

But what about the really hard stuff? Either of you, I'm going to ask, I told you, I asked you some questions. Have either of you rapidly clicked a link in the browser in your testing work in the last six months?

Like click it as fast as you can. Many, many times.

Marcel

Is it like how, how this happened?

Like, how I'm saying,

Jason Arbon

like, this is what, I think there's a bias with testers say, Oh, well, that would just do the basic stuff.

I'm telling you there, this generates a test that said click, click the store link multiple times rapidly. And I'm like, I hadn't thought about myself, but have you, have you done this test?

Marcel

No, no, no, no, not really.

Marcel

I wouldn't, I wouldn't even consider that. But the idea here is that usually the, the, the, the complex scenarios are like, like,

Jason Arbon

so this is where this is where we part paths for a quick sec though. You deal with the reality of this all the time though, right?

I mean, with with your own company and people, people's expectations, I don't know how to oh here, hold on. Oh, I don't know how to cancel the full screen recording.

Marcel

Don't left, don't left the call.

Juraj

Yeah, by a share you can stop it.

Marcel

So you know what? There's a stop sharing on the on the below. Below there is a button. No stop sharing.

Jason Arbon

Oh, boy. There is a bar. Just like Google meet, just like Google meet. Yeah, I'm an idiot. Okay, done.

Yeah. So let me challenge you back real quick on this. Okay. So you've had the initial tester reaction. This is why I also have been engaging a lot lately is that you immediately go like, Oh, but but there's things only things that I could do.

You've looked at it for five seconds. It's come up with something you hadn't thought of. I can tell you technically, I think it's an interesting case. And you didn't even see the rest of it scrolling through there.

How do you know there's not complex flows and scenarios in there? I don't know. You're right. There's 20 different bots at the top there. I'm saying one of them is guess what it's called? User flows.

Yeah, I see. Guess what it can define? Yeah. And then the other feature. And you can say, the little text box, bear with me, hold on before we get the biological reaction real quick. You can say, "make them really complex".

And it's very technically difficult. And guess what the AI is doing right now? It makes it pretty complex user flow. And by the way, faster and cheaper than you would have done it.

Marcel

Yeah, that's pretty cool.

What you did now, you just inspired me to retest that plug -in because our extension or whatever they call it in Google. Because I was playing with that. If you remember, I sent you some feedback on that.

Jason Arbon

It was good.

Marcel

And what about the? And I have the opportunity now to ask the same question. Any updates on this storing the interesting ones? Because I feel like there is great brainstorming. And then I would like to store some set of top 15 or something like these other ones I would like to pick up.

And be in my folder. Sorry.

Jason Arbon

We're going to have fun here today. You kind of made me demo. So I'll give you the full demo back. So yeah, you had a great suggestion on adding history. And I think you also were talking about, can you just export a test, either as to a test case management system or something, or a copy paste it easily, which it can.

Marcel

But or the other one was export the Selenium for it or something like that. Yeah, the automation stuff. Yeah, the automation stuff. So number one. So here's the funny thing where I think testing is going.

Jason Arbon

And this is what I'm focused on, at least. You know, the whole phrase, like if you ask somebody, they want just faster horses or whatever, instead of cars. That's kind of where I feel we're at right now.

So, you as a human want the history. AI doesn't have a problem. They remember everything. They can get to it almost instantaneously. The AI is not asking me for a history context. Yeah, yeah, I know.

Marcel

In testing, and you might say I am wrong, but what we do usually, there are two approaches. One is exploratory. And sometimes you have some set of test cases you want to repeat again and again and again.

Jason Arbon

And maybe this is, yeah. So for that approach, where I want to repeat and to be consistent. Ah, you want to. So here's what I will say, though. So this is experimenting to see how the community interacts with this stuff.

Marcel

They've asked for stuff like that to help them. Guess what else? I would appreciate some reciprocation and help. You've had some feature requests. But also, guess what? A lot of the feedback is, not just to be very frank, a lot of the feedback from human testers is, they say they want something to help them, but they don't really engage in stuff.

Jason Arbon

You can talk to this, Marcel, a little bit, but use it Use some optimism. If you want to test it, it will never be perfect. Humans aren't perfect. They kind of suck, actually.

Marcel

But this is what I found when you talk to the testers, because I'm building also a tool.

Jason Arbon

Yeah, yeah.

Marcel

So the first reaction from the tester is, I will test it, and I will report you all the bugs. I'll tell you what's wrong with that. By the way,

Jason Arbon

I know it's a hundred and a half notes. And nobody buys a bug report, really, at the end of the day.

So the question is, I am happy, obviously, to spend even my free time, paid time, really, even, to work on this. I actually really want to help manual testers, exploratory testers with AI. But that needs to be reciprocated, to be frank.

Because do I wake up every morning, and when I wake up in, because I've got to wake up every morning, by the way, every AI company, including yours, has got to answer this question, which is, who am I going to try to help?

The tester that files me a bunch of bugs, and then just assumes immediately that, oh, well, I won't do complex scenarios, or I can't do this, or I wouldn't do this, rather than even giving it like a 20 -second try.

Or the AI, because here's a funny thing, in that extension, what if it didn't wait for the human to click the button to generate the more test cases, and it just executed them? So the demand from the human side has got to be pretty strong to warrant taking something that works in the order of hundreds of milliseconds, and slow it down to like humans, who, they go home at night and sleep.

So I think this is what I mean. So I think there's going to be, like, they talk about digital, I think there's going to be an AI divide coming. This is my, and that's a concern. I'm actually very worried about it, is that people, humans that should, benefit from AI won't try it out.

They just won't get in the car. They're just gonna stay on their horse. They'll never get in the car because it's dangerous and loud and noisy. You know, and they just won't be driving. And they'll, you know, people live their life and never having been in a car.

It's okay.

Marcel

This is something that I really, really like. You are saying I don't care about the ones who are skeptical. They could stay with their horsey.

Jason Arbon

I'm not gonna want to. I'm gonna get some like dramatic gestures and holding my heart.

I want to, but I can't, really like, you can't drag a horse and make it drink. I can bring it to the water. Like right now, what I just did, well, and this is a minor part of what I'm building, actually.

Other stuff is just fully automated. Why have the human loop? I'm trying to let the humans into the loop. But if they won't dive in, I can't help them. I can't make them dive in. I can't make them dive in.

Even with you're like an AI nerd, AI developer, like working on automation tools and all stuff. And you're going like, you know, you see, what about that? What about that?

Marcel

Like, uh, but I know these questions I'm getting as well.

Jason Arbon

What about these complex scenarios? How you would do that or that? So I'm just passing down the same. If the AI can just do it all in a loop, like leaving the Codium stuff, forget what I'm talking about.

The Codium stuff at the API kind of unit test level that we wouldn't have, like everyone wants to ignore that. Everyone in the community, no one in the testing community will follow up on that paper.

It's published and sourced to you, but no one will follow up because they're worried about it. So I generally worry about those people, like because they say they're stressed, but they won't take any medicine, but I worry about them because they're stressed, but I can't also reach out through the ethernet and grab them and help them, make them install an extension, make them give feedback, say, Hey, what about, even if there were no complex scenarios?

Jason Arbon

Well, guess what? If somebody sent me a doggy treat and maybe it's a Starbucks coupon, I could probably get a bot that would do what they want. But just going to talk.

Marcel

Last question. Last question.

Jason Arbon

This is where I'm going to get banned from the internet.

Marcel

I was on that

last question this morning. Last question in this topic. And

Jason Arbon

And then talk about sharing information within the organization. Yeah. Not, but they're worried about AI, but the only thing was talk about how to build reports.

Marcel

Go ahead. Sorry.

I see. So what are the next plans of this? Is it voluntarily like a hobby project? Or some

no, no, no, no, no, I'm this because you know, I'm, I'm 100%. I'm working more now than I did at test.AI .

And I know that right now my bots deliver more value than I built with $30 million from Google at test AI. And but I, but the funny thing is, is I don't, I have an interesting thesis, I don't, don't think the world will actually take this incrementally.

So I have to think about go to market, fundraising, building, all that stuff. And I want to get all the technical major hurdles out of the way before I hit the gas this time , last time I didn't.

Juraj

And it's purely your project?

Jason Arbon

I want to figure out what's going on and then hit the gas, with go-to=market money.

Marcel

So are you working solo on that or?

Jason Arbon

No, we're 20 other people. But they're not a LinkedIn, they're LLMs.

Marcel

They're not people then.

Jason Arbon

No, I gave last year, I told myself I would only collaborate with, I sound like a crazy, I should be in a cabin somewhere, writing a manifesto. But I committed myself to only work with robots, with AI all last year, all calendar year.

And I didn't want to be influenced by humans that want. history, browsing, and copy and paste export, and

Marcel

I wasn't so bad on you, I was just asking some typical testers questions and it looks like...

Jason Arbon

I'm giving you trouble, you're giving me trouble, I'm giving you trouble, it's, it's, because you know the problem, right, you're seeing the same things. I just respond to them a little differently, I'm just going, I'm gonna work with AI to solve the problems, and if AI collaborates more with me, I'm happy to collaborate with AI-- because it can give me feedback too.

Juraj

Yeah, and may i will ask for the next steps, but you know, what is the main goal, what you would like to achieve, you know, that now you're the next... What is the main dream? What is the main dream with the check, you know?

Jason Arbon

The same dream I've always had, which is probably ridiculous, it's the same one as ever. It's every testers dream, everyone's thought the same thing, very simple, what if the just code just tested itself, that's it, it's pretty simple, it's hard to do.

Marcel

Oh, okay. I would close it here, with that statement, I think this is a good...

Jason Arbon

But we'll see, check back in, check back in three years, I'll be manually testing for a local VHS video store company. But we'll see, I'll try.

Marcel

No, no, I can't do this. Okay, we can move forward to the next or the last, you know, section and this is promotion. And if you would like to promote something, or celebrate, now is your time.

Jason Arbon

I don't actually...

When it's ready, it'll be ready. My goal is just test, my goal has always been, it's stupid, I say it publicly, no one listens to me, but it's to test the world's software. I just think...

Marcel

I heard that

Jason Arbon

I just kind of did.

Jason Arbon

And then we tested, I'll say this, this is maybe a self-attay-boy thing then. A little AI-panda.

Juraj

AI panda!.

Jason Arbon

I talk about myself and my accomplishments. I actually like him. I gave him a hug in the food line at a conference recently.

Jason Arbon

But no, at Google we had software. So the problem was anyway, what happened to test.ai ? I was just focusing on this meta goal. We suddenly tried to make a business out of it. But we were trying to figure out the hard problem first.

Google was a patient investor for that, which I appreciate him for. But with pre-GPT technology, it's a game changer. We built a lot of cool stuff. We would test thousands of apps, Android, web, thousands of them a day, lights out lab, like three nines, and then stuff just retried and it worked.

So really kind of almost 100, like almost infinite nines. And we sold that to Microsoft and Google. We were doing stuff that Microsoft and Google couldn't do. So that's not, and that's with earlier AI stuff.

So my point is that we were getting close to that objective. But as soon as I got pulled toward selling to people that just wanted to drag and drop and needed a UI to custom-craft individual test cases, which were not always that interesting, even if you looked at them, you're like, ahhh, maybe you shouldn't be.

I think there's better things to do with your time. Or just manually check it, which is where the word comes from. Actually, I'll tell you. Sorry. So that testing at scale had a first go at it, and I'm making another go at it.

But that's what you should try. But it's legit stuff. These people have paid for it and invested in it. So the fun thing is this. I'll leave you to this one. You guys will get this. So why did I call the company Checkie?

There's a lot of reasons. I'll list a couple of them, not all of them. One was that whole debate between just a lot of people think that code can only check. It can't test. And it's just so embarrassing after GPT that that's not embarrassingly not true.

And it can generate better test cases than people on this call have come up with. So that's kind of an inside joke a little bit. And then I'll leave it at that. There's like four other ones that are pretty funny.

Marcel

I think. OK. Maybe let's now close the ....

Juraj

Yes, now is your time. And maybe you can promote something.

Marcel

Yeah, I would just like thank you, Jason, that he joined us today. And I'm really happy that we had that.

Jason Arbon

We finally got the chance. Yeah, I'll probably take a nap and then try to send you a thousand dollars not to publish this.

Marcel

It'll be great. It'll be OK. It'll be good.

Jason Arbon

All right.

Marcel

Thank you very much for joining us.

And we could stop recording.

Juraj

Yeah, thank you very much, Jason.

Jason Arbon

All right, thanks, guys. Appreciate the patience.