Write Fewer Tests! From Automation to Autogeneration – David Khourshid – React Rally 2019

Write Fewer Tests! From Automation to Autogeneration – David Khourshid – React Rally 2019


>>Thank you! So by the way, let me just clear something
up, my last name is not piano, so I just wanted to make that clear. I’m really excited to be at React Rally. This is the fourth time that I’ve been here,
attended twice and I talked in 2017 about an equally boring topic, which was state machines,
but thankfully I might or might not talk about state machines again today. We’re talking about the terrible concept known
as testing. I think I could speak for everyone or mostly
everyone, in saying that testing is terrible, and it’s a painful thing that none of us want
to do, and apparently it’s a controversial topic, too, but at least I’m not talking about
CSS so we can just move on to testing and it’s that a lot of people have really strong
feelings about testing and my theory is that we really have strong feelings about writing
tests, not necessarily testing itself. So I put out a poll and I wanted to know what
people disliked most about testing applications that they’ve written and the majority of people
just absolutely hate writing tests and a close second is updating and maintaining those tests,
as well, and so what I want to talk about today is how we could fix this first point. And not by making testing more enjoyable to
write, but just eliminating it altogether and at the same time we’re going to be covering
updating, determining why tests fail, and discovering edge cases, as well. So if you take a look at your codebases, you’ll
see that as requirements grow, your lines of code grow, too, and it is a mathematical
fact that the lines of code needed to write tests are so much greater than the lines of
code that you write in your application, because if this is not the case, then you are missing
tests. So as requirements grow, your tests grow,
I wasn’t able to fit it on the screen, but they grow exponentially. And the problem with this, too, is that requirements,
you see this curve, it slows down, because they say, all right, they have enough tests
to write a lot of code to write, and so you’re not able to get features and instead you’re
spending time writing like all these lines of code, instead of doing all the feature
work that you should be doing. So there’s two schools of thought for this. You could either write a ton of tests, as
many as possible, I call this the Write Tests Frequently technique and it’s by Dan so you
know it’s a good technique, or you could be like Ryan Florence and just write code perfectly
the first time, which apparently is what he does. Or like Michael Jackson who wrote the Expects
library, you could write a unit testing library and encompass techniques like red, green,
refactor or well, because it’s Michael Jackson, it’s black and white refactor instead. Funny enough, his repo is broken, but that’s
just human nature. I’m sorry, I’m sorry. I see some of you walking out now, I totally
get it. So here’s what we were doing. Before test automation became a thing. We manually tested the code. A lot of you remember the pain of this. We would write the code and then we would
look at the business requirements and we would manually execute our application clicking
things and typing things, and just making sure that our code worked, or better yet,
we would hand it off to QA, who always found the way to break our app, no matter how good
our code was. Thankfully, this got a bit easier when we
started automating our tests, because instead, we could write these unit and integration
tests for our code and hopefully have them write. But what we’re going to talk about today is
modeling the code so that we could automatically generate exhaustive tests for every possible
thing the user to do in the app. And this isn’t the only level. There’s something called formal methods where
you could formally prove that it’s impossible for your code to have bugs. This is actually a thing, and some companies
do this. But the highest level of enlightenment which
maybe I’ll talk about one day is just delete your code, because —
[laughter] Then you don’t have to write any more tests. So this is a testing trophy and it talks about
the importance of tests. We have static tests where, you know, this
comes from types, whether you’re using TypeScript or if you’re use Flow for whatever reason,
I don’t judge you. We see that integration and end to end tests
are the most important tests. Ideally this is what our test structure looks
like, but in our apps it’s less after testing trophy and more of a testing ribbon, because
it’s more like “you tried.” What I find a little ridiculous too, is the
fact that the code that we write, we’re writing the tests for our own code he so we’re sort
of validating ourself which doesn’t make sense we’re patting ourself on the back and I can’t
think of any other industry where mission-critical safety code that might endanger the lives
of others is tested by the exact same people who wrote the code. [laughter]
I can’t — I’m sorry, that might be too soon. I didn’t know if that was gonna fly. [laughter]
Sorry. So here’s my thinking. Instead of writing all these tests, what if
they could be generated instead of being written? And what if these tests can automatically
regenerate themselves without us having to rewrite the tests? So this is the idea behind model based testing
and just like state machines, this is not a new idea. it’s a bit younger than state machines, it’s
about 20 years old give or take where the first research paper of 1999 was written. And ever since other industries have adopted
it in some level. There’s open source tools available for this,
but just like state machines it’s something that we’re not too familiar with in the frontend
and frankly the native Mobile Web dev technology industry that we’re in. So let’s do a puzzle over here, how many of
you have seen Die Hard. So you might remember this puzzle from 3 and
5 gallon jugs, how do you measure 4 gallons exactly? You could power water into one, you could
transfer water from one to the other, it’s a nice tricky puzzle. So let’s try to figure out a solution. What could we do? We could — let’s fill three up, right? And then we could transfer that to the 5-gallon
jug and then we could fill three up again. This is just one of the solutions that I,
you know, sort of came up with. And then I was thinking, we could take the
3-gallon jug, put it in the 5, which gives us 1 gallon in the 3, and then we could empty
the 5-gallon jug, power the 1-gallon into the 5-gallon and then some of you see it coming,
we have three gallons and we power it into the 5 and there we go, we have 4 gallons. So speaking of water. Water break. What does this look like? It looks like a state machine, right? I mean, I don’t know if I’m taking crazy pills
or not, but that definitely, you know, looks like a state machine and in fact we could
represent this entire thing as a state machine. So if you didn’t see my talk two years ago,
quick review: You could represent your app or your components or whatever as a state
machine in which we have an initial state, multiple states, events that trigger transitions
between states, and possibly final states, too. So that was my, you know, last talk summed
up in 20 seconds. All right, so this jug game can be represented
as this finite state machine. And because of this, the solution is just
following the path that leads us to one of the final states. And so this is one of two solutions for this,
and you could expand this to similar puzzles of that type. So the important thing is realizing that a
lot of things, whether it’s business requirements or your component logic, or even your app
logic at a higher level, is a state machine. Even biological process, even you are a state
machine. It’s a very important realization. So let’s talk about actually turning these
business requirements into a sort of state machine. You might have seen given, when, then as sort
of a format for writing these requirements, such as given a user is on the home page when
they click the dashboard link, they should be on the dashboard, so we have a precondition,
being on the home page, an action, which is clicking the button, and a post condition,
which is being somewhere else. So this could be represented exactly like
a state machine and so all we have to do is turn that state machine into executable tests
by adding assertions on each of the preconditions and post conditions, and adding some sort
of execution mechanism for the action, like basically clicking a button or typing in some
text in an input. So here’s how we do model-based testing in
five easy steps. First step is you create a model of your application,
and so this is going to be the most important step and it’s useful not just in model-based
testing, but in application design in general. So represent your app as these dots. Call them testing dots, call them Kent C.
Dots, whatever you want. All right, I’ll stop with the puns, sorry. From there you generate abstract tests and
all this is doing is path-finding, so we’re finding all the paths from one state, our
initial state, to ever which other state in our application. From there you make them real. You associate transitions with action, such
as a mouse click, keyboard input, some time delay, or anything else. So we have a precondition, the action happens,
and then we test the post-condition. All these need to be within the tests because
we need to actually execute them and make them real because our abstract tests do nothing
until we add these concreatizations of the tests. So then we execute the tests. The tests will quite possible fail: What this
means is one of two things. Either our model is wrong and doesn’t describe
the application exactly or our application is broken and so it’s up to you to figure
out which one it is. But then, if your tests pass, once you fix
everything you could seriously profit and what I mean by this is you could literally
save money. Because you could change the model. When that happens you don’t need to rewrite
any of the tests, the model will regenerate the tests for you, because again it’s just
an algorithm that traces each state in each transition and tries to go to every other
state. So let’s do a really quick example over here. Let’s say I have this very simple feedback
app where if you click good, it goes to things for your feedback, if you click bad, it goes
to this form where you know, you have to enter why your experience was bad and then when
you click submit, it goes back to that thanks for your feedback screen. So this could be done manually, and I’m using
the amazing testing library of React’s library by Kent C. Dodds, as well, so we’re pulling
things from the testing library and we are doing three things: Remember we’re doing our
precondition which is making sure we’re in a certain screen, we’re doing an action which
is making sure that an action that takes us to our post condition is executed, and then
we’re actually checking that post condition. But this is a lot. It’s like we have two full flows and that’s
already like 30 lines of code, just to do two flows you know, making sure we go to the
think screen and this one is going to the form screen, as well. So instead I’m proposing an alternate approach. What if we represented this as a finite state
machine and I’m using my X did library to do it
so when I’m in the question state and I click bad, it takes me to the form state and so
anyone whether you’re technical or not, can understand this state machine and be able
to trace through and verify that the state machine matches the business requirements
of your app. So then we get to the fun part, which is generating
all of these different paths that take us from one state to the other through our application. In this case I’m generating the shortest paths
and you could imagine a big long JSON output that outputs all of these for each state,
and so for example, there’s two ways to get to the thanks screen, you could click bad
and or. Because we’re representing it as a graph,
we are exhaustively checking all the different ways we could get to each of these states. And so, what we could do now is instead of
using this graph library to generate these, I’m going to use ex state test which is a
brand new library that I literally just released this morning, deploying it on Friday is a
bad idea. I don’t want any bugs or anything. We could add each of these assertions to each
of the states and as you can see, there’s one to the question state and that’s just
verifying, are we really on the question state? So with react testing library it becomes a
nice one-liner where you could use expect, I’m using assert. I’m just making sure we do have the question
screen on there. Now all we have do is create a model using
this testing library and that model is going to do all the complicated algorithmic work
of generating all of our paths for us, but one important step: We have to make sure we
could make the events real and we could associate, for example, click good, we associate that
just to clicking the good button. And so we could also specify test cases, as
well, if we want to do things in the future, such as validation, like not letting them
submit if they’ve entered nothing in. And so using this testing library, what we
could do is generate test plans that give us all of the paths for each one of the states
that we’re trying to go to. And then we could iterate through those paths,
do the setup, and then test those paths. So I’m actually going to show you what that
looks like, but of course the final important thing is making sure that we reach every single
state, and that’s what this that test coverage function is for. So let’s do a demo. So I have my feedback app over here, and I’m
doing exactly the code that you saw on the screen, because I literally copy and pasted,
so it should be the same code, and so let me just show you what this looks like real
quick, just to convince that you it is actually a real app. So we have a very simple screen and of course
it does exactly as we expect. But this is us manually testing. Using React Testing Library instead, we can
verify over here, and using the testing library it’s going to generate those paths and test
every single state, and test every single transition, as well, making sure that we could
reach all of these states, and just to prove that this does in fact reflect our application
business logic, let’s say that we forgot to add the submit button on the form. Now it’s going to fail. Don’t mind this. This is just barfing all over the place. But look up ahead! [laughter]
We had the path and we could tell exactly where it failed. So you could copy and paste this and you instantly
have a test reproduction that you could share with anyone who cares to see that reproduction. So when you’re on the form state, you could
see that that submit button, that events failed to execute, and so when we add it back in,
the tests will pass again. So
[applause] So there’s many ways to generate the paths. There’s shortest paths, which are literally
just the quickest way to get from one state to the other. But just like you might know from driving
or anything, the shortest path is not necessarily the shortest distance. In fact, we could represent shortest paths
by weight and we could, for instance, determine that users take a different path more often,
and so we want to place more importance on those paths for testing and you could use
something like Dijkstra’s algorithm. I typed this like five times before I finally
got it right, by the way. Or we could do simple paths which are much
more exhaustive. Simple paths are all the pacts that it takes
to get from one state to another state without repeating cycles. So in this case there’s a few of these. So you also have to do is trace a path and
you’ll see. So this is much more exhaustive and it’s going
to create many, many more paths. So, you might be thinking, like, all right,
you know, I see it in the tests, but is this limited to just, you know, testing within
React Testing Library and the answer is no. If we take that exact same testing model and
instead, add, say, Puppeteer to it. Let’s see what happens. I hope this works. So instead of doing React Testing Library,
it’s going to open up a version and literally go through every single one of those paths,
executing them on a real browser and so this is going to be your full experience, and we
have all the tests passing. So…
[applause] All right. So you could sort of think of this as time
travel debugging, I know it was talked about, but this is a little bit different. Instead of going back in time, now we’re seeing
all possible future events, and so this is really powerful. [laughter]
All right, so what does this have to do with React exactly? Absolutely nothing! And that’s the point. You could do this in Vue, Angular, it doesn’t
matter what you write in our application in, we are describing our application in this
abstract model that is implementation agnostic, which means that we could use the same testing
model for anything we implement our application in. And so this is a pretty powerful concept. And real quick, I wanted to have fun with
how many of these paths are generated, just because over here it’s generating 12 paths
because we have a fairly simple application, but what if we have an application that’s
more in depth? So over here, I modeled a tic-tac-toe game. The machine is over here. It’s a pretty simple machine. This is what it looks like. Don’t know if you can see that, but we have
three states playing winner. And so you know, we have that, oh, is this
going to tie? Yeah, it’s gonna tie, but anyway, you get
the idea. Now, because I have this state machine, I
thought, can I use this same machine to both power the app and power some — I don’t want
to call them tests, because it’s going to get a bit crazy, I’ll show you. But what I did was I took this machine, and
I mapped each one of the events, the player events, to actually clicking a square. So whatever the value is, whether it’s the
0, 1, 2, 3, 4, 5, 6, 7, 8, square I’m just finding that square and clicking it in Puppeteer. I narrowed it down, I said I want all the
paths where the state matches winner, there’s more than five moves, and the winner is X. So … hopefully … it will do that. And so it’s going to launch the app and generate
all those paths and then we’ll see. if it loads. There we go. So it’s going to find every single combination
AUDIENCE: Whoo! [applause] All right, let’s stop this before it gets
out of hand. All right. So … I released X state test for this, and
of course no documentation yet, but there are other similar tools for this that have
existed in the past. I don’t know how up to date they are, but
there’s Simulato, which has similar ideas, and also graphwalker, and that’s for Java,
not for JavaScript. I did the recruiter move, I’m sorry, that’s
for Java, not for JavaScript. [laughter]
So I’m going to be honest with you, model based testing is harder, I don’t mean in terms
of effort, I mean in terms of mental energy, because there is a learning curve in terms
of how you model the application and just learning the art of modeling so you could
learn your application in these state machines or state charts, but it’s also better, faster
and stronger for many different ways, it’s better because you’re going to cover many
more edge cases than a human could even determine manually in their lifetime. It’s also going to be faster, because you
no longer need to update tests or write tests. all you have to do is update the model it
needs updating and in case you want to support different browsers, different frameworks,
you just update the model tests and so these are going to result in stronger tests, because
you are covering all of those edge cases, you are tweaking how fine-tuned and how much
coverage you want your tests to have. And this is possible with model-based testing
because we’re not writing tests. It’s the entire point. So these Xstate tools are part of this whole
idea of model-driven development and that’s why I say that Xstate is so much more than
just another state management tool. How many of you use Gatsby? All right, a fair amount of you, so Gatsby
actually uses X state under the hood in order to manage processes like file managing and
stuff like that. Also React training uses it, I use it at Microsoft,
because I work at Microsoft and I could do that without them knowing and it’s also used
at other companies which I’m not allowed to say for whatever reason. And it’s also used for this really fun game,
some of you might know it. It’s called service workees, and this is a
really fun game with lots of levels and complex animations which are driven by X state and
by these state machines and you get this really great experience which is complex and also
helps you learn things like service workers. So X state is evolving. I just released X state test and I’m coming
soon with so many more tools for X state. Also, there is X state React, and it looks
very similar to the use reducer Hook, all you do is you pull this use machine hook and
you put a state machine instead of a reducer and you have the same state and dispatch but
they’re called current and send and it’s a really powerful paradigm that you can use
to power your React applications. So basically I want to make your code model-driven
and I want you all to make your code model-driven, because this allows you to do so many more
things, it allows you to generate tests, generate documentation from those tests, it allows
you to even generate prototypes and visuals, so in short, I want to make your code do more. Thank you, React Rally. [cheers and applause]

Leave a Reply

Your email address will not be published. Required fields are marked *