May 18, 2018
The Agile Toolkit Podcast
I always say that DevOps in one sense is really an extension of agile principles out to everybody on the ship. -Jeffery PayneBob Payne chats with Jeffery Payne, Founder of Coveros, at Lean+Agile DC 2018. The Payne Cousins (not really) chat DevOps and tips for pairing developers and testers. The discussion covers moving toward a generalized specialist model, testers showing up like a demolition crew, and the true meaning of pairing. [caption id="attachment_7988" align="alignnone" width="2024"]
Jeffery Payne sits down with
Bob Payne (not cousins..)[/caption] TRANSCRIPT Bob Payne:
[00:00:02] Hi I'm your host and technical idiot, Bob Payne. Just
struggling with the equipment there for a little bit, making making
the the the big newbie mistake of hitting play instead of record.
So I'm here at Lean + Agile DC 2018 and I'm here with Jeff Payne of
Coveros. Jeffery Payne: [00:00:25] Your cousin right. Bob Payne:
[00:00:26] Yeah. Cousin Jeffery Payne: [00:00:27] Yeah. Bob Payne:
[00:00:27] Yep. So Jeff what what are you talking about here today
since I am out here in the hall and not not in the talks. Jeffery
Payne: [00:00:38] Yes I'm talking about dev test pairing. Okay so
trying to get developers and testers to work together better. We
find that that's one of the biggest issues we see on teams when it
comes from engineering perspective. Bob Payne: [00:00:52] Yeah I
mean I think the early agilists were a lot of XP teams that sort of
did away with testers because everybody was considered to be a
tester. I think it was also sort of a chemistry of the particular
group of folks that were on that first team. And you had folks like
Elizabeth Hendrickson, Lisa Crispin. A lot of folks sort of brought
testing back into the Agil fold. Yeah what do you think the biggest
problems you see with testing and agile teams or trying to get
testers and coders to pair? Jeffery Payne: [00:01:31] Yeah I think
obviously one of the biggest problems is that they historically
haven't worked well together. They're kind of on different sides of
the fence as a check and a balance in some organizations right.
Jeffery Payne: [00:01:42] And and a lot of organizations even they
prefer that their testers not even talk to their developers they
want them to be independent speak because they think it's kind of
like an editor if if you haven't seen it and then you review it
another set of eyes you're not you know you're not influenced by
the development. The other sort of clean room actually that's the
traditional approach. Of course it's always been very late
lifecycle and very manual right. None of those things work well on
edge. All right. Bob Payne: [00:02:11] Well none of those things
actually work well in life. It's not just an agile thing. Jeffery
Payne: [00:02:16] So you know how do we change that? Bob Payne:
[00:02:17] Shoot, it's not secure and doesn't scale. I'm glad we
have 12 hours to fix this before production. Jeffery Payne:
[00:02:22] Yeah, Exactly. Here you go have it done by tonight. So
yeah. And so what we try to help teams fix that. Bob Payne:
[00:02:30] Yeah. Jeffery Payne: [00:02:30] Address those issues.
Bob Payne: [00:02:33] What are you what do you think has been most
beneficial recently for helping you in that in that quest of
getting folks to pair together. Jeffery Payne: [00:02:42] Well we
have some techniques and approaches that we like to use to try to
get them to work together and also learn from each other because
you know if you're moving toward a generalized specialist model on
your teams we like that model. Yup and you want collective code
ownership and you want a whole team quality all these you know
motherhood and apple pie concepts that we espouse too. You've got
to get everybody productive during the entire Sprint or whatever
you're doing story development or whatever. And the only way you
can do that is that people start learning from each other and cross
fertilize. Historically you know I was a developer developers
aren't great testers for a number of reasons. Jeffery Payne:
[00:03:20] Just you know out of the gate they're not very good
testers and testers oftentimes particularly if they are manual
testers they don't have a very strong technical background they
don't know code they can't write automation right. Those two things
together don't work very well. So we've found that by pairing Dev
and test they can help learn from each other and become stronger
teammates and collectively on the code better. Bob Payne:
[00:03:43] Now do you find that tools like cucumber or other. I
don't know if you're running into teams using fitness but are early
on fitness is one of those tools cucumber most recently specked
flow help bridge that gap so that testers can blow out those
scenarios a little more directly after the fixtures are done or
even before the fixtures are done. Jeffery Payne: [00:04:09]
Definitely. Yeah I mean the the BDD oriented. Bob Payne: [00:04:12]
Yeah Jeffery Payne: [00:04:12] Cucumber with Gherkin, kind of
natural language approach is a great way to start moving
particularly manual testers toward understanding how to automate
without having to dive right in and start like you know trying to
write good maintainable selenium scripts for instance or whatever.
I mean it's hard to write maintainable any kind of scripts. Bob
Payne: [00:04:33] Write would be better then record -those are a
nightmare to maintain Jeffery Payne: [00:04:39] No doubt, or record
any test is a bad idea because that's how they're sold often so.
Bob Payne: [00:04:44] Right. Jeffery Payne: [00:04:45] That's how
you know people think you're supposed to use those tools. We
definitely like those kinds of tools that we think they help move a
a tester toward being more capable of providing automation support.
Bob Payne: [00:04:57] What sort of behavioral, I mean, You
mentioned the word pairing. What does that mean when you say that
because I see a lot of I see a lot of misuse of the word. I'm
assuming you're not but the mis use of the word pairing Jeffery
Payne: [00:05:09] I Might be, who knows. Maybe you'll tell me i'm
wrong, Bob. Bob Payne: [00:05:11] And TDD, I see a lot of people
misusing or not really understanding TDD. That's most common but
Jeffery Payne: [00:05:17] Yes. Yeah. So I mean to me I'm basing it
off of the definition of pair programming. Go you know getting two
people together to work together collectively on some task. When
you talk in dev test you're really either talking about those two
people working together on code almost pair programming and one of
our techniques is to use a dev test to pair program yet which is a
little different right because one of them maybe doesn't actually
know how to write code. So what does that mean. Right. In pairing.
The other thing we use it for is to review each others tests. So if
you're going to ask developers to do a unit test you want them to
learn how to write good unit test meaning think through not just
happy path but you know the errors and boundary conditions
exceptions and all those kinds of things they usually inherently
don't know how to do that a tester can by working with them help
them understand how to do that better. Second if you're asking your
testers even if it's manual to create tasks for integration for
system for you know kind of the combinations of things across use
cases and your business flows they often don't they often won't
load the design. Well enough particularly if they haven't been
involved in those activities they should be but often aren't.
Jeffery Payne: [00:06:34] Yeah and the developer can help them
think through and understand how does this software all pieced
together to meet the you know the flow that we're looking for in
our application and how users use it so they can help each other
from a testing perspective we found. Bob Payne: [00:06:47] And one
of the other things that I think a lot of a lot of testers can help
with as well is what are the business rules like oh yeah if you're
doing an under UI test which quite often happens in the developers
domain you know what are the what are those conditions you know the
happy path is easy and that's usually where developers go because
they know the happy path works but they don't necessarily test
those boundary conditions as or that or the business rules right if
I had a whole bunch of J rules or other stuff I wouldn't test that
through the UI right. Jeffery Payne: [00:07:26] Yeah no doubt. Bob
Payne: [00:07:28] Yeah. Jeffery Payne: [00:07:28] And to your point
about a happy path. The other thing we've seen is not every
developer's like this but you know a lot of developers consider
what they're building to be a work of art. Right. They're like
Michelangelo creating the Sistine Chapel in their in their mind.
Yeah and they're all about creating this beautiful incredible thing
that's going to last forever and just people are going to you and
all over it even if it's just their peers. Bob Payne: [00:07:49]
Yep. Jeffery Payne: [00:07:50] And then the tester shows up testers
like a demolition crew. Bob Payne: [00:07:52] Yeah Jeffery Payne:
[00:07:53] Right. They're trying to poke holes in it and figure out
what's wrong with it and it's kind of like calling your baby ugly.
If you're asked to test your own code because you know you might
have every intention but in the back of your mind you might be
thinking I don't really want my Sistine Chapel to have problems in
it or look bad and changing that mindset is part of getting Dev and
tests to work together to understand the best way if you want to
build something great is to find any issues as fast as you can see
eradicate them. That's really about what it looks like when it gets
delivered yet not what it looks like. You know while you're making
the sausage right. Bob Payne: [00:08:27] Yeah. I find a lot of
people use the term Pairing and they're really talking about
working together on just acceptance criteria or something like that
that's necessary but not sufficient. I think that deeper level of
the deeper you can go in interaction and an understanding the
better off your team is clearly Jeffery Payne: [00:08:52] We've had
good success getting developers involved in doing some exploratory
testing as well. Bob Payne: [00:08:57] Sure. Bob Payne: [00:08:57]
You know a lot of times testers get together and do you know
session based exploratory testing across stories or whatever. What
about the idea of just getting the Dev and test together for a
story they're working on and having an exploratory testing session
where they work together and explore the product and talk about it
and identify bugs. Again that gets the developers a little bit more
comfortable doing testing and knowing what to look for thinking
critically about the app. And of course it helps the tester better
understand the app because if they're they don't understand
something about what they're testing they've got the developer
right there they can ask Hey what was this supposed to do or how
was this supposed to work. Jeffery Payne: [00:09:32] Now I think
the story is maybe vague did we really build the right thing or are
we testing it properly. That dialogue's very helpful. Bob Payne:
[00:09:38] Yeah. What else is exciting in your your world right now
Jeffery Payne: [00:09:42] Nothing Bob Payne: [00:09:42] No? Jeffery
Payne: [00:09:42] Nothing. Well as you know we do a lot of DevOps
work. Bob Payne: [00:09:47] Yeah sure. Yeah it's the new edge
issue. Jeffery Payne: [00:09:51] Yeah exactly. Bob Payne:
[00:09:52] Yeah. Actually you know we were going to be talking
later with some folks talking about sort of you know in many ways
Agile is sort of hit a ceiling and I'm hoping this will open up
gaps where we can get to real real agility and real cause. All too
often it's seen as a fix for the delivery team not right. Not a
systemic change that can build better value faster. Jeffery Payne:
[00:10:23] Yeah and I totally agree. I mean I think one of the
mistakes that the founding fathers of Agile made is you know they
were all about collaboration getting everybody to work together.
But they forgot a key piece of the lifecycle which was delivery and
release and production and production oriented. Bob Payne:
[00:10:41] And actually intake in the business side. Jeffery Payne:
[00:10:45] Exactly. You know it's funny this group that was all
about collaboration and getting everybody on the same page left all
these people out right by mistake. Obviously they were creating it
as they went so I understand. So I always say that dev ops in one
sense is really an extension of agile principles out to everybody
on the ship you know involved in the software delivery process in
the full lifecycle software. Bob Payne: [00:11:09] Yeah and agile
and dev ops are both the you know great grandchildren of lean which
was all about that base that whole process right. Jeffery Payne:
[00:11:21] Yes. Bob Payne: [00:11:22] Yeah. You know this
reintroduction of the concept of value streams and value team and
stuff - It's like back to the future. Jeffery Payne: [00:11:32] I'm
sure you've studied up on the history you know all the way back
through Demming and you know all the way back to you know
statistical process control and even beyond that I mean it's
clearly standing on the shoulders of giants like everything we do.
It's amazing how many people don't understand that or take the time
to find that out or understand. Bob Payne: [00:11:50] And the idea
that that actually Devops, Yeah there's a whole bunch of cool
technical stuff going on, but it's about closing the loop to be
able to learn. And my favorite Demming quote about that was
learning is not compulsory. Neither is survival. Jeffery Payne:
[00:12:07] Here's some great pithy comments. You know we're in
this. You know there was an article I read that compared it to an
extinct extinction level event you know where we've got you know
Internet of Things and big data and and organizations being able
competitors being able to go extraordinarily fast and learn and
reintegrate that learning. The end for the many organizations that
will that will mean their doom and not going to pretend that DevIps
or Agile is any silver bullet in allowing them to survive. But I
just know the status quo is not the strategy I would take. Jeffery
Payne: [00:12:56] Yeah. Well yeah I mean if software is really
eating the world which I think we would agree it is then you'd
better figure out how to optimize how you build deploy deliver and
feedback information fast because otherwise you are going to be out
of business. Yeah eventually. Bob Payne: [00:13:15] So what's
happening over at your company Coveros. Jeffery Payne: [00:13:18]
Coveros, yes! So We're busy little busy little beavers helping
people with Agile and devops just trying to get it right. And when
we focus more on the engineering aspects of both of those things
but I often get asked to you know help pull teams together and
figure out how to make it all work. Bob Payne: [00:13:36] Yup
Jeffery Payne: [00:13:36] But we really like the the engineering
aspects as I call it you know Automation doesn't solve all your
problems right. I always say a tool with a fool is still a fool.
Right. So you have to know what you're doing and you have to
collaborate work together. But automation can help and as long as
you take that philosophy you can leverage test automation and then
you see ICD Automation and other types of automation effectively.
If your view is that automation somehow solves all your problems
it's a magic bullet right. And it all you know takes culture or you
know magically make it all work then you're going to be really
upset right because it's not going to work so that's kind of what
we're focusing on. Bob Payne: [00:14:16] Magical thinking is a
strategy has also proved to be not the greatest.. Jeffery Payne:
[00:14:21] Hope. Hope is not a strategy. One of my favorite sales
books and I use that a lot. Yeah everybody says it's not grounded
in reality I would say just remember hope is not a strategy. Bob
Payne: [00:14:30] Yeah yeah yeah exactly. Well great. What what's
exciting you coming up. What do you see coming down the pike in the
next. You know what I know prediction is tough especially about the
future. Jeffery Payne: [00:14:47] Yeah the future because if I
could I wouldn't be in this business or I'd be retired long ago.
Bob Payne: [00:14:52] Yeah exactly. Jeffery Payne: [00:14:53] Well
I am I'm excited about. Bob Payne: [00:14:54] What do you actually
see that's here. Jeffery Payne: [00:14:56] Well I was very
skeptical at first but I am a little bit excited about what's going
on with integration of A.I. into Dev and test. There are some
interesting things going on around how you can leverage AI
capabilities to build better tests for your applications. Do
testing in a better way. So what actually look interesting. Are
they going to scale or are they going to work right we've been
talking about AI and you know robots take over the world forever
which of course is not going to happen. Bob Payne: [00:15:30] The
joke is AI is the next big thing and always will be. Jeffery Payne:
[00:15:34] Yeah it's very true because you and I we probably are
same same relevant age and we were coming up through the techie
ranks. AI got really hot for a while. Bob Payne: [00:15:43] I was
in the computer architectures for AI master's program so Jeffery
Payne: [00:15:47] Yeah! It was hot hot hot, VR - the first VR
systems came out and everyone was talking about these awesome
things and how we were going to live in alternative worlds. And all
that stuff and of course then like a lot of things that it didn't
really happen and kind of disappeared but it bubbled along and now
it's kind of popped its head up again. Bob Payne: [00:16:05] And so
I'm not familiar with the uses that folks have been you know the
application in the testing area what is the is this especially for
like I mean if you look at big data you don't know what's in there
necessarily. So you don't know know what to test for like where's
the where's the current application of. Jeffery Payne: [00:16:33]
Well there's a couple. One is of course everybody's trying to
figure out how to even test AI-based systems whether it's B.I or or
whatever it is you know how do we know the answers right. Right.
That's the age old problem in the systems is you know how do you
actually know whether what you got is true or not because you kind
of need that testing right. But the other side of it which which
we're more focused on is other ways to build better approaches to
automation that analyze the product analyze what you're building
and not completely write the scripts for you but take a step toward
providing you test automation capabilities and scripting without
having to do that on yourself. There are some new tools out on the
market really small startup stuff that's trying to take a different
look completely at how we create automated tests and how we maybe
do that automatically. Yeah and the software is a really hard
problem. Bob Payne: [00:17:37] Yeah I can I can I can
extraordinarily easily imagine doing like really good deep
progression by looking at sort of big data. Big data user behavior.
You know we've kind of done that to heatmap. You know we really
need this piece to be bulletproof because of risk. I'm sure there
are folks out there that are mapping the the usage. But I could
also imagine very easily just observe what folks are doing and and
learn from that. I mean it's the way to go. [00:18:20] You know Al
p haGo learned how to play go and meet you know with you know the
vast majority of the learning in a system like that is not from the
ruleset right. The initial ruleset it's actually playing another
copy of itself Veriga and and and going through the database of
previous games which for go is actually harder than chess but
apparently it never played go. But yeah it sounds easy it's go go.
How hard could it be. Jeffery Payne: [00:18:53] Just go right. Just
go. So what. What's up with that. Just sounds a lot harder. four
letters. Just kidding but Bob Payne: [00:19:04] It is four letters
is twice as many. Jeffery Payne: [00:19:08] That's fine. We're just
having a great time here right. Bob Payne: [00:19:15] Yeah. Jeffery
Payne: [00:19:16] So yeah that's what what I'm interested in that
is just you know trying to take the dev ops concept to the next
level. You mentioned round trip. Right. Which is you know a lot of
people spent their early instantiations of automation just focusing
on how do I get code you know from a change in their production as
fast as possible with quality and stability as well. You have to
balance those. But now I think the more sophisticated companies are
saying OK well it's great to get there but what happens if you get
there and something's wrong. What's the fastest roundtrip approach
to fixing that and addressing that. Is it rolling back. Is it going
roundtrip and coming through. You know because the the other thing
that's and people say why is that important if we're not the kind
of company like you know say and Amazon who's pushing code out
every 11 seconds right. Jeffery Payne: [00:20:05] Why do you need
that we need that for security and stability and performance
service level agreements. I mean if you've got a problem in
production it cost you money every minute every second it's down or
that there is a risk out there with a security perspective you've
got to figure out how to round trip change as fast as possible. And
that's an exciting area I think has been under looked at. You know
it hasn't really been the focal point of house is now I think
starting to be. I mean this it is really ironic that the safest way
to go is to be able to go fast. Bob Payne: [00:20:41] I mean
Jeffery Payne: [00:20:41] Oh yeah. Bob Payne: [00:20:43] I mean the
level you know I remember those days where company would have to
fail over to their dark side and emphasis on fail right because it
would be days hours just downtime before they could you know oh
shoot the Oracle logs didn't replicate. Yeah. Or whatever. And in
like extreme programming and some of the techniques there early on
they were seen as risky and the real practice in the same way that
drove up seems risky. If you're doing it the way you and I think
they should be doing it. It's actually the least risky way of
behaving Jeffery Payne: [00:21:37] Right. Yeah it is. Yeah of
course there are some apps that you'd like to be able to push into
production quickly but maybe can't ever fail. So you know you can't
you know this you know the Amazon concept of roll something out
there doesn't really work. Jeffery Payne: [00:21:53] Roll it back
and tune it roll back out and you're kind of using your customer to
test test and give you some time to live life critical for that. So
there are certain ones that you need. You know just double down on
your assurance process during your dev ops capability because it
can't fail on the field.. For a lot of others you know. Bob Payne:
[00:22:10] Well one of them one of the things that I've been
thinking about because I quite often talked about high quality and
the key is and someone came up to me and said what you're really
looking for is expected quality. So and he had an example that was
was a big oil and gas company and one of the things that they said
is your labels are too good. He's like What do you mean said we
need the labels to start to deteriorate immediately said we do not
want to see someone pouring a lubricant into a cooking pan in
Africa or in some other area where this is unfortunately a common
practice with a brand spanking new company logo on the outside of
that thing said is we actually need that to deteriorate. And I
start to think about that because as you mentioned you know some
fine you know I may not have critical transactions push something
around or find a roll it back. You know that might be fine. You
know canary roll out on Spotify right fine right. Jeffery Payne:
[00:23:39] Yep. Bob Payne: [00:23:40] Canary Roll out on the
firmware and in a medical device maybe not so fun. Jeffery Payne:
[00:23:46] Yeah Bob Payne: [00:23:46] Because the Canary dies
Jeffery Payne: [00:23:48] And it's a big Canary. Bob Payne:
[00:23:48] . Oh yeah yeah Jeffery Payne: [00:23:55] Yeah. No. No
doubt Bob Payne: [00:23:56] Yeah. Jeffery Payne: [00:23:56] And
that and that is something that I think people misunderstand about
dev ops. [00:24:00] You know when I speak about DevOps at
conferences I always well attended everybody's interested in the
topic because it's hot Bob Payne: [00:24:06] Right. Jeffery Payne:
[00:24:07] People have this perception and unfortunately senior
management does that Dev ops means speed and speed alone. The goal
no fast can I push things into production. Bob Payne: [00:24:17]
But imagine a life critical system where you could have test
automation every single infrastructure. Code line Change is
auditable in and you can get that level of safety. We used to put
two you know extraordinary manual testing. Jeffery Payne:
[00:24:44] Yes it was very expensive. Bob Payne: [00:24:45] And
it's prone to possibility of non repeatable results. Somebody makes
a mistake. Somebody configurations off. And now with you know with
tools that where you have immutable infrastructure you have
software configured network you can actually know to some a greater
degree of certainty than we were able to in the past that you have
a Conformance Test system. And that adds a lot of safety. Jeffery
Payne: [00:25:24] It does and it helps with regulatory is yes
right. I mean the one of the under the under represented aspects of
dev ops is CM Bob Payne: [00:25:34] Right. Jeffery Payne:
[00:25:34] Because if you're doing it right everything you're
dragging your entire manifest of your software your test your
environments your even your rollback your recovery procedures your
monitoring capability. Dragging that all the way through production
in a way that you know where everything came from and everything
takes and ties together. And that's what regulators want. Right.
Bob Payne: [00:26:00] Those that know they actually want safety
they don't care about the stack of documents they use sadly to
hopefully inspect that you knew what you're doing. Jeffery Payne:
[00:26:08] Want you to demonstrate that you have a process that
delivers quality and they want to see that there's relationships
between the various things that you're using to do that. And dev
ops gives you all that if you do it right. Bob Payne: [00:26:20]
Yeah. Jeffery Payne: [00:26:21] If you do it wrong it just you know
throws your code down through there and everything around it is
changing constantly and you're never really going to get the speed
or quality that you want. Bob Payne: [00:26:30] Yep well great so
anything you'd like to close out with Jeff for Jeffery Payne:
[00:26:36] Well just thanks for the chance to talk. I know you've
been doing this a long time and it seems like a great podcast and
we're really enjoying the conference. Looking forward to the rest
of it. Bob Payne: [00:26:49] And if you can stand to hear me talk
then they listen to some of the older ones I think Bob Payne:
[00:26:55] Definitely. Jeffery Payne: [00:26:56] Ok cool Bob Payne:
[00:26:56] I'll get some popcorn and listen to early one's .. I
wish you had started it maybe five years earlier than that right. I
mean. Bob Payne: [00:27:03] Yeah yeah Jeffery Payne: [00:27:03] If
you had started like right around 2000. Bob Payne: [00:27:05] Yeah
Jeffery Payne: [00:27:05] Then Bob Payne: [00:27:06] Yeah. Jeffery
Payne: [00:27:07] You know you would have had some interesting..
Bob Payne: [00:27:08] There's a there was some gap years as well.
Jeffery Payne: [00:27:12] But Well thank thank you very much for
having me. Bob Payne: [00:27:14] Thanks.