Hashing It Out
Hashing It Out

Episode 35 · 2 years ago

Hashing It Out #35: Constantinople Postponement - Trail of Bits & ChainSecurity

ABOUT THIS EPISODE

Back with season 2 of Hashing It Out, and we have a doozy! This Episode features Trail of Bits and ChainSecurity to talk about their amazing last-minute catch of EIP-1283's impact on the Constantinople Ethereum hard fork. We go over how they found it, what recommendations they made, and how the hard fork was postponed to quickly. We also discuss the tooling behind analyzing such a vulnerability, the true impact had it been released, and how the processes around hard fork release candidates could be altered by this detection.

Links: - empire hacking videos - EIP-1283 Analysis - contract upgrade anti-patterns - How contract migration works - Blockchain security contacts - securify - slither - eveem.org

Now entering kindtwork welcome to hashing it out, potgasp orretalk to the ATTECH intovators behind blocked in intrastructure anddecentralized networks. We dive into the weeds toget at Wyan how peoplebuild this technology, the problems they face along the way o'me, Listengand learn from the best in the business. You can join oir racks, all right, Woki back to hashing it out.Thiss is episode. Thirty five. As always, I'm your host Doctor Cory,petty along with my cohost calling froche, say: Hello, everybody, hello,everybody, and today's episode. We went on a nice,long break for e holidays, but yeah hi is Al. The season too. I call theirseason too yeah yeah no year and to bring us back. We have quite a show Um our guests, Torday, our Joclin and Danfrom trail of bits and Matius from change security. Would you like to giveus give yourselves a quick intratuction, starting with a ban to Joslin Samesiuson who you are, how you got involved into the space and what you carentlywork on? Yeah sure, Hey, I'm Danguido, the CEONcofounder of trailbits we're a seven year old, soffar security ran defirmthat I founded and have now grown up to about forty people. We specialize in Bocjain security outof the interest of our own employees, wasn't really a planned decision. I Hjust something that came up about two to three years ago, because we saw itwas a really interesting setoftechology and I wanted to play with it andthought the skillsy hid would be useful. So I have Joscelin here from our team,he's one of the leaders behind the tools that we, right by knowledge, thatwe push out and the audits that we do so Jocelin CID tells by yourself thinx for that tradition. Yes, so I'mjustnfized, IAM securtosuch a toobit. My background is mostly hoering up.Tection and explortation. I've been in in the SE in the TOMICAsystem, since something like two yeah, something like that and I've been doingmostly Adat, building off two like slite Ottra IG, thanks tous, about yourself, yeah HAP, to be here, I'm Ateasim, N, COOand train security. So we are compared to the young company. Um definitelyfocused on Itheorium mut contract auditing. We came out of M Universityhere when we started developining Tois, whichautomatically scans smart contract for security, vulnerability still as likemastertetisis then later during PhDs, and then these methods became veryapplicable for etherium, obviously, and then foundet chain security like alittle bit more than a year ago. U started out in February two thousandand eighteen with our first an employee and then grew to currently fourteenpeople, Um Astian Zuric M and my background is Um. I T security fromthe more practical, eectical engineering side of things, Um and I became active in a space quite sometime ago, but as a user back when eterium wasn'n etherium yetor wasn't there yet but um, then joint traing security beginning of past year and have been active in that space. Since then, all right, that's a quite the paneltoday, especially based on the topic we'd like to start with, and that is recently inside of a theram. We triedto do a hardfork and a bug with one of the IPS that was being rolled out wasfound by trained security, which they disclosed a few days before thehardfork Um. Can you give us, I guess, a Rapup ofthe article that you produced and then we can start talking about itsimplications and consequences after the securty community got a hold of it sure happy too. So, when Um Etherean does its network upgrades orhot forks, then what happens really? Is that the routsof etherium change and and h new things are allowed and Um? That M can have securityimplications. Obviously, usually it means we need to look out for newthings.

New Instructions are possible.Customers would come with new things. So when we looked into whatConstantinople allows people to do, Um we saw that m things get cheaper. Nowthings get cheaper is generally good because you can compute more onshamebut um as a company was kind of created whenthe first really big issue with smart contract security happened at Dabarkback like a two two, an a half years ago.Now, an the problem of how that was mitigated wasmm, always that you need to prevent reentrances, obviously reentrancies,meaning you give some other contract some other code, which youdon't control some control over your own Cote in oneway or the other, and and you need to carefully control what that other cotcannot do. One way it was done in the aftermonth of the Dowhack was limit thepossibilities they can do very generally by giving them very diffrengas, as in etherium being the Um fuer you need, and if you run I o gas, youget immediate in concert. Your transaction gets reverted, nothing that can happen now. This hard fork allowed some operationsto be cheaper. Most importantly, so coled store operations operationswhich change state on the block, Chaim, which can save, which can change thisglobal state which etherium is about and that allows to attack Ma contracts which, in the past,had this protection against m. These state changes just because you wouldnot a give. You would not give a long enough gas for like foreign, uncontrollable, smartcontract in your own cout, but now this little gas which was passed along isactually Nogh to do state changes and Um. With that, this classicalreentrancy attack was something which could be now applied to smart contracts,which were considered secure, odtby by USS Auditors, because weexpected certain m cause to to pprevent these kind of reentencyattack, while after that Hotfok they wouldn't do it anymore and Um. Then we m brought down this. I westarted researching them and um saw that this is in practice possible,brought up a postwhich Um. We shared with Bakponti of the therium foundationand Um. It got published in the security mailing list and Um veryquickly. Then the following activities happen, which I'm sure we'll talk about,so s that that basically m w t thebeginning of e last beginning of this week. So we have a mexed audience from: U Xdesigners, but mostly they're engineers. Can you maybe explain re entrancy as weknew it, and what this newreenticy kind of attack is. I don'teven know if it's new, it's just different. You know something theydidn't consider like what are what is reentrancy really like? What is it?What actually makes a reinteresty attack and what what happened when youlowered the gas price to actually h force EP twelve, eighty three to kind of likecome under question, so actually, what a reantency really isis being discussed thereis, a very technical description we are currently down on, which means you give someother smart contract control and after what you make some formor state changeHo. This doesn't help a lot. I think for most of the audience, what a Um most codes, no O, think of the programsas lineor or most delinear execution. Theyknow what happens perses and this, and if no, this and other things happeninon, but they kind of have alinea trace to the coat and with etherium in certain points,especially if you want to Um transfer money to some other accountmoney, so being it either or you want to interact with some othercode, you are calling it I it's similar like in normal in other languages, ifyou, if you call the library or call an API now, imagine you call an API. Let's say you...

...some some website, you load somewebsite and you include the response into your own programm and say like based on this. We will now interact differently right. Basically,the external call has impact on your own hote Anatoann, an injection attacksimilar to an injection attack. Now the crazy thing here is um reenciencyattacks areum more powerful than an injection attack because they allowthese outside code to call into Yourcod. Again it's like if the external websitecould directly go into your ebserver and quickly call some, let's say afunction in there, which then has impact on how you continue. With that request, you mate and, of course thissubly allows Um changes into in that smart contract. We are currently inwhich you didn't think of Sudday. They might change variable, which is Um,considered, not changeable during that line of Coud,and that will um mean okay. Now now some assumptions youmay perhaps you checked that you have enough tokens before you make a calland then ye you'll find you happy enough, but this reangency and alsother co to with far ail tokens, and now you don't have them anymore, butyou still think you have it because you sing in this linarly way or first Icheqk, then it's okay, then I do something and then I save something butm. In the meantime, this Attecha Cote could have changed an his own and sopreviously, before this particular hardfork. Many of the calls that couldhave been susceptible to an attack like this Um were too expensive. So youdidn't ou just basically assumed that, based on the way these calls alreadyworked, you couldn't do reinterancy, because you would never have enough gasto do a state change ecause. Every time you do anything on the EVN, a cost gas,especially in it's operation dependent and with certain calls, whether itbeing see and transfer. I think it senti transfers that right, O rect yeahthat you'd never have enough gas. It would be too expensive O to continuedoing this reentative attack, so they were considered secure and thisImpiteci they did was make it less expensive to do these types of thingsfor these two functions, which then opened up all of the previouslydeployed contracts who had a pattern like this um to possibly be vulnerablebecause they were now like that assumption of they'll never have enoughgas was no longer true after this eip change and so ye a great story. It'syou know all. Let's just make everything on a Theorio, more efficient,we'll lower the gas costs. Everything will run faster and nobody looked at itand thought about it for like two seconds and said. Well, can't we dependon that for reenticy protection, the Jake security people thought aboutthat for like ten seconds and were like. Oh, this is a bad thing. We should telleverybody about it. Um Yeah the layment's terms here: Is itjust loosens the restrictions on reangrancy, so there's a tiny windowopportunity and this thing kind of blew it open a little bit wider so that moreattacks were possible than before. So actually, that's that's kind of a pointthat I'm kind of curious about perslally is why why they held it.Anybody think about this until, like twenty four hours before the release-hey so mmatius, and I and everybody else is doing security have so muchwork on our hands that we can't sit around on police every single EIP. Youknow even Martin at the eterium foundation, like Martin's got hishandful just trying to make sure we don't have an unintentional fork,because different clinof limentations diverge Um. So you know we're alltrying to run businesses. Everybody's got to keep the company afloat withmoney, to pay her employees and nobody's really paying us to look atevery single Yo, ight be so what this really does is it reinforces that itdepends on our own personal interests and the hobby time that we have, whichis very small, to take a look at one of these ips in detail and the realanalysis behind the Um kindo security infact of this particular Ip took along time like I had to pull Joscelyn and Stavo and Evan and like half myteam off projects, they were working on to scan the whole block chain, figureout all the repercussions try and come up with remediations and honestly like it's, it's not a zerocost. It's actually a negative cost. Like I lost money by trying to helpwith this incident, Um Sai Ontater donattier et the dress which we illsure.I did whatever I'll take it, I'm not...

...going to refuse anybody's money, butwel I'l. take that back. I will refew some people's money, but not going with any scam coins today, um,but like this. This really brings up the point that t t there's very littlesecurity expertise around, and if you want that kind of input, you need to pull that input in somehow and since itwasn't done in this case, you know we get the Chan securitypeople who who decided to start poking around some area of Urium, for you knowno reason at all and then j St Discover this issue. That really ought tave beenobvious. So you're, it's like we'll get into some of the moretechnical details in a moment, but I just want to frontload with with this.You are the CE of a company. You are doing this literally Progono for thebenefit of the community. What can the community do to make notonly your life easier but perhaps build a business or an ecosystem aroundsupporting this open source software? That people are protocol that peopleare? Are you know making proposals for R and changing like? Do you feel likethereis there's something? Maybe we can do better as a community to facilitatecatching these kind of bugs sooner rather than later? Yeah I mean it's alittle bit of a tragecommons kind, O kind of deal 'cause. It's not anyoneindividual's responsibility to police the EPS to come out. It's it's. Thewhole community is right Um. So it's it's not necessarily a business, but II think the theory of foundation probably needs to set up a moredetailed approval process for yypees, where you can't pass go unless certain things get reviewed by asecurity x, Cork that they trust Um a and that that comes at a cost right,like somebody has to have to pay for that Um. What we decie, what werecommended- so I was going to this test year is blockpost hat has a lot ofhis desils and description and the and the white paper or security analysis Yehas put out ECAUS. I MEA a ASE amazing recommendations in it and hop arightyeah. That's what I'm skipping ahead to a little tiny bit because m! You knowyou probably don't need to review every single Yuk. There are a lot of EIPs upthere that don't change anything security critical, but there are a lotof things that you could just keyward, search on and say: Ah That's going tobe a bad one right like things that affect contract up great ability,things that affect gas cost um certain like. If I, if you H, modifythe synantics of an existing instruction Um, like those things, aredangerous and require a lot more study than things that don't do those thingsright. So you could imagine even like a Giha Bot hat just flags, certain issues,Ays wow. This is like security review required and then the EtheriamFoundation has to pull in somebody whether it's someone like Matius orsomehone, like I or someone like Joselin Um, to provide some kind ofstatement of like you know what we don't think you should do this orhere's how you might do it better. These are the use of the implicationsof doing this across the Bloci hew an that Howi curen exists. I don't. Idon't want to scream dom and gloom his entire time, because the UM totalamount of damage associated with deployed contracts that this would havehad what W, who this would have impactedhad this gone through, isn't incredibly large that we know of so far. Right hasonly two tests contracts. I think that were on the chain. They got impacted,they weren't even like real live. I don't causethere's a lot of there'sgoing to be a lot of media associated with m. The postponement of you knowhow terrible Itheorum is or how we don'thave our shits together and so on and so forth, and I would like to at leastgive reality of the potential F Restaura, like the the known risk ofdoing this type of thing, I think, more importantly, th. What we've learned from entire processis what you just described, is figuring out how to go about N, looking at EIPs appropriately for thechanges that they're making so that we can incentivize people to look at them.The correct way before throwing hem on the main net. Kiptain Nell talk aboutthat just a little bit. I think we are Super Lucky that this problem wasn'tworse than it is. However, however, I have to copy outthis and sure mtius wants to say this too. The result of our analysis was notconclusive. It's onexhaustive, like the tools that chain security has thetrailbits has, and the ones we pulled in from the EVM folks do not guarantee that we have a preciseanalysis of the entire Blatchang hit. Could be people affected right now thatwe don't know about, and that's another kind of you knowdiscovery here, that we don't really have the tools to effectively monitoreverything that goes on in the block jam. So it's probably worth it to notchange, not pull the rug out from underneath people when they deploy asmart contract and expect that it works...

...a certain way. Um there's anotherdiscussion here around: What is the mutability of certain ways thatEtherian works like Cen, the hard fork or the hard forks that we plan on afuture change the behavior of existing contracts, even in minor ways, becauseeven if they don't introduce a reentcioncy attack, they might modifybusiness logic or the way the application works, and that might bejust as bad depending on the scenario. So all this, I think, really came uptoday or like yesterday Um, but again it's one of these things. That'sobvious in retrospect, you're writing these tiny little computer programsthat you depend on. Ultimately, if, if you're, considering that thatcode will never change shouldn't the blocchain, never change to Um, so I don't know what the answer that is, II think on should never change. I thought you, you guys in your in youranalysis, put put out some pretty good recommendations in that you just don'tchange old stuff, but you can implement new stuff mean you. Don't change Ha s store, butyou could create a new type of store which is cheaper and maybe morerestrictive or has different security requirements around it tot you canprevent this kind of stuff. You talk a little about your recommendations. Yeah. The main challenge here really isthat etherium will keep moving and U, as Umsomeone who has seen this one, both sides of where Ike stuff, hards stoorined, because you are in this, what I call Jinger coding mode, where you'reafraid to move any peace, because you're not under control anymore, towhat kind of implications it has and basically the whole project is notmoving along anymore versus the one where you know. What's going on whatkind of implications it has, and you can do- changes that's very importantand think Um excee, this current discussion, where we started to lookinto Im, utabilities and stuff which we would consider changeable in the future.This is important for us is auditins, because we can advise our clients onsave now, but not guarante to be safe in the future and m. It was brought upats by U Bitalic himself. Very shortly after this Um um discussion about what to do now that Um ges cost suddenly issomething we need to be considering critical right as wasn't consideredthat critical before? U and N Nowadays. Clearly, so when we start to add a lot more complexity into ITERION,which people implicitly base their security on top of, then we would stophaving any things which we can move on any more.It will be an immutable dog chain. The only thing we even even changes- wil,we say future upcodes future instructions should I be possible, but never change existing ones,actually changes existing once for reentrancies, because an attackercurrently within reangence he has certain gas costs on their own side andwhich have a certain amount which you cannot get rid of, but new attack codewhich cheaper instructions potentially can get rid of it. So I think th, the Kiger of this discussion wasvery good, where we can think about what will the Etheria community think of almost or basicallyunchangeable things, and where do they say? Look in the future? We expect thatto change so better make sure your smart contract will stay, save thatthey, the gold global block gas, limit right m. that's something where we knowit's going to change eventually. Can it what woull happen if it goes lower alot? He are a lot of smart contracts, don't work anymore because they need alimit. They need some gas for sure, but then we expect that never to golawer orways to WIR. Do we really atust an implicit distumption to I reallylike to have that discussion going forwork, which is topping othersecurity auditors in the end, because we know what to rely on and to flexsmart contracts which might be vulnerable if they ever changed that,and if then, we see oh, it's going to be changed. We can at least warn we can effectivelyscan for it. We know the imclications and for other stuff, where we know itimmutable. We can get very strong garanteesbecause to me initially, the huge appeal of iteriumwas that it can um encode certain like real word things into smart contracts,which then will guarantee that for the forciable future and withthane security.Of course, we we are very much into this form of verifacation into...

...proving that certain properties hold. This relies onthe rules, don't change, or at least some rules, don't change. So we reallylike to to tell perhaps some Um Authoritygovernment authority that okay, these people will never be able to stealthose funds, because we mathematically prove that, but that relicence thingswhich should not change it would be good if you know that those areconsidered almost immutable. I want to just highlight something that wasburied in there. That Hatte has mentioned so r right before he started speaking. Youmentioned oh well, you know we're not going to change existing function, outywe're just going to add new ones, W we're only going to add newinstructions and the rabbit hole here goes super deep, like even when youmake these minor modifications like let's make a new instruction. That'sjust like call, but instead of call it's cheaper than call wellthat enablesnew attacks to since attacks that were possible. Thatcaused a certain amount of gas now might become possible because youlowered the the requirement to run an instructionwith nearly identical functionality. So even these tiny little modificationsend up having these ripple effects through the behavior of and thepossibilities of calling into existing contracts Um. So I'm with Etus here,like we do lots of verification forward projects as well, and it makes it areally hard target or a unit like how am I supposed to say that this goade issafe if the underlying execution model can change six months from now, itSoems like a time based approach to Sakeness 'cause, like if you thinkabout that from I guess more higher perspective. If you only make forchanges, then noin the Bolderanis exofist you're, making h the attackersstronger while making the defender weaker. In some cases, yeah Te, the thebig bran answer is well. I guess you Nedo security on it. Every six monthsnow ell- I guess maybe E L, th t t thats- speaks to another point thatthere's not a lot of like wonct. You deploy something typically, there's nota lot of monitoring of the thepd contracts. reprospect o changes in tooling an ifrestructure around thatwhatsoever, and and it's clear that we're a we're still an experiment likethe thermediais is still learning grolling becoming morkers. We look atthe changes on the horizon to make it more scalable at Laar, zero or one orwhatever. You want to call it. That's clue, they're going to happen and haveeffects and and think drinanapolis speaks about thisconcept quite a bit in reference to the bickcloin block chain. The osificationof the protocol allows you to more effectively build layers on top of it,and I don't see the hardening of a protocol happening at ha time soon,especially with a lot of the changes, a e trying to make to make it moreefficient so that it can scale and people shouldn't expect it to there'sno reason to expect it to just kind of have to experiment with ituntil we get to a point of scale, that's reasonable when we say all right,this is good enough and then we can start making these hard assumptions onthings that won't change and can change and should change so on and so forth.But until then I don't, I don't see a need to do that. You feel the same way. Anyone! So when you show Li people arealready builing business and application on top of, like the currentstate of that to Um. So we can, you can need to have somekind of Guamantye and to have some kind of way to unchow the code for thisapplication. It's not like tone is EXPLA Montan, but is using it for anywheel, case application and anything because peore ar building the wilapplication on top of it. So I dknow like Ho how long they couldwait t something Mo machoits Io, I'm not saying weight, but build it withwith the fact in mind that things will change. You need to keep that in mindand and Maye elocante resources to have constantly monitoring, really take alook at how to like migrate contracts and up mature they're uppratable andfor future cases that you may see coming. Don't don't make hard set boundaries onwhat works and what doesn't work or what secure and wasn't secure right now,because there will be changes and that's, I think, that's just a mindsetor mentality. We should have when to point smart contracts. The questionthat I ask is what, if there's a change that impacts the upperaidt ability ofyour contract like to me, it seems as though Um it seems as though we are we are I liket y. You definitely mede it clear again t this is an experimental chain and itwas always meant to be an experimental chain. It's a proof of concept I feel,and that there's still a lot of work to be done to figure out what the whatgood looks like you know what I mean, and you know right now. What this hasshown me is that a mutability is a lot more valuable than I thought. Um and-and that's you know: If people are...

...going to be building business processes,it needs to be a final form ultimately now, for now I mean obviously we're notgoing to have that. But this is not the final form we don't. We don't know whatthe final form looks like. Yet we have issues with scillability in terms oftransaction speed. We are adding features like state channels, becausewe n notice the need forlay or two solutions, and now we realize that wedon't have the ability to change the protocol very much once it's released.So, of course, the very first version of this release, which is that you knowthe one we have might not be optimal. For you know and future proof, so I don't know I still feel like we. You can't expect developers to knowthese details if you're going to be in the final form, you can't expect themto worry about. You know nobody worries if the TCP, ifts, if Y I be four cha changes, they just know that they have to migrate Ipsix or Use Nat, but they don't. They don't want to concern themselves. Withthe details of I of the IP stack on their onthe ssthey. Don't need to knowthis this stuff, they need to know. How do I build stuff and how many Haw Igelt it safely, and they can't just monitor all the security security mailings to see if their apbreaks oreven expect hem to understand it. If, if that happens, it needs to be a lotmore open than it currently is and Um, so the hardening might, you know,might be a different shame. I don't know so on this like when you start likesaying veryvaluing utibility a lot more Um. I think that is m quickly, leadingto not moving on anymore and um in the real world. When you say like okay, Istart my company. I am like the laws of Switzerland, the lawsof the. U S apply to me, you know they are going to change and you are goingto adapt so um in addition to the monitoring. Basically, what you need toknow is when you I need to adopt, things are going to change for sure.Don't expect that this thing will wrung forever unchanged, nor that upgradesmight come that you need to do something if you ant ongoing activebusiness. It's your duty to say technologically fine business, wisefing with the processors nowwhen. There was the buck about Umthe prointo processors like what was itcalled? I spectre then yeah, no suddenly things whichwere considered safe yesterday are insave anymore, doesn't mean the end ofthe world or that we should not ever upgrade or processes anymore to fromthe ones we consider safe right Um. So to this regard I would more say: Yes,we need monitoring, we need to know. What's going on, we need a processaround informing people about, but Longa term cycles for testing, forthat each company, who runs something onmain at Etheoream, also runs it on Robston on some test net, which getsupgraded, usually earlier to see. What's going on there, that the Etheriacommunity itself Um more encourages the okay thisp's goingto change the following things, not only in the technical sense, but alsoin the practical sens. like look exchanges. This is something whichmight affect you, especially and so on, so that, like, like lawyers, would do for theirclients for big companies when they say. Oh, this upcoming regulation change.This will impact US um. So then we need to figure out. Is it thejob of of the government, basically of the tereompondation to to do this, toencourage that or is that something where each company needs to employsomeone like m Dan or as to keep monitoring this to tell them about itboth might work just fine? Let me put that thet around on you,okay Youre business, and you are running your business on this blotching.It is a protocol, and there is literal value in money being generated fromthis this this blotching, your you need to know that thatbusiness model that Youve set up has assumptions which you can guarantee forthe remainder of your business. So, let's US say they decide to increasethe gas price of something for one reason or another that could impactyour business line right. It's not really a good idea to set yourbusiness on a shaky foundation where...

...you can't kind of have certainfundamental guarantees about how you are able to manage your finances andyour money external to you now. Obviously, there's always going to besituations like that. But, like my question is Um, do you really feel like? Maybe this is so I I think the pointI'm trying to really make Hareis not don't stop innovation, that's notsomething we need. We need to do, but I'm questioning whether or not um we'regoing to see the final form of what this kind of Proyou know. Kind ofsystem would look like, you know, emerge from a theoryum or, if we'rejust still learning that t the things we need to know to build that system,if not, if you're shocking about, if they're about building a business on atheory um whether or not it's a smart idea, it's sure Osparan idea just yeeto be aware of what can change and and that's more clear. You can make betterdecisions around how you build your business, don't automatically assume,because you say the word block Chane- that nothing will ever change, that'sjust on how things were and if just just be smart about what you're,building and more along, like the the relative timeguarantees of when things can change like no, how things can change and andhow long, and how long of I guess Aguarantee you have on those types ofthings. If you build your business that way, then it's smart and you can youcan you can adapt like Wat ys as saying so as long as you're you build with theability and knowledge of being able to adapt, and you should be OK. 'casethings are going to change, regardless of what you build on. There is no suchthing as perfect to be te bility forever. Witen, the BIKWAN blockchanges. That's that's supposedly, like you know the most I I would call e the most stubborninterms of change. I cannot mention a small little shotout here, so I don't think we're going to resolve this discussion. This is Ishoe on the pot CAS now, so I think in the meantime, yeah companies shouldprobably be familiar with how contract upgrades and contractmigrations work and, if there's one authoritative reference for that, it'sthe two set ofblock post, the Jocelyn Rote Um. So you know in the meantime,we're not going to get a fully hardened block jain that never changes tomorrow.At the same time, the changes that we have in the futureare you know, against the best efforts of Mateus and I and everyone elseworking on security on the etherium back Thain were going to have anincident like this again guaranteed Um, there's like as you're changing thiskind of functionality. There's no way that unintentionally, at least onceyou're going to encounter an issue. My dog is in the room, so it's really incumbents on peopledeveloping smart contracts right now to ensure that they can run an upgradewhen they need it, and that means also including a migration Um. So if, if that's a thing that you'renot familiar with, yet you should check out our blog and Goslin's papers onthat exact subject, just to put it lad on that. So where do we go from here like what l?What do you want to talk about? Is there something that has been pressingon your mind that these things brought to light other such of things wehaven'tgotten to? I mean, there's a million othersecurity issues in in a theorin that we could talk about earlier. This morningwe published a list of recorded videos from an event that we held in Decemberwor. We got together. Some of the best experts in eperium security dependwatchin security, more generally, to speak about all the problems thatthey're working on. In that they've observed. We have an event in New Yorkcalled Empire. Hacking runs every other month and in December I do a half theyeventane kind of a Mani Conference, so I teemed it around blocktnd security.This time and um t there are some greay talks up there. If you, if reader H, iflisteners whowould, I like to take a look I'll, think that ision no as mall, so the things that that that kind of has beenal my mind as well is thenature of of smart contracts in etheorium. Do you feel like we needfultering, complete smart contracts? Obviously, that makes your job harder.If you can't use poral verification techniques on on these things, would it be better? Do you think if he,if, if we weren't adopting such a flexible smart contract system andreally just had what we basically needed for a a rout chain and delegatedall the more complex operations off chain to to lay or two solutions e, soI I think what strikes me is that Um in a lot of safety, criticalenvironments- like you know, embedded systems, airplanes, things that actually affect human life. There are really strict requirements:uround eliminating certain kinds of functionality from the code that getswritten like eliminating a recursion, because while you can use formalmethods to prove or disprove that certain things can happen, it's harderto do so um in these smart contractenenvironment, it seems like be willing to take those steps. WED prefer to havethe flexibility rather than limit...

...people to say, like a the main specificlanguage that lets you do a lot of things, but doesn't let you do some Um? I think, there's a really strongargument for following the safety critical kind of approach, but you knowagain, this is something that, like I'm armchair, you know commenting there, there's notmuch control. I have over how itteorium does or does not work with smartcontracts or any other Blotcang. Unless I go and build it myself Um. So I think the reality is that we wehave to deal with this kind of complexity and that's why companieslike trail of bits and Chan security have invested in a lot of this reallyheavy weight. Automated reasoning 'cause. It's the only way that we canget any kind of understanding about what what goes on at a little level. Ithink it's also like pointed to put out that, like the security communityoverall is relatively new rete to the same, and it's and it's quite small umlike Ti. For instance, this last fcon was the first ive con, where weactually had a security track, and this last year probably marked rapid growth in the securitycommunitity over all, but it's still relatively small to where it should bebased on its importance in in what a theorium is used for or what thelotchains are used for. Well, you know what I actually SD, I'm going to saysomething positive. If you look any other like technologyarea, if you, if, like W, we're reviewing a kolnel driver or um a pieceof windows software or like a web application, or something like that, we're not using automated reasoningtechniques, we're not using formal methods, we're not providing ourclients, proofs, R or guarantee, is that their code works correctly. Theydon't have a SPEC they're, not testing ou, against properties like, eventhough the security community for Atheorium and for smart contracts andfor block Jane stuff is just is small. The kinds of techniques that we couldbring to bear on these problems is extraordinarily better than forsecurity. More generally, so I don't. I actually think that it'sa bad thing if the SECURTY ommunity gets to be too big. What I'd ratherhave is I'd rather have a small number of people with really pointy stickslike I really want to have very effective techniques that begin deploythat scale well and that provide like guarantees that provide reallystrong assertions but the codeworks. Well, what I don't want is, I don'twant an army of people rolling around pointing out. Oh, that's a reanciency,that's a reantrancy like ane at a time th! That's that's like a Tistopia in inin my mind, so I gotto give e smartcontract worldcredit here. Instead of starting at at like floor,one we've kind of entered the discussion at like the tenth floor Um because fromthe getgo companies like Jane, Scurit and trelibits have have brought thesereally heavy weight techniques m into a practical newshcase, so that that soI'm I'm curious. How did you guys do your analysis? What tools are you usingand like this is a lot of work to be done in a very short period of time?What does your system look like? What does your architecture look like? Howdid you get this? How did you get Tis done so damn past and like how Ho whattools are you actually using, and how can somebody W as curious about thisstart using those schools, it's she'll, tive, let's go! I want to hear Matius First, who Abou O Tto, who ponipactorstentered so I'll, actually um give him the go for that one toto so that I don'tblunder it by saying some words and Etal O familee was in the wrong way. IOto do the same thing of Jocelin, yeah think so. First of all um I I know this was a moment ago, but Ihave to follow up on what they ansaid the the block Horst on migration andand CONTRAC APRATA, really excellent. I can. I can recommend them also so, butback to the question on, on which truth we are using so um in general am tofind the UM I mean in general, were using allkinds of toods, but I guess your question was more tagger than at whicht tt we were using to find volunable contracts for, for this particularreentrantive, fo consertenurable. I guess right, yes, Thats Strip, yeah! I want to knowwhat you tools. I want to know how you did this like. How did you accomplishthis? No so quickly, I uess give it overall framework or like I guess U orlike a menta model for people this. This was chaing security released an articlethat showed a possible new rgeter attack and then that hit the ecommunity. Maybe forty hours roughly forty hours before the hardfork was aschedule to be to hit and I'd say within twenty four hours. They turnedaround an article that that gave quite...

...here wasn't exhaustive, but it wasquite extensive in terms of the range of analysis that they were asking now.What tools did they use to get the snauses done so quickly? Yeahmyeto from outside, I mean first ofall, um it ECAD. It wasn't exhaustive, so we we first focused on very valuablecontracts and Heh autantradobit had slightly different approaches, so wecompie different list of of contracts, which h was good also because we werelooking through different things, but so yeah we were. We were first tryingto figure out what contracts a are most important to look at than what are welooking for in this kind of contract, and we are looking for very certainconditions, very specific conditions right, so that this reentrance you canappear sog. First of all, the most basing codition is there need to be acall to a different adres. Otherwise you cannot have areentrancy at all. Then then, following this call has to be some state changeand there has to be a separate function, which Um does some kind of state update, and thistate up hed has to be very cheap, so edely g discussd before the whole pointof this terentraty s, that there's is now cheaper option to do this update. So wehave to find a function like that that that does does it so, basically, firstwe we downloaded Um the contra contract. We adentified thevaluable countric with on other contract. We, I e found that M, whichones contained these relevant parts, so to figure that out we we use differentkind of tools. One was Um too much, I'm probably Mbaranzi hisname but too much Culinko, who has themdodork Evemdodord dcompita day, which we used to identifycertain contracts that that might be vulnerable and we use all kinds ofother tools. U T to figure out which might becandidate and in the end we we looked at them manually. We one of the twos wealso used was our securifid tol, which is open towarde as well B yeah, becausethis can identify one one of the two Um pre requisitesquite well and quite officienly, because not everyone might know it o the tool is called securify and youwill like find it on security, FELAT CG to easily give it a spin, but with thehelp of the eterium foundation, actually, who gave us a grand lastsummer to continuously develop that and to the open sourcet. We were able tospend some time and M released it publicly. Now. Also so, Tha h, the fact Yo brought anytheor up the Etheriam founation kind of Um thereis. Another question I have,which is: What is what was it like reporting this? That's they and use different tool than onehear: whaut Daus, okay, Sigo Goad sojust like macheus, I'm going to letJoscelyn explain so Joctlin's the primary author behind slither, which isourestatic analyzer. I actually think it's quite funny that Jane Security andtrailabits ar are on these two parallel tracks of development, where we havevery closely related tools but just work. A little tindy bit differently inthe details, itwas helpful in the scenario I mighmmya yeah. No, certainlyit was extremely helpful in this. In this this scenario, we got someadditional coverage m. We were able to look at some areas. They weren't. Theywere able t look in a lot of areas we couldn't so it was. It was good, butyes, s, N Jocelin tell us how so lither works on what you did Yeh. I followlike Apot, which is kind of similar to en trasicat. He did so. I honi bytargeting, like the high profile contlact, that you can find on on a tes Caen or like the contentiesHenyouan amount of tosection Minanta, most of e R, and I workd detecormenciter, which is Asolititis Taticanaliza. We kind of detect this very particularpattern of Wen onti that can be affected by this Contantedi MaberHartholk, so Um after Wevie ving e Wazeer Fal like high profine target. I continue to extandexanolyis to any kind of contract. I could find fromfrom a Qare on it TA, scame and Um talking, which is so inresing that MTerto, who site woking a Tcien and for...

...this particular wiinancy. We need O,really precise model of what is a gas cost of you, conflact, so kind of to to be able to analyze this. With my two, I have to combine liter,which Wi h, another aniser, which is Montico, which is a Sombodi executionengine which arose me to kind of have a precize information about the gas coastof the function, don't make sense: Enin Reca, not the highest evelpossible. This IP allowed for a specific type of pattern to exist thatallowed for the B Abov to hapen. You codified. This pattern somehow searchedacross all the BLOCKCA Marka. You can get a hold of m ISO they're, looking atsolidity code and the other tools looking at mvm code, an a lot ofinstances for this particular pattern that gave you a set of possiblecontracts that could have this pug and then you mainly reviewed those to lookfor whether or not it was actually vulnerable. Yes, anwhat a's nice about this- isthat I hats what I'm saying earlier is that because they work slightlydifferently, you can you can get came up, possibly with different setsor or reverified results from PR from eachother sets, so that you had a stronger guarantee that there was a volmabilityin a given contract or got a larger search. SPASE N! That's exactly true, and it'sactually what in eterium happens, often rigd with paroty and ges. We havelike different companies doing very simular things, but covering each otherand terro of bits and train security are similar. In that sense too, forexample, thelither is Um, analyzing solidity, so the high level languageand U Um, and it can then be very specific towards certain securitieschecks which can be found there. The securified tool looks at the Eviambidecode level, so it can analyze all smart contracts which are out there,even if we don't know the source coat, which is, of course powerful, because, as we figured out during this searchwas that for a lot of high values, Mart contract, Ote, which was a lot ofeither wellthey, don't publish a source code also for reasons to a little bithigh into what's going on, th Theye, usually Mortisic, wallets and now certain checks are way harder towrite this way, just because the EBM level is not as easy to understand thanthis lidity code. U So there, you always have this trade of usability,where it is like completeness amounts of checks for its politives and so on,and as what I like about, like you see, oh they're, going this irection. Let'scover a different direction to have as complete a picture as possible D is not only us right. Th. TheSecurity Community is four very interesting people who arereally really good at what they do, often working alone from somewhere andthen building towards which they open sorcs and which thenhelp everyone. Travel pit is a great example for being able, as like an alllike old and also, let's say nicely fondd company who can go on and theythey actually open source most of their work and we can built on top of it. So it's not like Chang security is notusing some of the touds of traid off bits and I'm Sur trn of pitzis usingthe toets. We like too, were still whenever weekend, Um that that definitely huge strength ofthe Etheriam world. Where you you have these different clients, defferentcompanies, different security, Tois to look at things, wron, verydifferentperspectives to cover as much as possible. Well, I want to continue onthat line. I want to say before we move on that Um part of the reason for the postponementnot all of it, but part of it like during the call when, when we weretrying to decide whether or not what what to do base on what was going onwas the lack of information we weren't sure of the total risk or surface of vulnerability of all thecontracts bast on as changed, and because of that lackof information. Itwas best a opponent to to to hear back from the analysis that we that we gotum and I think that's that's an important thing. Is that one? If this,if we can find ways to incentivize looking at these things earlier, we can.We CA, make better decisions quicker, so we don't have hardfork postponsentsin the future and too. I think it was a very good decision to postpone, basedon the lack of information um and...

...that's that's it's something that Ithink will get a lot of critical eyes on the community, because we don't haveour shit together, but it was the right decision at the time to not introduce changes that couldpossibly have massive consequences d and just postpone something instead yeah. I do agree that the to oncommunity did the good decision in the short amount of time. That's that Eyhad so god to see like they are going to like ust good process of decision. So when they decided to postpone didn't beit wasn't a code already demployed on most most notes to actually do thehardfork m. How did that postponement happen? Oh Man, it was like I don't. Idon't know the story behind that and that's what I was going to ask thisearlie like when you reported this eetherum group and they made a decisionto postpone the the t e, the H, hardfor wha out of the Braksworkoso. I was in alot of the communications channels and an initiatives that sprung outimmediately after the decision decision was made. Um a credibly, fast report onwhat the state of things were. The postponement was drafted with a lot andpart due to mycripto holding out of that and Hutsor Jamison sperheadingAporshon, O that there's a block post. That whateveryone saw basically is theresults of this Um article that came out within hours of the decision beingmade, which gave instructions on what the people whowere compacted needed to do. That was mostly exchanges, note operators and Umand minors, and so what basically, what they needed to do was that GEF andparity released the releases that fixed the issue because, like you said,a lot of people already had their your infrastructure set up to handle this.This hardfore change automatically as it was supposed to roll out what theyneeded to do as then upgrade to a duversion that didn't have this happenat basically definitely postponed the hartforechange, and so a huge communications campaign happened in order to reach outto all of the people who needed to do this, such so that the hard fork didn'thappen, which was g reaching all the exchanges, largnote operators,steakholders and D and minors, and that was a massive email campaign totecampaign. Social Meta campaigned from the people, er involved. To get thatdone, and I was I was. I was quite impressed with how fast it happened andhow efficiently it happened. Well, all right M that, by the way,incredibly impressed Um can we use Thi soucan opportunity to dash doubt that etheiumit' Lieso, a lot of naysayers onthe otherium network, say that tms actually centralized, because you knowthey they're being driven by a core group of people who make the ultimatedecisions for things, and I know that's a bunk argument, but ite uses anopportunity to say that's, actually not what happened here 'cause, even though they they were ableto stop this hard fork so quickly. It wasn't a centralized process. They hadto literally reach out to people or inorganically, communicate that theyneeded to adjust what they were doing. Um It just to me, was pretty prettyimpressive that the community was so responsive and reactive. The concern Ihave is that if this particular situation happens again, ane theory ofnetwork is much larger. It might be more difficult to contain such ascenario worse of invencions for that matter. Yes, so what? What is yourorpits cantentious exactly? So? What? What are your thoughts with regard tothat guys? I mean obviously th the philosophy behind this is that we don'tcare, but if we don't care also, it's self defeating that's kind of a problemlike it's, not that we don't care. It's just that Y Ow, the community decides acommunity makes cicizens people decide that their network does, but at the same time, it's also likethere's this trade off with that and that, if things are too unwieldy orlarge, if there's too many people to reach out to, if you can't get thenotice out quick enough and you have a zero day vulnerability, which isintroduced through a protocal change M, you know you have you have zero day tofix it like you, gotto get on that. So what? What? What? What are yourthoughts with regards to that and how we can maybe mitigatethese kind of concerns going fover yeah? The ability to coordinateincidents like this is, is a et's say a week, but um rapidly avolving capability for thecommunity. So we ran into this ourselves a few weeks back when we weretrying to help levelqu report, their gas consumption, gas. Token relatedsecurity issue Um, where we needed to track down the security coninformationfor every single exchange and dialed...

...back about two three months, and thatwasn't anything that was possible. Nobody had Madea list and therecertainly wasn't going to be any kind of you know a nice essentialized systemthat allowed you to contact all these people. We didn't even have emailaddresses Um, so we tried our best to trind out the situation by puttingtogether a set of best practices for incident coordination and for security,contect information through a getorepository. We call it block chanedsecurity contects, and I am pretty sure that the etherium foundation at leastreferenced that when they were trying to find people and how to reach out tothem. Yes, that's that was part of the list of people to contact and who werewho like how to get a hold of them. Yeah S, this kind of basicinfrastructure. You know it's frustrating, but most people inmost companies don't create the stuff until a securty incident happens. You unfortunately learn by getting afew scars and Um. It teaches you what you need to do for the next one, so I I expect that to happen here,we'll probably see a lot of rapid infrastructure and like capacitybuilding after this event, but Um, I'm sure it was really difficult overthe last couple of days to figure a Co talk to Yeahn, there's an oppositeproblem too Um. This is one the community hasn'tfigured out yet, where so contract upgrades right like you'rein exchange, you decide to list an asset on your exchange. It's in theyear C. Twenty token, an ERC twenty token is not like a flat digital asset.It's a smart contract and smart contracts can change in the future. Soif somebody does a smart contract upgrade and changes the logic behindthat ear, see twenty token, should the asset be listed or delisted from theexchange that ave previously approved it, and how will the exchange find outabout it? I have no idea 'cause. Most companies that producesmart contracts are not announcing to the world that they have made acoachchange. So it's kind of on exchanges to monitor that informationand there's no way like how do they ask? Is that an authorized change? 'CAUSE,you know. Maybe you broadcast like hey, were upgrading this, but that broadcastitself might be malicious. Somebody could have taken control of whateverkeys or Webserver or emale address that you have Um. So the whole kind ofsupply chain here also exposes a lot of these same riskswhere communication is not trustworthy, Um and there's a lot of potential hissues with coordination that I see possibly cropping up in the future.Well, what you just did as gave me an idea for Neip, where, if somebody wasto actually do a smart contract upgrade, it must emit an event so that theexchanges Coul just monitor Echane, and if they see that event, then they haveto change their change. You know do whatever their their response is, forthat particular event. Well, the event's not itself enough right,because someone can broadcast like hey, I'm going to upgrade, but you stillneed to get some kind of confirmation from a person behind it, but like thiswas intentional hers. What it does you need to describe it Um 'cause, I you K W. It may change theunderlying fundamentals of that of that asset. Yoare Olyng abuilt proceduresaround that, meaning that if they see that event and they haven't preapprovedit, then it gets belisted on their exchange, the ASERT they es. You havesomething there yea. We definitely agree with the Danestatement there. So actually, some of our clients Um, have upgrade featureswhere if they want to uprade th the contract, it goes through some greatperiod, so they can only upgrad the contract by kind of announcing anupgrade and then two weeks later they can actually upgrade it. So I think that's something I mean,even though I I would like for such things to be more stendardized Um. I think that's something that peoplecan adopt fairly quickly. It comes Toi obvis disadvantage that if you have todo some emergency upgrade, then that won't be so fast but um to me. Mthe upgrade really conflictwith that whole immutability idea and therefore I think they should berestricted somewhat significantly. So I m I'm definitely in favor of of somegood handling and in theory then then very stanardize process can happenright until you can say. Okay, first of all, I'm announcing this upgrade Nowatim going to review this code and then approved that this particular Cotagh,and now that we have soon excot Hash in as an as in idiaminstruction. Then after the UPRAT happened, the theexchange will actually check that the uprade happened correctly and thenafterwards, otherwise we're delist te token. So ther interioor things couldbe nice but yeah. I guess, wear we're bit further away from it. I think apart of this that I want to bring up...

...that like a were dealing with permissionaassystame.We can't enforce at the technical level any of thesethings. People could do whatever the Holl they want and w were another experiment in this.In the entire thing is social consensus and what we choose as the right thingto do or m safe thing to do at a social level and more maybe even appropriatelyleveld socially emerged consensus. What exchanges choose to list and how theyTus Soliti, is up their opinion and then how the community interactuollythat thing Kinda at the end of the day gives us an idea of what everyone isdoing is the right thing to do. Ou, know kind of, but, like that's part ofthe experiment, these things, because Thei'r permissionless is that anyindividual or company can do whatever they want. It's up to the entirecommunity to decide. What's the right thing to do and that' he usually shone through where money flows and howit flows so I'll mention that one of the best references for building asystem for time locked up grades is the GEM anie dollar. It's one of the talksthat was given our December Empire Acting Wer. We have some video andslides and I would love it f. More companiesbuilding upgrade systems reference that in their own design. This we have the ongoing discussionabout how much to share how much to publish Um the good thing aboutthetherium is like you. Have this Er cs right year see twenty, it's not an EIPtenty that is um, something which is a standard now which people agreed on byshowing that it's actually a very good thing to do, and I'm actually in favor of havingsomething like that for opperatability just by practice just by people sayinglike Luk. This is the standard we follow, not because the EBM forces as to do butbecause the UM consensus for it good cod is like that. I know to hat:Everyone follows it: they adfunctionality to it so and so forth.It's more like a standardized minimal functionality in practice exactly and then othe other ones mightmight be more about Um practices on how Ho? How do you have tohandle your keys right? No one talks about this Um, it's it's. Of course. Wesometimes hear from people who get hucked just in the very traditional way,and then you ask him well what did you do with your private keys? Well, I havethis computer and they own computer, but this is Um. Security doesn't stoponly on this level and what's coold practice Aa stop there, and I like that. A lot of these things aren't enforcedon the very lowest level of the EBM, but thatsensible projects have to figure that out D and are doing it and um greatly improving overtime, IAM, of course, infavor of publishing all security findings to say ey. This is what wefound trade of bits. Does that to regular blockposts? We do it topopulishing most of our audits and so that people see look. This is what canhappen. This is how to handle it, and what the scan showed is that a lot ofpeople follow best practices because best practices around smart contractdevelopment did actually prevent this reintancy to be attackable in manysmart contracts and- and we found some motcontect a vulnerable, but so forign,no major token, no major crowd say no major. Won't you Sak Wallet, because thepeople follow good practices and development which which had to developover time and which we have to continuously share as good at W as we are at publishinginformation, both chain security and trail bits. It's it's reallyunfortunate, but there's still so much hidden knowledge kind of locked upinside the auto reports that we both publish. If you really wanted to get to be aexpert in smart contract security, you'd go and dig through all ovour pastour CID fom published documents, there's some hidden gems still in therethat I don't think the hest of the Communiti is caught onto yet. I wouldagree- and I think that's a great way to kind of rap up. thes stepisode isthere anything that we should have asked you all or gotten to that wedidn't get a chance to starting with Joson. I think we we did. I pretty good toofAl Wisound, Utoah secuitcal system I can, I can think of Da and also please feel free to takethis time t to sheld whatever you do. I ge be ever H, yeah. So if there's two things thatpeople do after listening to the stock, um it'sprobably check out our blog,because we have a great set of videos. We learn more about security intheorium en wain security, and then too is you. Should PIP installslither, analyzer and securify really,...

I mean there's no downside to instillinmore than one tool, but I think these things are out there and it's reallyimportant that people start to use the tools and knowledge that our companieshave put out Um, it's UH. It's only going to happen. Ifpeople start like taking some effort and that's the easiest thing you can do, yeah though luckily then already didthe shilling for so I can s so I think Um. We I mean we. I think we characterizethe current situation quite well, but we also said we don't know the foodstory yet so I mean Um. I think we should a we are, even thoughwe all agree, it was the right decision to postpone Um. I think it might stillbe interesting to see eventually what what's going to come out, because Ithink we still potentially can can find out. More can can find more contractand if anything, this shows adented thatmaybe we need Ome superpowerful too. That really can can show us what theeffect are on the whole droctrain and and what Um Yeah? What? What reallychanges when we change thish when we introduce the new ERP, so I think Umtherethere can be some exciting things coming out of this little mishap and Um,but yeah Um. The last thing I want to say is Um. I it was touched on beforein terms of communication wheewhere also, I was quite impressed with thecommunication, but I was also soperimpressed with the quick release.Peed of PAROTY Goith Heran travel all of theseprojects that were just super fas to release the new clinds. I would definitely agree with that.Having up been involved with those channels and and and Sau like peoplereally really really work and spent a lot of time trying to make sure thathappened quickly. I was very impressed with the the quickness of response andand quality of it Um, and I think that about wraps it upaudience listeners if you enjoy this click. The like button share it withall your friends. Tell your dog et Cetera. Well, be you can find a suthashing it out otstream or the BICKON PoAs tcom. You can join the slack and join us in conversation to talk aboutwhat we're to talk about next or give us ideas or even thorow US money. Ifyou want to Um speaking of money, you know we are always open to sponsorships.We look herwere we're trying to make this at least pay for itself, so feel Yo, F,free to reach out to us, Pettiot houshing it out out stream and callingand hashing it outotstreams Corria passing about thatchterm. Is it I'mpretty sure it's petty you're, right, ATS, petty it's Qit for every otheremail that I have. I don't know why OA tpettyit you coald also reach out to uson twitter at Colin Cuche that collincuse and Cory Corypcor Petty Cor,petty, and we are also hashing it out, pod ontwitter as well so tice guys. Thanks for all the work youdid to help us mitigrate this problem, then I afford to continue using theirquality tool. Gentleman UPORIAN, FO, great hatest,.

In-Stream Audio Search

NEW

Search across all episodes within this podcast

Episodes (108)