Hashing It Out
Hashing It Out

Episode 35 · 4 years ago

Hashing It Out #35: Constantinople Postponement - Trail of Bits & ChainSecurity

ABOUT THIS EPISODE

Back with season 2 of Hashing It Out, and we have a doozy! This Episode features Trail of Bits and ChainSecurity to talk about their amazing last-minute catch of EIP-1283's impact on the Constantinople Ethereum hard fork. We go over how they found it, what recommendations they made, and how the hard fork was postponed to quickly. We also discuss the tooling behind analyzing such a vulnerability, the true impact had it been released, and how the processes around hard fork release candidates could be altered by this detection.

Links: - empire hacking videos - EIP-1283 Analysis - contract upgrade anti-patterns - How contract migration works - Blockchain security contacts - securify - slither - eveem.org

Entering forecast work. Welcome to hashing it out, a podcast where we talked to the tech innovators behind blocked in infrastructure and decentralized networks. We dive into the weeds to get at why and how people build this technology the problems they face along the way. Come listen and learn from the best in the business so you can join their ranks. All right, welcome back to hashing it out. Mr Is episode thirty five. As always, I'm your host, Dr Corey Petty, along with my cohost, Callin Cuche. Say Hello everybody. Hello everybody, and today's episode. We want on a nice long break for a holidays. But yeah, it's called the season two. We'll call that season two. Yeah, yeah, New Year, and to bring us back, we have quite a show. Our guests today are Jocelyn and Dan from trailer bits and the Tis from change security. Would you like to give us give yourselves a quick introduction, starting with a dand to Jocelyn from a Tis, on who you are, how you got involved into the space and what you currently work gone? Yeah, sure, Hey, I'm Dan Guido, the see and confounder of trail bits. We're a seven year old software security R D firm that I founded and have now grown up to about forty people. We specialize in blockchain security out of the interest of our own employees. Wasn't really a planned decision, just something that came up about two to three years ago because we saw it was a really interesting set of technology and wanted to play with it and thought the skills we had would be useful. So I have Joscelyn here from our team. He's one of the leaders behind the tools that we write, but knowledge that we push out and the audits that we do. So, Jocelyn, go ahead, tells about yourself. Thanks. Without tradition. Yes, so I'm just my fhisht. I'm a security with such a lot would have bit. My big one is mostly undering up the detection and exportation. I've been involved in the secuity in the two ecosystem since something like two years, something like that, and I've been doing mostly audit and building off two like slitter or display. All Right, thanks. Tess about yourself. Yeah, happy to be here. I'm atias'm and COO and chain security. So we are, compared to the young company, definitely focused on ethery and smart contract auditing. We came out of university here when we started developing to its which automatically scan smart contract for security vulnerable. It is still as like masterly. This is the late up during PhDs, and then these methods became very applicable for you, theium, obviously, and then I found it chain security, like a little bit more than a year ago. Started out in February two thousand and eighteen with our first employee, and then grew too, currently fourteen people based in Zurich, and my background is it security from the more practical electrical engineering side of things, and I became active in the space quite some time ago, but as a user back when etherium wasn't the theorium yet or wasn't there yet. But then joined chain security beginning of last year and I've been active in that space since then. All right, that's a quite a panel today, especially based on the topic we'd like to start with, and that is recently, inside of a Theorem, we tried to do a hard work and a bug with one of the IPS that was being rolled out was found by chain security, which they disclosed a few days before the hard work. Can you give us guess a wrap up of the article that you produced and then you can start talking about it's implications and consequences after the security community to get ahold of it. Sure happy to. So. When ethery and does its network upgrades or Hart forks, then what happens really is that the roots of ethery and change and new things are allowed and that can have security implications. Obviously. Usually it means we need to look out for new things. You instructions are possible. Customers will...

...come with new things. So when we looked into what Constantinople allows people to do, we saw that things get cheaper. Now things get cheaper is generally good because you can compute more on chain. But as a company was kind of created when the first you really big issue with smart contract security happened at the hop, back back like what the two, two and a half years ago now, the problem of how that was mitigated was always that you need to prevent reentrancy. Is Obviously reentrancy is meaning you give some other contract, some other code which you don't control, some control over your own code in one way or the other end and you need to carefully control what that other code can do. One way it was done in the aftermath the Dow hack was limit the possibilities they can do very generally by giving them very little gas, gas any theory and being the fuel you need. And if you run a loop gas you get immediately cancer that your transaction gets reverted and nothing that can happen. Now this hard fork allowed some operations to be cheaper, most importantly so called store operations, operations which change state on the blockchain, which can save, which can change this global state which the Theorem is about, and that allows to attack smart contracts which in the past had this protection against the state changes just because you would not a give you would not give a long enough gas for like foreign, uncontrollable smart contracts in your own code. But now this little gas which was passed along is actually enough to do state changes and with that this classical reentrancy attack was something which could be now applied to smart contracts which were considered secure, also by by us as auditors, because we expected certain calls to to prevent these kind of reentrancy attack, while after that hot book they wouldn't do it anymore. And then we wrote down these we started researching them and saw that this is in practice possible. wrote up a post which we shared with the back bounty of the theory and foundation and it got published in the earth security mailing list and very quickly then the following activities happened, which I'm sure we will talk about soon. Yeah, so that that basically worm the beginning of last beginning of this week. So we have a mixed audience from UX designers but mostly their engineers. Can you maybe explain reentrancy as we knew it and what this new reentrancy kind of attack is? Or I don't even know if it's new, it's just a different, you know, something they didn't consider. Like what are what is reentrancy really like? What is it? What actually makes a reinteresty attack and what what happened when you lower the gas price to actually force PEU? twelred and eighty three two kind of like come under question. So actually what a reentrancy really is is being discussed. There's a very technical description we are currently down on, which means you give some other smart contract control and after what, you make some form of state change. I'll this doesn't help a lot, I think, for most of the audience. What a most code as know or think of their programs as a linear, or mostly linear execution. They know what happens vers this and this and if now, this and the other things happen and so on. But they kind of have a linear trace to the code and with etherium in certain points, especially if you want to transfer money to some other accounts, money, so being at either or you want to interact with some other code, you are calling it. It's similar like in normal in other languages, if you if you call a library or call an API. Now, imagine you call an API. Let's say you some...

...some website, you load some website and you include the response into your own program and say like, based on this, we will now interact differently. Right, basically, the external call has impact on your own code and will turn on in injection attack, Sim Mirror to an injection attack. Now the crazy thing here is regency attacks are more powerful than an injection attack because they allow these outside code to call into your code. Again. It's like if the external website could directly go into your web server and quickly call some let's say function in there, which then has impact on how you continue with that request you made. And of course this is suddenly allows changes into in that smart contract we are currently in, which you didn't think of. Suddenly they might change a variable which is considered not changeable during that line of code and that will mean. Okay. Now, now some assumptions you make. Perhaps you checked that you have enough tokens before you make a call and then yeah, you're fine, you have enough. But this regency allows other code to withdraw all tokens and now you don't have them anymore. But you still think you have it, because you sink in this linearly way of first a check and it's okay, then I do something and then I save something. But in the meantime this attacker could could have changed and so and so previously, before this particular hard fork, many of the calls that could have been susceptible to an attack like this. We're too expensive, so you didn't. You just basically assumed that, based on the way these calls already worked, you couldn't do reintency because you would never have enough gas to do a state change. Because every time you do anything on the evm that cost gas, especially in its operation, dependent and with certain calls, whether it be sent in transfer, I think it's sent a transfer set right. Yeah, that you'd never have enough gas, it would be too expensive to continue doing this reintency attack. So they were considered secure. And this input the CIPA did was make it less expensive to do these types of things for these two functions, which then opened up all of the previously deployed contracts who had a pattern like this the possibly be vulnerable because they were now like that assumption of they'll never have enough gas was no longer true after this EP change. And so, yeah, it's a great story. That's you know, Oh, let's just make everything on a theory of more efficient, will lower the gas costs, everything will run faster. And nobody looked at it and thought about it for like two seconds and said, well, don't we depend on that for reentrancy protection? And then the Jake security people thought about that for like ten seconds and we're like, Oh, this is a bad thing, we should tell everybody about it. But yeah, the Layman's terms here is it just loosens the restrictions on reentrance. See, so there's a tiny window opportunity and this thing kind of blew it open a little bit wider so that more attacks were possible than before. So actually, that's that's kind of a point that I'm kind of curious about personally, is why the why the hell didn't anybody think about this until like twenty four hours before the release? Hey, so, mattias and I and everybody else that's doing security have so much work on our hands that we can't sit around a police every single EIP. You know, even Martin at the etherium foundation, like Martin's got his handful of just trying to make sure we don't have an unintentional fork because different client implementations diverge. So, you know, we're all trying to run businesses. Everybody's got to keep the company and float with money to pay our employees and nobody's really paying us to look at every single EP. So what this really does is it reinforces that it depends on our own personal interest and the hobby time that we have, which is very small, to take a look at one of these dips in detail and the real analysis behind the kind of security impact of this particular eip. took a long time, like I had to pull Joscelyn and Stavo and Evan and like half my team off projects they were working on to scan the whole blockchain, figure out all repercussions, try and come up with remediations and honestly, like it's not a zero cost. It's actually a negative cost, like I lost money by trying to help with this incident. So it do that to your donateier eath address, which will sure I did whatever, I'll take it. I'm not going to refuse anybody's money, but are...

...well about take that back a little refew some people's money, but not going with any scam coins today. But like this, this really brings up the point that there's very little security expertise arounds and if you want that kind of input, you need to pull that input in somehow. And since it wasn't done in this case, you know, we get the chain security people who decide to start poking around some area of etherium for, you know, no reason at all and then discover this issue that really ought to been obvious. So You'R's like. We'll get into some of the more technical details in a moment, but I just want to front load with with this. You. You are the CEO of a company. You are doing this literally pro bono for the benefit of the community. What can the community do to make not only your life easier, but perhaps build a business or an ecosystem around supporting this open source software that people are protocol, that people are, you know, making proposals for and changing like. Do you feel like there's there's something maybe we can do better as a community to facilitate catching these kind of bugs sooner rather than later? Yeah, I mean it's a little bit of a tragedy at Commons kind of kind of deal, because it's not any one individuals responsibility to police the e Ip's to come out. It's it's the whole community's right. So it's not necessarily a business, but I think the etherium foundation probably needs to set up a more detailed approval process for e ips where you can't pass go unless certain things get reviewed by security expert that they trust and and that that comes out a cost, right, like somebody has to have to pay for that. What we decide, well, what we recommend it. So I'll excip it this EST here, as blog post has has a lot of these desails and the description and the and the white paper or a security analysis you guys put out, because I mean that is amazing recommendations in it and I hope you can that the say right, yeah. So that's what I'm skipping ahead to a little tiny bit, because you know, you probably don't need to review every single EIP. There are a lot of EIPs out there that don't change anything security critical. But there are a lot of things that you can just keyword search on and say that's going to be a bad one, right, like things that affect contract upgradability, things that affect gas cost, certain like if it if you modify the semantics of an existing instruction, like those things are dangerous and require a lot more study than things that don't do those things right. So you can imagine even like a Github bought that just flags certain issues as wow, this is like security review required, and then the etherium foundation has to pull in somebody, whether it's someone like Matthias or someone like I or someone like Jocelyn, to provide some kind of statement of like, you know what, we don't think you should do this, or here's how you might do it better, or these are the these are the implications of doing this across the blockchain and how and how it currently exists. I don't I don't want to scream doom and gloom. This entire time, because the total amount of damage associated with deployed contracts that this would have had, what with WHO this would have impacted had this gone through, isn't incredibly large that we know of so far. Right. Yes, I like to test contracts, I think that were on the chain that could impacted. They weren't even like real life. I don't care, because there's a lot of there's going to be a lot of media associated with the postponement of, you know, how terrible a theorem is or how we don't have our shit together and so on and so forth, and I would like to be at least give a reality of the potential risk, or like the the known risk of doing this type of thing. I think, more importantly, the what we've learned from his entire process is what you just described, is figuring out how to go about and looking at EPS appropriately for the changes that they're making, so that we can incentivize people to look at them to correct way before throwing them on the main net. Could Canal talk about that just a little bit? I think we are Super Lucky that this problem wasn't worse than it is. However, however, I have to covey out this, and I'm sure you, teas wants to say this to the result of our analysis was not conclusive. It's not exhaustive, like the tools that chain security has, the trail bits has and the ones we pulled in from the Evem folks do not guarantee that we have a precise analysis of the entire blockchain. So they could be people affected right now but we don't know about and that's another kind of, you know, discovery here that we don't really have the tools to effectively monitor everything that goes on in the blockchain. So it's probably worth it to not change, not pull the rug out from underneath people when they deploy a smart contract and expect that it works a certain way. So...

...there's another discussion here around what is the mutability of certain ways that etherium works, like can the hard fork or the hard forks that we plan in the future change the behavior of existing contracts, even in minor ways, because even if they don't introduce a reentrancy attack, they might modify business logic or the way the application works and that might be just as bad, depending on the scenario. So all this, I think, really came up today or like yesterday, but again, it's one of these things it's obvious in retrospect. You're writing these tiny little computer programs that you depend on. Ultimately, if you're considering that that code will never change, shouldn't the blockchain never change to so I don't know what the answer to that is. I think it should never change. I thought you, you guys, in your in your analysis, put out some pretty good recommendations and that you just don't change old stuff, but you can implement new stuff. Not Change Behavior of a store, but you could create a new type of store which is cheaper and maybe more restrictive or has different security requirements around it. You can prevent this kind of stuff. He talked a little about your recommendations. Yeah, the main challenge here really is that ethery and will keep moving and as someone who has seen this from both sides of where like stuff holds to a grind because you are in this what I call Jenga coding mode where you're afraid to move any piece because you're not under control anymore to what kind of implications it has. And basically the whole project is not moving along anymore, versus the one way you know what's going on, what it kind of implications it has and you can do changes. That's very important and think actually this current discussion where we started to look into immutabilities and stuff which we would consider changeable in the future. This is important for us as auditors because we can advise our clients on safe now but not guaranteed to be saved from the future. And it was brought up also by Vitelli himself, like very shortly after this discussion about what to do now, that guess cost suddenly is something we need to be considering critical. Right. Gas Wasn't considered that critical before and now it is clearly so. When we start to add a lot more complexity into ethery, in which people implicitly base their security on top of, then we will stop having any things which we can move on anymore. It will be immutable block chain, the only thing we even even changes, where we say future opcodes, future instructions should be possible, but never change existing ones. Actually changes existing ones for reentrancies, because an attackle currently within reentency has certain gas costs on their own side which have a certain amount which you cannot get rid of. But new attack code which cheaper instructions potentially can get rid of it. So I think the kickoff of this discussion was very good, where we kind of think about what will the ethery community think of almost or basically unchangeable things, and where do they say? Look in the future, we expect that to change. So better make sure your smart contract will stay safe. Let's say the go global block gas limit, right, that's something where we know it's going to change eventually. Can it? What would happen? And if it goes lower, a lot, a lot of smart contracts don't work anymore because they need a limit, they need some gas, for sure. But then we expect that never to go lower, always to go high, do we really? Is that just an implicit assumption? So I really like to have that discussion going forward, which is helping us as security auditors in the end, because we know what to rely on and to flag smart contracts which might be vulnerable if they ever change that. And if then we see, oh it's going to be changed, we can at least warn, we can effectively scan for it. We know the implications and for other stuff where we know it's immutable, we can get very strong guarantees because to me, initially the huge appeal of eitherium was that it can encode certain like real word things into smart contracts, which then we'll guarantee that for the foreseeable future and with change security, of course, we we are very much into this formal verification, into proving that certain properties hold. This relies on the rules don't change, or...

...at least some rules don't change. So we really like to to tell a HALP, some authority, Government Authority, that okay, these people will never be able to steal those funds because we mathematically prove that. But that reliesnse things which should not change. would be good if you know that those are, at considered, almost immutable. I want to just highlight something that was buried in there that Mattia has mentioned. So right before we started speaking, you mentioned, Oh, well, you know, we're not going to change existing functionality, we're just going to add new ones. We're only going to add new instructions. And the rabbit hole here goes super deep. Like even when you make these minor modifications, like let's make a new instruction that's just like call, but instead of call it's cheaper than call. Well, that enables new attacks to since attacks that were possible that cause a certain amount of gas now might become possible because you lowered the requirement to run an instruction with nearly identical functionality. So even these tiny little modifications end up having these ripple effects through the behavior of and the possibilities of calling into existing contracts. So I'm with Matthias here. Like we do lots of verification forward projects as well, and it makes it a really hard to target for you to like, how am I supposed to say that this code is safe if the underlying execution model can change six months from now? It seems like a time based approach of to say this because, like, if you think about that from my God, I guess, more higher perspective, if you only make forward changes, then in the bouldering and sexist you're making these the attacker stronger while making the defender weaker in some cases. Yeah, the big brain answer is, well, I guess you need a security on it every six months now. Well, I guess maybe, like, well, that's that's that speaks to another point that there's not a lot of like once you deploy something, typically there's not a lot of monitoring of the deployd contracts with respect to changes and in tooling and infrastructure around that whatsoever. And and it's clear that we're a we're still an experiment, that the Etherym Vum it's still learning, growing, becoming more efficient. We look at the changes on the horizon to make it more scalable, at layer zero or one, or whatever you want to call it, that's clear. They're going to happen and have effects and and I think under a sense, and Apolis speaks about this concept quite a bit in reference to the bitcoin block chain, the Ossification of the protocol allows you to more effectively build layers on top of it and I don't see the hardening of the protocol happening anytime soon, especially with a lot of the changes are trying to make to make it more efficient so that it can scale. And people shouldn't expect it to. There's no reason to expect it to. It just kind of have to experiment with it until we get to a point of scale that's reasonable when we say all right, this is good enough and then we can start making these heart assumptions on things that won't change and can change and should change to one and so forth. But until then I don't I don't see a need to do that you feel the same way anyone. So when you sure that of people are already building business and application on top of like the con set of was, I too. So we can, we can need to have some kind of qualanty and to have some kind of way to unchoose. I could falls a supplication. It's not like it to whom is explainment on? No, but you is using it for any weak cares, application and anything, because people don't bullion low with application on top of it. So I don't know, like whole how long is it could wait a twelve some saying mom shown it's I'm not. I'm not saying weight. But build it with the with the fact in mind that things will change. You need to keep that in mind and maybe allocate resources to constantly monitoring, really take a look at how to like migrate contracts and up and make sure they're upgradable and for future cases that you may see coming. Don't don't make hard set boundaries on what works and what doesn't work, or what secure and wasn't secure right now, because there will be changes and that's I think that's just a mindset or mentality we should have. When the point smart contracts. So the question that I ask is, what if there's a change that impacts the upgradability of your contract? Like to me, it seems as though. It seems as though we are we are. I like you, definitely made it clear again this is an experimental chain. It was always meant to be an experimental chain. It's a proof of concept, I feel, and that there's still a lot of work to be done to figure out what the what good looks like. You know what I mean and you know right now. What this has shown me is that a mutability is a lot more valuable than I thought and that, you know, if people are going to be building business...

...processes, it needs to be a final form ultimately. Now, for now, I mean, obviously we're not going to have that, but this is not the final form. We don't we don't know what the final form looks like yet. We have issues with scalability in terms of transaction speed. We are adding features like state channels because we know notice the need for layer two solutions, and now we realize that we don't have the ability to change the protocol very much once it's released. So, of course, the very first version of this release, which is the you know, the one we have might not be optimal for you know and future proof. So I don't know, I still feel like we you can't expect developers to know these details. If you're going to be in the final form. You can't expect them to worry about you know, nobody worries if the TCP, if TCP, if Ip for change changes. They just know that they have to migratee iep six or use neat. But they don't. They don't want to concern themselves with the details of the I of the IP stack on there's on their system. They don't need to know this this stuff. They need to know how do I build stuff and how am I how do I build safely? And they can't just monitor all the security stuff, security mailings to see if their APP breaks or even expectable understand it if that happens. It needs to be a lot more open than it currently is, and so the hardening might, you know, might be a different change. I don't know. So on this like when you started like saying very value in mutipability a lot more, I think that is quickly leading to not moving on anymore. And in the real world when you say like okay, I start my company. I like the loss of Switzerland. The Laws of the US apply to me. You know they are going to change and you are going to adapt. So, in addition to the monitoring, basically what you need to know is when do I need to adapt? Things are going to change, for sure. Don't expect that this thing will run forever and changed. Know that upgrades might come, that you need to do something if you want ongoing, active business. It's your duty to day, technologically fine, business wise fine, with the processes. Now, when there was the buck about the pre into processes like the what was it called Specter? Then? Yeah, now, suddenly things which were considered safe yesterday are in safe anymore. Doesn't mean the end of the world or that we should not, i. e. A upgrade or processes anymore to from the ones we consider safe right. So, to this regard I would more say, yes, we need monitoring, we need to know what's going on. We need a process around informing people about perhaps longer term cycles for testing. For that each company who runs something on Mainet etherium also runs it on Robston, on some test net which gets upgraded, usually earlier, to see what's going on there. That the etherium community itself more encourages this. Okay, this earp is going to change the following things, not only in the technical sense, but also in the practical science. Like, look exchanges, this is something which might affect you specially, and so on. So that like lawyers would do for their clients, for big companies, right, they say, Oh, this upcoming regulation change, this will impact us. So then we need to figure out is it the job of the government, basically of the etherium foundation, to do this, to encourage that, or is that something where each company needs to employ someone like Dan or us to keep money towarding this, to tell them about it? Post might work just fine. Let me put that that around on you. Okay, your business and you are running your business on this blockchain. It is a protocol and there's literal value and money being generated from this. This this blockchain. Your you need to know that that business model that you set up has assumptions which you can guarantee for the remainder of your business. So let's just say they decide to increase the gas prices something for one reason or another. That could impact your business line. Right. It's not really a good idea to set your business on a shaky foundation where you can't kind of have certain...

...fundamental guarantees about how you are able to manage your finances and your money external to you. Now, obviously there's always going to be situations like that, but, like my question is, do you really feel like maybe this is so? I think the point I'm trying to really make care is not don't stop innovation. That's not something we need to we need to do, but I'm questioning whether or not we're going to see the final form of what this kind of prode, you know, kind of system would look like, you know, emerge from etherium, or if we're just still learning that the things we need to know to build that system. If, if you're talking about them, about building a business on a theorium, whether or not that's a smart idea. Of It's Shures a smart idea. You just need to be aware of what can change and that's more clear, you can make better decisions around how you build Your Business. Don't automatically assume, because you say the word blockchain, that nothing will ever change. That's just not how things work. And if just just be smart about what you're building and more along like the relative time guarantees of when things can change, like know how things can change and how long and how long have, I guess, a guarantee you have of those types of things if you build your business that way that it's smart and you can you can, you can adapt, like Mathias was saying. So as long as you're you build with the ability and knowledge of being able to adapt and you should be okay, because things are going to change regardless of what you build on. There is no such thing as perfect mutability forever, even in the BITCOIN blockchain changes. That's that's supposedly like to know the most. I would call it the most stubborn in terms of change. I can't not mention a small little shout out here. So I don't think we're going to resolve this discussion. This is issue on this podcast. So I think in the meantime, yeah, companies should probably be familiar with how contract upgrades and contract migrations work, and if there's one authoritative reference for that it's the two set of blog posts the Joscelyn wrote. So you know, in the meantime we're not going to get a fully hardened blockchain that never changes tomorrow. At the same time, the changes that we have in the future are, you know, against the best efforts of the TS and I and everyone else working on security on the etherium blockchain, we're going to have an incident like this again, guaranteed. There's like as you're changing this kind of functionality, there's just no way that, unintentionally, at least once, you're going to encounter an issue. My dog is in the room. So it's really incumbent on people developing smart contracts right now to ensure that they can run an upgrade when they need it, and that means also including a migration. So yeah, if that's a thing that you're not familiar with yet, you should check out our blog and Joscelyn's papers on that sect subject, just to put a lid on that. So where do we go from here? Like what? Like, what are you want to talk about? is or something that has been pressing on your mind? At these things brought to light other sepes of things we haven't gotten to. I mean there's a million other security issues and in a theory and that we could talk about. So earlier this morning we published a list of recorded videos from an event that we held in December. Or we got together some of the best experts in atherium security and blockchain security more generally to speak about all the problems that they're working on and that they've observed. We have an event in New York called Empire hacking runs every other month and in December I do a half they event. It's kind of a mini conference, so I themed it around blockchain security this time and there are some great talks up there. If reader listeners who would like to take a look, I'll think that mission out as well. So, yeah, one of the things that that kind of has been on my mind, as well as the nature of of smart contracts in etherium. Do you feel like we need fultering complete smart contracts? Obviously that makes your job harder if you can't form use formal verification techniques on these things. Would it be better, do you think, if the if we weren't adopting such a flexible smart contract system and really just had what we basically needed for a root chain delegated all the more complex operations off chain to to layer two solutions? Well, so I think what strikes me is that in a lot of safety critical environments, like, you know, embedded systems, airplanes, things that actually affect human life. There are really strict requirements around eliminating certain kinds of functionality from the code that gets written, like eliminating recursion, because while you can use formal methods to prove or disprove that certain things can happen, it's harder to do so in these smart contract environments. It seems like be willing to take those steps. We prefer to have the flexibility rather than limit people to say,...

...like a domain specific language that lets you do a lot of things but doesn't let you do some. I think there's a really strong argument for following the safety critical kind of approach. But you know, again, this is something that, like I'm arm chair, you know, commenting. There's not much control I have over how atherium does or does not work with smart contracts or any other blockchain, unless I go and build it myself. So I think the reality is that we have to deal with this kind of complexity and that's why I companies like trailer bits and chain security have invested in a lot of us really heavyweight automated reasoning, because it's the only way that we can get any kind of understanding about what goes on at a low level. I think it's also like pointed to put out that, like the security community overall, is relatively new, to new to the same and it's it's quite small. Like take, for instance, that this last Def con was the first can we actually had a security track and this last year probably marked a rapid growth in a security community overall, but it's still relatively small to where it should be based on its importance in in what the Theorem is used for or what blockchains are used for. Well, you know what I actually took. I'm going to say something positive. If you look at any other like technology area, if you if like when we're reviewing a kernel driver or a piece of windows software or like a web application or something like that, we're not using automated reasoning techniques, we're not using formal methods, we're not providing our clients proofs or guarantees that their code works correctly. They don't have a speck. They're not testing against properties like even though the secure dirty community for atherium and for smart contracts and for blockchain stuff is just as small, the kinds of techniques that we can bring to bear on these problems is extraordinarily better than for security more generally. So I don't. I actually think that it's a bad thing if the security community gets to be too big. What I'd rather have is it rather have a small number of people with really pointy sticks, like I really want to have very effective techniques that we can deploy, that scale well and that provide like guarantees, that provide really strong assertions that the code works well. What I don't want is I don't want an army of people rolling around pointing out, oh, that's a reentrancy, that's a reentrancy, like one at a time. That that's that's like a dystopia in my mind. So I got to give the smart contract world credit here. Instead of starting at at like floor one, we've kind of entered the discussion at like the ten floor because from the get go, companies like Jain Security and trail abates have brought these really heavyweight techniques into a practical use case. So that that so I'm curious. How did you guys do your analysis? What tools are you using and, like this is a lot of work to be done in a very short period of time. What is your system look like? What is your architecture look like? How did you get this? How did you get it done so damned fast, and like how? What tools are you actually using and how can somebody who's curious about this start using those tools? It's she'll tie. Let's go. I want to hear Materia's first. So, who about our Tito, who found that bact just entered. So I'll actually give him the go for that one to to so that I don't blunder it by saying some words I'm not totally familiarly, was in the wrong way. I'm going to do the same thing justly. Yeah, thanks. So, first of all, yeah, I know this was a moment ago, but I have to follow up on what then said. The block post on migration and contract upgrade really excellent. I can I can recommend them also. So, but back to the question on which tools we are using. So in general to find the I mean in general we're using all kinds of tools, but I guess your question was more targeted at which tooths we were using to find vulnerable contracts for this particular re entrance. You fol concert to Nerpal, I guess right. Yes, that's correct. Yeah, I want to know what you tools. I want to know how you did this, like how did you accomplish this field so quickly. I said, to give it over, offering work or, like I gues said, or like it's a model for people. This this was chain security released an article that show a possible new re entestry attack and then that hit the same community maybe forty hours, roughly forty hours before the hard work was a scheduled to feed to hit. And I'd say within twenty four hours they turned around an article that that gave quite a there wasn't exhaustive, but it was quite extensive in...

...terms of the the the range of analysis that they did and we're asking now what tools that they use to get this analysis done so quickly. Yeah, yeah, yeah, so from our side, I mean first of all, as you said, it was not exhaustive. So we first focused on very valuable contracts, and here also and treadle bits had slightly different approaches. So we compile different list of contracts, which was good also because we were looking through different things. But so, yeah, the we were we were first trying to figure out what contracts most important to look at. Then what are we looking for in these kind of contracts? And we are looking for very certain conditions, very specific conditions. Right so that this reentrancy can appear. So first of all, the most basic condition is there needs to be a call to a different address, otherwise you cannot have a reentrancy at all. Then then, following this call, has to be some stage change and there has to be a separate function which does some kind of state update, and this state uphead has to be very cheap. So, as we did get discussed before, the whole point of this new enterity is that there's this now cheaper option to do this update. So we have to find the function like that, that that does the does this. So basically, first we downloaded the contrary contract, we identified the valuable contracts. We download a contract, we have found that which ones contained these relevant parts. So to figure that out we use different kinds of tools. One was them too much. I'm probably Miss Browsing his name, but too much Colinko, who has a evm dot org, so ev w e M Dotorg, the compiler day, which we use to identify certain contracts that might be vulnerable, and then we use all kinds of other tools to figure out which might be candidates. And in the end we looked at them manually. We one of the tools we also used was our securified tool, which is open source as well. Be Yeah, because this can identify one one of the two present requisites. Quite whether I'm quite efficiently, because not everyone might know it. The tool is called securify and you'll like find it on security fight at thh to easily give it a spin. But with the help of the theory and foundation actually, who gave us a grand last summer to continuously develop that and to open source it, we were able to spend some time and released it publicly now. Awesome. So that the fact you brought in the theory up, the theorum foundation kind of is another question I have, which is what is what was it like reporting this? That's Dan use different tool that one here. What Dandies? Okay, so go ahead. So, just like Matigus, I'm going to let Joscelyn explain to Jocelyn's the primary author behind slither, which is our static analyzer. I actually think that it's quite funny that chaine security and drill a bits are on these two parallel tracks of development where we have very closely related tools that just work a little tiny bit differently in the details. It was helpful in the scenario. M Yeah, yeah, I know. Certainly it was extremely helpful in this and this this scenario. We got some additional coverage. We were able to look at some areas they weren't. They were able to look at a lot of areas we couldn't. So it was it was good. But yeah, Jocelyn, tell us how slither works. On what you did yes was okay. I thought of like a push, which is kind of seemed out to one Chet Siod you did. So I started have by targetting like the high profile contract that you can find on on a test Cann so like the contract with anyone about amount of those action minimutum oft of eare and I've wrote a detective in slitter, which is a solidity, is that you can alyzer to kind of detect this really particular pattern of fortrancy that can be affected by this constantatium WORB outfolk. So after we viewing the wizard for the like high profile target, I continue to expand, like the analysis to any kind of contract I could find from from a core on a scane and sting, which is also in interesting. That solity is slitter was...

...working at the seldity level and for this particular we in tryancy. You need to really precise model of what is a guess cost of your contract, so to kind of to to be able to analyze this. With my tool. I also combined slitter, which with another anither, which is Monty Care, which is a symbodic execution engine, which alluse me to kind of have a precise information about the gas cost of the function. Don't extense. Yeah, so I then ified to recap, and not at the highest level possible. This EP allowed for a specific type of pattern to exist that allowed for the book above to happen. You codified this pattern somehow searched across all the blockchain sparkracs you can get ahold of with. So they're looking at solidity code and the other tools looking at evm code or a lot of instances for this particular pattern. That gave you a set of possible contracts that could have the bug and then you manually review those who look for whether or not it was actually vulnerable. Yes, it was nice about this. Is that like? It's? What I was saying earlier is that because they work slightly differently, you can you can get you came up possibly with different sets or re verified results from pre from each other sets, so that you had a stronger guarantee that there was a vulnerability in a given contract or you got larger search space. Yeah, that's exactly true and it's actually what in etherium happens often. Right, we have parity and get we have like different companies doing very similar things but covering each other and turnut of bits and chain security are similar in that sense to, for example, of the Sliza is analyzing solidity. So the high level language and the and can then be very specific towards certain securities checks which can be found there. The security fight to news at the EVM byte code levels, so it can analyze all smart contracts which are out there, even if we don't know the source code, which is of course powerful because as we figured out during this search was that for a lot of high value smart contracts out there which hold a lot of either well, they don't publish a source code also, for reasons to a little bit hide into what's going on, they usually multi sick wallets. And now certain checks are way harder to write this way, just because the evm level is not as easy to understand then the solidity code. So there you always have this trade of USABILITY, whereas like completeness, amounts of check sports positives and so on. And that's what I like about like you see, oh they're going this direction, let's cover a different direction to have as complete a picture as possible. And it's not only US right there. The Security Community is full very interesting people who are really, really good at what they do, often working alone from somewhere and then building tools which they open source and which then help everyone try up. It is a great example for being able as like an old, like older and also, let's say, nicely found company who can go out and they actually open source most of the work and we can build on top of it. So it's not like rand security is not using some of the tools of trade off bits and I'm sure tryal it's is using the tools we like to spurs to whenever we can. So that's definitely huge strength of the ethery and world where you you have these different clients, different companies, different security tools to look at things from very different perspectives to cover as much as possible. Well, I want to continue on that line. I want to say before we move on, that part of the reason for the postponement. Not all of it, but part of it, like during the call when when we were trying to decide whether or not what would what to do based on what was going on, was the lack of information. We weren't sure of the total risk or surface of vulnerability of all the contracts based on this change, and because of that lack of information, it was best up opponent to hear back from the analysis that we that we got, and I think that's that's an important thing. Is that one if this, if we can find ways to incentivize looking at these things earlier, we can we can make better decisions quicker so we don't have hard work postponements in the future. And to I think it was a very good decision to postpone based on the lack of information, and that's that's a it's something that I think will get a...

...lot of critical eyes on the community because we don't have our shit together, but it was the right decision at the time to not introduce changes that could possibly have massive consequences and just postpone something instead. I do agree that they too, uncommunity, did a good decision into shot the moot of times. That's so it's good to see it likes going to our like she's a good process of decision. So when they decided to postpone, didn't the wasn't the code already deployed on most most notes to actually do the hard work? How did that postponement happen? Oh Man, it was like, I don't I don't know the story behind that, and that's what I was going to ask this earlier, like when you reported this the etherium group and they made the decision to postpone. The the the the hard work. What? How do the brakes work? How the hell? Okay, so I was in a lot of the communications channels and initiatives that sprung out immediately after this decision. Decision was made, a incredibly fast report on what the state of things were. The postponement was drafted with a lot and part due to my crypto holding onto that and Hudson Jamison spearheading up portion of that. There's a blog post that would like. Whatever one saw basically is the results of this article coal that came out within hours of the decision being made, which gave instructions on what the people who were impacted needed to do. That was mostly exchanges note operators and and miners, and so what basically what they did to do was pet death and parody released the releases. That fixed the issue because, like you said, a lot of people already had their infrastructure set up to handle this, this hard fork change automatically as it was supposed to roll out. What they needed to do is then upgrade to a new version. That didn't have this happen. That basically definitely postponed the hard fork change, and so a huge communications campaign happened in order to reach out to all of the people who needed to do this such so that the hard fork didn't happen, which was reaching all the exchanges, largemote note, operators, stakeholders and and miners. and that was a massive email campaign, twitter campaign, social media campaign from the people involved to get that done. And it I was. I was, I was quite impressed with how fast it happened and now efficiently and happened well, all right, that, by the way, incredibly impressed. Can we use this as an opportunity to dashed doubts that etheriums. That's like. So a lot of naysayers on the etherium network say that a theiums actually centralized, because you know they're being driven by a core group of people who make the ultimate decisions for things, and I know that's a bunk argument, but maybe use as an opportunity to say that's actually not what happened here, because even though they were able to stop this hard fork so quickly, it wasn't a centralized process. They had to literally reach out to people or in organically communicate that they needed to adjust what they were doing. It just to me was pretty pretty impressive that the community is so responsive and reactive. The concern I have is that if this particular situation happens again and the etherium network is much larger, it might be more difficult to contain such a scenario, or sousint interest for that matter. Yeah, so what is your or, if it's contentious, exactly? So what what are your thoughts with regard to that, guys, I mean, obviously the philosophy behind us is that we don't care, but if we don't care also as selfdefeating, that's kind of a problem. Like it's not that we don't care, it's just that, you know, the community decides, community makes decisions, people decide what their network does, but at the same time it's also like there's this tradeoff with that and that if things are too unwieldy or large, if there's too many people to reach out to, if you can't get the notice out quick enough and you have a zero day vulnerability which is introduced through a protocol change, you know you have you have zero day to fix it. Like you got to get on that. So what? What? What? What are your thoughts with regard to that and how we can maybe mitigate these kind of concerns going forward? Yeah, the ability to coordinate incidents like this is is a, let's say, a week but rapidly evolving capability for the community. So we're in into this ourselves. A few weeks back when we were trying to help level K report their gas consumption gas token related security issue, where we need to track down the security contact information for every single exchange and dial back...

...about two three months, and that wasn't anything that was possible. Nobody had made a list and there certainly wasn't going to be any kind of, you know, Nice decentralized system that allowed you to contact all these people. We didn't even have email addresses. So we try our best to try and help the situation by putting together a set of best practices for incident coordination and for security contact information through a GITHUB repository we call blockchain security contacts, and I am pretty sure that the etherium foundation at least reference that when they were trying to find people and how to reach out to them. Yes, that's how that was part of the list of people to contact and who were who like how to get all of them? Yeah, so this kind of basic infrastructure. You know, it's frustrating, but most people in most companies don't create this stuff until a security incident happens. You unfortunately learn by getting a few scars and it teaches you what you need to do for the next one. So I expect that to happen here. We'll probably see a lot of rapid infrastructure and like capacity building after this event. But I'm sure it was really difficult over the last couple days to figure out. To Talk to you, there's an opposite problem too. This is one the community hasn't figured out yet, where so contract upgrades right, like you're an exchange. You decide to list an asset on your exchange, it's in the Arc Twenty token and your c twenty token is not like a flat digital asset. It's a smart contract and smart contracts can change in the future. So if somebody does a smart contract upgrade and changes the logic behind that EARC twenty token, should the asset be listed or delisted from the exchange that are previously approved it? And how will the exchange find out about it? I have no idea, because most companies that produce smart contracts are not announcing to the world that they have made a code change. So it's kind of on exchanges to monitor that information and there's no way, like how do they ask is that an authorized change? Because you know, maybe you broadcast like hey, we're upgrading this, but that broadcast itself might be malicious. Somebody could have taken control of whatever keys or web server or email address that you have. So the whole kind of supply chain here also exposes a lot of these same risks where communication is not trustworthy and there's a lot of potential issues with coordination that I see possibly cropping up in the future. Well, what you just did is give me an idea for any ip where if somebody was to actually do this smart contract upgrade, it must admit an event. So the exchanges could just monitor the chain and if they see that events then they have to change their change, you know, do whatever their response is for that particular event. Well, the events not itself enough, right, because someone can broadcast like hey, I'm going to upgrade, but you still need to get some kind of confirmation from a person behind it like this was intentional, here's what it does. Need to describe it because you know it may change the underlying fundamentals. So that of that asset. Yeah, we build procedures around that, meaning that if they see that event and they haven't preapproved it, then it gets delisted on their exchange. Yeah, sure, yes, she has something there. Yeah, we definitely agree with the DN statement there. So actually, some of our clients have upgrade features where if they want to upgrade the the contract, it goes through some great period so they can only upgrade the contract by kind of a nowncing an upgrade and then two weeks later they can actually upgrade it. So I think that something I mean, even though I would like for such things to be more ended, that I think that's something that people can adopt fairly quickly. It comes with the obvious disadvantage that if you have to do some emergency upgrade, then that won't be so fast. But to me the upgrades really conflict with this whole immutability idea and therefore I think they should be restricted somewhat significantly. So I I'm definitely in favor of some some good handling. And in theory then then very standardized process can happen. Right, in theory you can say, okay, first of all, I'm announcing this upgrade. Now someone's going to review this code and then prove that this particular Code Hash. And now that we have soon xcode Hash in the as an as an vm instruction, the after the upgrade happened, the exchange will actually check that the upgrade happened correctly and then afterwards, otherwise we're delicted token. So their in theory things could be nice, but yeah, I guess we we're bit further away from that. I think a part of us that I want to bring up that like a dealing with permission, a systems we can't...

...enforce at the technical level. Again, these things people could do whatever the hell they want, and what we're like another experiment in this in this entire thing's social consensus and what we choose as the right thing to do or safe thing to do at a social level and at more, maybe even appropriately levels, socially emergent consensus. What exchanges shoes to list and how that shoe celested is up to their opinion, and then how the community interacted. That thing kind of, at the end of the day, gives us an idea of what everyone is doing. Is the right thing to do. Kind of, but like that's part of the experiment these things, because their permissionless is that any individual or company can do whatever they want. It's up to the entire community to decide what's the right thing to do, and that's usually shown through where money flows and how it flows. So I'll mentioned that one of the best references for building a system for time locked up grades is the Gemini Dollar. It's one of the talks that was given our December empire hacking or. We have some video and slides and I would love it if more companies building upgrade systems reference that in their own design. You know this. We have the ongoing discussion about how much to share, how much to publish. The good thing about etheroreum is like you have this e ORC's right. Your C twenty. It's not an EP twenty. That is something which is a standard, no, which people agreed on by showing that it's actually the very good thing to do. And I'm actually in favor of having something like that for upgradeability, just by practice, just by people saying like look, this is a standard we follow, not because the EVM forces us to do but because the consensus for good code is like that. And the kind of though that everyone follows it, they add functionality to it, so on and so forth. That's more like a standardized minimal functionality in practice exactly. And then other other ones might might be more about practices. On how how do you have to handle your keys? Right, no one talks about this. It's of course. We sometimes hear from people who get hacked just in the very traditional way, and then you ask him, well, what did you do with your private keys? Well, I have this computer in their own computer. Right. This is security doesn't stop only on this level and what's good practice doesn't stop there, and I like that a lot of these things aren't enforced on the very lowest level of the EVM, but that sensible projects have to figure that out and are doing it and greatly improving over time. I am, of course, in favor of publishing all security findings to say hey, this is what we found. Trade of bits does that to regular block posts. We do it to publishing most of our audits and so that people see, look, this is what can happen, this is how to handle it. And what the scan showed is that a lot of people follow best practice. Is because best practice is around smart contract development did actually prevent this reentrancy to be attackable in many smart contracts and and we found some smart contracts of a vulnerable but so far and no major token, no major cross say and no major going to take wallet because the People Follow Good Practices and development, which which had to develop all the time and which we have to continuously share. As good at we as we are at publishing information, both chain security and trail a bits, it's really unfortunate that there's still so much hidden knowledge kind of locked up inside the audit reports that we both published. If you really wanted to get to be a expert in smart contract security, you'd go and dig through all of our past archive a published documents. There's some hidden gyms still in there that I don't think professed the community is caught onto yet. I would agree, and I think that's great way to kind of wrap up this episode. Is there anything that we should have asked y'all or gotten to that we didn't get a chance to? Starting with Jocelyn N, I think we did pretty good two of Ourso with sound. But if I my Shin in to Security Systema and I can think of Dan and also, please feel free to take this time to to shill. Whatever you do, it get people con come here. Yeah, so if there's two things that people do after listening to this talk, it's probably check out our blog, because we have a great set of videos for you to learn more about security in theorium and blockchain security. And then, too, is you should pip install slyther analyzer and securify. Really, I mean there's no down side to installing...

...more than one tool, but I think these things are out there and it's really important that people start to use the tools and knowledge that our companies to put out. It's only going to happen if people start, like taking some effort, and that's the easiest thing you can do this. Yeah, so luckily then, already did the shilling for us. So I can see much. So I think we, I mean we, I think we characterize the current situation quite well, but we also said we don't know the full story it. So I mean I think we should all. We all, even though we all agreed was the right decision to postpone, I think it might still be interesting to see eventually what's going to come out, because I think we still potentially can can find out more, can can find my contract and if anything this shows, as then said, that maybe we need some super powerful tool that really can can show us what the effect are on the whole rock chain and and what. Yeah, what what really changes when we change this? When we introduced the new EARP so I think they're there can be some exciting things coming out of this little mishap. And but yeah, the last thing I want to say is it. It was touched on before in terms of communication, where we're also I was quite impressed with the communication, but it was, as a super impressed with the quick release speeds of parity. Go with you and traffle all of these projects, that we're just super fast to release the new clients. I would definitely agree with that, having been involved with those channels and and Saul like people really, really really work and spend a lot of time trying to make sure that happened quickly. I was very impressed with the quickness of response and quality of it, and I think that about wraps it up. Audience listeners, if you enjoyed this, click the like button, share it with all your friends, tell your dog, etc. Will be you can find a SOT hashing it out dot stream or at the Bitcoin podcastcom. You can join the slack and join us in conversation to talk about what we could talk about next or give us ideas or even throw US money if you want to. Speaking of money, is you know, we are always open to sponsorships. We look for we're trying to make this a at least pay for itself. So feel you know, feel free to reach out to us. Petty at Hashing it out stream and Colin at hashing it out dot streams corey at passing it out that stream. Is it? I'm pretty sure it's petty. You're right. As Petty, it's cory for every other email that I have. I don't know why did that. Yeah, it's petty. It as it. And then you can also reach out to us on twitter at Colin Cuche, that Coln Seu see and Corey at Corey Core Petty Corp Etty, and we are also hashing it out pod on twitter as well. So thanks, guys, thanks for all the work you did to help us mitigate this problem and I look forward to continue using their quality tools. Y'All, make your call great. PODCAST.

In-Stream Audio Search

NEW

Search across all episodes within this podcast

Episodes (133)