Conf42 Rustlang 2022 - Online

Pushing Rust to the limit in a Blockchain Environment

Video size:

Abstract

Rust’s greatest strength is arguably its support for zero-overhead abstractions. This was crucial when creating a blazing fast smart contract framework on whose shoulders people entrust millions of dollars worth of assets. We almost broke Rust in the process. Almost.

Summary

  • Andre Marinika is from the Internet scale blockchain. This presentation is aimed more at rust enthusiasts and rust programmers. So, without further ado, let's get going.
  • The title of today is pushing Rust to the limit in a blockchain environment. This presentation will have three parts. The first part is a primer on what we're trying to achieve. The second is a collection of fun little tales about stuff we encountered. The third part is about solutions we came up with to fix problems.
  • Elrond is a super fast and cheap layer. One blockchain. We have a lot of very cool features. Something we pride ourselves very much in is the great Dapps, both mobile and on the web. We hope we're going to end up being on every person's cell phone or web browser in the world soon.
  • We are trying to be the fastest blockchain on the planet. We opted instead for Webassembly. Contract size for us is crucial, aka byteco size. You want to build a smart contract that is as small as possible. Here is a high level overview of how to build an efficient smart contract framework.
  • The second thing you want to do in a framework is make it pretty. High level code has to be pretty. The programmer has to look at the code and say, yeah, that's an addition. It's actually one of the most important things when you're designing a framework.
  • When designing frameworks, what abstractions do you choose? For instance, here we chose to make all the contracts traits. Now contracts can call each other as if nothing happened. And now another short story about the first arguably useful smart contract.
  • So if contracts can call each other so elegantly, why not call them from outside of the blockchain? Rust is magic enough that you can do that. And now I want to talk to you about some even more interesting stuff, some stories, some stuff that we just happened along the way. There's going to be three exhibits.
  • In rust, you just use a result. Results are super cool. There's a problem, though. We didn't want the result at all, actually. We wanted to have something that replaces the result completely. And we cut 20% of some of the smart contracts with this.
  • There's a lot more to be talked about, but of course I had to keep it within reasonable bounds. You can find me on telegram, you can email me. Just email me directly and I hope to see you soon. Have a nice day.

Transcript

This transcript was autogenerated. To make changes, submit a PR.
Hello, fellow Rastasians. My name is Andre Marinika and I'm from the Internet scale blockchain, and I'm very excited to be today here to have this presentation, which is a bit unusual for me. I haven't done anything like this ever before. This one is going to be something aimed more at rust enthusiasts and rust programmers, unlike the more usual presentations that we do for the blockchain crowd. So, without further ado, let's get going. The title of today is pushing Rust to the limit in a blockchain environment. And it's really a story. It's a story about how we came to create some of our products and how we discovered Rust, how we learned about rusts, and how we eventually pushed it to its limits and we almost broke it. Not quite. Rust is a resilient riddle crab. First off, who I am, my name is Andrei Marinica. As I said, I've been an engineer at Elrond for almost four years and a proud rustician for about three. I've always, always been very passionate about programming language frameworks, language models, program models and all that. And I'm probably going to keep doing this for the rest of my life, for reasons you will see later. My favorite emoji is the axe, but also, of course, our little friend there, the crab. This presentation will have three parts. The first part, I have to give you a primer on what we're doing. What is it actually that we're trying to achieve? Because otherwise the rest don't make much sense. I'm going to try to keep it short, though. Now, the second one is the juicy bit. It's a bit of like a manual. Have you ever wondered how it is to write a framework? Spoiler alert. I do not know the exact number of steps involved, but it's many. And again, I'm not going to present everything that we ever did, but I'm going to tell you about the fun bits. I'm going to try to keep it fun. And then the third part is a collection of fun little tales about stuff we encountered and crazy solutions we came up with to fix problems which none of them would have been possible without the generous support of the rust language. So let's keep going. A short crash course in Elrond architecture. So, first off, what is Elrond? Maybe you've heard of it? Maybe not. Elrond is a super fast and cheap layer. One blockchain. And if you ask me, the coolest out there. Don't let anybody else fool you. We have a lot of very cool features. Sharding is one of the big ones. We have really fast smart contract, and something we pride ourselves very much in is the great Dapps, both mobile and on the web. And we hope we're going to end up being on every person's cell phone or web browser in the world soon. So stay tuned for that. We have our own coin, our own token called egold, with very cool tokenomics and a growing ecosystem which we are very, very grateful for and trying to cherish and grow. And I could be going on about all of these subjects on and on forever, but I really want to focus on one thing and one thing only today. And it is the smart contracts. You see, we are trying to be the fastest blockchain on the planet. Yeah, I'm not sure if there yet, but we're trying to get there. And the thing is, with the fastest blockchain is that there are nothing without the fastest vms, which in turn are nothing without the fastest smart contract. Right? Because no matter how fast you can churn out transactions, if the smart contract is a bottleneck, I mean, that's where the interesting parts are. And so nowadays everybody is trying to go for this or that copy of solidity and of EVM. And from the get go, we felt it wasn't really quite the thing for us, so we opted instead for Webassembly. And I must tell you, I don't know how many of you have worked with Webassembly. I love webassembly really, really much. It's so clean, it's so portable, it's so nice and sandboxed. I feel it's the perfect match for the blockchain and was an underlying technology. We chose Wasmer, which is awesome technology. It's a webassembly executor, which we've built our products around. But we're not talking about wasmer today. We're talking about how we integrate contracts into Wasmer and how we run them. And this is a helpful chart. Maybe it's a bit much at first glance, but really it's nothing very complicated going on right here. So we have a blockchain or a node. Well, it's a machine somewhere in your cloud most of the time. We have a vm on top, which we lovingly named Arwen. And then on top we run the Wasmar engine, which takes the smart contracts, compiles them into some machine code, and then runs them and does all those kind of magic tricks. Simply put, just think of these three levels. So there's a programmer writing a smart contract in a high level language, obviously, and then they're going to compile it down to something like to webassembly. The webassembly gets saved on the blockchain. It's portable, it's super clean. And then whenever you need to run that smart contract code, it will get just in time compiled. It gets cached into machine code and run. Right. So why am I talking about all of this? It's because of our objectives. Contract size for us is crucial, aka byteco size. You want to build a smart contract that is as small as possible. We're talking kilobytes. We even have smart contract that are smaller than one kilobytes, that are like 800, 900 bytes, which is kind of crazy. And this is very important for two reasons. One, the just in time compilation is actually not so cheap. I mean, it's not so expensive either. But if you're trying to run thousands of contracts per second, even with caching, it can get in the way. And also, blockchain storage is expensive because you have to replicate this for everybody that's in the network. And so we want these things, and we also want speed. We want a bit fast because we want to again run thousands of them per second. And in the process, we also don't want to burden the smart contract developers with all the low level details. And to be honest, they cannot be trusted. I cannot be trusted. I do not want to know how to optimize a smart contract. I need a framework that does it for me. And looking around, we realized it's really only one language that can cut it, and we all know what that language is because otherwise we wouldn't be at this conference. So without further ado, I'm going to present you a very, very high level overview of how to build a smart contract framework. Obviously, in rust, which satisfies all this criteria, I'm going to give it a bridge because it's been three years of work. So first things first, you got to make it fast. As I said, you want your smart contracts to be as small as possible, as smart as possible. This means smart contracts cannot be bothered with a lot of the stuff your regular programs are bothered. For instance, big numbers are the bread and butter on the blockchain. You can't be bothered with defining addition and stuff like that. You also need to use all kinds of crypto functions. You don't want to implement any of your code in every smart contract. And oh boy, don't get me started about memory allocation. Memory allocation is great. But when you're running on such a tight schedule, you want everything out. So there's this trick we've been using, you see, as in very helpfully displayed in the table to the right. Smart contracts are supposed to be the brain. They're supposed to be the what, what we are doing. And then all the muscle stuff, it shouldn't be in a smart contract. So we put it in the vm, and our vm is doing actually a lot for the smart contract. Not crazy smart stuff, mind you, but crazy powerful stuff. Like it's doing all the big number arithmetic, addition abstractions, powers all the functions, everything. And there's a helpful example here. So this is a function that's very simple function. You will see in a moment. This is extremely, extremely sanitized, the webassembly. So don't be scared about it. It's pretty much assembly, but check out what we're doing here. So we have a function which is the smarts. It's the smart contract doing the smarts. And then we just want to get some arguments and then add them together and return them back to the user. That's it. And we don't want the contract to do that. There's no logic for that. And all of those functions like add two big ins, that's actually in the VM. The VM actually does it. And the way it communicates with a smart contract is it gives it some handles. It's like a pointer, but it's not a pointer, just an integer that says, hey, I'm going to add big int number five with bigging number seven. Can you do the math for me? And the beautiful thing is, the smart contract never even sees these numbers. The smart contract is like the recipe. It's not the food, it's just the recipe. It just tells you what you do. There's no food there, there's nothing to eat. The VM does everything. So that's first thing that we have to do. We thought, how many things fast and small? And that's probably the smallest thing you can do because you can do all these huge operations in just a few lines of assembly. But obviously nobody wants to write code like red. I definitely don't want to do. I guess you don't either. So, tada. The second thing you want to do in a framework is make it pretty. And when I say pretty, I'm not only talking about aesthetics. I mean, aesthetics are nice. Everybody wants to look at some piece of code and say nice. But that's not really the point of it. The point of it is these little smart contracts, they're not your average programs that do average program things. There's millions of dollars flowing through them. So you want to make sure you don't have bugs. You really want to make sure you don't have bugs. And the only way to do that is to write some really, really high level code. And high level code has to be pretty, has to be readable. The programmer has to look at the code and say, yeah, that's an addition. It shoulders be bothered with how the addition was performed. So it's actually one of the most important things when you're designing a framework. This should have probably been the first, but the segu wouldn't have flowed so well anyway. And the point is, what makes rust so great is that rust makes this possible. All this crazy stuff with numbers that are not even in memory, that are not even regular things. And still you can override operators and you can have these two beautiful big and objects, and you can add them and get a result, and you don't care what happens behind. And as we'll see later, you don't even know what happens behind. But it's beautiful. It's pretty. So next thing on list, a smart contract. How it looks like this is one of the really simple ones. And I just wanted to post this here to show how pretty it is. Not only that, but I also wanted to talk about another thing. When designing frameworks, what abstractions do you choose? For instance, here we chose to make all the contracts traits. They could have been modules, they could have been just random functions thrown in, but we chose traits for several reasons that we're going to see in a moment. And one of them is you make a framework, you want your stuff to be testable, right? You write a smart contract, you want to test it. So first thing, you write a smart contract, you ship it to Webassembly, you convert it into a smart contract, and you run it on the blockchain. But it's hard to test. It's a complex scenario. You want to write unit tests, you want to write integration tests. So what you do, you make this straight. You change its implementation completely. The implementation is auto generated. The developer doesn't care really, but you change its implementation and now it's a library and you can run it and you can run tests on it. Yeah, we got it. Now the next thing, you want to make it interoperable. Interoperability is not easy. See, these smart contracts, they don't live in a vacuum. They're not everyone alone in their tiny little rooms. They have to talk to each other because otherwise where's the fun? And thing is, they're not regular programs, they're not even regular functions that you can call. So calling them is a bit weird. You have to pass through all these gates, you have to put them on a blockchain, you have to make a transaction from another transaction and stuff like that. You have to format transaction strings, you have to interpret results. No programmer wants to do this. Programmers just want, I want to call this function, I want to call this endpoint as I would be calling a normal function. So a good framework would do that for you. You want to call a contract from another contract. It should be like, oh yeah, sure, this is contract, this is an object, just call method, add on it. And you can see that add with bun of five. And of course in between the call and the receiver, it's going to be a lot of magic. But how do you do that? Magic? Well, we're in rusts, we can do anything, don't we? So there's a thing called a proxy you want to call something. So it's like a facade, a proxy, what will you. So for this contract, for instance, for the add method, you can have another trait that is an interface to your trait. So yeah, it's traits all the way, I guess. And then you can call this add function in the trait and have some ugly auto generated code to do all the magic for you. Right? Cool solution done. Well, not quite done because we're in rusts. We can go even further than that. I mean, look at that other proxy, it's beautiful. But we had to write by hands and we're lazy, aren't we? So we're going to auto generate the proxy too. And now we have an auto generated proxy with an auto generated implementation. That was a lot of magic behind it. And now contracts can call each other as if nothing happened, as if they're not even on a blockchain, almost. And now another short story about a thing that happened. So we were writing actually the first arguably useful smart contract. And you know that pesky time when you just started programming and you suddenly realize your source code is 2000 lines long and it's all in one file and you start hating your life? Yeah, well, we've been there. So in order to fix this problem, we had to find a way to cut smart contracts into pieces. And we call them modules. So you have a smart contract, you have some module that was some code and you just import it and it's as if they become one. And it seems easy, you make another trade. We have super traits in rust, which is really handy. They work a bit like inheritance, so that's really cool. And it all seems easy and dandy, but then you realize, oh no, I auto generated a bunch of code. And now I don't only have to compose my main trait that I wrote, I also have to compose all those auto implemented things that I wrote. And I didn't even talk about half of them, but just the proxy, think of it, it's like auto generated thing calling another auto generated thing, and they have to know about each other because they have to be accessible to one another. So a big, big bunch of code, but you toil away, you work on it and you manage, you just manage at some point to write all this auto generated code. But guess what? There's another problem, and I guess you won't be thinking about it, but in the process you've been having all these modules put together and you've been generating all these kinds of c type endpoints that end up being endpoints in webassembly. And you realize that actually these endpoints are not generated based on your super duper inheritance. They're always generated. If they're in a crate, they're always generated. And we call them stray endpoints. And they're a pain. They're a real pain because imagine you have this beautiful library with 20 modules and you want only one of them, and guess what? You got entire family with you, and surprise, you're on the blockchain and somebody can call a meta that you didn't even know existed in your contract, which sucks. So another story about how we went about it is we had to go like really meta, I mean really meta. So brace yourselves, there's going to be pictures coming soon. So the first attempt is you want to create this thing called an API, and Abi is just a description of your public functions. That's it, nothing fancy. But you can't just generate it because you get modules and you don't have access to those modules. So you have no idea what those modules contain. So zero, it's failure, doesn't work. So what you actually want to do, instead of creating the ABI in macros, you generate a generator of an abbey in the macros, and then the abis call each other, and then you have a metacrate that puts everything together, and then you generate some more code that will generate then your webassembly, and then you take a deep breath and wait for the next slide, which is the picture. So I hope this makes it a bit clearer. So we have a contract with a module. The contract gets an ABI generator generated, which calls the generated module generator and so forth. And so you get a nice ABI, and then you have a nice webassembly smart contract. And what if you have this sort of module that is astray lying around, is not included anywhere? Well, because it is not included in the API generator, it will also not be included in the webassembly in the end. Tada. All you needed to do was tons of metacode and metaprogramming. But hey, that's okay, because we're in rust, we can do everything we want. And now the final, final thing, we had these beautiful smart contracts living in the blockchain, like in a cage. We can make them escape, right? We have so much magic, we know how they call each other. So if contracts can call each other so elegantly, why not call them from outside of the blockchain? Rusts as elegantly, why not use these smart contract, like, I don't know, some web weird services and call them directly from the outside? And we can do just that because we just made a bunch of proxies, a bunch of facades, a bunch of magic code. So you can go almost like in a database, like, oh, just query this function. Oh, make a transaction to this function. And as you can see, all you got there is like a function call, like call, sum, call, add. That's it. And rust is magic enough that you can do that. So to recapitulate, all these 300 plus easy steps in three years can be thought of as make it fast, make it pretty, make it testable, make it interoperable, make it composable, and finally make it escape. And not necessarily in that order. And they're not in one particular order or another. Anyway, so there we go. We have a framework. We have a rust framework. We can write smart contracts. We can test them. And now I want to talk to you about some even more interesting stuff, some stories, some stuff that we just happened along the way. And there's going to be three exhibits. There could be more, but three exhibits, it is of fun problems, fun solutions, and I hope they inspire you. And if not, at least that you have a little bit of fun. So exhibit a, how a sane person would write a decentralizer trait. So this is like really simple stuff. You want to decode stuff, you want to handle errors. And everybody knows in rust, you just use a result. Results are super cool. You know what error you got super elegant. There's a problem, though. We started investigating the bytecode because we were crazy, crazy, crazy about performance. And what do you know, the actually innocent result turned out to be producing a lot of jumps, a lot of ifs, because every time you get out of a method, you have to check for errors, and if there's an error, you want to jump out and stuff like that. And the thing is, we didn't really care about it, because you're on a blockchain, if something goes wrong, you wanted to go wrong fast, you wanted to crash ASAP. So we didn't care about these results, but still, we didn't want to write some horrible piece of code. So what did we come up with? This thing, whatever this is. So we have an error handler. An error handler is static, is generic, and there's a thing called an handled error. And the thing is, we didn't want the result at all, actually. We wanted to have something that replaces the result completely. And, oh, boy, did we try some ugly solutions before we came up with this. The trick is the rust compiler does a very, very smart thing. So if you convince it that your error type is a thing called the Never type, which, by the way, is a very cool thing, it's written as an exclamation mark, and it's a type that can never be called that nobody can call. It's a sign of code that is unreachable. And rust knows this. And if you pop it in with your generics, the never type rust will know that the result that has a never error is basically not a result. It's always success. So it will just replace it with the actual result you have. And we cut, I think, like 20% of some of the smart contracts with this. So that's the first story we got for you. In case you were wondering how to get rid of result, that's the way. Now, exhibit B is what we would call Varag madness, or monomorphization madness, you call it. So, the thing is, you might have noticed in one of the previous slides, we have this really nice contract, and the contract just takes some arguments, and the framework does everything for you, just gets them off the blockchain, it prepares them for you and all that. But you want VAR arcs, you want variable number of arguments. And to be honest, the first real smart contract that we ever wrote relied heavily on them. So it was a problem from the get go. And the thing is, again, we're obsessed, obsessed about performance. So if there are no varags, you just want to hard code arc number zero, arc number one, and so forth. If there are varags, you want to have a loop going through them, right? Right. So who should decide which one is which? Well, the macros can do it because we don't want to pollute with annotations and stuff like that. So the compiler should know, right? It should know. And it can know based on type, because we know which types are varags and which types are not. But how do you do it? Because think of it, you have to decide whether you have varugs or not on the first argument. But the first argument might not be relevant. The last one is probably going to be the varg. So you have to kind of make the compiler peek into the future and now start, just think for it for a moment how you would do it. You would probably start try to do something with, I don't know, tuples and stuff. But then how do you peek into the future? How do you make sure everything is at compile time? Okay, spoiler, whatever this is, you can do sort of like a functional style fold. And just look at the magic there. There's let value of value of value in a tuple, in a tuple, in a tuple, in a tuple. Turns out you can nest tuples forever. Turns out you can make lists of tuples that go on forever, right. Just like a list in a functional programming. And then you can have them as generics, and then the outer tuple can look at the inner tuple and so forth, and can actually interrogate at compile time if it's a varac down the line or not, and can decide what to do about it with not. And this pattern, by the way, works in this beautiful let that rusts very helpfully decomposes for us in this beautiful explicit type definition, and also as an argument, because we need an argument for the error reporting. So there goes infinite tuples forever. Rusts can do that. Exhibit C. Now this is a bit of a weirdo, but I will have to let you bear with it for a moment. We all know that the bread and butter of rust is ownership. If you don't know what ownership is. And remember again, in one of the previous slides we were talking about these managed types, which they are orchestrated by the smart contract, but they're actually in the vm, they're not there. And these handles, these types, they actually act like pointers of something that rusts, isn't there. And think of this big int. This big int is just not there. But we use it as if it's there and the variable X has ownership over it, even though there's no memory there, and then you can take references to it, and then in the bytecode, you'll have a reference to an in 32, and then you can clone them and whatnot. But now think about what happens in this scenario. You have a vector that is also managed somewhere in the vm that contains begins, and you want a reference to it. You want a reference to whatever random element, element I, and you get an item out, and that is a reference to a position in a vector that is not there. What type do you make it? What kind of a reference is that in rusts that is not there. And you also want to dereference it and actually use it as a regular reference. And how do you do that? Because there's nothing there to reference. And the solution is, well, of course we had to create our own type because you can't just overwrite references in rust. It would be fun, but probably horribly unsafe. So we did write our type, which is manage rusts, but it also has a lifetime. But it's not really the point. How do you work with it? And it turns out, drumroll. It's transmutation. So apparently you can take this type and you can create a sort of weird zombie reference out of it. By transmutation. You can trick rust into thinking there's something there where there's nothing there, so you can work with reference to nothing. And I think that's the coolest thing in rust, and I think it's also the closest we got to actually breaking it and actually getting to the limits of what rust can and cannot do. And luckily, we did not quite break it, and we're still using it, and it's great. So I hope you enjoyed the ride. There's a lot more to be talked about, but of course I had to keep it within reasonable bounds. I'm probably going to try to make a blog post out of it that's a bit more detailed. So that's actually the first time this kind of stories surfaced into the world. I hope you enjoyed it. You can find me on telegram, you can email me. We have some telegram channels that we use for all kinds of questions, but especially for the crazier questions. Just email me directly and I hope to see you soon. It was a pleasure being here with you and, and onwards to the questions and the discussion. Have a nice day.
...

Andrei Marinica

Core Team Software Developer @ Elrond

Andrei Marinica's LinkedIn account Andrei Marinica's twitter account



Awesome tech events for

Priority access to all content

Video hallway track

Community chat

Exclusive promotions and giveaways