# Stateless Ethereum call transcript: Sync call #0 **April 21, 2020** 00:00 Piper: Me from Trinity Tim, Rai another consensus folk that I assume are representing Pegasus Alexi from turbo geth, Martin from geth and... do we have have anybody from nethermind? Doesn’t sound like it. okay, um who does not have a good idea or is would anybody here like me to go over the kind of general high-level concept. 00:34 I'm gonna just not even ask that question and then go for it anyways just to make sure that we're all in the same page The general goal of this in my like my goal is to take a hard look at sync what all the clients are doing, try to carry on kind of some of the conversation that was begun while we were in Paris. 00:57 And with the express goal of either figuring out if there's a version of sync that works well for all of us that we could actually move towards together as a group, or what the other option is assuming that we can't satisfy that condition. The goals for that actual sync mechanism are to come up with something that does not depend on essentially that breaks the current model on the current model being that. 01:32 We are heavily dependent on sort of a super majority of full nodes existing on the network in order for syncing to function. So like sync as it exists today depends on leaning heavily on the already fully synced nodes to provide data. In order for nodes that are newly syncing to sync. So that means something akin to bit torrent style-ish swarming behavior, um, so that we can have a minority of full notes and still be able to sync reliably. 02:16 And then there's another goal that's sort of buried in there, which is to change the sync protocol so that it resists being abused to implement stateless clients. That's sort of a protection mechanism that I think the network needs. We've shown that we can use GetNodeData to do that to abuse the sync protocol to build a stateless client using GetNodeData and I don't know enough to say definitively but my guess is that you can probably do a similar thing with. 02:59 I suspect that similar things could be done with snap specifically because it doesn't actually get rid of get node data. I think snap if I'm correct still depends on using get node data to patch up the tree at the end. Yeah and even still the fact that you can fetch ranges of accounts probably still lets you do some of the just dynamic state retrieval on demand. 03:34 And since the stateless ethereum client road map. Has removed the ability to fetch, you know, data live over the network on demand as like a explicitly supported feature for like the first rollout, It means that if we don't kind of cripple the ability to do stateless clients over the state sinc protocol, then we're still likely to -- essentially we're going to still be leaning on the gentleman's agreement to not abuse the state sinc protocol to build a stateless client. And well that might be viable to be honest like we all kind of know each other and have some level of social reputation and I think that if somebody's client was crushing the network because it was doing that that like, Maybe we'd be able to talk it through and get it worked out um, but I don't think any of us like the idea of having the health of our network protocol be dependent on an agreement to write clients responsibly. 04:46 Alexey: So well there's there's another facet to it is that so my suspicion is that one of the reasons why the LES for example does not take on with the incentivization. I mean, they might be some technical issues but I also see that why would anybody use LES if you can use get node data and get everything the other way? And you like when you roll out the instance of LES like the question is that is anybody going to use it because you can get exactly the same functionality to get node data. And that is a shame because if LES you incentivize LES would be the very important that interesting kind of experiment towards the more sustainable network where there's more kind of money circulating not through the not just to the miners but with another operators and that's why I see this as the this kind of it's LES, it's like a light ethereum, whatever something yeah, it's like it's like a the protocol which have supports life coins, that's what I mean, just there was a the question so essentially what I see here is that. 05:56 We would not want to yeah to keep providing get node data as the requirement for the stateless clients, so we need to think about something else, which. A yeah that's my point so incentivization would be important Piper: And the other big thing here is is Martin you can maybe speak to this from the get side Alexey I know that you've said that get node data is just really hard to implement with the flat database layout and so the other piece here is that getting rid of get no data makes the transition to like kind of a global adoption of the flat database layout a much simpler proposition, does that that sound accurate? 06:49 Martin: I if the only estimated solves underlying problems and with the flat database there are some additional problems. Yeah, I'm just not on board with the whole abandoning get node data… Piper: Thats it's fair. Is some of that just skepticism on that we like can actually do it or not find that it's necessary that the benefits outweigh the cost? 07:31 Martin: Well it's made it because when we implemented flat… -- something we haven't really considered but we discovered along the way is that when you have data corruption It might take you. I mean regularly if you have data corruption in your tie you get that block, you notice immediately that there's consensus store. If you get the data corruption in your flat database, you won't notice it until maybe a thousand blocks later that maybe my data doesn't match the state root. if I try to sum my data up it doesn't match. So somewhere in the last thousand blocks or whatever it was corrupted and it’s a pain in the assto try to to I mean, it's a pain in the ass to as today, to dig into the reason a consensus error occured. But about any of you also to narrow down like the last a thousand blocks where and at any point that could have happened. And that's yeah when I realized that I got a bit scared scared about [removing get node data]. Piper: And get node data helps just because it lets you patch these things up is that the idea? all right Martin: Well it could do that so that's I that was not actually what I meant, although yes, it's pretty nice if you have if you can patch up your database if it's corrupted but I meant was that if your try where your you can most more easily reconcile the state from your flat database with the state in your trie, and if you have the trie you can deliver get node data. 09:12 Piper: Got it. Any other comments on that specifically it's not exactly a subject that I know enough about because I have an implemented flat layout and so I just don't have to the personal basically the Alexey: we did try to we did something to implement to get node data in some form, but it turned out to be quite resource intensive. 09:37 I mean, we have it enabled by default now, but so even if you have a trie which is what we do in like in memory and we are trying to create which we created some kind of cache which will respond to some of the get node data. Basically by inflating our memory because we don't really need this. 09:56 We just use it to provide others with the data which becomes quite expensive, but also we do not actually have the get-node data for even like for some of the “blocks minus one” or “block minus two” because they're simply not in memory, so that's because get node data one of the issues that it doesn't tell you where to look for the data, it just tells you oh here's the hash go look everywhere. 10:20 Is it history? Is it current state? I don't know where you where you need to look for it. Just somewhere. If at least, It told me like okay this needs to be in a block of X and this is where you know, that would be much better. So at the moment, I don't know where what to look. 10:34 Martin Yeah, yeah, but this is the same with geth. and we all have the most recent 128 tries. older than that we start proving. So there's like yeah, that's all really. Piper: So I'm gonna kind of set this one aside for for a little bit here the takeaway that I have right now for get node data is the maybe there is still a good use for it, but maybe the use for it we can supplement with a with a different primitive that does a similar thing but isn't just the blind hash retrieval which is I think something that's been talked about before, you know, changing the parameters so the rather than requesting by hash you request by like, State root and tree path or something like that, but. 11:31 if it's deemed that we still need some mechanism like this maybe we can look at doing it a different way at least that is Alexey: yeah and this is why this is one of the reasons I like the something like LES much more because in all the LES requests it does actually tell you where to look for data. 11:47 It tells you the block actually tells you like this is the account that I want rather than just seeing give you the unit hash. So that's why I prefer the LES approach if you were to to you know, you can use that to patch your database and this is where I was advocating for let's say that people want. 12:06 To build the is whatever beam sync or stateless miner, okay, you might guess which do it by LES because you're if you want to run stateless miner sure spend a bit of money pay the providers for the immediate access to LES and you should be alright the same thing for beam sync if you want a you know, your your note fire up straight away just pay a little bit of money and you will get that then you don't need to go into the sync network which will merry go round if you want everything just right now. 12:38 So that's why basically I'm thinking that immediacy in this case is what people paying for not for like availability for information, but for getting it right now rather than waiting like a few hours. 12:54 Piper: So as far as Mary go around goes, which is the generic term like I'm still intending to mean “merry-go-round” to mean of like an utterly generic term for an idea for saying, um. I have like taken a stab at writing up a version of that, um, Alexey it's my understanding the you have kind of like a fundamentally different idea that you want to look at I know that you said that you're not maybe you're not ready to like present it but I'm not ready. 13:31 Alexey: I'm ready not ready to present it but we made some steps towards it towards so we started to prepare some fundamentals yes, Piper: okay, can we talk about it like, can you give us? Alexey: So okay so basically I already started this right up maybe a few days ago to try to explain to the to the my team members about what I actually mean by by this version of the Mary go around and it does differ from your proposed version in the the main difference is that I'm proposing to sync the state always in the order in the specific order rather than using some sort of randomization. 14:12 So essentially starting from the from the very low addresses or address hashes ending with the very high address hashes. And basically even though our combined state trie with all the kind of storage nodes are is very uneven, we still trying to slice it up by the by the weight essentially. So we're slicing the things by the way, we give it we are given the essentially the duration of the cycle: one cycle is like when you've synced entire state And the duration of the cycle is split into ticks what I call ticks, for example, it could be like one block interval or something like that and then you're figure out what is the total weight of the the total of weight of that of the state and it that includes not just the not just the account and balances and stuff like that, but also storage and also code byte code as well, so you have to calculate the weight we're using the like you've taken into account all this kind of stuff so then. 15:19 You split up the the state in the chunks of or the relatively equal, like similar weight. And then you generate the schedules. So all the the seeders who have the state, they are they should have information available some sort of weight index which will allow them to very quickly actually generate the schedule because they can walk the weight index which is basically overlaid upon the the weight index is overlaid upon the state and so it allows the seeder to to split the state into the similar weight to chunks, and generate the schedules the sync state or ahead of time. And that sync schedule is essentially the sequence of a intervals for each tick. Let's say that if we're gonna have like 4,000 ticks or 5,000 ticks that means that we generate a list of 5,000 items and each item essentially is like “what is the the the prefix or the hash of the address at the beginning of the day of this, you know of this slice and then well, what is the end beginning the end?” 16:29 So they pre generate this and it is stored somewhere -- it's not gonna be super, you know, it could probably be generated quite fast we haven’t measured it yet -- it could also be stored quite efficiently and then basically as the the the sink the the sync progresses all the seeders use this schedule to deliver to to distribute the slices, so they know exactly from which the point to which point to distribute and what happens then is that because the state also changes during the the, you know, during the cycle, 17:04 They will be delivering what the current they won't be delivering like they won't be trying to sort of sort of freeze the state in this in a past because it's quite an efficient they will be delivering whatever they have at the moment within those boundaries of each slice plus they will also be patching delivering the data to patch up the old thing which we can get from the block witness and that's the current thinking. so the we have started to prepare for this to to get some quantitative data and one thing we did is we first of all we have merged our account bucket and storage bucket into one bucket so that they basically now go in exactly the order that we want them to be in this weighted, you know state. We also completely now pretty much completely separated all the the merkle hash information out of the bucket. we even removed the account state storage routes into that separate bucket, which will also allow us to actually support hexary binary trees at same time. Because we basically remove all the relevant information from the our flats bite so essentially all the merkle will be separate. And the next thing we will do is that we will try to slice up the state, like maybe the current state or some really recent state, slice it up into certain number of pieces, append some of the proof parts which are necessary to prove it because the ideas that you have a slice and you also have a proof of the slice so when you deliver it, it's actually proven to be a rooted in certain group and we can see like what is the overhead of the proof of the slice like how big this slices are like, you know, we start getting some feeling for it so so so far we've just been preparing the the data structures for that. So that's basically it now. 19:06 Piper: Or I have a question or two that may be premature but I'll yeah sure yeah it'll ask away um, so the reason that I wrote up my previous one was because it was the only way that I could figure out how to start like understanding the hidden problems or like some of the things and one of the ones that I found that seems problematic is kind of the coordination problem, so so it's sort of like saying when does the sync -- I'm gonna call it an epic the you know, one run through the schedule the like, Some of all of the ticks and the schedule one sync epic on right around the merry-go-round. 19:48 How do you like do you have any ideas on like how the clients figure out when to start one of those and like, you know, one client thinks that it's four thousand ticks and one things that it's three thousand nine hundred ninety nine Alexey: yes, so there is a two ideas at the moment. one probably which is going to be used for initially so when we just start up these networks and maybe the second idea will be used later when we we have more established protocols, So the first idea is we already it was proposed by Vitalik. essentially to have multiple merry go rounds with the multiple with a few different settings, for example, if we say that first merry go round will be four thousand blocks cycle another one with five thousand blocks and so forth if we start multiple of them and in this case if you if you fix the the cycle that means that it's very easy to coordinate. because you just have the module or you know, you have a block number which is module all of five thousand then you get the beginning of the cycle. okay, and so what we do before we have any sort of formalized coordination. We just basically by some experimentation we figure out a bunch of numbers that will be selected let's say for very slow clients we're going to use like 10,000 or 20,000 for fast ones we're going to use five thousand, and then people will basically gravitate towards these few of these networks. And then they will be coordinating off chain like which is the hot hottest one at the moment like which one should I go to oh yeah, you should go to that five thousand one and then once we have established maybe what kind of constitutes the the good choice we might actually start.Putting this coordination into the headers and miners will basically help us to coordinate. so if so then we can include this number somehow include this number into protocol and make the miners to rubber stamp so okay now the the, you know, the the cycle length should be five thousand. okay after certain growth of the state it's with too large like hey, let's say six thousand. So I don't know how we're going to do it, but I suggest we postpone this until we have played around with the multiple merry go round verion. 22:12 Piper: Ok so to say that back: to make sure that I understand yeah the idea is that we've got a number of different parameters that affect what the cycling length would be so: client bandwidth, client processing power, state size, right and like all -- the location the location of the person where are they in a fast network or they slow network or something like that yeah, -- so we've got kind of like hardware, network, and then the actual state size and all of those things affect and move like what's a reasonable schedule and we reasonable cycle side here right and so the idea is that like if we set these like kind of like the cycle size may only be one of these magic numbers, right and you're like yeah, you know, maybe like a logarithmic scale or just an evenly spaced set of things or whatever we figure out. The idea is that like the servers can advertise across some set of like cycle links and say I'm willing to serve these and then yeah, Alexey: so yeah, there will be populated certain length and I'm sure for example if we you know, if this these kind of additional structures for the schedules will be super small, then I can to totally see like our client servant 10 different networks. right different cycles, you know what why not? Piper: sure sure because in theory you could chunk the bigger cycles into the smaller cycles or yeah, Alexey: so yeah, you can do that well but yeah, so that's why I think it's it's a interesting idea to start with so that it doesn't actually pay into the corners or anything like that, so get the quick. 23:58 Flexible yeah. 24:03 Piper: Anybody else want to dig in there ask questions clarify? Rai: yeah I had a question about like in general which types of the merry go round are we drilling into? So have we have we given up on any kind of like dynamically deciding like how fast to sweep through the the trie or like finding either finding a schedule and advance like kind of what you were Alexey: so it's I think it's kind of the more meta question is that we have discussed a little bit in Paris is that my proposal is not to decide anything too soon. like where are we going to because what we want to do? I think at the moment is to pursue multiple variants of it, we don't want to force everybody to into the same direction yet because nobody has implemented nobody has basically figured out the whole thing yet so once we got some of the proposals which are basically figured out in full, pretty much there, they're pretty much working where they conceptually working then. 25:08 I think we can we can start having this conversation about making decisions so So far see this what I used to call it hackathon. is that you know, we just bring up the ideas will discuss them if we implement them that's the important bit as well, so. If you if you want to choose certain like whatever you like you what what what you basically think is reasonable because we all smart people so we have our bets and okay. 25:35 I bet this is gonna work but I bet so if you bet on something just do that and if you're right, you look better than the right horse right Rai: okay, let me rephrase my question and -- that that all makes sense -- but let me refuse my question to is anyone looking into a scheme in the family of merry-go-round algorithms that has us sweeping through the address space kind of dynamically instead of like, you know a family of algorithms. 26:01 Where you decide on a fixed epoch size or you calculate the schedule in advance? Alexey: Well, I actually looked I thought about whether it's possible kind of mathematically and I sort of without proof I gave up I figured out it's not possible so if you figure out how to do that dynamically, you know, please tell let me know but so far. 26:23 I don't know how to do it, Piper: okay, so to clarify Rai, what you're asking is are we giving up on the idea Of the sync algorithm essentially doing a clean one-touch. Like beginning to end pass through, the state? is that is what you mean? Rai: So not that not that you're guaranteed that you'll do it in one pass but that you don't know the rate at which you're going you don't own advance the rate at which you're gonna be coming through the trie you're determining that based on what prefixes people are advertising to you and like setting and using that to set the rate Piper: okay, um, I think that the method that so my alternative which we maybe need to come up with names for these to reference them... but the one that I wrote up that you asked questions about. I think it's satisfies that property of you not really knowing what rate you're going to make it through you just sort of get on and then you start to discover how fast you can pull and how fast the data becomes Rai: that's because of the flexibility thing you were talking about? Piper: flexibility and essentially randomization so like Alexey’s approach is ordered and you get some upfront knowledge that you. 27:54 You end up depending on the full notes to give you that knowledge what's the schedule and then you you know, get on the merry-go-round and. Alexey: Well actually no no no the the the leechers don't need to know the schedule they just hop on and get the data right they start not receive this schedule Piper: yeah okay so but they could I mean assuming the schedules being published like you could look at it right? like a literature could get on and say A: yeah, I mean, nobody prevents the theaters to publish this schedule, but it's not normative. 28:27 John: So on the Pipers version, there's there's some kind of coordination that's happening so you know what you're getting and then on it's pretty much what it's ordered. Piper: I think that Alexey’s version has a front coordination at like a sort of global level and my version has like, no up front coordination, and minimal coordination at the client level so clients do a little bit of coordination with each other. 29:00 John: Like the protocol has built-in protocol coordination for Alexeys version and then your version has clients sort of do a little bit of talking and then they know what to do like they're like what the clients themselves are coordinating Piper: Quick show of hands. Has in it who like has not read or isn't familiar with the version of this that I wrote up. 29:27 I think it would be good if you just go over it because yep all right, Piper: um, so alright so we've got this one version that kind of sweeps this state beginning to end on a on an upfront schedule, so I came at the problem from a different direction and said, For like we'll define an epic during which like a section of the tree is considered hot. We can use something like the block hash to determine with where that section is and so at for any epic there'd be some you know slice of the tree where it's considered hot and the width of that slice is not really even defined at the protocol level so what happens is at the beginning of each epic or around the beginning of each epic, 30:16 Clients would advertise to each of their clients who are like connected to them. I am willing to serve you the state inside of this prefix inside of this slice and so that is a way for the servers to essentially publish what information they're making available And on the and the intent there is for people who have part of the state to not be bombarded with requests for state that they don't have so if I have some of what's inside the hotspot. 30:51 I can broadcast just the parts that I have. Then clients to start asking the servers for the information that they need from inside of them regular state Across a number of these epics which would be relatively short. I think 16, 32 blocks something like that, that hot spot moves around on the tree and over some number of them you end up covering the entire view of the tree. This removes any need for coordination and it allows clients to pull the data at the rate that they're capable of pulling it and it also still allows clients to serve other clients. So to partially sync notes can still serve the other sync nodes and it. They would just only advertise the data that they actually have. And the idea here is that there's no firm restrictions on the width of those slices. 31:47 So for a tree that's very small, somebody serving data can broadcast a very wide range like a very broad prefix that they're willing to serve because they're just isn't as much data under it. Or if the tree is very deep and heavy the server who has the knowledge about that can actually restrict their their slice of the tree and say “I'm only willing to serve from the standard slice. 32:14 It doesn't swarm in the sense that you can just blindly pass the information around the network between the different clients, but it does swarm in the sense that all of the clients that are partially synced are still fully capable of being first class servers to other clients who just don't have as much data as they have. 32:37 In Alexey’s model. It's a fully push-based thing, so the servers are pushing the data out and the clients are passively receiving it. The current models of sync are all fully pull-based so that clients are just asking for the data that they need And this one is a sort of hybrid of the two where the servers broadcast what data they're willing to serve and then the clients are about the pull but only from that slice of data. 33:13 I think that that covers the like general just a bit of a at a high level questions or clarifications. Jason: And so during during that window the data that you serve is pinned to the stateroot from the beginning of that window, so it also implies that you're using witnesses or some flood witnesses. 33:33 Piper: Yeah. There's a there's an ambiguity there that says like since you're serving data from the beginning of the window it still doesn't fix the need to patch up the tree in some sense because you won't be getting a consistent view, so either you're using get node data or a new primitive or you're using witnesses to patch up the data that you have already pulled so that at the point where you've traversed the whole tree you're just done. 34:00 The get node data version says that the point where you've got the whole tree you've got a disparate set of trees and you do what snap does and use something to patch up the tree after that. So, There's ambiguity there in exactly what the mechanism is but either way there has to be a mechanism to patch up the tree whether or not you do it while you sync or after. 34:26 And I think that that's global to just any sync mechanism because we're always going to sync somewhere behind HEAD And I think it's-- like other than something like fast-sync, that like eventually turned like I think that even fast sync at some point would hit a point where you just couldn't enumerate the database and get to a recent head fast enough if the tree was large enough or something, so like I think any of these sync algorithms we're going to end up in a spot where we're sinking to an old head that still needs to be patched up. Jason: Yeah, the only the only alternative I can think of which is not a good one is something like warp sync where you're sinking the whole data set to a particular head and then you full sync forward from there right is problems with that yeah we've seen that that when breaks at certain network conditions. 35:21 John: Go just I'm trying to make sure I understand this because I'm writing these notes that there's sort of an emerging coordination that happens by clients saying what they what they're what they're what they're willing to have and then there's event like eventually you'll get the data but you don't know when the when you'll get it all. Like it will happen at some point. 35:40 Piper: Right there could be a really unfortunate situation if the network parameters are tuned funny where you need stuff from one section of the tree and the block hash just doesn't land anywhere near there for a while. And so either the protocol is really restricted and says, well you just have to wait and that's sort of unfortunate or we leave the protocol loose enough so that a client could. 36:09 Advertise data, that is far away from the hot spot To a client that it, through some heuristic or whatever, recognizes that they still need that information. So oh and that's the other piece and I can't remember if this is written if this is in the write up or not but there can be a two-way broadcast so clients can also in theory advertise what information they have which gives servers a cue to know what they need. So that kind of broadcast of like “what information am I willing to serve” 36:46 Also comes into play or like essentially informs it can inform both directions. Rai: Yeah that was that was kind of how I was envisioned envisioning what you were talking about where like the block hash is just determined like some slice of the try right you're like this is this is around where we're sinking. And then you because your clients can advertise to you the prefixes they have you essentially get like ideas of like lines radiating near that that represent like the nearest places to that line that that your peers don't have and so you can start serving away from that distance and getting further further and further out. 37:29 Piper: I like the idea of the protocol being loose and allowing servers to play around with different versions of that behavior. It I think allows for like a kind of experimentation in the same protocol that's good but it also might leave too much room for freedom and there could be like abuse vectors hiding in there so I --So we need to have like a minimum of width that a good honest peer would like serve or something -- maybe, or it might even just be a like extra like it's the server is allowed to maybe broadcast whatever they want and if the server wants to say you can ask me for anything and that's okay in the server could choose to do that but since everybody gets to set like broadcast the restrictions on what they're willing to serve, then it then it automatically builds like let's click the server change with those parameters are based on their ability, size of the tree, the clients they're connected to whatever so it kind of leaves some freedom there that I like. Jason: It just occurred to me the goal to have sinking clients served data, I think is only possible if they are actively keeping up with witnesses? -- yes -- right so so the idea of patching it up later doesn't work doesn't work if you want your clients to re-serve the data Piper: that is correct. I have not thought about that so so in theory both of these I think are actually dependent on us having witnesses flowing around the network Alexey: well actually I what I thought about this from the beginning and in my yet unwritten proposal what we do is that when we when we start serving in certain slice we actually because we serving in them at the tics which are kind of a corresponding to the block numbers, so it's actually the deterministic like which block number corresponds to which slice what you can do is you can basically superimpose the slice on the witness. essentially witness becomes attached to the slice as one piece of thing it's like it's going to be like a one slice plus witness thing and then this is what you're actually distributed and this is what everybody is distributing the same piece of information so that don't they don't even need to while they syncing they don't need to listen to the witnesses. They they can but you know, they don't need to so essentially that protocol automatically includes the witnesses. 40:16 Again, we need to figure out what is the overhead of this because we know that witnesses have lot of huge hashing overhead like you know, if you start doing that thing so we need to see on the data like how much overhead does it introduce. is it even viable. or what the have you. so yeah, that's why I'm I'm wanted to prepare less to make the judgment based on data. 40:41 Piper: That's fair. Martin I have a question for you if you're still around which is like I know that you guys have put a lot of work into snap and it sounds like a nice protocol might intuition both and thinking about the write-up that I made and Alexis proposed way is that like a lot of the work that I think you guys heard that. 41:08 I'd like to suspect you guys had to do for snap and translate well into one of these new primitives because we're still just talking about sending contiguous slices of the leaves across the network from a kind of like snapshotted point in history because we've got like this kind of like epic concept of some sort. 41:34 Does that sound accurate to you or do you feel solved after Martin: I missed a lot of the conversation because my daughter's making dinner on the fire alarm went off... Wow. Yeah. Wonderful. But yeah, it sounds like it's okay, you know that was in the details. Piper: Yeah, yeah, absolutely and like I am really happy that you guys put a lot of work into like this new protocol so like there's a little bit of like guilt on my side even from like say and let's come up with a new one now. 42:09 That you guys got something, you know new figured out that's better than fast sync. So I'm certainly sensitive to that and Martin: oh no, don't be I mean, yeah with with we kind of went from what can we do to improve? Yeah. I mean everyone should follow the path. I think that’s beneficial. 42:33 John: This is the the one that you guys released like a couple weeks ago, just so I understand we're talking about. --No,? -- or there was like a post about it. And then it was like something Piper had worked on. I mean a Peter had worked on. Martin: well yeah with what we released was the underlying database modeling gap, which allows us which will allow us to ship a new protocol on top of that later on. 43:04 John I I've also curious what um, if everyone at anyone else wants to know this might give a thumbs up but then like what is the path that you guys took to get to where you were because I'm wondering if there's some hidden lessons and stuff that like how is it that you got to where you were and why is it that you got there? Some of the underhood under the hood description of that would be really interesting. Since we're all sink. Martin: So at some point we're gonna make a big post about the whole the whole thing is gonna be a long post. There are I mean, it's so much things that had to be built and built around accommodating this model and this sync experience. 43:47 Um, Yeah, it's huge. Alexey: Yes, I can totally understand that it's because there are so yeah, you know, if you start doing this flat models, you just yeah, it's not as straightforward as ….. Yeah, it's always the reorder that's like things. Piper: So at this stage, um, I am inclined to kind of dig into next steps like viable way forward as well as sort of just like straw man poll of everybody here of like what they do what whether or not this is a an idea that they want to continue to pursue. 44:35 So. The way that I would say this going forward is that there's the quantitative research side that needs to be done. Alexey, it sounds like you and your team are are kind of like looking at the way forward for the approach that you've looked at you started writing it up just sort of internally and I would probably be looking at doing a similar thing on our side despite probably having some infrastructure issues that we'll have to work out to get there, but in general for like the way forward for this it feels like something that needs to just actively start getting experimented with. And then at some point if we're Actually going to do something together and do a sync protocol together. The the flaw that I've seen us like fail at in the past is is multiple solutions that take different trade-offs but are approximately equally as good in the end and teams getting committed to the one that they worked on the most and us ending up with different solutions and disparate end results because we don't come together early enough. And so that is the one like it's the way that I see like I see this failing in two ways One we just never go anywhere with it and don't get momentum to go forward or Two we go in two different directions for too long and we're unable to come back together, so those those are sort of. 46:23 Alexey I know I I think that we probably don't have enough history to generalize on this things..our our kind of joint history is too short --but isn't that like the history to date like right now is that like we had like fast sync and warp sync -- that kind of got like developed well that was before us that was way before we came to the scene right --but then like you made red queen and we made fire hose and likely they were they were close -- we started implementing we we tried to cross breed them that's what we're Andre was doing and then we started to implement implement the crossbred version, but then when after certain amount of time with the with the stateless ethereum, we just realized that the complete something so much more simple and better. because like with a fire like with a fire hose the issue I realized that there was just basically too many primitives yeah. When we looked at the witnesses like okay, so we just need one primitive and then it went even further obviously we didn't have time to implement this but now I'm actually just like ditch all that to do the these the the merry go round thing yeah, Piper: So my my intent in all of that rambling was just to say there is a failure mode in which we like just never make a decision because we have a couple of comparable options and so I just wanted to put it out there. 47:58 I don't have a solution that I'm proposing to that. I'm just stating something for us to be aware of as if we continue to move forward Alexey: well, you know what one of the one of the biggest motivations for so for us to prioritize this work to prioritize the sync is that you know, something that we figured out on the in in in the summit in Paris is that actually the this particular this type of a sync network would actually allow us to facilitate the much simpler transition to of the of the merkle tree to whatever you want. From binary tries to code merklization. so forth and the reason being is that because if you have a heterogeneous network where some of the implementations are kind of more aligned to produce this kind of data quickly, for example, I would say tubo-geth yet is much more aligned to to to support both hex and binary trees. 49:00 We already pretty much kind of almost do that because we separated the completely separated the merkle data from the normal code data, so but we if we have this sync network we can you know, if it works we can simply provide all the states in whatever format you want we can binary hexary, whatever you want in a very efficient way so that instead of everybody trying to rewrite their database which I know is super inefficient, they simply just download it. like whenever they want. And this is this motivates my desire to make this kind of much more predictable like the the sinc cycle needs to be predictable so that you know that if you wanted to you can obtain the the binary version of the more cool tree in this amount of hours. So that once everybody's comfortable with this idea then we can say okay, so then we can figure out when the hard fork is going to be because from now on we have a working network which can deliver your data this data whenever you want it to, and everybody even if they haven't quite finished their migration process inside their clients, they can just simply jump on this network, make this one time transition, like download everything and forget about the hexary transition.. They can skip the entire kind of this keep the entire like engineering effort of trying to migrate their own tries because basically turbo-geth is just gonna do it one time service where it will blast away these binary trees and after hard fork everybody will have binary trees so after that, it doesn't really matter anymore, so that's why I see how it could save like everybody tons of engineering effort. 50:46 Piper: So next steps seem like: Narrowing in on more concrete definitions of these proposals and the kind of quantitative research of like, actually doing either like minimal implementations or even actual implementations of these things of whatever the leading merry-go-round concepts are maybe we can like pick different types of like animal carriages? Rai: So one one way. I was visualizing it is like the tri-space has different colors like it's a rainbow from like, you know red to purple or whatever so if you're going through you're kind of like traversing the rainbow whereas like pipers yours is like yeah randomly, so I guess I'll call the pipers like rainbow and the other one. 51:41 Maybe mine's like the tie dies like yeah, oh tie dye yeah well I Alexey: I thought about this visualization which essentially is, you know, we usually draw the state rises like these like triangular structures right but what you can do is you can just like stretch them out like in a circle like the polar coordinates and so they end up being like stretched out in the circle and if we also apply some cleverness in terms of a kind of weight like but you stretched it by weight you can actually see clearly how the merry-go-round is going to work because they're just going to progress through certainty, I mean, it's my version, of course, --yeah no I be really cool to do with colors too you can see that the red portion is like wide so you see oh the lower prefix keys are like a really big part of the schedule or something right at the unique visualization and Piper: Frankly I really like that idea of a predictability. I'm a little skeptical on the coordination but but not but I haven't thought about it enough to be legitimately skeptical. I'm just skeptical because it's typically a hard problem. Okay, so the way that I see this going is more concrete proposals more concrete qualitative research on actually playing around with implementing some of these things and trying them out in the live network and then like Alexey is talked about having some sort of hackathon. And that doesn't necessarily mean coming together any sort of like physical but like sort of. Coming together on a schedule and I'm like, all right we're like going for this now and here's like our timeline and we're gonna try to have you know, cross-client sync, you know network stuff going on and kind of like an active working on it at some point when we say this is the approach that we feel good about as group and even though you know, I might have you know, we may have some opinions about like which set of tradeoffs are better, my hope is that we can just say... We're just we're picking this one at some point and pick one together and and kind of like put our you know, subjective things aside and actually like get something like this rolled out. I'd say by the end of this year would be really like it feels reasonable in terms of timeline and that puts us queued up to actually do the binary tree transition that we need for stateless ethereum in an in a good timeline as well. 54:25 John: As far as calls by this seem really really good so maybe with I we should do another one yeah. Alexey: I mean yeah the calls usually oh good. I mean the I we always want to do things we always define something interesting to talk about yes, yeah. -- So that does that mean we it's productive to have them again? -- 54:50 Piper: I would say something like a month from now assuming that we've got... New information which assumes like a write-up a better write-up some actual data to to report on that would be my kind of like next steps so if we're gonna have another call I would target it for about a month from now and that feels like the right amount of time for us to dig in and and have some some new information to share. Cool. Questions or anything else from anybody before calling it quits slash a win? All right. Nice office Alexey. Like all the light. Thanks!