Archives

MAT Working Group session
.
28th October 2020
.
3:00pm (CET)
.

NINA BARGISEN: Hello everyone.

BRIAN TRAMMELL: Hi, I am Brian Trammell, this is Nina Bargisen, we are co‑chairs for MAT Working Group at RIPE 81 on the 28th October 2020. We have a fun and action packed agenda for you today. Four presentations. Generally I guess when we do this in person, we have about 90 minutes, today we have sort of the 45 to 60. So, we will get right into it with Andra Lutu.

ANDRA LUTU: I am going to try to ‑‑ okay, the hardest part, publish my screen.
.
Hi everyone. Thank you so much. And thank you for joining me for the session today.
.
I'm going to be talking a little bit about how cellularity relies on mobile international roaming and how things are actually roaming. So I'm going to explain what we did in this work with a joint collaboration between Telefonica Research and North Western University.

What do we mean by things roaming? I want to clear this from the beginning of the talks. With things, I actually mean any device that is not a smartphone that people will use on a day‑to‑day basis. So, you can think hereof multiple different IoT things we call them, from connected cars to coffee vending machines, connected points of sales, the things you pay with your card in shops, smart meters, for example, energy or gas meters, and even your wearables, devices that you carry with you and move around with you.

So obviously not all these devices actually move so they are not mobile but they actually need some sort of seamless connectivity. They need connectivity from mobile operators and that's why we have been focusing on this. Here, I'm not talking about your vacuum cleaner in the house, but I'm cellularity devices that need to operate anywhere in the world and they need the sort of connectivity.

Even if they don't move, they connect to mobile networks, what we are seeing today is actually they are roaming.
.
So now, I am going to talk a little bit about the interest in roaming. Why is roaming actually a solution foreseen or the and why did this become a mainstream sort of trend that we're going to uncover today? So, you know, mobile users meant actually more roaming, back in the day when we used to travel, before the situation now we're in, we're in a DPS they were carried to go around the world and you know receiving after your nice holiday in the islands, you are going to receive a huge bill. So a lot of regulators actually put a lot of effort into removing this danger, and, for example, in Europe, we saw roam like at home in the regulation that allows you to take your mobile phone and just use the traffic in the world in the same way that you would do in your home operator.
.
Obviously, you know, increased mobility meant increased usage of this once known as a niche service to users, and this also basically allowed for new industry trends to appear, for example Internet of things.
.
So what we see today is that infrastructure that became more mature to allow people to take advantage of this actually opens up to IoT devices. So, in a sense, IoT devices roam internationally much in the same way that people do.
.
So, I'm going to just ‑‑ before I dive into all the analysis and I'm going to show you numbers and stuff I want to just quickly refresh with everyone here what does international roaming mean and what do we need in order to be able to take our phone and travel anywhere in the world?
.
So basically what we have here is the need for any network to connect to any other network ‑‑ by a network to connect to any other mobile network. So we do this using a service that is offered by an IP exchange provider, so again not to used with IXP, that's different, that's for a public Internet. The IP exchange providers actually work on top of global carriers, and they offer this service for inter‑connection for data roaming. So an operator would make a service charge to an operator provider, then there is a global capability to other operators in the world. Obviously this just interconnects, however an operator also has commercial agreement with the other operators, we call them roaming partners, so, this commercial agreement, this roaming partnership basically regulates the tariff that an home operator would have to pay any other roaming partner that behaves as a visited operator for any of its customers. So, when I travel from Spain to the Netherlands with my phone, my home operator will actually have to pay the visited operator in the Netherlands interoperator tariff for allowing me basically to connect to their network.

So, we have two important notions here. The inbound number, so for visiting ‑‑ if you take the other point from the visiting network operator side, we're going to see the forum users as inbound roamers, so me with my SIM card from Spain, I will be an inbound user in the operator in Netherlands and also the people travelling from Netherlands to Spain are going to be outbound roamers. So these are more or less, this is the language we are going to be using today and this is the basis that basically allows us to travel around with our phones.
.
Now, you know, all this infrastructure actually allowed for roaming of things and how this came about was with the appearance of M2M platform, it's machine to machine platforms. So machine to machine platforms actually have commercial agreements, offer deals to different IoT verticals for connection. That means when you buy your car, for example, from Germany, it might come actually with a SIM card from German operator but you get to drive it like I don't know in the Netherlands or in the UK. So this happens because this particular vendor of cars, you know, went to an interim platform, bought this service from the interim platform with a SIM card from a home operator that you know is integrated, it has two ports basically on the same platform.

Much of the perks of you driving your car across the world come from these existing roaming partnerships that I just explained before. So the home operator that basically supports the interim platform will exploit its roaming partnerships across the world in order to allow the IoT verticals to function anywhere in the world. This is actually very important because you know if you think about connecting cars they actually need this sort of similar connectivity and to be able to function anywhere where you night distribute as a vendor, and also it allows for ‑‑ it makes sense in the commercial sense, because you don't negotiate with other every country a separate agreement, you go to an Internet platform, afterward you have the agreement and you can deploy your car, basically.

So, you had to what we're going to be doing is try to understand you know, how persuasive of these types of services of M2M services and how many devices can we even say something about the number of devices that you know use this sort of end‑to‑end services in our roaming.
.
So we are going to take two different points of view, if we look at the this ecosystem I'm showing here. We're going to look at, you know, the breadth of one end‑to‑end platform. So we're going to look at an operational end‑to‑end platform and we're here just taking a sample of about 100,000 devices that are operational on 4G, and we look at their whereabouts for about 11 days in November 2018. But also, we look at the point of view of a visited mobile network operator in a sense, so we look at the population, the entire population of a mobile network operator, and the question we ask is, okay, can we see how many IoT devices are actually connecting to this operator's network that are inbound roamers? So they are not their own users, right.
.
So, let's take it from the second point that I explained. So, here, for this mobile network operator we looked at population of about, I think, 40 million devices. It's an operator from the UK. And we basically looked at about three weeks in 2019, and what we did is break down this population of devices on roaming status. So you have, on each of these rows would be a roaming label, we call it, and all on the columns you are going to have the date. So in terms of roaming labels what you see in the upper two rows are the non‑roamers, so the native devices this operator sells but within the subscriptions but also you know, virtual operators.

Then we have the inbound roamers, which can be also national or international, right, and we see these two labels here, and then we have outbound roamers which are actually very few in this case.
.
Then we're going to be looking at inbound roamers, so, the devices that are not native to this particular operator, but, you know, they connect to the greater network of this operator in the UK.
.
We see that in this example that we analyse, they represent about 17% of all this for the median population of devices, and this is pretty constant in time.
.
So obviously the first question we did okay, can we tell apart, are they things, are they people? How can we classify them? Obviously, you know, this is standard IoT, it's not straightforward to classify them, it's not very easy. So what we did was to try to use different information that we could gather from these devices, from the GSME DB devices.

What we find in the blue heat map here is that we've basically broken down into different categories, what we have look, we were able to tell apart which were the feature phones, which are, you know, very simple voice phones that some people use, so they are still primary devices to people. From smartphones, which were the primary devices for smartphone devices, and the rest would be the population. We were able to tell apart the end time population and the classification that we created for the information that I mentioned before.
.
And now basically if you add up these columns, you should add up to 100%, right. So what we tell from this heat map is that, you know, about 74% of all the devices we classified as M2M IoT devices are actually inbound roamers. So the devices ‑‑ majority of devices that we saw are inbound roamers.
.
Actually, if we just check the distribution of all inbound roamers in these categories, so out of all the inbound roamers this operator sees, we only see that 27% of them are actually smartphones. Very little are actually smartphones.

Obviously, the next question was, where are these devices coming from? Which is the home operator of this devices in the sense of the home country?
.
Obviously they are M2M devices so they are not going to be travelling the same sense of people. That means that the home operator basically supplies them that similar and they operate in the UK. Right. So in the upper histogram we basically see the breakdown of all inbound roamer per home country. We see the majority, more than 30% of all the inbound roamers come from the Netherlands and then on the lower heat map, we just broke them down on the three categories I mentioned before, feature phones, IoT devices and smartphones. And we see that more than 50% of the all the devices roaming from the Netherlands are actually M2M devices. And there are other countries here. So quite a bit of mixture. We have a lot of devices coming from different places roaming in the UK.
.
Now, why are these things roaming? So why is there this possibility for these things to be roaming in the UK? And obviously this brings us to what I mentioned before about the M2M platforms. So what happens is that because, you know, we have this infrastructure that is getting more mature, we have now not so many boundaries in terms of commercial agreements, the M2M platforms actually gain traction and they have multiple clients, multiple IoT verticals that take advantage of this particular ecosystem and, instead of purchasing SIM cards in different countries where they might operate, they just have one agreement with the M2M platform. They go the SIM cards, for example, from the Netherlands and then, you know, they deploy their smart meters, which work quite a lot in the UK.
.
The obvious second question that we kind of answered from the point of view of the M2M platform was to try to understand a little bit what is the spread, what is the breadth of this, of their coverage? So what we did was take ‑‑ well, change points of view here and we looked at this question from the point of view of the M2M platform and we actually leveraged here the dataset I mentioned before of 100,000 devices and we see that this particular M2M platform, operational M2M platform, operates in different markets and gets supports from four different home mobile network operators. By "home", I mean that that sells the IoT ‑‑ the SIM cards to the IoT devices and that gives the service.
.
So we see that they operate in Mexico and Argentina, where there is not much roaming. However, the situation looks quite different in Europe, where using SIM cards from Germany is big. We see that there are a bunch of devices operated in Spain or Germany, France, US, and so on. So we see that, you know, there are ‑‑ there is a coverage of about 77 countries and 127 visited network operators. So, again, because of the possibility of taking advantage of the roaming agreements, we see that the M2M platform can actually cover quite ‑‑ well, has basically a global coverage.

BRIAN TRAMMELL: Sorry, can we go onto the conclusion? We're a little bit tight on time.

ANDRA LUTU: Sure, sure. What I wanted to say, the direction of these devices is actually 2G devices, they don't do a lot of traffic. They don't generate a lot of traffic. Also they are not moving much. They are basically stationary, as we see in a lot of smart meters, what we see here compared to normal smartphones in terms of activity. But they are ‑‑ actually occupy the network for much longer, so, on average, they are four times as long connected to the radio network, and they are very, very small consumption of data. They don't generate a lot of revenue for the visiting operator, so it's important for the visiting operator, for any operator to be able to distinguish this.
.
In the same time, they generate quite a lot of signalling so they do occupy these resources, the radio resources, while they don't read too much in terms of profit to the visiting operator.
.
Obviously, you know, we have different types with different behaviours, so some patterns defer here in terms of smart meters and connected cars.
.
But I'm just going to skip forward here to what I wanted to say in terms of conclusion.
.
So we saw, basically, what we offered here was the first characterisation in terms of the support that roaming gives to M2M communications and what it actually meant in terms of the population of a visiting network operator. So it's important for operators to kind of understand how they give service to these devices and break them down and try to understand how they can, yeah, increase their profits here.
.
So, yeah, this is kind of where I leave it. Thank you.

BRIAN TRAMMELL: Thank you very much. We have time for about a quarter of a question. You can find Andrei in the virtual hallway. Thanks a lot for this. This is a topic that we haven't looked at a lot in MAT Working Group but it's a very interesting sort of view of, you know, how the Internet traffic is changing on, you know, from the mobile operator standpoint. So thanks a lot for that. It's cool.

We are a little bit short on time. I'll go ahead and ask Raffaele to come up and present his work on UDP options.

RAFFAELE ZULLO: Hi. I hope you can see me and hear me. I am going to talk about UDP options and how we can overcome the limitations today in deployment. This is part of my work at the University of Aberdeen, but I am currently affiliated with the network security. We're going to talk about the UDP options and we're going to introduce our solution, the checksum compensation option, describe our measurements and results and also explain how you can redistribute our measurements.
.
Basically, UDP options is an recent extension to UDP, it's a very simple protocol and very essential with a fixed ‑‑ but the extension leverage is the redundancy to create an option scenario at the end of UDP payload that will be filled with these options, they are encoded with a type encoding seen here. This is a very small change, but it actually affects several fields, including the UDP checksum. There are already a number of use cases for UDP options, you can find them in the draft for example DNSSEC is one application that can benefit from the options. Bull we want to focus on if UDP options can be safely deployed, because every time we change something, we introduce something new, we usually face protocol situation.
.
So we looked for a simple analogies limiting UDP options deployment and we have found two of them. One is related to UDP and checksum validation with the four checksum schemes observed in the wild for UDP options packet. Another one is related to the length consistency check with sum boxes with actually checking the measure between a UDP line in case of mismatch. These issues can be found in middle boxes but also in end hosts because of the checksum off‑loading mechanism.
.
Why UDP checksum is problematic? The picture shows how UDP checksum is computed and it involves three length values, in the header, and as checksum coverage length. Now, the header contains, by definition, the correct length, but it's possible that implementations instead did use the other two lengths from the IP header instead from the UDP header and this unfortunately is what we have observed. We have two variables with two possible values so we have observed four schemes.
.
Obviously all the schemes generate the same checksum for UDP but generate four different checksums for UDP options. And one of the schemes has obviously been in and the other is not and going to discard packets with the correct checksum and, surprisingly, the most used one is the wrong ones, based on the upload LAN.
.
So, we focused on the difference between correct checksum and the IP payload checksum in case of options and we try to understand if you could manipulate the packet, add in something to the packet to make these two schemes generate the same value, and we actually succeeded to do this with the checksum compensation option, the 2‑byte field computed as in the definition, with a few minor complications to keep the alignment. However, the most important thing is that this option compensated the data between the correct checks and the IP checksum. If you include this option in UDP option packet, it can transfer ‑‑ it can be valued regardless of the scheme that is implemented in a middle box.
.
So we have done our measurements more extensively using a suite of ‑‑ these without options and seven with the options and we have tested the paths to UDP and HTTPS service. Basically we send real application packets to UDP server and, based on the subject of the replies that we received, we can infer which packets have actually reached the destination.
.
In this case, we can see that there is a middle box implementing wrong checksum, and so when we send a standard request encapsulating our UDP options packets with correct checksum, we don't see any response, we see a response when we send the UDP options packet with our option. We have also tweaked this methodology for HTTPS servers and cannot rely on application layer response, but we observed when one third of the HTTPS servers that we considered replied to UDP packets with the ICMP reportable messages so we can leverage this to understand which packets have reached the other end. Obviously there are a few complications, for example, due to...
.
These are the ten packets that we send in our measurement. Even from the overall traverse result, we can see that the traversal rate of the packet, according to the original specification, is very low and well below the packets using the IP load checksum and the other checksum for using a CCL. If we mix together the results of all the tests for each path, we can characterise the paths in terms of pathology, and we can see that the paths are affected by the term in pathology, in light green are very few, while the IP payloads in checksum pathology in red is widespread. There is also a very weird case with dual validation, paths on which both checksums are accepted, but we will not have the time to do it today.

BRIAN TRAMMELL: I don't hear that weird echo, for what it's worth.

RAFFAELE ZULLO: Is everything okay? Basically, if we check all the paths on which there is this one device implementing the wrong checksum, it's up to 80% of the paths traversed by UDP options. If we group together paths on which UDP options and paths which were necessary to use, CCO, we reached these other paths that shows clearly how CCO can significantly increase UDP options and for the first two categories, the incremental CCO is even larger than the original traversal rate. We can also do this analysis on spaces and we can see that the CCO contribution is red and still clearly visible. There is also a portion of ASs and a portion of paths on which UDP options that don't work even using CCO and this is still to boxes implementing a length inconsistency check and these boxes need to be updated to allow the use of the CCO. Then also other analysis can delve into the genesis of the pathologist, also having some feedback from network manufacturers, and I say a few words about the tools we have developed and trace more and we have released this tool so you can reproduce our measurement.
.
We have also released a basic script to generate a whole different checksums and using a precompute CCO and that included a specific probe in a mobile text‑box, so you can download trace more at this address, what you can do is end‑to‑end ‑‑ for example, you can check that to which packets you can receive a response in this case a DNS response from a destination server but you can also use it in a traceroute way, for example, these traceroute showed that presumably at the second node, there is a box implementing the wrong checks and using the traceroute approach you can ‑‑ other than checking you can see the options that work in your network and you can also pinpoint the interfering node. If you want to check from an Android device, UDP options, you can use instead this application. And basically we have talked about UDP options and described the limitations that deployment. We have also shown how our checks and compensation option again increase the chances of a UDP options deployment, and we have described very briefly the tools that you can use to reproduce our result.
.
We wanted to test our results on a larger dataset. We have partially done it without finding any surprising result. And we also want to test more thoroughly the UDP options in clients' network and this is why we have included that probe.

If you want to take a look at the paper or ‑‑ you are free to ask any questions.

BRIAN TRAMMELL: Thank you very much. Do we have any questions? Thank you very much. Super interesting work. With that I'll ask Alex to come up and talk to us.

ALEX ULMER: Now you get a little bit different presentation because I will present to you visual analytics tool, it's a progressive tool that approximates the geographic path of BGP updates and, to have a little bit more time for the core part, I will skip a little bit over the first slide. So first I want to give you an introduction who I am because I'm not from this domain of the Internet routing or network security.
.
Then I show you a little bit related BGP update visualisations and then come to the core part of what our tool does and how is it programmed, and, at the end, give you a short demo of the website, and how it's used.
.
So my name is Alex Ulmer. Like I said, my core domain is information visualisation and visual analytics, but for the past four years we are also working in cyber security visualisations because we joined Athina programme, that's a national research centre in Germany.
.
So, the goal of this presentation is, I want to present to you our new prototype and how we visualise this path, but because of my limited knowledge in this field, we made some assumptions which you need to make if you do approximation. It would be great if someone is interested in a collaboration and discuss these things, how to improve this approach, because this is an ongoing work and currently fully active working on this.

So a BGP update visualisation very fast, what we are doing is we want to visualise this AS path. This is a BGP update which you might know, and we want to see how this is going over the world geographically. So there was already previous work, you may already know the BGPlay tool, it's from RIPE Stat, where you have a node link diagram where you can see the nodes as the ASs and the links how the path is going.
.
There were other tools in research community. Here, we have already geographic annotation, but on the country level where you can see how much traffic was sent over a specific country. Another related work is the work where we can see a polygon of the announcements, where you can immediately see in which country some IP prefixes are announced.
.
So, why do we want to do this geographically? So, it's like you have maybe seen in the two examples. Path changes are easier to understand if you have just a node link diagram. We can't know all the AS numbers, who they belong to. And also, you can immediately spot unusual detours on the world map.
.
So how do we do this more accurate as the previous related work? So, we use the geo IP data with a city precision, and this is also one thing we also highly dependent on the accuracy of this database, so there might be some inaccuracies in the visualisation. Then we thought how do we start to visualise these update paths because you have these huge boxes and it's difficult to to see. We want to proximate the internal routing in an AS and the routing between the ASs. Then of course we have to think about how can we visualise internal AS routes, therefore we need some knowledge, and there is only a few providers that share like a network map where we can really see the data centres and the connection between them. So we tried ‑‑ we had some other network stuff from bigger provider like Hurricane Electric an we tried to proximate as close as possible to these images. Of course next steps would be to gather more numerical data to have really bench marks to run, how good our approximation is.
.
So this is a visualisation of all the geo IP data from MaxMind, you can see it's a lot of white dots. It's also interactive. You can zoom in and click on every dot. But this is a different project. So, based on this, we thought how can we cluster these IP blocks because we know where the IP blocks is from the database and we know which providers own these IP blocks. So, we made some assumptions, of course, of course some good, some are weaker, but they are the main feedback what I want to achieve from this presentation and get in context for this.
.
So first what we do, we take these geo IP data and cluster them so that we can proximate where data centres of this provider are. So we have these assumptions, I think because of the time I cannot go through every one, but the slides are also online. We ran into some problems with different algorithms like the K‑d‑tree and the K‑d‑cell. You can see, for example, it's a cluster centre may get on a spot that's not inside the cluster, we get a cluster data centre approximation where there is a strange cell, so we created a hybrid solution that eliminates all these problems, so, this is for example the IP block location just for the AS 174 from Cogent, and, with our algorithm, we approximated these data centres. You can see there are a lot of them because they are little little points here which we didn't want to filter out. We were, at this point, not sure where the filter threshold should be, but this is like the first shot we took.
.
And then the next step is, okay, now we approximated the data centres, but how they are interconnecting? So we again made some assumptions that short connection are prioritised. Connections between big data centres are prioritised and data centres should reach other data centres with lower amount of hops.
.
So therefore, we used the ‑'s minimum spanning tree and add edge to reduce the distance between data centres. And the feedbacks also are welcome again. Now again we can see the data centre and with our algorithm we came up with this internal connections. So if we compare this to the info graphic from Cogent, we can see that we have an approximation of some data centre here that are not really existent, and in Europe there are a lot of data centres here in the centre but we only aggregated some. So it's very difficult to, like, to fine‑tune the algorithm, the model behind the algorithm, so it represents all the dense areas and really sparse areas fine.

And this is ongoing, we have to improve this.
.
And then to the biggest part. Now we have these connections from all the ASs that we have computed, now we can visualise the AS path. What we do there is we have assumptions that we prioritise long paths in the AS network and that are neighbouring ASs appeared ‑‑ that where the peering is happening, that they have to have a data centre close by.
.
And so, based on this, now we just use the geo IP data, but now we are using the BGP update data, the raw data. So how do we get them as this is a progressive tool that's always up to date?
.
If you type in the prefix and the time frame which we see in the demo, all the archives are ‑‑ the links are expected from the BGP screen broker, so we know this is all the download links, then we download all the files this this time period and process them with the BGP scanner. Then we filter out the requested prefix and run the approximation algorithm for each AS in the path and then we cache all the archives, so if you called us again in the future it doesn't take so much time.
.
So now I will go over to the live demo. Here you have the interface, put in the time. It's a maximum of two days and there is a lot to download. You all might know a large BGP update archives are so I have this in cache, so I can basically load this data and this is a time span of almost eight hours of BGP updates, and when I run this tasks without caching, the first update path is already visualised after 20 seconds and getting all the data for the eight hours took like eight minutes, so, now you can just hover over these paths and see how they are going. We have also down here a bar chart with a histogram where you can just filter out time and slide through, so then that you have less over‑plotting and analysed paths. For example, I take this path here, you can see more detailed info about the update here and this is the AS path. You can have additional information about the ASs, the holder, the country, when the registration was done, and here you can see we have this AS path, it goes from 3402 to the origin, or the other way around, so we have the origin here, and the next step is it hops over to ‑‑ the number is just not visible right now, and then it goes over to the other AS number and basically here we enter the Cogent AS and here we use the internal routing approximation tool to this, the location where the next hop is from, the BGP update.
.
So you can view this, and also you can always input an AS number here and view how the approximation was for the internal network, also dependent on the geo IP data from the various dates. Currently we only have weekly updates because the maximum database is just updated like this.

And with this, thank you for your attention, and I am open for questions.

BRIAN TRAMMELL: Cool. Thank you very much. So, I'd actually say if we take the questions to the mailing list because I have done a very bad job of managing questions in other places as well, but Daniel, I'll give you 15 seconds for your question.

DANIEL KARRENBERG: Thank you very much for this very nice talk. I really appreciate it. The stuff looks really cool. You might want to look at the RIS data that's now in BitQuery which might actually speed up your way of getting the BGP data. And the one thing I was wondering is, you talked about determining which data centres are big, how do you do that?

ALEX ULMER: So basically, we have been ‑‑ the prefix size, we know how much IP addresses are in this block and, based on that, we determine if there should be a big data centre there in this area.

DANIEL KARRENBERG: I think a lot of discussion needs to happen to give you some more domain knowledge, so if you have time for that, I'm quite sure if you just subscribe to the mailing list of the MAT Working Group, there might be some people giving you good idea.

ALEX ULMER: Yeah, that would be great.

BRIAN TRAMMELL: Thanks very much. Let's do take the following discussion on this to the MAT mailing list. One of the reasons we, as Chairs, wanted to bring this discussion in is, or this presentation, is, in here, we have, you know, sort of investment in the visualisation, the user interface and there is people in MAT that are working on the accuracy of the underlying data and sort of starting a conversation between those two things, it's like good UI plus improving geolocation data gets better information for all.
.
So, with that, I will ask Robert Kisteleki to come up and give us a tools update and a celebration of RIPE Atlas instead of the coffee break. So, take it away, Robert. You have the rest of the time, use as much as you can.

ROBERT KISTELEKI: So I can use up people's coffee time. I will try to do my best to speed up as much as possible and skip slides as much as possible as well.
.
I am Robert. I am usually the usual suspect to give this presentation about various stuff; for example, research. We have published a couple of articles as usual on Labs ‑ please check it out ‑ and Labs has been ten years last year, as you might know.
.
Atlas is ten years this year. I gave a lightning talk yesterday, please look up the details. This was the vision that we wanted to achieve, and I think for the most part we did achieve that.

After ten years, we are here, and we will be there. Please look at the slides. And for more details, here is the presentation.

More importantly, Atlas and BigQuery, now if you want to, you can access our data on Google BigQuery, dig in, try to melt the system if you can. I challenge you. This is how you can do it. Go there and knock yourselves out.

RIPE Stat, there is another collaboration. Various versions of the systems exist.

There will be a new UI. Please come tomorrow at noon into the main room, we will give a demonstration about how it looks, and you will be amazed.
.
We still support the widget API. Don't worry, your widget will not go away.

There are some features that we have updated recently. Theres a looking glass which is pretty realtime for BGP. MaxMind has changed, we have a new geolocation service. Also, RPKI histories and some other widgets that have been contributed by APNIC.

Thank you very much.

BRIAN TRAMMELL: That was impressive. Okay. So, thank you all very much. It was awesome to see Robert do it with slides this time. We will come up with some other sort of difficulty to throw in front of Robert for the next MAT Working Group meeting. Nina and I have to go and figure out what that's going to be.
.
So, with that, I declare MAT Working Group for RIPE 81 closed and post‑MAT‑Working‑Group coffee for RIPE 81 open. We will see you all next time in the ether or in a conference room or in both, so, thank you all very much. Have a good day.
.
(Coffee break)