{"id":75553,"date":"2024-02-20T17:25:54","date_gmt":"2024-02-20T09:25:54","guid":{"rendered":"https:\/\/version-2.com\/?p=75553"},"modified":"2024-02-07T17:33:56","modified_gmt":"2024-02-07T09:33:56","slug":"libdrop-file-sharing-through-nordvpn","status":"publish","type":"post","link":"https:\/\/version-2.com\/zh\/2024\/02\/libdrop-file-sharing-through-nordvpn\/","title":{"rendered":"Libdrop: File sharing through NordVPN"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
\u00a0<\/div><\/div><\/div>

The Libdrop library allows NordVPN users to share files over Meshnet<\/a>. In this article, we explain how we developed our file transfer system and the role Libdrop plays in it.<\/p>

<\/p>

What is Libdrop?<\/b><\/h2>

Libdrop is a cross-platform library developed in the Rust programming language. It is compatible with Windows, MacOS, Linux, iOS, and Android. File sharing within the NordVPN environment is facilitated by the Libdrop library, which is available as an open-source resource on GitHub<\/a>.<\/p>

The goal of Libdrop implementation is to allow smooth and secure file sharing between users over Meshnet. The library should be easily integrated into the NordVPN application so API users can issue transfer requests, with the rest of the processes being carried out in the library.<\/p>

Libdrop protocol<\/b><\/h2>

The Libdrop protocol enables peer-to-peer file sharing via both IPv4 and IPv6<\/a>. In this process, the sender presents files to the receiver, who then selects specific files for download. Downloads are then initiated.<\/p>

The transfer is live until one of the peers goes down or the transfer is explicitly canceled by either of the peers, after which the files are no longer available for download. This provides the user with a time window where they can decide which files they want to download now and in the future while the transfer is still up.<\/p>

<\/p>

High-level overview of communication between two peers.<\/p><\/div>

Communication and low-level details<\/b><\/h2>

Let\u2019s take a closer look at the technical details of the communication process, and how we developed our current setup.<\/p>

gRPC<\/b><\/p>

At first glance, it seemed evident that our easiest course of action would be to focus on the HTTP server and client because this is very easy to use and understand, as well as being a time-proven technology. We could make a REST endpoint and just proceed with a regular HTTP download.<\/p>

To enhance speed and control, we opted for gRPC. Because gRPC is a binary protocol it has less overhead. It is also strongly typed, making errors harder to introduce. gRPC technology automatically generates the code needed for both the client and the server, making it an excellent fit. In fact, Libdrop was originally built on gRPC.<\/p>

Initially, it was very comfortable to use \u2014 both the client and the server code just worked. We could issue a certain call via the wire and expect the appropriate function to be called on the peer.<\/p>

However, as time went on, we found that debugging gRPC presented some challenges, and the “black box” nature of it began to concern us. The generated code also had little control over the socket itself because it was abstracted too far away to gain direct access. Consequently, we transitioned from gRPC to WebSockets in pursuit of a more adaptable solution.<\/p>

WebSockets<\/b><\/p>

Unlike gRPC, WebSockets is not strongly typed, which offers a degree of flexibility. This flexibility comes at the cost of making it easier for bugs to appear. However, there’s no automatic code generation, which is a plus.<\/p>

The ability to easily introduce versioning was another advantage. We just need to have the URL in the form of “ws:\/\/{addr}\/drop\/{version}\/query.” It also helped that WebSockets is a fairly easy-to-understand technology that works in tandem with HTTP so the traffic can be inspected easily as well as debugged.<\/p>

Choosing WebSockets turned out to be a wise decision. It led to a reduction in code complexity and greatly enhanced our understanding of the data flow. Plus, having written the code ourselves, we felt fully in control of the system.<\/p>

<\/p>

Simplified representation of backward compatibility between Libdrop versions.<\/p><\/div>

Rust and Tokio<\/b><\/p>

Due to the nature of the Libdrop library\u2019s heavy IO and event-driven architecture, the codebase contains a lot of asynchronous flows which could have been a tough problem. However, Rust\u2019s great implementation of async alongside the Tokio library proved to be a great combination in dealing with this and avoiding potential crashes.<\/p>

Rust shines because the borrow checker is really persistent about lifetimes and safety while developing because it prevents you from compiling incorrect code that breaks ownership rules.<\/p>

We are also fairly safe from panics as we spend most of the time in Tokio tasks and those are executed in catch_unwind<\/a>. This means that if the Tokio task panics it will simply yield an error instead of tearing the whole thread down.<\/p>

Still, not every place in the codebase runs in a Tokio task, and so for those cases where a Tokio task is not involved, we tune Rust linter to detect unwrap() calls in the codebase that could potentially invoke a panic handler.<\/p>

NordVPN uses Rust in numerous libraries and panics are handled in custom panic handlers. These handlers wrap the error and emit it via callback so the API user receives it and can properly log it.<\/p>

API and the dilemma: To block or not to block?<\/b><\/h2>

We\u2019ll now explore the choices we made around our API.<\/p>

SWIG<\/b><\/p>

For the API we used SWIG, which was already battle-tested and proven by libraries such as Libtelio. SWIG automatically generates FFI binding code for all target platforms, but it\u2019s not without limitations. While it\u2019s very easy to pass primitives such as integers and strings, higher-order structures are not that comfortable. In a compromise, we accept certain parameters as JSON strings.<\/p>

JSON strings, while slightly less optimal, are a great solution to the problem. All mainstream languages know how to parse it or have a popular library ready to do so. The downsides to JSON strings are less type safety and a need for greater control to avoid breaking the conformity.<\/p>

Event-driven architecture and reporting<\/b><\/p>

One question that arose around the API was whether or not we should block it. Based on the API users we opted to not make the API block and communicate via events. This provides more complexity on the API design side but it provides an event-driven API and means that API users don\u2019t need to care about threads. App developers are usually experienced in working with callbacks so this architecture suits them well.<\/p>

Callbacks are used for event notification and reporting so the API user can receive reports and log them where appropriate. Events are for reporting. Both events and reports are passed on as JSON-encoded strings.<\/p>

Errors are reported when the parameters to the API are incorrect or when a runtime error is encountered.<\/p>

Types of events<\/b><\/h2>

Events are emitted for various milestones:<\/p>

A transfer was requested.<\/p>

The transfer was successfully queued (the API returned no error) and contains all the paths collected.<\/p>

A file upload\/download was started, finished, or failed.<\/p>

A file upload or download progressed.<\/p>

User experience and history tracking with SQLite<\/b><\/h2>

When considering how to track transfer records and states, our team opted for a local SQLite database that users can easily inspect.<\/p>

We chose SQLite for its flexibility and cross-platform availability, and because it offers a strong query system that makes it user-friendly.<\/p>

The widespread use of SQLite in various applications gave us added confidence in its reliability and performance, making it an easier choice over alternatives like JSON files or custom binary formats.<\/p>

Database limitations: A read-only resource<\/b><\/h2>

The SQLite database does not control Libdrop’s operations in any way. Its role is purely read-only. The SQLite database serves to offer our users a convenient API for accessing transfer histories and logs, without impacting the underlying functionality of Libdrop.<\/p>

In cases where we fail to open or migrate the SQLite database successfully we can remove it entirely and try again. If it fails again we can then use an in-memory database that provides proper functionality while the app is alive.<\/p>

Security and validation<\/b><\/h2>

Security in Libdrop has several key focuses:<\/p>