We built Egnyte Connect - a secure distributed file system that combines Hybrid Sync with Cloud File System to answer content collaboration needs of businesses in wide variety of business needs. In 2007, businesses had started to become more distributed customers were using multiple devices to access their files and there was a need to make this experience as smooth as possible. It can be accessed from a variety of endpoints like FTP, WebDAV, mobile, public API, and browsers and features strong audit and security components. It is built for hosting a variety of file services and to cloud-enable their existing file repositories. What is your system used for?Įgnyte Connect as the content collaboration and data management platform is used by thousands of customers as the single Secure Content Platform for all of their document management needs. You can find more about these in the For The Techies section at our blog. CFS(cloud file system), EOS (Egnyte object store), Content Security, Event Sync, Search Service, User behavior based recommendation service form major parts of the system. Getting To Know You What is the name of your system and where can we find out more about it?Įgnyte Connect is the content collaboration and data management platform. Millions of desktop clients accessing files interactively Millions of desktop clients syncing files with the cloud for offline access Multiple terabytes of extracted content indexed in Elasticsearch Tens of petabytes of data stored in our servers and other object stores such as GCS, S3 and Azure Blobstore Native desktop and server hosted clients that allow both interactive as well as hybrid sync access to the entire namespaceģ primary regions with one in Europe connected to respective GCP regions using Google Interconnectĥ00+ Storage nodes powered by Tomcat/Nginxĥ0+ Text extraction instances (autoscaled) Python used to power client-side code, certain microservices, migration scripts, internal scripts ReactJS/Backbone/Marionette/JQuery/npm/Nightwatch In general, Egnyte connect architecture shards and caches data at different levels based on: To add elasticity, reliability and durability, these data centers are connected to Google Cloud platform using high speed, secure Google Interconnect network.Įgnyte Connect runs a service mesh extending from our own data centers to google cloud that provides multiple classes of services: Collaboration Egnyte Connect PlatformĮgnyte Connect platform employs 3 data centers to fulfill requests from millions of users across the world. In this article, I will share the current architecture and some of the lessons we learned scaling it along with some of the things we are looking to improve upon in the near future. To help our customers manage ongoing data explosion, we have designed many of the core components over the last few years. Over time, costs for S3 and GCS became reasonable and with Egnyte’s storage plugin architecture, our customers can now bring in any storage backend of their choice. Our only option was to roll up our sleeves and build basic cloud file system components such as object store ourselves. Let’s take this problem and transform it to a cloud-native file system used by millions of paid users spread across the globe and you get an idea of our roller coaster ride of scaling this system to meet monthly growth and SLA requirements while providing stringent consistency and durability characteristics we all have come to expect from our laptops.Įgnyte is a secure Content Collaboration and Data Governance platform, founded in 2007 when Google drive wasn't born and AWS S3 was cost-prohibitive. It is limited by the disk space it can’t expand storage elastically it chokes if you run few I/O intensive processes or try collaborating with 100 other users. There are a couple of downsides though in case you are looking to use it to support tens of thousands of users working on hundreds of millions of files simultaneously containing petabytes of data. Your Laptop has a filesystem used by hundreds of processes. He and his colleagues spend their productive hours scaling large distributed file systems. Egnyte is a Secure Content Platform built specifically for businesses. This is a guest post by Kalpesh Patel, an Engineer, who for Egnyte from home.
0 Comments
Leave a Reply. |