Visit our repo on Github if you encounter any issues while using our framework or have suggestions for new functionality. We encourage active engagement and welcome contributions, so if you are interested in becoming a contributor, we would be delighted to have you join our community!
EdgePersist
EdgePersist is a package of components that enable edge storage for IoT and smart device edge networks. It consist of three main components:
- Edge Storage Component: the core component that provides the edge storage capabilities based on MinIO, K3s and Prometheus.
- Edge Localized Docker Registry: the localized docker registry that uses the edge storage component as a back end for real-time and proactive docker image migration and replication.
- Edge Registry Sync Daemon: a daemon process that syncs the localized registry to one or more remote ones based on a Kafka message bus.
Features
EdgePersist aims to provide a reliable, fast, stable and secure shared storage engine, accessible by all devices and users in an edge/cloud continuum. Furthermore, it needs to be extremely lightweight since it is created for edge devices with extremely limited resources, like Raspberry Pies or other micro-computer devices.
Inherently decentralized and highly scalable P2P solution
An open-source framework, called MinIO is utilized. MinIO is an inherently decentralized and highly scalable P2P solution, allowing us to deploy it freely on usable nodes. It is designed to be cloud native and can run as lightweight containers managed by external orchestration services such as Kubernetes.
Object storage over block storage
MinIO uses object storage over block storage so it is in fact a combination of the two systems, preserving the lightweight distributed nature of block storage while providing the plethora of metadata and easy usage of the object storage.
Web-based GUI and an AWS S3 compatible API library
MinIO offers a web-based GUI and an AWS S3 compatible API library for easy accessibility and integration. This allows users to interact with MinIO using familiar S3 API commands and provides a user-friendly interface for managing and accessing stored data.
Datashim Integration
The Kubernetes Dataset Lifecycle Framework provided by IBM’s Datashim is employed on top of MinIO, allowing the edge storage component to be used as a file system folder, which is useful for applications that we cannot or do not want to integrate with the Restful API of MinIO.
Localized Docker Registry support
A Localized Docker Registry is provided using EdgePersist as its file storage backend, in order to move application images closer to the edge and limit network traffic and image download times.
Registry sync daemon support
A daemon process that syncs the localized registry to one or more remote ones based on a Kafka message bus. It aids in the pre-loading and replication of application images from remote registries or filestorage systems into the Localized Docker Registry.
Semi-automated deployment and off-loading
A set of bash and yaml scripts have been developed that handle all the configuration, installation and deployment processes that need to be contacted before and after the MinIO workers are deployed. This ensures that scaling can be performed seamlessly on each cluster, regardless of the underlying physical machines that act as nodes.
Harmonization of IoT network diversity
K3s is used as an orchestrator which is compatible with most devices that run windows or unix based operating systems. This enables to create highly configurable generalized deployment scripts that handle the deployment, configuration, un-deployment and re-deployment of the storage workers. In addition, Dataset Lifecycle Framework provides a uniform way of accessing the data, using the local file system of each device, eliminating the need of customized solutions for each new device.
Our Team at EdgePersist
EdgePersist is an open source project welcoming contributions of enthusiastic volunteers that want to be involved in the project.
EdgePersist is a joint research effort of HUA – Department of Informatics and Telematics, and ICCS/NTUA - Institute of Communications and Computer Systems.
Key people that are currently involved in the active development and maintenance of the tool are reported below.
Antonios Makris is a Senior Researcher at the School of Electrical and Computer Engineering, National Technical University of Athens. He received his BSc Degree in Computer Science in 2013 and MSc Degree in Web Engineering in 2015, both from Harokopio University of Athens. In 2022, he received his PhD in the area of Distributed Systems from the same department. His main research interests include Distributed Computing, Edge and Cloud Computing, Machine/Deep learning, AI Robustness, Efficient Computing, Big Data Management and Analysis, and Spatiotemporal and Trajectory Analysis. He has participated in numerous EU funded and national projects.
Antonios Makris
PostDoc Researcher
Evangelos Psomakelis is a PhD candidate at the department of Telematics and Informatics of Harokopio University of Athens and a Research Associate at the Distributed, Knowledge and Media Systems (DKMS) of the National Technical University of Athens. He is a graduate of Telematics and Informatics of the Harokopio University of Athens, holding a Master in Computer Science degree from the same university. His main research interests include machine learning, data mining and cognitive systems. He was involved in numerous EU funded projects.
Evangelos Posmakelis
PhD Candidate
Konstantinos Tserpes is an Assistant Professor at the School of Electrical and Computer Engineering, National Technical University of Athens. He holds a PhD in the area of Distributed Systems from the school of Electrical and Computer Engineering of the National Technical University of Athens (2008). His research interests revolve around cognitive software infrastructures for novel application classes, working at the intersection of research areas such as cloud/edge computing, distributed systems and ML/AI. He has been involved in several EU and National funded projects leading research for solving issues related to scalability, interoperability, fault tolerance, and extensibility in application domains such as multimedia, e-governance, post-production, finance, e-health and others.
Konstantinos Tserpes
Assistant Professor
EdgePersist in Action
The EdgePersist framework is currently being utilized in two European Union’s Horizon 2020 projects, named ACCORDION and CHARITY.
GitHub
Contact
Our team is committed to enhancing the performance of EdgePersist as well as enriching it with new amazing features.
We welcome collaborations with both industrial and research partners to further develop EdgePersist.
If you want to know more, please contact us!
Our Address
Omirou 9, Tavros, Athens Greece, GR17779
Email Us
{antoniosmakris,vpsomak,tserpes} at mail dot ntua dot gr
Call Us
+30 210 9549 413