FS24
Challenge Task Winner Distributed Systems FS24
The winning group is: Alice Glaus, Mischa Binder, and Nils-Robin Grob
Co-Puzzle Web App is a collaborative puzzle using websockets, deployed interally at OST: https://dsys-puzzle.network.garden. For the load balancer they used Traefik, everything dockerized of course, for the frontend/backend they used React with next, and for the database, they used MongoDB. Congratulations, very impressive work!
The price is sponsored by Axelra, a tech venture builder that accelerates digital business models.
Distributed Systems (DSy)
This lecture series focuses on distributed systems, which are computer systems made up of multiple autonomous components, or nodes, that are linked and communicate with one another to achieve a common goal.
By the end of the series, you should have a good understanding of the various components and technologies involved in distributed systems and blockchain, as well as their implementation and deployment considerations.
Lecture Topics
Nr | Date Rapperswil | Date St.Gallen | Topics |
---|---|---|---|
01 | 19.02.2024 | 21.02.2024 | Admin, Introduction / Motivation, part 1 |
02 | 26.02.2024 | 28.02.2024 | Introduction / Motivation, part 2 |
03 | 04.03.2024 | 06.03.2024 | Load Balancing |
04 | 11.03.2024 | 13.03.2024 | Containers and VMs |
05* | 18.03.2024 | - | Categorization, Debugging Containers |
06 | 25.03.2024 | 27.03.2024 | Monorepos / Polyrepos |
05* | - | 03.04.2024 | Categorization, Debugging Containers |
- | - | - | - |
07 | 15.04.2024 | 17.04.2024 | Interview with Simon Tuck, Authentication, |
08 | 22.04.2024 | 24.04.2024 | Protocols |
09* | 29.04.2024 | - | Web Architectures |
10 | 06.05.2024 | 08.05.2024 | Deployment, Exam preparation |
11 | 13.05.2024 | 15.05.2024 | Blockchain, Bitcoin, Ethereum Introduction, Smart Contracts |
09/12* | - | 22.05.2024 | Mail Infrastructure, Web Architectures |
13 | 27.05.2024 | 29.05.2024 | Challenge Task Presentations |
14 | 03.06.2024 | 05.06.2024 | Challenge Task Award Winner Announcement, Q&A (no exercises) |
Online/Offline Lectures & Exercises
Due to student requests, the Distributed Systems (DSy) lecture will pre-recorded. The lecture videos will up uploaded before the lecture. The length of the videos will be ~ 80 - 100 minutes (2 x 45 min. lectures) divided into segments. The news segment (~10 min) will cover news around the topics distributed systems and blockchain and are not part of the exam. It should serve as on overview of current topics in this field.
I will be present in the lecture room 3.113 from 08:10 - 09:50 in Rapperswil on Mondays, on Wednesdays in the lecture room FZ0232 from 13:10 - 14:50 in St.Gallen. If you have not watched the lectures, you can watch it in this lecture. During exercises (10:10 - 11:50, 15:10 - 18:40 in Rapperswil; 15:10 - 16:50 in St.Gallen), you can work on your challenge task. You do not need to be present on Monday or Wednesday, but it is highly recommended to discuss your ideas, issues, and problems with the challenge task. On request, you can also join the exercises remotely via MS Teams.
The online lectures and PDFs can be accessed from outside the OST network (also on Youtube), the exercise and admin material only with VPN.
Lecture
Lecture 1
The admin part gives an overview over this lecture and presents the challenge task for this course (Slides: FS24-DSy-01-admin.pdf). The second video (news) reports on blockchain news. The first article is about a paper analyzing the feasibility and costs of attacks on Bitcoin and Ethereum, introducing the Total Cost to Attack (TCA) metric and noting past attacks on smaller chains like Ethereum Classic. The paper concludes that attacking these major blockchains is economically impractical, underscoring their robust security. The next article discusses the independence movement of Nginx development through freenginx.org after its acquisition by F5, reflecting on the importance of maintaining the project's openness and integrity. Lastly, two articles touches on the significance of fault tolerance in distributed systems, citing advancements in ECC RAM for desktops and strategies for enhancing Raspberry Pi reliability, emphasizing the ever-present risk of hardware failures and the importance of preparing for them (Slides: FS24-DSy-01-news.pdf). The third video show the first part of the motivation why distributed systems are necessary (Slides: FS24-DSy-01-intro1.pdf). [104min]
Lecture 2
The admin part is about group forming (Slides: FS24-DSy-02-admin.pdf). The second video (news) reports reports on distributed systems and blockchain news. The first article showcases a Java optimization effort, illustrating how leveraging advanced coding techniques, e.g., the use of sun.misc.Unsafe, can dramatically reduce data processing times from 71 seconds to 1.7 seconds. The next article shifts to a court case revealing emails between Satoshi Nakamoto and Adam Back, which brings new insights into the early development of Bitcoin but leaves Nakamoto's true identity a mystery. The next topic is about scalability challenges faced by large language models (LLMs), emphasizing the difficulties of running these sophisticated models on consumer hardware and the critical importance of distributed systems. The final article serves as a reminder on the vulnerability of hardware failure, highlighting the importance of distributed systems in mitigating the impact of such failures (Slides: FS24-DSy-02-news.pdf). The third video show the second part of the motivation why distributed systems are important (Slides: FS24-DSy-02-intro2.pdf). [73min]
Lecture 3
The admin part is about group forming and lecture content (Slides: FS24-DSy-03-admin.pdf). The second video (news) reports on recent incidents and developments. The first article is an Economic Denial of Sustainability (EDoS) attack, exemplified by incidents at Netlify and Vercel, where scalability features led to unexpectedly high bills for users due to DDoS attacks, raising concerns about platform security measures. The next topic is the alleged sabotage of underwater internet cables by the Houthi movement, affecting data traffic between continents and showcasing the vulnerability of critical infrastructure. Lastly, Bocek touches on malware targeting GitHub dependencies through repository confusion attacks, stressing the importance of secure development practices and the potential benefits of developing within Docker to mitigate risks (Slides: FS24-DSy-03-news.pdf). The third video introduces load balancing (Slides: FS24-DSy-03-load.pdf). [110min]
Lecture 4
The admin part is about answers questions from the exercises (Slides: FS24-DSy-04-admin.pdf). The second video (news) reports on various topics in his lecture series on distributed systems and blockchain, including the results of a 1 billion row challenge showcasing the performance of different programming languages, where Java, Rust, C, and Golang demonstrated comparable speeds. He reports on the impact of broken sea cables, likely caused by an anchor attack by Huthi rebels, on internet traffic and mentions a significant Meta outage affecting services relying on Facebook login. The series also covers the European Commission's scrutiny of Apple's web app policy for iOS, leading to a reversal on home screen web app functionality in the EU. Additionally, this video shares insights on personal projects involving the deployment of fiber optic cable at home and setting up a home lab, emphasizing the fun and educational aspects of such endeavors, and briefly mentions innovative projects like dockur for automating Windows installations in VMs and an alternative method for running containers without Docker (Slides: FS24-DSy-04-news.pdf). The third video dives into VMs and containers (Slides: FS24-DSy-04-container.pdf). [129min]
Lecture 5
The first admin part is about the lecture content (Slides: FS24-DSy-05-admin.pdf). The second video (news) reports on a continued critiqued for Apple's continued monopolistic practices in app distribution. Other topics include how to update online services without downtime, using the example of Erlang/OTP, and Meta (formerly Facebook) building huge computer systems to work on advanced AI projects. Figma, a design tool, had to make big changes to its database to handle more users, showing how companies need to grow their systems as they get more popular. Another article talks about how PostgreSQL, a database program, is getting better at handling different kinds of data work, which could simplify tech setups for many people. There's also a mention of a video that explains containers and virtual machines, tools for running software efficiently. The Tor Project's new WebTunnel feature is designed to help users bypass internet censorship by disguising their internet traffic. Lastly, a court case confirmed that Craig Wright, who claimed to have created Bitcoin, was not the real inventor, according to a judge (Slides: FS24-DSy-05-news.pdf). The third video finishes the introduction with a categorization and definition of distributed systems (Slides: FS24-DSy-05-cat.pdf). The last video (2nd admin video, use VPN!) shows how to handle issues with docker and docker compose (Slides: FS24-DSy-05-debug.pdf). [100min]
Lecture 6
The first admin part is about the lecture content and exercises (Slides: FS24-DSy-06-admin.pdf). The second video (news) reports on on how web bloat impacts users with low-end devices, examines the benefits and constraints of passkeys as a secure authentication alternative, debates the efficacy of HTTP/2 versus HTTP/3, and shares thoughts on terminal latency improvements. Additionally, an article is presented that shows an analysis on the cost-efficiency of on-premise data centers over cloud services and a critique of new EU anti-money laundering laws is mentioned for their potential impact on financial privacy. (Slides: FS24-DSy-06-news.pdf). The third video discusses the difference between a monorepo and polyrepo (Slides: FS24-DSy-06-repo.pdf). [50min]
Lecture 7
The first admin part is about the lecture content and exercises (Slides: FS24-DSy-07-admin.pdf). The second video (news) reports on significant tech developments, including the release of Podman 5.0.0 which enhances compatibility and performance for Mac and Windows, and discusses the occasional challenges in transitioning from Docker. Next, the universal need for Kubernetes is questioned, noting its operational complexity despite its scalability benefits. A critical security flaw in XZ Utils, identified as CVE-2024-3094, which permits remote code execution is discussed. Additionally, challenges in handling JSON numbers across different programming environments are touched, along with improvements in Docker caching strategies in GitHub Actions, and a notable decrease in GNOME 46 terminal latency. The news video concludes with insights on the rapid development of Meta's Threads app, which leverages the existing architecture of Instagram. (Slides: FS24-DSy-07-news.pdf). The third video discusses authentication (Slides: FS24-DSy-07-auth.pdf). The forth video is an interview with Simon Tuck from TrustSquare where we talk about Artwork.ID within the Certify project. [107min]
Lecture 8
The first admin part is about the lecture content and exercises (Slides: FS24-DSy-08-admin.pdf). The second video (news) reports on significant tech developments, including the recent Bitcoin halving, which halves the block rewards for miners every 210,000 blocks as part of the protocol to limit Bitcoin's supply, historically leading to price surges due to anticipated scarcity. He also discusses the One Billion Row Challenge, highlighting a significant speed improvement in data processing using CUDA compared to traditional C++ implementations, which shows not all tasks are suited for GPU acceleration. Additionally, Thomas Bocek contrasts the programming philosophies of "think twice, code once" versus "think once, code twice," emphasizing the value of rapid prototyping in revealing unforeseen issues and practical constraints that theoretical planning may miss. (Slides: FS24-DSy-08-news.pdf). The third video discusses protocols (Slides: FS24-DSy-08-protocols.pdf). [89min]
Lecture 9
The first admin part is about the lecture content and exercises (Slides: FS24-DSy-09-admin.pdf). The second video (news) reports on the vulnerability of even robust systems, highlighted by hardware failures on the Voyager 1 spacecraft, emphasizing the importance of durable design. In cybersecurity, he notes significant findings on password security, revealing that modern GPUs can crack simpler passwords rapidly, advocating for stronger security measures. Lastly, he covers the legal issues surrounding cryptocurrency mixers, such as the case against Samourai Wallet for facilitating illegal transactions, underscoring the clash between privacy and regulatory enforcement in crypto activities. (Slides: FS24-DSy-09-news.pdf). The third video discusses web architectures (Slides: FS24-DSy-09-arch.pdf). [88min]
Lecture 10
The first admin part is about the lecture content and exam preparation (Slides: FS24-DSy-10-admin.pdf, Exam: FS23-DSy-exam.pdf). The second video (news) provides a weekly roundup on distributed systems and blockchain. The economic vulnerability of cloud systems, termed "Denial of Wallet," where cloud resources can be exploited to cause significant financial losses is discussed. Highlighting security challenges, the risks associated with development environments, comparing security and simplicity in containerized setups and emphasizing the dangers of malicious repositories in Docker is explored. Finally, contrasting legal outcomes in the cryptocurrency sector, detailing the substantial differences in prison sentences between Sam Bankman-Fried and Changpeng Zhao, which underscores the clash between the severity of charges and regulatory enforcement in crypto activities are covered. (Slides: FS24-DSy-10-news.pdf). The third video discusses deployments (Slides: FS24-DSy-10-deployment.pdf) [97min]
Lecture 11
The first admin part is about the lecture content and challenge task presentations (Slides: FS24-DSy-11-admin.pdf. The second video (news) discussed recent developments in the blockchain and distributed systems: a significant coronal mass ejection (CME) has triggered potential severe geomagnetic storms, with visible auroras possible in central Europe. The discussion around database technologies has highlighted the benefits of API database architecture using PostgREST with PostgreSQL. The Open Source Security (OpenSSF) and OpenJS Foundations have issued warnings about social engineering attacks targeting open source projects, exemplified by an attempted backdoor in the XZ Utils project. A Visa co-developed study indicates that over 90% of stablecoin transactions might not involve genuine users, challenging the potential of stablecoins to revolutionize the payments industry soon. (Slides: FS24-DSy-11-news.pdf). The third video introduces blockchains, Bitcoin, and Ethereum (Slides: FS24-DSy-11-bitcoin.pdf) [111min]
Lecture 12
The first admin part is about the lecture content and challenge task hand in (Slides: FS24-DSy-12-admin.pdf). The second video (news) discussed recent developments in the blockchain and distributed systems: UniSuper experienced a major service disruption due to a Google Cloud misconfiguration, underscoring the importance of robust backup strategies like 3-2-1. Cloudflare's new HTTP proxy, Pingora, offers improved performance and resource efficiency, providing developers with customization flexibility. Traefik's latest version addresses websocket issues on macOS, enhancing its usability and robustness for load balancing. In crypto news, Alexey Pertsev, the developer behind Tornado Cash, was sentenced for money laundering, raising concerns about developer liabilities in open-source projects. Finally, the leader of the LockBit ransomware group was identified, demonstrating ongoing efforts to combat cybercrime despite persistent threats. (Slides: FS24-DSy-12-news.pdf). The third video discusses mail infrastructure (Slides: FS24-DSy-12-mail.pdf) [56min]
Lecture 13
The first admin part is about the challenge task hand in (Slides: FS24-DSy-13-admin.pdf). The second video (news) is about Ethereum co-founder Vitalik Buterin who discusses the similarities and differences between Layer 2 solutions and execution sharding, highlighting their use of ZK-SNARKs and Data Availability Sampling (DAS) for computation and data verification. Layer 2 solutions, managed through smart contracts, offer flexibility but require better coordination, which EIP 7683 aims to address by improving interoperability. Next, we revisit an SMTP bug where missing periods in emails were traced to line length limits and improper character escaping, with significant real-world impacts like incorrect financial statements. A fascinating tutorial shows how modifying QLC SSD firmware can transform it into an SLC SSD, increasing durability and speed at the cost of storage capacity, though the method involves dubious tools from unofficial sources. Finally, we cover a Google Cloud network issue caused by an automation bug, affecting services for nearly three hours, and a Microsoft incident that disrupted web search functions for several hours, underlining the complexity and interdependence of distributed systems. (Slides: FS24-DSy-13-news.pdf). [12min]
Lecture 14
CT/News (MP4)
This week's summary of Distributed Systems and Blockchain news highlights several key topics, and this week's focus is on the challenge task where students must implement a dockerized distributed system to pass the course. The winner of the challenge task is the Co-Puzzle Web App from Alice, Mischa, and Nils-Robin, a collaborative puzzle using WebSockets, React, Next.js, MongoDB, and Traefik, deployed internally at OST. Each winning student receives 50 CHF in Bitcoins. Congratulations! This news segment will resume in the fall with the blockchain lecture.. (Slides: FS24-DSy-14-news.pdf). [7min]
Challenge Task FS 2024
This semester's challenge task (CT) is the design and implementation of a simple distributed system (of your choice) where a service instance can fail. The system needs to have the following components:
- Simple Frontend (e.g., HTML, Vue, React, Svelte)
- Loadbalancer(e.g., traefik, nginx, HAproxy, Caddy)
- Two instances of a service (your choice), and during the challenge task presentation, one instance will be shut off.
- Scheduled Task that does something every minute (not more not less)
- Database (persistent storage, does not need to be scalable, but you can build it scalable if you want)
Requirements
All requirements below must be met in order to pass this lecture.
- Load balancing with multiple instances with failover of a service instance
- Dockerized
- Simple frontend
- Scheduled task
- Persistent storage
- Use latest stable releases of chosen libraries and frameworks
- The solution may use existing libraries and code, but those must open software software
- You are allowed to use any language, framework, and platforms. However, the supervisors are familiar with those: Java, Golang, JavaScript, Linux.
Deliverables
Hand-in 1: 14.04.2024, 23:59 (CET) - initial version of your project.
Final hand-in: 26.05.2024, 23:59 (CET) - well documented infrastructure, presentation (slides) of the application, also showing the architecture and design decisions via email to thomas.ost-at-bocek.ch or via a repository invite. The code and configuration should be easy to read and/or well documented, the presentation (slides or text) should show the architecture, components, and design decisions. On the 27.05.2024 in Rapperswil and 29.05.2024 in St.Gallen, you will present and demo your solutions onsite.
Groups
There will be groups of 2 - 3 for the challenge task. During the challenge task, the group shall meet every week during exercise hours to work on the task and discuss the next steps. The groups shall utilize their homework times to work on the challenge task, besides the exercise time slots assigned on Tuesday. You do not have be present at the exercises. The groups shall determine and set-up an internal projecO plan and shall distribute the workload so that each group member gets a fair load of work. Your submitted deliverables will be only accessible via VPN.
For a reference of previous work, go to FS23.
Nr | Name 1 | Name 2 | Name 3 | Project | Final Submission | Presentation Date |
---|---|---|---|---|---|---|
01 | Dominik R. | Stefan M. | Nicolas R. | TaskTracker | ✓ | slides, code |
02 | Lukas A. | Silvan K. | Sara O. | listify | ✓ | slides, code |
03 | Jan M. | Thomas R. | Patrick W. | Matricula* | ✓ | slides, code |
04 | Nils-Robin G. | Mischa B. | Alice G | Puzzle | ✓ | slides, code |
05 | Micha H. | Matthias H. | Tom S. | Crypto-Tracker | ✓ | slides, code |
06 | Philipp F. | Christoph B. | Simon A. | Todo List | ✓ | slides, code |
07 | Tobias K. | Noah F. | Pastebin | ✓ | slides, code | |
08 | Simon P. | Jonas K. | Isaia B. | v-manager | ✓ | slides, code |
09 | Roger M. | Leonardo R. | Tseten E. | wördle | ✓ | slides, code |
10 | Nico F. | Kyra M. | Kevin K. | MarketMinds | ✓ | slides, code |
11 | Claudio D. | Philip S. | TimeTracker | ✓ | slides, code | |
12 | Sabrina F. | Elena G. | ElSa | ✓ | slides, code | |
13 | Marco S. | Matteo G. | Philipp H. | Browser Fingerprinting | ✓ | slides, code |
14 | Valentino D. | Andrin K. | Daniel S. | Site Pulse 360 | ✓ | slides, code |
15 | Roman C. | Dejan B. | Mino P. | Mail (MOST) | ✓ | slides, code |
16 | Ali A. | Eric H. | Valerio F. | Todo | ✓ | slides, code |
17 | Martyn F. | Yannick S. | Livio M. | resource-monitoring | ✓ | slides, code |
18 | Fabian I. | Noël J. | R/Place | ✓ | slides, code | |
19 | David T. | Karim H. | Niklas K. | Clipper | ✓ | slides, code |
20 | Patrick S. | Anja F. | Mona P. | minute-by-minute | ✓ | slides, code |
21 | Lukas B. | Tim E. | SharedDocs (CRDT) | ✓ | slides, code | |
22 | Aziz H. | Fiona P. | JokeFactory | ✓ | slides, code |
*) Additional requirement: Kubernetes deployment
**) Earlier dates are possible