Lectures Publications Theses Events

FS23

Challenge Task Winner Distributed Systems FS23

alt text

The winning group is: Manuel Metzler, Reto Furrer, and Sascha Häring

Their web application, designed to share collections of pictures, featured automated tagging through PhotoPrism. For storage, they used ScyllaDB, a high-performance, fault-tolerant database compatible with Cassandra. Everything is self-hosted with their own Kubernetes cluster using MetalLB.

The price is sponsored by Axelra, a tech venture builder that accelerates digital business models (Linkedin announcement).

Important dates

Distributed Systems (DSy)

This lecture series focuses on distributed systems, which are computer systems made up of multiple autonomous components, or nodes, that are linked and communicate with one another to achieve a common goal.

By the end of the series, you should have a good understanding of the various components and technologies involved in distributed systems and blockchain, as well as their implementation and deployment considerations.

Lecture Topics

NrDate LectureTopics
0121.02.2023Admin, Introduction / Motivation
0228.02.2023Introduction / Categorization
0306.03.2023Monorepos / Polyrepos, Protocols 1
-14.03.2023-
0421.03.2023HTTP/3 practical examples, Protocols 2
0528.03.2023Authentication, Web Architectures
0604.04.2023Load Balancing
-11.04.2023-
0718.04.2023Containers and VMs
0825.04.2023Deployment (Docker Swarm, Kubernetes)
0902.05.2023Distributed Applications: Mastodon, Distributed Applications: Tor
1009.05.2023Blockchain, Bitcoin
1116.05.2023Ethereum Introduction, Smart Contracts
1223.05.2023Challenge Task Presentations I
1330.05.2023Challenge Task Presentations II
1406.06.2023Challenge Task Award Winner Announcement, Exam preparation, Q&A - 08:10-09:50 (no exercises)

Online/Offline Lectures & Exercises

Due to student requests, the Distributed Systems (DSy) lecture will pre-recorded. The lecture videos will up uploaded one week before the lecture. The length of the videos will be ~ 80 - 100 minutes (2 x 45 min. lectures) divided into segments. The news segment (~10 min) will cover news around the topics distributed systems and blockchain and are not part of the exam. It should serve as on overview of current topics in this field.

I will be present in the lecture room 3.113 from 08:10 - 09:50. If you have not watched the lectures, you can watch it in this lecture. During exercises (10:10 - 11:50, 13:10 - 18:40), you can work on your challenge task. You do not need to be present on Tuesday, but it is highly recommended to discuss your ideas, issues, and problems with the challenge task. On request, you can also join the exercises remotely via MS Teams.

The online lectures and PDFs can be accessed from outside the OST network (also on Youtube), the exercise and admin material only with VPN.

Lecture

Lecture 1

Admin News
Introduction

The admin part gives an overview over this lecture and presents the challenge task for this course (Slides: FS23-DSy-01-admin.pdf). The second video (news) discusses the regulators and Binance, shows the CDN comparison of Fastly and AWS CloudFront from Backblaze, reminds of important Linux commands, shows what can go wrong with Vercel, if you have an infinity loop, and gives an update on Do Kwon (Slides: FS23-DSy-01-news.pdf). The third video gives a motivation why distributed systems are necessary (Slides: FS23-DSy-01-intro.pdf). [118min]

Lecture 2

Admin News
Introduction / Categorization

The admin part is about group forming (Slides: FS23-DSy-02-admin.pdf). This video (news) discusses the server failures of the city of Wuppertal, gives an update on the Let's Encrypt Stats page, shows how a million of web sockets can be kept open on a consumer grade laptop from 2013, shows an infinity glitch and an Interview with Guilherme Sperb Machado, then talks about broken undersea cables, why latency matters, and gives an update on Do Kwon and the Swiss bank (Slides: FS23-DSy-02-news.pdf). The third video discusses definitions of distributed systems and tries to categorize them (Slides: FS23-DSy-02-categorization.pdf). [70min]

Lecture 3

Admin News
Monorepos / Polyrepos Protocols 1

The admin part is about group forming and lecture content (Slides: FS23-DSy-03-admin.pdf). This video (news) discusses FTX and Binance, talks about expensive mistakes, reports on the attack on Quad9 and discusses self-hosting solutions (Slides: FS23-DSy-03-news.pdf). The third video the difference between a monorepo and polyrepo (Slides: FS23-DSy-03-repo.pdf). The fourth video starts with the protocols (Slides: FS23-DSy-03-protocols1.pdf) [83min]

No Lecture

News

The video (news) explains whats happening currently with the de-pegging of USDC and why it happened. (Slides: FS23-DSy-04-news.pdf). [10min]

Lecture 4

News
HTTP/3 Protocols 2

This video (news) follows up on the USDC peg, talks about the Docker Hub issue, talks about the fly.io outage, and that Instagram is disabling Its NFT features (Slides: FS23-DSy-04-news2.pdf). The third video shows practical examples with HTTP/3 and shows how HTTP/3 can be used in the challenge task (Slides: FS23-DSy-04-http3.pdf). The forth video finishes with the transport protocols (Slides: FS23-DSy-04-protocols2.pdf). [102min]

Lecture 5

Admin News
Authentication Web Architectures

The admin part is about group forming and lecture content (Slides: FS23-DSy-05-admin.pdf). This video (news) reminds you that access and refresh tokens should be kept secure, an update on Do Kwon, how to de-cloud your apps, and mentions that the SEC is threatening Coinbase for their unregistered staking services. (Slides: FS23-DSy-05-news.pdf). The third video shows existing authentication mechanisms, and highlights JWT, an authentication mechanisms that can be used for stateless services. (Slides: FS23-DSy-05-auth.pdf). The fourth video explains different web architectures that could be used for the challenge task. (Slides: FS23-DSy-05-arch.pdf) [111min]

Lecture 6

Admin News
Load Balancing

The admin part is about lecture content and the first challenge task milestone (Slides: FS23-DSy-06-admin.pdf). This video (news) talks about Trump Digital Trading Cards, about file system limits, ChatGPT, about Bittrex shutting down US operations, and an article to avoid HTTP/3 for the moment (Slides: FS23-DSy-06-news.pdf). The third video explains load balancing and shows load balancing with Traefik, Caddy, and Nginx (Slides: FS23-DSy-06-load.pdf). [88min]

No Lecture

News

This video (news) gives an update on Googles 5m file limit, talks about ERC-4804, an URI adaption to Ethereum, a rant about npm dependencies and the Ethereum Shanghai update. (Slides: FS23-DSy-06-news2.pdf). [7min]

Lecture 7

Admin News
Containers and VMs

The admin part is about lecture content and the first challenge task milestone (Slides: FS23-DSy-07-admin.pdf). This video (news) mentions the faster boot time of Podman, talks about how the SEC are defining DeFi, and gives and update on Tornado Cash from an EFF's perspective (Slides: FS23-DSy-07-news.pdf). The third video dives into VMs and containers (Slides: FS23-DSy-07-container.pdf). [91min]

Lecture 8

Admin News
Deployment

The admin part is about lecture content, the first challenge task milestone, and the reminder of the talk "From digital crimes to court in the UK" with Philip Anderson (Slides: FS23-DSy-08-admin.pdf). The video (news) shows an intro video to VMs and containers by IBM, talks about the MiCA regulation, about KDF and crypto detectives, and about local-first software (Slides: FS23-DSy-08-news.pdf). The third video gives a high-level overview over deployments (Slides: FS23-DSy-08-deployment.pdf). [50min]

Lecture 9

Admin News
Distributed Applications: Mastodon Distributed Applications: Tor

The admin part is about lecture content (Slides: FS23-DSy-09-admin.pdf). This video (news) reviews articles on distributed systems and blockchain topics. The first article illustrates various load balancing algorithms, such as round robin, weighted round robin, and least connections. The second article delves into self-hosting emails and mechanisms like SPF, DKIM, DMARC, and DANE. The third article compares REST and gRPC protocols for API development. The fourth article presents a map of Starlink, Oneweb, and GPS satellites, discussing their orbits and latencies. The final article evaluates Google Authenticator's synchronization feature and its security risks (Slides: FS23-DSy-09-news.pdf). The third video talks about the distributed application Mastodon (Slides: FS23-DSy-09-dist-app-mastodon.pdf). The fourth talks about the distributed application Tor (Slides: FS23-DSy-09-dist-app-tor.pdf). [103min]

Lecture 10

News
Blockchain, Bitcoin

No admin video for lecture 10. In this episode of Disputed Systems and Blockchain in the News, the use of Time-based One-Time Passwords (TOTP) for authentication, highlighting two open source TOTP applications: Aegis Authenticator, and FreeOTP+ is discussed. Next, an article by Amazon's CTO discussing the importance of choosing between microservices and monolithic architectures based on specific use cases, as there is no one-size-fits-all solution is mentioned. Lastly, the European banking sector's recent phenomenon of "bank walk," a slow but notable outflow of customer cash is mentioned (Slides: FS23-DSy-10-news.pdf). The second video introduces Bitcoin and its mechanisms (Slides: FS23-DSy-10-bitcoin.pdf). [93min]

Lecture 11

Admin News
Ethereum, Smart Contracts

The admin part is about lecture content (Slides: FS23-DSy-11-admin.pdf). In this news episode, Thomas Bocek discusses a bit flip issue in a Merkle tree from a Chromium mailing list that impacted a certificate transparency log, suggesting the use of ECC RAM for servers. He introduces a site explaining SSL/TLS handshakes and shares a report indicating that nearly half of all internet traffic comes from bots. Finally, he mentions Sam Altman's cryptocurrency project, Worldcoin, designed to distribute cryptocurrency globally for free, and its potential implications for the future (Slides: FS23-DSy-11-news.pdf). The third video introduces Ethereum and smart contracts (Slides: FS23-DSy-11-ethereum.pdf). [74min]

Lecture 12

Admin
News

The admin part is about lecture content and the challenge task handin and deadline (Slides: FS23-DSy-12-admin.pdf). In this news episode, Thomas Bocek discusses various developments in the realm of distributed systems and blockchain: cryptocurrency wallet provider Ledger introduced a seed phrase recovery service, raising security concerns due to their past data breach. Discord migrated its data management from Cassandra to ScyllaDB, addressing performance issues, reducing the number of nodes, and improving latency and resilience. A recent post argued that due to advancements in modern hardware, scaling up could be more cost-effective, reliable, and simpler than scaling out strategies. Lastly, Airbnb improved their page performance by implementing HTTP streaming, sending data in chunks as soon as they're ready and reducing the first contentful paint time. (Slides: FS23-DSy-12-news.pdf). [20min]

Lecture 13

Admin
News

The admin part is about the challenge task handin and deadline (Slides: FS23-DSy-13-admin.pdf). In this news episode, Thomas Bocek discusses articles related to distributed systems and blockchain. The first article compares memory usage for 1 million concurrent tasks in different programming languages. The second article highlights challenges with large language models, including limited context windows and slow API calls. The need for distributed systems and horizontal scaling is emphasized. The last article introduces Podman, an alternative to Docker, and its new GUI version. (Slides: FS23-DSy-13-news.pdf). [11min]

Lecture 14

Exam preparation
News

The first video is the exam preparation (Slides: FS22-DSy-14-exam.pdf). In the news video, Thomas Bocek concludes a lecture series on distributed systems and blockchain. The highlight of this conclusion is the announcement of the challenge task winner, wherein students were required to build a fully scalable distributed system capable of serving HTTP/3 requests. The students presented many different ideas, which not only reflected the depth of student creativity and technical knowledge but also provided interesting insights into their technology preferences. (Slides: FS23-DSy-14-ct.pdf). [44min]

Challenge Task FS 2023

This semester's challenge task (CT) is the design and implementation of a simple distributed system (of your choice) where a service instance can fail. The system needs to have the following components:

  1. Simple Frontend (e.g., HTML, Vue, React, Svelte)
  2. Loadbalancer(e.g., traefik, nginx, HAproxy, Caddy)
  3. Two instances of a service (your choice), and during the challenge task presentation, one instance will be shut off.
  4. One storage backend

Requirements

All requirements below must be met in order to pass this lecture.

  1. Load balancing with scalable service
  2. Failover of a service instance
  3. Dockerized and Frontend connects via HTTP/3
  4. Persistant storage (storage does not need to be scalable, but you can build it scalable if you want)
  5. Use latest stable releases of chosen libraries and frameworks

Deliverables

Hand-in 1: 17.04.2023, 23:59 (CET) - initial version of your project.

Final hand-in: 22.05.2023, 23:59 (CET) - well documented infrastructure, presentation (slides) of the application, also showing the architecture and design decisions via email to thomas.ost-at-bocek.ch or via a repository invite. The code and configuration should be easy to read and/or well documented, the presentation (slides or text) should show the architecture, components, and design decisions. During the weeks 23.05./30.05., you will present and demo your solutions onsite.

Groups

There will be groups of 2 - 3 for the challenge task. During the challenge task, the group shall meet every week during exercise hours to work on the task and discuss the next steps. The groups shall utilize their homework times to work on the challenge task, besides the exercise time slots assigned on Tuesday. You do not have be present at the exercises. The groups shall determine and set-up an internal project plan and shall distribute the workload so that each group member gets a fair load of work. Your submitted deliverables will be only accessible via VPN.

For a reference of previous work, go to FS22.

NrName 1Name 2Name 3ProjectSourcePresentation
01Sascha H.Reto F.Manuel M.Gotographygroup1.zipgroup1.pdf
02Josip D.Joshua B.Oliver D.Image puzzlegroup2.zipgroup2.pdf
03Momoko W.Marc H.-Learning cardsgroup3.zipgroup3.pdf
04Dante SJanosch B.-AI-NoteToolgroup4.zipgroup4.pdf
05Dario B.Fabrice-Ronny B.-AI-Health-Coachgroup5.zipgroup5.pdf
06Lucien Z.Florian R.-Project Banggroup6.zipgroup6.pdf
07Lukas G.Anian G.-Activities in Zurichgroup7.zipgroup7.pdf
08Simon H.Luca K.-Bites & Sipsgroup8.zipgroup8.pdf
09Vina Z.Vanessa G.Lukas M.TasKinggroup9.zipgroup9.pdf
10Gian-Luca V.Nick G.Felix T.OST Placegroup10.zipgroup10.pdf
11Jann F.Lukas S.-Vote Systemgroup11.zipgroup11.pdf
12Noah S.Joel S.-Simple Cloudgroup12.zipgroup12.pdf
13Fabian F.Claudio K.-Twitter@Homegroup13.zipgroup13.pdf
14Carlos C.Joshua C.-Distribution of network device configsgroup14.zipgroup14.pdf
15Lukas D.Kaj H.Andri J.StudiAppgroup15.zipgroup15.pdf
16Fabio S.Michael E.-Noteivategroup16.zipgroup16.pdf
17Jan P.Miles S.Michael S.Online Art Gallerygroup17.zipgroup17.pdf
18Simon L.Davide F.-Boulder Path Appgroup18.zipgroup18.pdf
19Jonas M.Patrick L.-LFT Servicegroup19.zipgroup19.pdf
20Timo K.Fabio E.-Instagram clonegroup20.zipgroup20.pdf
21Thajakan T.Corsin S.-HA-Chatgroup21.zipgroup21.pdf
22Nicolas G.Pascal L.-FiTrackgroup22.zipgroup22.pdf
23Andrew W.Lucas von N.-Todo listgroup23.zipgroup23.pdf
24*Dario G.Nick W.Samuel M.Spontaneousgroup24.zipgroup24.pdf
25*Jasmin F.Mirio E.-Spontaneousgroup25.zipgroup25.pdf
26Ramon E.Marc K.-Live-Votegroup26.zipgroup26.pdf
27Natalia G.Kailing P.Linus F.Image managergroup27.zipgroup27.pdf
28Kevin LMarco K.-Digitale Kundenkartegroup28.zipgroup28.pdf

*) this group works on the SE project, and cannot win the CT price. Furthermore, they have 2 additional requirements: use a distributed database on at least 2 nodes, and integrate JWT authentication.