FS22
Challenge Task Winner Distributed Systems FS22
The winning group is: Caspar Martens, Simon Kindhauser, Lara Gubler
Caspar Martens, Simon Kindhauser, and Lara Gubler implemented Amy Gdala, a web application to memorize information by talking about your day (writing a diary) and to search for all your stored information with natural language queries. Their techstack included GPT-3, Kibana/Elastic Search, Coqui-STT/TTS, Python for micro-services, Kubernetes, and Istio for gRPC load balancing.
The price is sponsored by Axelra, a tech venture builder that accelerates digital business models (Linkedin announcement).
- Lecture dates: 21.02.2022 - 11.06.2022
- Exam sessions: 20.06.2022 - 16.07.2022
- Deadline Challenge Task: 12.05.2022, 23:59 (CET)
- Exam: 12.07.2022, 15:10 - 16:10
Distributed Systems (DSy)
| Nr | Date Lecture | Topics |
|---|---|---|
| 01 | 25.02.2022 | Admin, Introduction / Motivation |
| 02 | 04.03.2022 | Introduction / Categorization |
| 03 | 11.03.2022 | Distributed Applications: Tor, WebSockets, Protocols 1 |
| 04 | 18.03.2022 | Monorepos / Polyrepos, Protocols 2 |
| 05 | 25.03.2022 | Containers and VMs |
| 06 | 01.04.2022 | Load Balancing |
| 07 | 08.04.2022 | Application Protocols and RPC, Deployment (Docker Swarm, Kubernetes) |
| - | 15.04.2022 | - No Lecture (Spring Vacation) |
| 08 | 22.04.2022 | Classic Consensus |
| 09 | 29.04.2022 | CRDT, Deployment (Docker Swarm, Kubernetes) |
| 10 | 06.05.2022 | Blockchain, Bitcoin |
| 11 | 13.05.2022 | Challenge Task Presentations I |
| 12 | 20.05.2022 | Challenge Task Presentations II |
| - | 27.05.2022 | - No Lecture |
| 13 | 03.06.2022 | Exam preparation, Ethereum Introduction, Smart Contracts |
| 14 | 10.06.2022 | Challenge Task Award Winner Announcement, Q&A - live via MS Teams 08:10-09:50 (no exercises) |
Online lectures
The Distributed Systems (DSy) lecture will be held online. I'm preparing online videos for the lectures on Friday 08:10 - 09:50. The length of the videos will be ~ 80 - 100 minutes (2 x 45min. lectures) divided into segments. For the exercises, I will be available online on Friday from 10:10 - 11:50 and 13:10 - 14:50 on MS Teams. Team code: 2tbddgw. During exercises, you have time to work on your challenge task, ask questions about the challenge task or the lecture, or just to saying "Hi" :)
The online lectures and PDFs can be accessed from outside (also on Youtube), the excercise material only with VPN.
Lecture 1
The admin part gives an overview over this lecture and presents the challenge task for this course (Slides: DSy-FS22-01-Admin.pdf). The second video (news) discusses crypto miners and GPUs, common NGINX configuration mistakes, a reminder that networks can fail and an interesting development in deanonymizing coin mixers (Slides: DSy-FS22-01-News.pdf). The third video starts with the introduction into distributed systems (Slides: DSy-FS22-01-Intro.pdf). [120min]
Lecture 2
The admin part shows the current lecture plan and gives an update on the challenge task group forming (Slides: DSy-FS22-02-Admin.pdf). The second video (news) discusses neon gas shortage, high crypto currency demand due to the SWIFT ban, and how to bring Internet to Ukraine (Slides: DSy-FS22-02-News.pdf). The third video finished the introduction by discussing categorization of distributed systems (Slides: DSy-FS22-02-Categorization.pdf). [55min]
Lecture 3
The admin part shows the current lecture plan and gives an update on the challenge task group forming (Slides: DSy-FS22-03-Admin.pdf). The second video (news) discusses Internet connectivity and services in Ukraine/Russia, latency numbers, and the SSD report from backblaze (Slides: DSy-FS22-03-News.pdf). The third video is about tor, how it runs and how it can be setup (Slides: DSy-FS22-03-Tor.pdf). The fourth video is about Web Sockets, which is a requirement in the challenge task (Slides: DSy-FS22-03-Websocket.pdf). The fifth video is the first part of the protocol lecture (Slides: DSy-FS22-03-Protocols1.pdf). [90min]
Lecture 4
The admin part shows the current lecture plan (Slides: DSy-FS22-04-Admin.pdf). The second video (news) discusses the Tor rewrite in Rust, decentralized distribution of F-Droid, how to make docker images smaller, a new podman feature, BBR vs CUBIC benchmarks, and finally, the short-selling of USDT (Slides: DSy-FS22-04-News.pdf). The third video is about project setups, it is about monorepos and polyrepos (Slides: DSy-FS22-04-Repo.pdf). The fourth video is the second part of the protocol lecture (Slides: DSy-FS22-04-Protocols2.pdf). [108min]
Lecture 5
The admin part shows the current lecture plan (Slides: DSy-FS22-05-Admin.pdf). The second video (news) discusses the virtual Heineken beer, TCP and TLS with Wireshark, HTTP proxies, TLS 1.3 protocol tinkering, Vitalik about Crypto's Future, and another negative Web3 article (Slides: DSy-FS22-05-News.pdf). The third video is about VMs and containers (Slides: DSy-FS22-05-Containers.pdf). [117min]
Lecture 6
The admin part shows the current lecture plan (Slides: DSy-FS22-06-Admin.pdf). The second video (news) discusses complexity vs simplicity, an new Spring Core vulnerability?, STACKIT - an new cloud provider, how to reduce docker image size, the lost 38 Starlink satellites, DNSSEC adoption, and the Axie Infinity Hack (Slides: DSy-FS22-06-News.pdf). The third video is about load balancing (Slides: DSy-FS22-06-LoadBalancing.pdf). [97min]
Lecture 7
The admin part discusses WebSocket design choices, a problem with passive health checks with caddy, and shows the current lecture plan with a reminder of the first hand-in (addendum: there is an option based on d) that fulfills the CT requirements, and is now my favorite solution: create a single WebSocket instance (with the necessary REST endpoints) that only deals with WebSockets, while the other services instances can still be scaled). (Slides: DSy-FS22-07-Admin.pdf). The second video (news) discusses Intel's new Bitcoin ASIC, the new crypto assets EU rule, Spring4Shell, and performance tuning with kTLS (Slides: DSy-FS22-07-News.pdf). The third video is about application protocols, from protobuf to JSON, and Apache Thrift to gRPC (Slides: DSy-FS22-07-App-Proto.pdf). [74min]
Lecture 8
The admin part shows the current lecture plan and briefly shows the topics for distributed systems at other universities. (Slides: DSy-FS22-08-Admin.pdf). The second video (news) discusses the delays with Ethereums PoS, giving talks in North Korea, the Hetzner Cloud data loss, and the DeFi Beanstalk hack. (Slides: DSy-FS22-08-News.pdf). The third video is about classic consensus (RAFT/Paxos) (Slides: DSy-FS22-08-Consensus.pdf). [74min]
Lecture 9
The admin part shows the current lecture plan. (Slides: DSy-FS22-09-Admin.pdf). The second video (news) discusses crypto hacks, Nikes NFTs, ECDSA issues in Java, and the EU discussion about banning Bitcoins. (Slides: DSy-FS22-09-News.pdf). The third video finishes classic consensus with CRDT (conflict-free replicated data type) (Slides: DSy-FS22-09-CRDT.pdf). The fourth video is about deployment with Docker Swarm and Kubernetes (Slides: DSy-FS22-09-Deployment.pdf). [80min]
Lecture 10
The admin part shows the current lecture plan with a reminder to the hand-in of the challenge task on the 12.05.2022, 23:59 (CET). (Slides: DSy-FS22-10-Admin.pdf). The second video (news) discusses some blockchain hate and some blockchain love, the project loom C5M project, and the Backblaze drive statistics for Q1. (Slides: DSy-FS22-10-News.pdf). The third video starts with Bitcoin and Blockchains (Slides: DSy-FS22-10-Bitcoin.pdf). [105min]
Lecture 11
The news video briefly discusses cloud pricing, the two generals problem, and the evolving topic around the de-pegging of the TerraUSD from the USD, followed by more negative news from the crypto world (Slides: DSy-FS22-11-News.pdf). [27min]
Lecture 12
The news video briefly discusses the Ethereum merge, another algorithmic stable coin that lost its peg, affordable GPUs, and a followup on Terra/Luna - errata: there will be no algorithmic stable coin in the new fork (Slides: DSy-FS22-12-News.pdf). [12min]
Lecture 13
The admin part discusses the previous exam. (Slides: DSy-FS22-13-Admin.pdf). The second video (news) discusses the TerraUSD de-peg, an analysis that algorithmic stablecoins do not work, a discussion about SPA vs MPA, and Graviton 3 processor benchmarks. (Slides: DSy-FS22-13-News.pdf). The third video finishes the blockchain part with a high level introduction into Ethereum (Slides: DSy-FS22-13-Ethereum.pdf). [118min]
Lecture 14
The first video reveals the challenge task winner of this course (Slides: DSy-FS22-14-CT.pdf). The second video (news) mentions another UST analysis, the 8 fallacies of distributed systems illustrated, DDR5 prices going down, the published HTTP/3 standard, a broken sea cable, and that file sharing is becoming popular again. (Slides: DSy-FS22-14-News.pdf). [12min]
Challenge Task FS 2022
This semester's challenge task (CT) is the design and implementation of a simple distributed system (of your choice), with at least one service with a websocket, and where a service instance can fail. The system needs to have the following components:
- Simple Frontend (e.g., HTML, Vue, React, Svelte)
- Loadbalancer(e.g., traefik, nginx, HAproxy, Caddy)
- Two instances of a service (your choice), and during the challenge task presentation, one instance will be shut off.
- One storage backend
Requirements
All requirements below must be met in order to pass this lecture.
- Load balancing with scalable service
- Failover of a service instance
- A websocket needs to send data from a service to the frontend. You need to implement (consume) a websocket (using e.g., Blazor that integrates websockets does not fulfill this requirement). You can produce websocket data in your backend or use 3rd party websocket services (e.g., crypto market data)
- Dockerized
- Persistant storage (storage does not need to be scalable, but you can build it scalable if you want)
- Use latest stable releases of chosen libraries and frameworks
- The solution may use existing libraries and code, but those must open software software
- You are allowed to use any language, framework, and platforms. However, the supervisors are familiar with those: Java, Golang, JavaScript, Linux.
Deliverables
Hand-in 1: 14.04.2022, 23:59 (CET) - initial version of your project.
Final hand-in: 12.05.2022, 23:59 (CET) - well documented infrastructure, presentation (slides) of the application, also showing the architecture and design decisions via email to thomas.ost-at-bocek.ch or via a repository invite. The code and configuration should be easy to read and/or well documented, the presentation (slides or text) should show the architecture, components, and design decisions. During the weeks 13.05./20.05., you will present and demo your solutions via MS Teams.
Groups
There will be groups of 2 - 3 for the challenge task. During the challenge task, the group shall meet every week during exercise hours to work on the task and discuss the next steps. The groups shall utilize their homework times to work on the CT, besides the exercise time slots assigned on Friday. You do not have be present at the exercises. The groups shall determine and set-up an internal project plan and shall distribute the workload so that each group member gets a fair load of work. Results will be only accessible via VPN.
For a reference of previous work, go to FS21.
| Nr | Name 1 | Name 2 | Name 3 | Code | Slides | Presentation | Project | CT passed |
|---|---|---|---|---|---|---|---|---|
| 01 | Adrian L. | Jason B. | - | group1.zip | group1.pdf | group1.mp4 | Filesharing | ✓ |
| 02 | Olivier L. | Jan U. | Benjamin P. | group2.zip | group2.pdf | group2.mp4 | Wordl | ✓ |
| 03 | Amanda W. | Liliana S. | André B. | group3.zip | group3.pdf | group3.mp4 | Excursion Management System | ✓ |
| 04 | Niklas K. | Jeremy S. | Fadil S. | group4.zip | group4.pdf | group4.mp4 | Sprüche Generator | ✓ |
| 05 | Rolf O. | Benjamin K. | group5.zip | group5.pdf | group5.mp4 | Collaborative Data Visuaalizer | ✓ | |
| 06 | Marco A. | Dominik E. | Mathias L. | group6.zip | group6.pdf | group6.mp4 | Todo/Kanban List (TFOS) | ✓ |
| 07 | Laurent D. | Damian D. | Carlo D. | group7.zip | group7.pdf | group7.mp4 | Battleship | ✓ |
| 08 | Thomas K. | Dominic W. | group8.zip | group8.pdf | group8.mp4 | Chatplattform | ✓ | |
| 09 | Lara G. | Simon K. | Caspar M. | group9.zip | group9.pdf | group9.mp4 | Text to Speach | ✓ |
| 11 | Florian B. | Leonard S. | group11.zip | group11.pdf | group11.mp4 | Fractal generator/viewer | ✓ | |
| 12 | Dejan J. | Ronny M. | Gianluca N. | group12.zip | group12.pdf | group12.mp4 | Draw together | ✓ |
| 13 | Alessandro U. | Svenja S. | group13.zip | group13.pdf | group13.mp4 | NetflixTinder | ✓ | |
| 14 | Serafettin T. | Thashvar U. | group14.zip | group14.pdf | group14.mp4 | Crypto Portfolio Manager | ✓ |